Human posture recognition method based on lightweight spiking neural network

By constructing a human posture recognition method of lightweight pulse neural network, using the leak-integration-ignition LIF explicit iterative model and feature extraction stage, the problem of high power consumption and accuracy loss in the mobile terminal of deep neural network is solved, and low power is achieved Human posture recognition with low computing capacity, suitable for embedded devices.

CN118762384BActive Publication Date: 2024-11-12CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411241191.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-12
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

In the existing image processing technology, the high power consumption and complexity of deep neural networks lead to limited applications on mobile terminals or microcomputers, and convolutional neural networks have problems such as loss of accuracy and waste of computing resources when processing spatiotemporal data flows. .

Method used

Using a human posture recognition method based on lightweight pulse neural network, a low-power Light-Energy SNN (LESNN) model is designed by constructing an explicit iterative model of leakage-integration-ignition LIF, combining feature extraction and dimensional splicing in four stages. , directly process spatiotemporal data flow using pulsed neural networks, reducing the amount of calculation and improving the recognition accuracy.

Benefits of technology

It realizes human posture recognition with low power consumption and low computing volume, reduces energy consumption by about 137 times, improves recognition accuracy and efficiency, is suitable for embedded devices, and solves the problems of resource waste and accuracy loss in traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762384B_ABST
    Figure CN118762384B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of image processing technology, and in particular to a method for human posture recognition based on a lightweight pulse neural network. It includes: photographing human postures in different postures, using the photographed images as data sets, and storing the spatiotemporal data flow information of the collected images in the form of comma-separated value files; constructing a leakage-integral-ignition LIF explicit iteration model; constructing a loss function, and adding a regularization term as a flexible restriction of the loss function; constructing a low-power lightweight LESNN network based on a pulse neural network, so that the feature map undergoes four stages of feature extraction and dimension splicing to obtain a lightweight pulse neural network model; setting the initial parameters of the model and training the model; inputting the spatiotemporal data stream to be tested into the trained model, and obtaining the posture state of the human body after detection and recognition. The advantages are: using a pulse neural network to achieve lightweight; introducing a regularization term to effectively alleviate gradient explosion and gradient disappearance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a human body posture recognition method based on a lightweight pulse neural network. Background Art

[0002] At present, the method of target recognition in the field of image processing mainly adopts the method of convolutional neural network, and the powerful GPU has also greatly improved the computing power and analysis efficiency of the intelligent recognition model. However, with the superposition of the depth of the deep neural network model, the accuracy does not increase linearly with the increase of the model depth. Therefore, most of the existing object recognition technologies use highly complex and expensive chips, which are difficult to achieve low-cost and low-power computing, resulting in a waste of resources. In addition, the high storage and high power consumption of the network model seriously restrict the application and promotion of deep neural networks in mobile terminals or microcomputers with limited computing resources.

[0003] In order to solve the high power consumption problem of deep models, the use of spiking neural networks (SNNs) can effectively reduce the power consumption of models. Spiking neural networks can well receive the spatiotemporal data streams generated by moving objects and efficiently process these changing information. However, there are some problems in training large-scale SNNs: First, the complex neural dynamics models in the spatial and temporal domains make back propagation difficult to understand. Specifically, neural activity not only propagates layer by layer in the spatial domain, but also affects the state in the temporal direction. Second, event-driven pulses are discrete and non-differentiable, which also hinders the implementation of error back propagation (BP) based on gradient descent.

[0004] In view of the above shortcomings, there are many related solutions at home and abroad, such as:

[0005] (1) In the invention patent application with publication number CN115810129A, publication date March 17, 2023, and invention name “An object classification method based on a lightweight network”, a lightweight network is constructed. By presetting multiple expansion rates, a hole convolution is performed on the image of the object to be identified to obtain different image features, and then the image features are weighted by an adaptive weight assignment method to perform feature fusion and output the classification results. This method can alleviate the model redundancy of the convolutional network, but the model and training strategy it adopts are relatively simple, and the recognition accuracy is not very high. In addition, the convolutional neural network is used, and the global information of the image is calculated during the convolution process, which wastes unnecessary computing resources.

[0006] (2) The paper "Fast-Classifying, High-Accuracy Spiking Deep Networks Through Weight and Threshold Balancing" reports the conversion of ANN networks into SNNs, and analyzes the impact of converting ANNs into SNNs on the selection of pulse neuron parameters (such as trigger thresholds and discharge rates) to reduce performance losses during network conversion. Since artificial neural networks (ANNs) have relatively mature technologies, most of the existing methods for directly processing spatiotemporal data streams use the method of converting ANNs into SNNs. Converting a trained ANN into an SNN can reduce training time and computing resources, and make it easier to adjust and optimize the model. However, the conversion process also loses the biological rationality of SNNs and the better interpretability of biological neural systems. At the same time, the ANN-SNN conversion strategy requires a larger time step and is no longer applicable to neurological hardware.

[0007] (3) The paper “ENABLING SPIKE-BASED BACKPROPAGATION FOR TRAINING DEEP NEURALNETWORK ARCHITECTURES” proposed an approximate derivative method to explain the behavior of LIF neurons, making it possible to directly train SNNs. However, when directly training SNNs, the gradient propagation showed great instability and tended to vanish in most cases, which hindered the construction depth of the SNN model and affected the performance of the method.

[0008] (4) The paper "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications" reports a lightweight convolutional neural network MobileNet, which can achieve real-time image classification and target detection on mobile devices. This method uses depthwise separable convolution to reduce the amount of computation. However, the use of 1x1 convolution to expand the channel dimension causes the information transition to be uneven, which in turn hinders the flow of information and affects the accuracy of network prediction and recognition.

[0009] Among the methods mentioned above, using convolutional neural networks for low-power and lightweight processing is not the optimal solution for resource conservation; directly converting ANN to SNN may result in accuracy loss and requires additional conversion steps, such as weight normalization and threshold setting, which increases the complexity of implementation and is not suitable for neurological hardware; improving the neuron model improves accuracy, but this operation is heavily dependent on the performance of the original network. Although the above methods can process spatiotemporal data flow information, whether it is the network architecture of ANN to SNN or the network architecture based on SNN, their model limitations make it impossible to achieve a balance between low power consumption and high accuracy.

[0010] In summary, the current mainstream object recognition processing methods all use convolutional neural networks to implement image processing, but this method will lose the time series information between image frames; another method is to use time slicing to flatten the time dimension of the spatiotemporal data stream. Although this method can use the time series information, it will also lose the time series information of pixels in a time window and does not fully utilize the spatiotemporal characteristics of the spatiotemporal data stream. In addition, the traditional convolution operation uses a pixel-by-pixel calculation strategy, which results in very redundant calculation overhead. Summary of the invention

[0011] In order to solve the above problems, the present invention provides a human body posture recognition method based on lightweight pulse neural network.

[0012] The present invention aims to provide a method for human posture recognition based on a lightweight spiking neural network, comprising the following steps:

[0013] S1. Establishing a data set: photographing human bodies in different postures, using the photographed images as data sets, and storing the spatiotemporal data stream information of the collected images in the form of comma-separated value files;

[0014] S2. Constructing the leakage-integration-ignition LIF explicit iterative model;

[0015] S3. construct a loss function and add a regularization term as a flexible restriction of the loss function;

[0016] S4. Construct a low-power lightweight LESNN network based on a spiking neural network, so that the feature map undergoes four stages of feature extraction and dimension concatenation to obtain a lightweight spiking neural network model;

[0017] S5. Set the initial parameters of the model and train the lightweight spiking neural network model;

[0018] S6. Input the spatiotemporal data stream to be tested into the trained lightweight pulse neural network model, and obtain the posture state of the human body after detection and recognition.

[0019] Preferably, the explicit equation of the leakage-integration-ignition LIF explicit iteration model is:

[0020]

[0021] ;

[0022] in, u is the membrane potential, n and l(n) represent the neuron number of the nth layer and the nth layer, respectively. is the synaptic weight from the jth neuron in the previous layer n to the ith neuron in the next layer (n+1), Indicates pulse output, for The next state of t represents the time step, is the given excitation threshold.

[0023] Preferably, the feature extraction and dimension splicing of the four stages in step S4 specifically include the following steps:

[0024] S401. Visualize the data flow of the input feature map and write it into the data preprocessing, and generate the voxel grid of the corresponding time window according to the set time step; the input feature map first passes through a downsampling module to reduce the height and width of the feature map by 2 times, and the channel dimension is changed from 1 to 16 to obtain the feature map ;

[0025] S402. Feature map Entering the LE Block1 in the first stage, the feature maps are obtained after passing through the residual path and the shortcut path respectively. and feature map ; In the channel dimension and Add together and get the shape Feature map ;

[0026] S403. The feature map enters LE Block3 and passes through the residual path of LE Block3 to obtain a shape of Feature map ; The feature map and feature map before sending to LE Block3 Adding in the channel dimension, the shape is The feature map of

[0027] S404. The feature map is sent to LE Block2 in the second stage. After passing through the residual path and the shortcut path, the residual path feature map and the shortcut path feature map are obtained respectively. The shortcut path feature map and the residual path feature map are added in the channel dimension to obtain a shape after downsampling by two times: Feature map;

[0028] S405. The third and fourth stages repeat the operations of steps S403-S404, and finally obtain a shape of The feature map of , where T, C, H, and W are 5, 128, 8, and 11 respectively;

[0029] S406. The feature map finally passes through an adaptive average pooling layer with an input of 128 and an output of 5 to complete the construction of the lightweight pulse neural network model.

[0030] Preferably, the residual paths of the LE Block 1 and the LE Block 2 are the same and both have two branches; After further downsampling through an LDCB block, the feature map height and width are changed to 60 and 87, and the number of channels is changed from 16 to 32, and the feature map is obtained. ; On the other branch, the maximum pooling operation is used to reduce the height and width of the feature map by 2 times to 60 and 87, while the number of channels remains unchanged, and the feature map is obtained. ; The characteristic graph passing through LDCB And the feature map after maximum pooling The feature maps are split into four groups in the channel dimension, and the four groups of feature maps are spliced ​​in the channel dimension to generate four groups of fused feature maps. ~ , the shape is , where T, C, G, H, and W are 5, 12, 4, 60, and 87 respectively; summing up in the grouping dimension realizes information fusion between different groups, and the generated shape is Feature map , feature map T, C, H, and W are 5, 12, 60, and 87 respectively; then the feature map Send it to the LCB block to further extract features and change the channel dimension from 12 to 16 to obtain the feature map .

[0031] Preferably, in the shortcut path of LE Block1, the input feature map After a maximum pooling operation, the height and width are reduced by two times to 60 and 87, and then a LCB block with a convolution kernel of 1 is used to obtain the feature map. , feature map Shape and feature map .

[0032] Preferably, in the shortcut path of LE Block2, the feature map is downsampled through a maximum pooling operation (maxpool), reducing the height and width from 60 and 87 to 60 and 44, and then passed through an LCB block with a convolution kernel of 1 to obtain the feature map and concatenate it with the original feature in the channel dimension, reducing the number of channels from 16 to 32.

[0033] Preferably, the initial parameters include: the time step is set to 5, the threshold voltage is 0.4, leakage constant Set to 0.25, gradient approximation Set to 0.5, batch_size per GPU is set to 64, the number of epoch iterations is 300, and the learning rate is 0.025.

[0034] Preferably, step S4 further comprises performing time-effective batch normalization on the feature map during the process of constructing the lightweight spiking neural network model. The mathematical expression is as follows:

[0035] ;

[0036] in, , ; In the above formula, express No. k The feature map of each channel, , and They are The mean and variance calculated based on the dimension statistics data, The setting in the code is 1e-5. 、β is a constant, p[t] is a trainable weight.

[0037] Preferably, the loss function of step S3 adopts mean square error MSE; the regular term expression is as follows:

[0038] ;

[0039] In the formula, Y represents the label vector, t represents the current time step, T represents the total time step, λ represents the regularization hyperparameter, M represents the prediction matrix where each prediction neuron is connected to a specific class, represents the prediction vector of the last layer at time step t.

[0040] Preferably, human body postures include walking, running, waving, squatting and turning; the data set includes a training set and a test set, and the ratio of the training set to the test set is 8:2.

[0041] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0042] The present invention designs a novel low-power lightweight recognition method (called Light-Energy SNN, LESNN model), abandons the concept of picture frames, can directly use spatiotemporal data streams for image processing, and realize the target's posture recognition function. Unlike convolutional neural networks, we use pulse neural networks (SNN) to process target information. The entire model uses deep pulse convolution to extract channel-by-channel information; then a special group-by-group summation operation is used to expand the feature information from the shallow layer to the deep layer representation, which greatly reduces the model calculation amount while ensuring accuracy, while broadening the model representation ability and promoting information flow; finally, a shortcut path based on membrane potential mapping is used to eliminate redundant multiplication and addition operations caused by non-pulse convolution, further reducing the computational overhead. After parameter calculation, the model LESNN of the present invention consumes only 0.02mJ of energy for a complete calculation, which is about 137 times less energy than the traditional convolutional neural network with the same architecture, and the model parameters are reduced from 128.04MB to only 2.24MB. This also makes it possible to achieve energy-saving and efficient downstream tasks in embedded devices. The present invention solves the shortcomings of the prior art, adopts the third-generation neural network SNN architecture, fully utilizes the spatiotemporal characteristics of spatiotemporal data streams, improves the accuracy and efficiency of target recognition, and greatly reduces computing overhead and energy loss. It is expected to be widely used in the field of image processing and bring more value to users and the market. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a method for human posture recognition based on a lightweight pulse neural network provided according to an embodiment of the present invention.

[0044] Figure 2 1. The spatiotemporal data flow and visualization effect diagram provided according to an embodiment of the present invention; (A) is the spatiotemporal data flow; (B) is the visualization effect diagram of the spatiotemporal data flow.

[0045] Figure 3 It is a diagram of the architecture of a lightweight spiking neural network LESSN model provided according to an embodiment of the present invention.

[0046] Figure 4It is a schematic diagram of the visualization results of the lightweight spiking neural network LESSN model recognition provided according to an embodiment of the present invention; (A) is a visualization effect diagram of the spatiotemporal data flow; (B) is a two-fold downsampling feature map; and (C) is a four-fold downsampling feature map. DETAILED DESCRIPTION

[0047] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0049] The present invention provides a human body posture recognition method based on a lightweight pulse neural network (the specific method flow chart is as follows Figure 1 ), including the following steps:

[0050] S1. Establish a data set: photograph human bodies in different postures, use the photographed images as a data set, and store the spatiotemporal data flow information of the collected images in the form of comma-separated values ​​(CSV) files;

[0051] Human body postures include walking, running, waving, squatting, and turning; the spatiotemporal data flow (spatial event flow) and visualization effects of walking actions are shown in Figure 2. Figure 2 As shown;

[0052] The dataset includes a training set and a test set, with training samples and test samples divided in a ratio of 8:2.

[0053] S2. Constructing the Leakage-Integration-Fire (LIF) explicit iterative model;

[0054] The LIF model is often used to describe the activity behavior of neurons, including the update of membrane potential and the release of spike potential. Its implicit equation is:

[0055] (1)

[0056] (2)

[0057] in u is the membrane potential, t is the time step, τ is the time constant, I is the presynaptic input, and is a given excitation threshold. Equations (1) and (2) describe the behavior of firing neurons in the form of a leak-integrate-fire mechanism; when the membrane potential reaches a given threshold, the neuron will emit a spike, and at this time the membrane potential u Reset to , otherwise the neuron receives presynaptic stimulation I , and updates its membrane potential according to formula (1).

[0058] The differential expression described above in the continuous domain is widely used in biological simulation, but it is too implicit for implementation on the mainstream machine learning framework; therefore, the iterative form of equations (1) and (2) is converted into an explicit iterative form, and the expression is as follows:

[0059] (3)

[0060] The presynaptic input I Expanded to linear summation ; Subscript j represents the presynaptic index, o(j) represents the corresponding presynaptic pulse; by After scaling and incorporating it into the synaptic weight W, we can get the following formula:

[0061] (4)

[0062] In the formula, u is the membrane potential, k represents the membrane potential attenuation factor (set to 0.2 in the code), t represents the time step, j represents the presynaptic index, o(j) represents the corresponding presynaptic pulse, W represents the synaptic weight.

[0063] We then add the trigger and firing mechanism to Eq. (4) and assume that the membrane potential after reset is is 0, and the final explicit equation is:

[0064] (5)

[0065] (6)

[0066] Among them, n and l(n) represent the neuron number of the nth layer and the nth layer respectively. is the synaptic weight from the jth neuron in the previous layer n to the ith neuron in the next layer (n+1), Indicates pulse output, t represents the time step, is a given excitation threshold. Equations (5)-(6) reveal that the pulse output The excitation activity affects the next state through the leakage-integration-firing mechanism . This can be interpreted as if the neuron fires a spike at time step t, then the membrane potential at time step t+1 will be determined by the term (1- ) clears its attenuation component On the contrary, if no pulse is sent out at time step t, the membrane potential is retained until time step t+1 and added to the accumulated membrane potential at time step t+1.

[0067] Through equations (5)-(6), the implicit equations (1)-(2) are transformed into explicit iterative forms, making them easier to implement in our framework.

[0068] S3. Construct a loss function and add a regular term as a flexible restriction of the loss function;

[0069] Specifically, the mean square norm (mean square error) MSE is used as the loss function, which is used to measure the mean square error between the average prediction result and the label vector Y;

[0070] The regular expression is as follows:

[0071] (7)

[0072] In the formula, Y represents the label vector, t represents the time step (the current time step), T represents the total time step, λ represents the regularization hyperparameter (used for flexible limit loss, set to 1e-3 in the code), M represents the prediction matrix where each prediction neuron is connected to a specific class, represents the prediction vector of the last layer at time step t.

[0073] S4. Construct a low-power lightweight (Light-Energy SNN, LESNN) network based on a pulse neural network, and make the feature map undergo four stages of feature extraction and dimension splicing to obtain a lightweight pulse neural network model ( Figure 3 );

[0074] The four-stage feature extraction and dimension splicing specifically include the following steps:

[0075] S401. Visualize the data flow of the input feature map and write it into the data preprocessing, and generate the voxel grid of the corresponding time window according to the set time step; the input feature map first passes through a downsampling module, and the height and width of the feature map are reduced by 2 times to 120 and 173, and the channel dimension is changed from 1 to 16;

[0076] Specifically, the height H of the input feature map is 260 and the width W is 346; the time step of the voxel grid is 5 and the number of channels is 1; the convolution kernel of the downsampling module is 3 and the step size is 2;

[0077] S402. Feature map Entering the LE Block1 in the first stage, the feature maps are obtained after passing through the residual path and the shortcut path respectively. and feature map ; In the channel dimension and Add together and get the shape Feature map , where T, C, H, and W are 5, 16, 60, and 87 respectively;

[0078] Specifically, there are two branches in the residual path of LE Block1, one branch After further downsampling through an LDCB block, the feature map height and width are changed to 60 and 87, and the number of channels is changed from 16 to 32, and the feature map is obtained. ; On the other branch, the maximum pooling operation is used to reduce the height and width of the feature map by 2 times to 60 and 87, while the number of channels remains unchanged, and the feature map is obtained. ; The characteristic graph passing through LDCB And the feature map after maximum pooling The feature maps are split into four groups in the channel dimension, and the four groups of feature maps are spliced ​​in the channel dimension to generate four groups of fused feature maps. ~ , the shape is , where T, C, G, H, and W are 5, 12, 4, 60, and 87 respectively; summing up in the grouping dimension realizes information fusion between different groups, and the generated shape is Feature map , feature map T, C, H, and W are 5, 12, 60, and 87 respectively; then the feature map Send it to the LCB block to further extract features and change the channel dimension from 12 to 16 to obtain the feature map ;

[0079] Specifically, in the shortcut path of LE Block1, the input feature map After a maximum pooling operation, the height and width are reduced by two times to 60 and 87, and then a LCB block with a convolution kernel of 1 is used to obtain the feature map. , feature map Shape and feature map ;

[0080] S403. The feature map enters LE Block3 and passes through the residual path of LE Block3 to obtain a shape of Feature map , where T, C, H, and W are 5, 16, 60, and 87 respectively; the feature map before being sent to LE Block3 and the feature map Adding in the channel dimension, the shape is The feature map of

[0081] Specifically, there are two branches in the residual path of LE Block3, one branch After an LDCB block (no downsampling required), the feature map is obtained ; On the other branch, the maximum pooling operation is used to reduce the height and width of the feature map by 2 times to 60 and 87, while the number of channels remains unchanged, and the feature map is obtained. ; The characteristic graph passing through LDCB And the feature map after maximum pooling The feature maps are split into four groups in the channel dimension, and the four groups of feature maps are spliced ​​in the channel dimension to generate four groups of fused feature maps. ~ , the shape is , where T, C, G, H, and W are 5, 12, 4, 60, and 87 respectively; summing up in the grouping dimension realizes information fusion between different groups, and the generated shape is Feature map , feature map T, C, H, and W are 5, 12, 60, and 87 respectively; then the feature map Send it to the LCB block to further extract features and change the channel dimension from 12 to 16 to obtain the feature map ;

[0082] S404. The feature map is sent to LE Block2 in the second stage. After passing through the residual path and the shortcut path, the residual path feature map and the shortcut path feature map are obtained respectively. The shortcut path feature map and the residual path feature map are added in the channel dimension to obtain a shape after downsampling by two times: Feature graph, where T, C, H, and W are 5, 32, 30, and 44 respectively;

[0083] Specifically, the residual path of LE Block2 is the same as the residual path of LE Block1. After the residual path, we get The feature map of

[0084] Specifically, in the shortcut path of LE Block2, the feature map is downsampled through a maximum pooling operation (maxpool), and the height and width are changed from 60 and 87 to 60 and 44, and then it is obtained through an LCB block with a convolution kernel of 1 and concatenated with the original feature in the channel dimension, changing the number of channels from 16 to 32;

[0085] S405. The third and fourth stages repeat the operations of steps S403-S404, and finally obtain a shape of The feature map of , where T, C, H, and W are 5, 128, 8, and 11 respectively;

[0086] S406. The feature map finally passes through an adaptive average pooling layer with an input of 128 and an output of 5 to complete the construction of the lightweight pulse neural network model.

[0087] Step S4 also includes performing temporal-effective batch normalization (TEBN) on the feature map during the construction of the lightweight spiking neural network model; the mathematical expression of the temporal-effective batch normalization is as follows:

[0088] (8)

[0089] (9)

[0090] (10)

[0091] In the formula, express No. k The feature map of each channel, , and They are The mean and variance calculated based on the dimension statistics data, is a very small number (approximately 0, the code is set to 1e-5), 、β is a constant, p[t] is a trainable weight.

[0092] S5. Set the initial parameters of the model and train the lightweight spiking neural network model;

[0093] Specifically, use adamW to train the model parameters and save the model parameters after reaching the specified epoch;

[0094] In a specific embodiment, the initial parameters are set as follows: the time step is set to 5, the threshold voltage is 0.4, leakage constant Set to 0.25, gradient approximation Set to 0.5, batch_size per GPU is set to 64, epoch iterations are set to 300, and learning rate is set to 0.025; after the initial 150 epochs, the learning rate is decayed to 0.1 times the original value after every 50 epoch iterations. The learning rate is initially set to a relatively large value of 0.025 in order to make the decline faster at the beginning of training, and the subsequent learning rate gradually decays to make the model converge as soon as possible. After multiple iterations using 8 NVIDIA GeForce RTX3090 GPUs, the loss value will tend to stabilize, and the network model parameters at this time are saved.

[0095] S6. Input the spatiotemporal data stream to be tested into the trained lightweight spiking neural network model (LESNN network model), and obtain the posture state of the human body after detection and recognition.

[0096] The advantages of the present invention include the following points:

[0097] (1) Use pulse neural network to achieve lightweight instead of traditional convolutional neural network: Traditional convolution uses a pixel-by-pixel calculation strategy, which consumes a lot of computing resources. Pulse convolution only calculates at the pixel position where the pulse is emitted. This sparse summation strategy greatly reduces the computing overhead and energy loss. At the same time, the pulse neural network also considers time series information to improve the model recognition effect. Therefore, we abandon the traditional convolutional neural network method and use the pulse neural network to build a more energy-saving and lightweight network.

[0098] (2) Combining residual neural network and lightweight network to build a model with deeper model depth: Residual learning has been proven to alleviate the model degradation problem in deep networks. Therefore, we draw on the idea of ​​residual learning to improve the feature details from shallow networks to deep networks, and stack LE Blocks according to actual needs to achieve higher-precision human posture recognition.

[0099] (3) Optimized the loss function: The regularization term is added to achieve flexible restrictions on the loss function, which effectively alleviates the gradient explosion and gradient vanishing problems and helps build a deeper and more stable model.

[0100] (4) TEBN replaces TDBN as the model normalization function: Based on TDBN, learnable weights are added to each moment, which helps stabilize the gradient norm.

[0101] (5) This invention firstly applies pulse deep convolution to realize feature extraction of single channel dimension, and then adopts a novel group-by-group summation operation to realize the fusion of channel dimension while reducing the amount of calculation, expand the model's ability to represent details, and promote information flow. At the same time, the model's dependence on the number of channels is reduced, that is, a model with a small number of channels can effectively trigger pulses. We set the channels to 16 and 32 respectively, and conduct training and testing on the data set as shown in Table 1. According to the number and type of operations of 32-bit floating-point operations in the prior art, the energy loss is estimated. A multiplication and addition operation (MAC) consumes 4.9pJ of energy, and an addition operation (AC) consumes 0.9pJ of energy; synaptic operations (SyOPs) are used instead of FLOPs to measure the computational complexity.

[0102] Table 1 Comparative test of different numbers of LESNN channels

[0103]

[0104] Compared with ResNet with the same architecture, our model saves 137 times more energy, as shown in Table 2.

[0105] Table 2 Comparative test of LESNN and ResNet with the same architecture

[0106] Model Number of channels Accuracy (%) SynOps (M) Synaptic computational complexity (M) Params (M) Model parameters (M) ResNet 61.64 - 11.17 2.74 LESNN 68.67 52.43 0.18 0.02

[0107] In addition, characteristic graphs are drawn according to the pulse rate, such as Figure 4 As shown, the change in color from black to white represents the change in the pulse firing rate.

[0108] In the present invention, firstly, we need to take actions such as walking and waving as data sets, and store the spatiotemporal data flow information in the form of CSV files; then construct the LIF explicit iterative model to realize the pulse communication between neurons; considering the signal communication between two convolutional layers, the neurons are normalized after executing LIF once; the whole model consists of four stages, and a different number of lightweight modules are stacked in each stage; then the spatiotemporal data flow information is input into the model, the number of model iterations is set, it is iterated according to the loss function, and the STBP is used for back propagation training to obtain our weight file; after completing the training of the pulse neural network, the spatial data flow to be identified is transmitted to the network to predict the human posture recognition result. The key points of the present invention include the following aspects:

[0109] (1) Using pulse neural networks and spatiotemporal data streams as input to reduce energy consumption: Different from traditional image processing, due to the lack of information between picture frames, we abandoned the picture frames as the input of the model and used spatiotemporal data streams as input, fully considering the time series information. Only focusing on the state information of the moving object can make the model respond faster without causing the loss of information of the moving target. More importantly, it can greatly reduce the consumption of computing resources and realize the lightweight and energy-saving model.

[0110] (2) Establishing the LIF explicit iteration model: The LIF implicit differential equation is derived into an explicit form so that it can be directly calculated and differentiated, which can better optimize the problem and make the code easier to write. Then the LIF module is embedded between neurons. By setting LIF model parameters such as threshold voltage and leakage constant, the LIF module is used as a leakage-integration-ignition mechanism to simulate the behavior of biological neurons.

[0111] (3) The normalization function TEBN effectively extends the depth of the SNN model: The normalization function TEBN is a normalization method for SNN neurons. It scales the presynaptic input with different weights at each time step to make the SNN smoother in time distribution, which helps stabilize the gradient norm and can effectively extend the depth of the SNN model.

[0112] (4) LESNN model (model established by the present invention): By integrating the residual idea of ​​ResNet and the deep separable idea of ​​Mobile Net, a residual deep pulse convolution block LE Block is constructed, so that the model can be built deeper and is lighter than other SNN networks.

[0113] In general, the present invention is a recognition method based on low power consumption and light weight. With the continuous advancement of science and technology, sustainable development has become the hottest topic today and is the "golden key" to cracking current global problems. At the same time, the demand for human posture recognition using embedded devices or mobile devices is gradually increasing, such as live video broadcasting, game entertainment, and some intelligent dialogue robot devices. Therefore, it is increasingly important to achieve more convenient and efficient recognition of human posture. Traditional neural networks usually have more parameters and deeper network structures, which makes them more difficult to train and optimize. This may require more computing resources and longer training time, and it is difficult to achieve real-time interaction on some real-time interactive devices.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0115] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A human posture recognition method based on a lightweight pulse neural network, characterized in that: The steps include: S1. Establishing a data set: photographing human bodies in different postures, using the photographed images as data sets, and storing the spatiotemporal data stream information of the collected images in the form of comma-separated value files; S2. Constructing the leakage-integration-ignition LIF explicit iterative model; S3. construct a loss function and add a regularization term as a flexible restriction of the loss function; S4. Construct a low-power lightweight LESNN network based on a spiking neural network, so that the feature map undergoes four stages of feature extraction and dimension concatenation to obtain a lightweight spiking neural network model; S5. Set the initial parameters of the model and train the lightweight spiking neural network model; S6. Input the spatiotemporal data stream to be tested into the trained lightweight pulse neural network model, and obtain the posture state of the human body after detection and recognition; The explicit equations of the leakage-integration-ignition LIF explicit iteration model are: ; in, u is the membrane potential, n and l(n) represent the neuron number of the nth layer and the nth layer, respectively. is the synaptic weight from the jth neuron in the previous layer n to the ith neuron in the next layer (n+1), Indicates pulse output, for The next state of t represents the time step, is the given excitation threshold; The feature extraction and dimension splicing of the four stages in step S4 specifically include the following steps: S401. Visualize the data flow of the input feature map and write it into the data preprocessing, and generate the voxel grid of the corresponding time window according to the set time step; the input feature map first passes through a downsampling module to reduce the height and width of the feature map by 2 times, and the channel dimension is changed from 1 to 16 to obtain the feature map ; S402. Feature map Entering the LE Block1 in the first stage, the feature maps are obtained after passing through the residual path and the shortcut path respectively. and feature map ; In the channel dimension and Add together and get the shape Feature map ; S403. The feature map enters LE Block3 and passes through the residual path of LE Block3 to obtain a shape of Feature map ; The feature map and feature map before sending to LE Block3 Adding in the channel dimension, the shape is The feature map of S404. The feature map is sent to LE Block2 in the second stage. After passing through the residual path and the shortcut path, the residual path feature map and the shortcut path feature map are obtained respectively. The shortcut path feature map and the residual path feature map are added in the channel dimension to obtain a shape after downsampling by two times: Feature map; S405. The third and fourth stages repeat the operations of steps S403-S404, and finally obtain a shape of The feature map of , where T, C, H, and W are 5, 128, 8, and 11 respectively; S406. The feature map finally passes through an adaptive average pooling layer with an input of 128 and an output of 5 to complete the construction of the lightweight pulse neural network model.

2. The method for human posture recognition based on a lightweight spiking neural network according to claim 1, characterized in that: The residual paths of the LE Block 1 and the LE Block 2 are the same and both have two branches; After further downsampling through an LDCB block, the feature map height and width are changed to 60 and 87, and the number of channels is changed from 16 to 32, and the feature map is obtained. ; On the other branch, the maximum pooling operation is used to reduce the height and width of the feature map by 2 times to 60 and 87, while the number of channels remains unchanged, and the feature map is obtained. ; The characteristic graph passing through LDCB And the feature map after maximum pooling The feature maps are split into four groups in the channel dimension, and the four groups of feature maps are spliced ​​in the channel dimension to generate four groups of fused feature maps. ~ , the shape is , where T, C, G, H, and W are 5, 12, 4, 60, and 87 respectively; summing up in the grouping dimension realizes information fusion between different groups, and the generated shape is Feature map , feature map T, C, H, and W are 5, 12, 60, and 87 respectively; then the feature map Send it to the LCB block to further extract features and change the channel dimension from 12 to 16 to obtain the feature map .

3. The method for human posture recognition based on a lightweight pulse neural network according to claim 2, characterized in that: In the shortcut path of LE Block1, input feature map After a maximum pooling operation, the height and width are reduced by two times to 60 and 87, and then a LCB block with a convolution kernel of 1 is used to obtain the feature map. , feature map Shape and feature map .

4. The method for human posture recognition based on a lightweight pulse neural network according to claim 3, characterized in that: In the shortcut path of LE Block2, the feature map is downsampled through a maximum pooling operation, changing the height and width from 60 and 87 to 60 and 44, and then passed through an LCB block with a convolution kernel of 1 to obtain the feature map and concatenate it with the original feature in the channel dimension, changing the number of channels from 16 to 32.

5. A method for human posture recognition based on a lightweight spiking neural network according to any one of claims 1 to 4, characterized in that: The initial parameters include: the time step is set to 5, the threshold voltage is 0.4, leakage constant Set to 0.25, gradient approximation Set to 0.5, batch_size per GPU is set to 64, the number of epoch iterations is 300, and the learning rate is 0.

025.

6. The method for human posture recognition based on lightweight spiking neural network according to claim 5, characterized in that: The step S4 also includes performing time-effective batch normalization on the feature map during the construction of the lightweight spiking neural network model. The mathematical expression is as follows: ; in, , ; In the above formula, express No. k The feature map of each channel, , and They are The mean and variance calculated based on the dimension statistics data, The setting in the code is 1e-5. 、β is a constant, p[t] is a trainable weight.

7. The method for human posture recognition based on a lightweight spiking neural network according to claim 6, characterized in that: The loss function of step S3 adopts mean square error MSE; the regular term expression is as follows: ; In the formula, Y represents the label vector, t represents the current time step, T represents the total time step, λ represents the regularization hyperparameter, M represents the prediction matrix where each prediction neuron is connected to a specific class, represents the prediction vector of the last layer at time step t.

8. The method for human posture recognition based on lightweight pulse neural network according to claim 7, characterized in that: The human body postures include walking, running, waving, squatting and turning; the data set includes a training set and a test set, and the ratio of the training set to the test set is 8:2.