Gesture recognition method based on balanced active learning and nuclear norm regularization model

Abstract

The invention discloses a gesture recognition method based on balanced active learning and a nuclear norm regularization model. The method is characterized by comprising the following steps: 1, constructing the nuclear norm regularization model based on a pulse neural network; 2, training the nuclear norm regularization model by adopting a preposed self-supervision task and an active learning task; 3, acquiring gesture image data by an acquisition module; 4, a preprocessing module carries out preprocessing operation on the gesture image data; 5, coding the standard graphic data b by a coding module; 6, the input layer of the trained nuclear norm regularization model carries out convolution operation on the coded data; 7, performing feature extraction operation on the convolution data by a feature extraction module; 8, the adaptive average pooling layer carries out average pooling operation on the feature data; 9, the first full-connection layer performs full-connection operation on the pooling data to obtain full-connection data; and 10, performing full connection operation on the full connection data by an output layer, and outputting a gesture recognition result h. The method has the effect that the gesture recognition accuracy of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of spiking neural networks, and in particular to a gesture recognition method based on a balanced active learning and nuclear norm regularization model. Background Art

[0002] In recent years, active learning has been successfully applied to solve the problem of high cost of labeled data in the field of artificial neural networks (ANN). It is mainly divided into two categories: uncertainty-based active learning and class-distribution-based active learning. These methods have provided ideas for solving the problem of high cost of labeled data. However, the solution based on class imbalance may cause the model to be confused about the data points near the decision boundary, while the uncertainty-based strategy may sample the overlapping parts of the dataset, which may lead to the selected data not representing the true data distribution of the whole dataset. In addition, training a spiking neural network (SNN) model requires a large number of labeled Dynamic Vision Sensor (DVS) datasets. However, it is very difficult to obtain data in the SNN field and there is a lack of large-scale dynamic datasets. More importantly, because the feature representations and information transmissions of ANN and SNN are different, these methods are not applicable to SNN. Unfortunately, there are only two studies on active learning in the SNN field, both of which are based on uncertainty-based strategies. Therefore, there is an urgent need to develop more new active learning strategies in the SNN field to solve the problems of data annotation and difficulty in obtaining data. Therefore, this paper aims to propose a new active learning strategy applicable to SNN to help the model select more valuable samples for training, achieve optimal performance within a limited label budget, and thus solve the problems of data annotation and difficulty in obtaining data.

[0003] In addition, the binary spike activation mechanism of SNN inspired by biology makes SNN more power-efficient, and also makes the directly trained deep spiking neural network have characteristics such as discontinuity, non-invariance, time dynamic characteristics, and time dependence. This leads to a mismatch between the gradient descent method used by the directly trained SNN and the loss landscape, and the generalization ability is low. Currently, a large number of researchers have conducted research on the characteristics of SNN such as discontinuity, time dynamic characteristics, and time dependence. Although substantial progress has been made, the problem of poor model generalization caused by the discrete non-differentiability of SNN remains the main obstacle to the development of spiking. Therefore, it is necessary to design a new loss function to alleviate the problems of gradient mismatch and low model generalization.

[0004] Disadvantages of the prior art: Spiking neural networks are often applied to gesture recognition tasks. However, the mismatch of the loss landscape and the low model generalization in the directly trained spiking neural network will systematically lead to a low gesture recognition accuracy; in addition, training an SNN model requires a large number of labeled dynamic datasets, and the cost of obtaining labeled data is relatively high. Summary of the Invention

[0005] A gesture recognition method based on a balanced active learning and nuclear norm regularization model provided by the present invention can effectively improve the accuracy of gesture recognition results and reduce the model training cost.

[0006] To achieve the above object, the key of a gesture recognition method based on a balanced active learning and nuclear norm regularization model provided by the present invention is:

[0007] Step 1: Construct a nuclear norm regularization model F based on a spiking neural network NNR , where the nuclear norm regularization model F NNR is provided with an input layer, a hidden layer, and an output layer connected in sequence. The hidden layer is provided with a feature extraction module, an adaptive average pooling layer, and a first fully connected layer connected in sequence;

[0008] Step 2: Use a pre - self - supervised task A and an active learning task B to train the nuclear norm regularization model F NNR to optimize the model parameters and obtain a trained nuclear norm regularization model F NNR ;

[0009] Step 3: A collection module collects gesture image data a in real - time and transmits it to a pre - processing module;

[0010] Step 4: The pre - processing module performs pre - processing operations on the gesture image data a to obtain standard graphic data b and transmits it to an encoding module;

[0011] Step 5: The encoding module encodes the standard graphic data b to obtain encoded data c and transmits it to the trained nuclear norm regularization model F NNR ;

[0012] Step 6: The input layer of the trained nuclear norm regularization model F NNR obtains the encoded data c, performs a convolution operation on it to obtain convolution data d, and transmits it to the feature extraction module;

[0013] Step 7: The feature extraction module performs feature extraction operations on the convolution data d to obtain feature data e and transmits it to the adaptive average pooling layer;

[0014] Step 8: The adaptive average pooling layer performs average pooling operations on the feature data e to obtain pooled data f and transmits it to the first fully connected layer;

[0015] Step 9: The first fully connected layer performs a fully connected operation on the pooled data f to obtain fully connected data g and transmits it to the output layer;

[0016] Step 10: The output layer performs a fully connected operation on the fully connected data g and outputs a gesture recognition result h.

[0017] The preprocessing operation is to adjust the spatial resolution of the gesture image data a to 48×48, which can avoid the memory consumption of the model caused by the excessive original data and improve the operation speed of the model.

[0018] Through the above design, the local density clustering distance is firstly introduced in the pre - self - supervised task to measure different data and solve the problem of data overlap; secondly, the balanced active learning strategy based on the spiking neural network is used to solve the problem of data confusion; finally, the training algorithm with nuclear norm regularization is introduced to effectively improve the generalization performance and accuracy of the model.

[0019] Preferably: in the step 1, the feature extraction module is provided with 8 feature extraction blocks, and the 8 feature extraction blocks have the same structure and are connected in sequence at the head and tail;

[0020] Each feature extraction block is provided with a first convolutional layer, a first batch normalization layer, a first spiking neuron layer, a second convolutional layer, a second batch normalization layer, a residual connection layer and a second spiking neuron layer connected in sequence.

[0021] The feature extraction module is used to extract the data features of the input samples; the adaptive average pooling layer is used to calculate the sufficient statistics of the features locally, so as to reduce the overall number of features, prevent overfitting and reduce the amount of calculation; and reduce the error by taking the average value, maintain the invariance of rotation, translation, scaling, etc., and reduce the increase in the variance of the estimated value caused by the limited neighborhood size during the image feature extraction process.

[0022] Preferably: in the step 2, the pre - self - supervised task A includes the following steps:

[0023] Step A1: The acquisition module obtains an unlabeled data set X with N samples and encodes it, and uses the data set X to train the nuclear norm regularization model F NNR for training;

[0024] Step A2: Calculate the local density clustering distance δ of each data point in the data set X in the nuclear norm regularization model F NNR and sort the data set X in ascending order according to the local density clustering distance δ to obtain an ordered data set

[0025] The pre - supervision task is mainly used to cluster data, and through the pre - supervision task, the model can better extract features.

[0026] Preferably, in the step 2, the active learning task B includes the following steps:

[0027] Step B1: Calculate the loss value LOSS of the nuclear norm regularization model F NNR after the r-th training, set the balance factor μ according to the loss value LOSS, and then generate a corresponding sub-pool through the balance factor μ

[0028] Step B2: Select K samples from the adaptive sub-pool for labeling according to the posterior probability of the r-th training;

[0029] Step B3: Repeat steps B1 - B2. When the number of labeled samples reaches the label budget, the training is completed.

[0030] Preferably, the nuclear norm regularization model F NNR is trained using the LIF neuron model. The forward iteration formula of the LIF neuron model is as follows:

[0031]

[0032] where l represents the number of neurons in the l-th layer of the spiking neural network, k τ represents the decay factor, represents the membrane potential of the i-th neuron in the n-th layer at time t, represents the output of the i-th neuron in the l-th layer of the spiking neural network at time t, represents the presynaptic input, t represents the time constant, and f() is the activation function; V th is the firing threshold. When the neuron membrane potential u exceeds the given threshold V th , the neuron emits a pulse and resets the membrane potential to zero; represents the synaptic weight, is the bias, and j represents the identifier of any data point;

[0033] From formula (2), we can obtain Therefore, the backpropagation calculation gradient formula of the LIF neuron model is as follows:

[0034]

[0035] where is the second factor, representing the partial derivative of the output of the LIF neuron with respect to the membrane potential; is the first factor, representing the partial derivative of the loss value loss with respect to the output of the previous layer of neurons; is the third factor, representing the partial derivative of the presynaptic input with respect to the synaptic weight; L NNR represents the loss value LOSS, Represents the labeled data p y,Θ The probability corresponding to the label y, p y,Θ Represents the labeled data, where T is the length of the pulse sequence Represents the calculation of the partial derivative

[0036] Preferably: The output layer calculation formula of the nuclear norm regularization model F NNR is as follows:

[0037]

[0038] The hidden layer calculation formula is as follows:

[0039]

[0040] Preferably: The hidden layer calculates and updates the neuron membrane potential through formulas (1)-(3). Each feature extraction block outputs features to the output layer, and finally the firing rate of each neuron is output through the output layer. The expression is as follows:

[0041]

[0042] Wherein, represents the probability corresponding to the label y i ; x i represents the i-th data point, that is, the fully connected data g

[0043] The output layer calculates the probability distribution values corresponding to various gesture category labels, and takes the gesture category label corresponding to the maximum probability distribution value as the gesture recognition result h for output.

[0044] Preferably: In step A3, according to the output result of the first fully connected layer, the calculation process of the local density clustering distance δ of each data point in the dataset X is as follows:

[0045] The dataset X of N points is represented as The distance threshold d c The calculation expression is as follows:

[0046]

[0047] Wherein, D i represents the distance between the point x i and its nearest neighbor. The density ρ of the point x i The calculation expression is as follows: i is as follows:

[0048]

[0049] Wherein, d ij represents x iand x j The Euclidean distance between;

[0050] For a given data set Point x i The local density clustering distance δ of i The calculation expression is as follows:

[0051]

[0052] Wherein, Is a mathematical symbol indicating arbitrary, Indicates that for any data point j, the formula (10) is satisfied.

[0053] Preferably: in the step B1, the nuclear norm regularization model F NNR The loss value LOSS calculation formula is as follows:

[0054]

[0055] Wherein, h = 4, λ represents a hyperparameter, L CE Is the cross-entropy loss, ‖‖ * Is a mathematical symbol for calculating the nuclear norm, Represents calculating the output value Of the nuclear norm; Θ represents the rotation angle;

[0056]

[0057] Represents corresponding to the label y i Probability of.

[0058] In order to improve the generalization performance of the model, a penalty term is added at each time step to constrain the output distribution at each moment.

[0059] Preferably: in the step B1, the ordered data set X S Generates a sub-pool The formula is as follows:

[0060]

[0061] Wherein, μ is a balance factor, L is the labeled pool in the r-th training, N is the total number of images in the data set X, R is the total number of training times, and the k-th sample in the ordered data set X S After sorting is represented as x k ;

[0062] The number of samples in each sub-pool is:

[0063]

[0064] Among them, the minimum value of μ is At least K samples are labeled each time of training;

[0065] In the first training, the sampler φ selects the first K data from the sorted unlabeled pool X S Among them. In the r-th (r > 1) training, the sampler φ selects K data with the smallest maximum posterior probability, which is calculated by of Therefore, the newly labeled data is

[0066]

[0067] Among them, is the output of the nuclear norm regularization model F NNR after the (r - 1)-th training, is the r-th sub-pool, and K is the number of data to be labeled in each cycle.

[0068] Advantages of the present invention:

[0069] 1. The local density clustering distance (LDCD) is introduced as an effective measure for the amount of information carried by data, providing important reference data for screening more contributing data for model training. By selecting a subset of data, similar performance can be achieved with part of the data as with the entire dataset, reducing the model's dependence on a large amount of training data and the training cost of the gesture recognition model;

[0070] 2. A balanced active learning strategy based on spiking neural networks is proposed. This strategy takes into account the diversity and uncertainty of data under different label budgets by setting different balance factors, and selects the data with the largest amount of information carried for model training;

[0071] 3. Aiming at the problems of loss pattern mismatch and low model generalization in directly trained spiking neural networks, the present invention introduces a penalty term of nuclear norm in the cross-entropy loss function for training, effectively enhancing the model's generalization ability and improving the gesture recognition accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a schematic diagram of the overall process of the present invention;

[0073] Figure 2 is a schematic diagram of the training process of the present invention;

[0074] Figure 3 is a schematic diagram of the active learning task in the present invention;

[0075] Figure 4It is a flowchart of the working process of the nuclear norm regularization training algorithm;

[0076] Figure 5 It is a comparison chart of test results with a medium label budget in the embodiment;

[0077] Figure 6 It is a performance comparison chart of various methods under the same labeling ratio in the embodiment;

[0078] Figure 7 It is a comparison chart of test results with a small label budget in the embodiment;

[0079] Figure 8 It is a comparison chart of the accuracy of test results between BAL-NNR and SAL in the embodiment;

[0080] Figure 9 It is an influence diagram of the BAL-NNR model on the loss curves of CIFAR10, DVS128 Gesture tests, and CIFAR10DVS tests in the embodiment;

[0081] Figure 10 It is a t-SNE visualization diagram of the sample distribution on the CIFAR10 dataset;

[0082] Figure 11 It is a feature diagram of the visualized SNN model without the nuclear norm constraint algorithm in the embodiment;

[0083] Figure 12 It is a feature diagram of the visualized SNN model with the nuclear norm constraint algorithm in the embodiment;

[0084] Figure 13 It is a comparison chart of the LDCD sorting and CDD sorting effects on the CIFAR10, DVS128 Gesture, and CIFAR10DVS datasets in the embodiment;

[0085] Figure 14 It is a comparison chart of the test loss values of the model with cross-entropy as the loss function and the model with the nuclear norm constraint loss function in the embodiment;

[0086] Figure 15 It is the structural schematic diagram of the nuclear norm regularization model F NNR of. Detailed implementation mode

[0087] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples. The following examples or drawings are used to illustrate the present invention, but not to limit the scope of the present invention.

[0088] As Figure 1 shown: A gesture recognition method based on a balanced active learning and nuclear norm regularization model includes the following steps:

[0089] Step 1: Construct a nuclear norm regularization model F based on a spiking neural network NNR , where the nuclear norm regularization model F NNR is provided with an input layer, a hidden layer, and an output layer connected in sequence. The hidden layer is provided with a feature extraction module, an adaptive average pooling layer, and a first fully connected layer connected in sequence;

[0090] Step 2: Use a pre - self - supervised task A and an active learning task B to train the nuclear norm regularization model F NNR , optimize the model parameters, and obtain a trained nuclear norm regularization model F NNR ;

[0091] Step 3: The acquisition module collects gesture image data a in real - time and transmits it to the pre - processing module;

[0092] Step 4: The pre - processing module performs pre - processing operations on the gesture image data a to obtain standard graphic data b and transmits it to the encoding module;

[0093] Step 5: The encoding module encodes the standard graphic data b to obtain encoded data c and transmits it to the trained nuclear norm regularization model F NNR ;

[0094] Step 6: The input layer of the trained nuclear norm regularization model F NNR obtains the encoded data c, performs a convolution operation on it to obtain convolution data d, and transmits it to the feature extraction module;

[0095] Step 7: The feature extraction module performs feature extraction operations on the convolution data d to obtain feature data e and transmits it through the adaptive average pooling layer;

[0096] Step 8: The adaptive average pooling layer performs average pooling operations on the feature data e to obtain pooled data f and transmits it to the first fully connected layer;

[0097] Step 9: The first fully connected layer performs a fully connected operation on the pooled data f to obtain fully connected data g and transmits it to the output layer;

[0098] Step 10: The output layer performs a fully connected operation on the fully connected data g and outputs a gesture recognition result h.

[0099] As Figure 15 (a) shows, the input layer is a convolutional layer, the output layer is a fully connected layer, and the feature extraction module is provided with 8 feature extraction blocks. The structures of the 8 feature extraction blocks are the same and are connected in sequence at the head and tail;

[0100] As Figure 15As shown in (b), each of the feature extraction blocks is provided with a first convolutional layer, a first batch normalization layer, a first spiking neuron layer, a second convolutional layer, a second batch normalization layer, a residual connection layer, and a second spiking neuron layer that are connected in sequence. Figure 15 Among them, Conv represents the convolutional layer, BN represents the batch normalization layer, LIF1 represents the spiking neuron layer, and ⊕ represents the residual connection layer.

[0101] The nuclear norm regularization model F NNR The specific network parameters are shown in the following table:

[0102]

[0103] Among them, conv1 represents the input layer, Block1-8 represents eight feature extraction blocks connected in sequence, AdaptiveAvgPool represents the adaptive average pooling layer, FC1 represents the first fully connected layer, and FC2 represents the output layer.

[0104] As Figure 2 shown: In step 2, the pre - self - supervised task A includes the following steps:

[0105] Step A1: The acquisition module obtains an unlabeled dataset X with N samples and encodes it, and uses the dataset X to train the nuclear norm regularization model F NNR ;

[0106] Step A2: Calculate the local density clustering distance δ of each data point in the dataset X in the nuclear norm regularization model F NNR , and sort the dataset X in ascending order according to the local density clustering distance δ to obtain an ordered dataset

[0107] In step 2, the active learning task B includes the following steps:

[0108] Step B1: Calculate the loss value LOSS of the nuclear norm regularization model F NNR after the r - th training, set the balance factor μ according to the loss value LOSS, and then generate a corresponding sub - pool through the balance factor μ

[0109] Step B2: Select K samples from the adaptive sub - pool for labeling according to the posterior probability of the r - th training;

[0110] Step B3: Repeat steps B1 - B2. When the number of labeled samples reaches the label budget, the training is completed.

[0111] The nuclear norm regularization model F NNRTraining is carried out using the LIF neuron model. The forward iteration formula of the LIF neuron model is as follows:

[0112]

[0113] where l represents the number of neurons in the l-th layer of the spiking neural network, and k τ represents the decay factor, represents the membrane potential of the i-th neuron in the n-th layer at time t, represents the output of the i-th neuron in the l-th layer of the spiking neural network at time t, represents the presynaptic input, t represents the time constant, and f() is the activation function; V th is the firing threshold. When the neuron membrane potential u exceeds the given threshold V th , the neuron emits a pulse and resets the membrane potential to zero; represents the synaptic weight, is the bias, and j represents the identifier of any data point;

[0114] From formula (2), we can obtain Therefore, the backpropagation calculation gradient formula of the LIF neuron model is as follows:

[0115]

[0116] where, is the second factor, representing the partial derivative of the output of the LIF neuron with respect to the membrane potential; is the first factor, representing the partial derivative of the loss value loss with respect to the output of the previous layer of neurons; is the third factor, representing the partial derivative of the presynaptic input with respect to the synaptic weight; L NNR represents the loss value LOSS, represents the labeled data p y,Θ corresponds to the probability of the label y, p y,Θ represents the labeled data, T is the length of the pulse sequence, represents calculating the partial derivative.

[0117] The output layer calculation formula of the nuclear norm regularization model F NNR is as follows:

[0118]

[0119] The hidden layer calculation formula is as follows:

[0120]

[0121] The second factor It is about Dirac's δ - function, which is a non - differentiable step function and is usually approximated by a differentiable curve, i.e., gradient substitution. Currently, there are various shapes of substitution gradients, such as triangular, exponential, and rectangular curves. In this embodiment, a rectangular substitution gradient is selected. Mathematically, it can be described as:

[0122]

[0123] where a is a hyperparameter that determines the width of h(·), and the available interval of the gradient can be expressed as a = 1.

[0124] The third factor is derived as:

[0125]

[0126] The hidden layer calculates and updates the neuron membrane potential through formulas (1) - (3). Each of the feature extraction blocks outputs features to the output layer. Finally, the firing rate of each neuron is output through the output layer, and the expression is as follows:

[0127]

[0128] where, represents the probability corresponding to the label y i ; x i represents the i - th data point, that is, the fully - connected data g;

[0129] The output layer calculates the probability distribution values corresponding to various gesture category labels and outputs the gesture category label corresponding to the maximum value of the probability distribution as the gesture recognition result h.

[0130] In step A3, according to the output result of the first fully - connected layer, the calculation process of the local density clustering distance δ of each data point in the dataset X is as follows:

[0131] The dataset X of N points is represented as The calculation expression of the distance threshold d c is as follows:

[0132]

[0133] where D i represents the distance between the point x i and its nearest neighbor. The density ρ i of the point x i is calculated as follows:

[0134]

[0135] Among them, d ij represents the Euclidean distance between i x j and x

[0136] The above formula only needs to use the neighborhood points with a radius of d c to calculate the local density of each point. This greatly reduces the amount of calculation and reduces the influence of the main cluster on the density of small clusters. The local density calculated by this formula is more reasonable.

[0137] For a given data set the calculation expression of the local density clustering distance δ i of the point x i is as follows:

[0138]

[0139] Among them, is a mathematical symbol indicating any, indicating that for any data point j, the formula (10) is satisfied.

[0140] The larger the value of δ, the greater the amount of information carried, that is, the most difficult samples to distinguish. Therefore, finally, the data set X is sorted in ascending order of δ to obtain the data X s .

[0141] In the step B1, the loss value LOSS calculation formula of the nuclear norm regularization model F NNR is as follows:

[0142]

[0143] Among them, h = 4, λ represents a hyperparameter, L CE is the cross-entropy loss, ‖‖ * is a mathematical symbol for calculating the nuclear norm, represents the calculation of the output value of the nuclear norm; Θ represents the rotation angle;

[0144]

[0145] represents the corresponding probability of the label y i .

[0146] In the step B1, the formula for generating the sub-pool S from the ordered data set X is as follows:

[0147]

[0148] Among them, μ is the balance factor, L is the labeled pool in the r-th training, N is the total number of images in the dataset X, R is the total number of training times, and the ordered dataset X S The k-th sample in is denoted as x after sorting k ;

[0149] The number of samples in each sub-pool is:

[0150]

[0151] Among them, the minimum value of μ is At least K samples are labeled each time of training;

[0152] In the first training, the sampler φ selects the first K data from the sorted unlabeled pool X S In the r-th (r > 1) training, the sampler φ selects K data with the smallest maximum posterior probability, which is calculated by of Therefore, the newly labeled data is

[0153]

[0154] Among them, is the output of the nuclear norm regularization model F NNR after the (r - 1)-th training, is the r-th sub-pool, and K is the number of data to be labeled in each cycle.

[0155] The present invention balances the uncertainty and diversity of sampling data through the balance factor μ. The present invention allows adjacent sub-pools to have overlap (μ < 1) or interval (μ > 1), as shown in Figure 2 (b). Therefore, the length of each sub-pool can dynamically meet the requirements of different annotation budgets. The balance factor μ is selected based on the maximum performance of the main task model F trained on n , and the expression is as follows:

[0156]

[0157] Among them is trained on , which is quite small, only 15% in ordinary experiments. Therefore, the balance between diversity and uncertainty can be easily achieved without too much computational effort.

[0158] Next, the corresponding active learning strategy is analyzed according to different balance factors μ. It can be seen from Equation (13) that when 2 < i < I, ends at the samples, and starting from the th sample.

[0159] Case 1: μ > 1. In this case, it can be obtained that So there exists a total labeled subpool and There is an overlapping part. Since some samples may have been labeled in previous cycles, it is necessary to delete from as shown in Equation (13). as shown in Equation (13).

[0160] Case 2: When μ < 1, there exists an interval and and Therefore, Equation (13) can be simplified to

[0161]

[0162] Case 3: μ = 1. In this case, there is and there is no overlap or gap between adjacent subpools, which indicates that is the subpool after evenly dividing the sorted dataset X S . Therefore, Equation (13) degenerates into an evenly divided subpool:

[0163]

[0164] where I is the number of cycles and N represents the number of images. The k-th sample in the sorted dataset X S is denoted as x k .

[0165] Next, various experiments were conducted with different label budgets and different angles to prove the effectiveness of the gesture recognition method based on balanced active learning and nuclear norm regularization model proposed in the present invention, and this method was compared with the random sample selection method, confidence-based active learning, ActiveFT, the current SOTA active learning model in the field of spiking neural networks, and the class balance-based active learning baseline.

[0166] (I) Experimental data: The experiments were conducted on static and dynamic datasets, including:

[0167] 1. Static dataset

[0168] 1) The CIFAR-10 dataset has a total of 60,000 color images with an image size of 32*32*3. The training set has 50,000 samples and the test set has 10,000 samples. CIFAR-10 has 10 categories, namely airplane, dog, bird, car, etc., and the span between categories is large.

[0169] 2. Dynamic Datasets

[0170] 1) The DVS128Gesture dataset contains 11 gesture categories from 28 subjects captured by a DVS128 camera, with 1176 training samples and 288 test samples respectively, and the captured image size is 128*128.

[0171] 2) The CIFAR10-DVS dataset is obtained by converting 10,000 frame-based images in the CIFAR10 dataset into event-stream-based images through a dynamic vision sensor (DVS), thus obtaining an event-stream-based dataset CIFAR10-DVS with medium difficulty. These event-stream data contain rich temporal information. Since the original CIFAR10-DVS dataset contains a large number of events in one event stream and has a resolution of 128×128, directly inputting it into the SNN without preprocessing will cause huge memory consumption and extend the training time. Therefore, the spatial resolution of the CIFAR10-DVS dataset is adjusted to 48×48, and the dataset is divided into 9k training samples and 1k test samples.

[0172] (II) Comparison methods: The BAL-NNR of the present invention is compared with the following active learning methods:

[0173] 1) Confidence: The uncertainty of a sample is predicted using the highest probability score.

[0174] 2) Random sampling: This is the simplest baseline.

[0175] 3) The SOTA active learning model for spiking neural networks: Xie et al. used the loss prediction module ActiveLossNet to extract features of deep SNNs and select valuable samples for training.

[0176] 4) ActiveFT: This method ensures class balance by selecting a data subset that is similar to the data distribution of the entire unlabeled pool and has sufficient representativeness.

[0177] 5) BAL: A balanced active learning strategy in the field of artificial neural networks.

[0178] (III) Experimental configuration: Use the hot start technique to mark the initial sample set and use ResNet19 as the backbone network. The experimental platform used is Intel(R) Core(TM) i9-14900K@6.0GHz, NVIDIA GeForce RTX 3090 24G*2, 64-bitUbuntu 20.04. In each cycle, the model is trained from the weights of the most recent cycle. The average accuracy of three runs is calculated. Other detailed configurations are shown in Table 1.

[0179] Table 1 Experimental parameters

[0180]

[0181]

[0182] 1) Hyperparameters: membrane potential time constant τ in the experiment m Set to 0.5, the neural firing threshold V th Set to 0.5. The resting potential V reset Set to 0.0.

[0183] 2) Proportion of labeled data: The criterion for evaluating the performance of active learning methods is to use a fixed proportion of labeled data to The accuracy of In the following experiments, the medium label budget will perform 7 rounds of active learning iterations. In each iteration, 10%, 15%, 20%, 25%, 30%, 35%, and 40% of the data will be selected from the unlabeled dataset and added to the training set as the input data for the next iteration model. The small label budget will perform 5 rounds of active learning iterations. In each iteration, 2%, 4%, 6%, 8%, and 10% of the data will be selected from the unlabeled dataset and added to the training set as the input data for the next iteration model. In each iteration, the SNN model is retrained.

[0184] (IV) Medium label budget: Figure 5 As shown, when performed on CIFAR10, DVS128Gesture, and CIFAR10DVS, where the ranking metric for both sub-pools uses LDCD, the adaptive sub-pool shows a large advantage.

[0185] As shown in Table 2, Figure 6As shown in the figure, the effectiveness of the present invention is tested under a medium label budget. The performance of various methods under different labeling ratios ω is compared. The results show that BAL-NNR is superior to other methods under a medium label budget, especially for dynamic data sets. On DVS128Gesture, the performance of BAL-SNN with ω = 35% exceeds that of Random with ω = 40% by 2.08%, is 0.11% higher than that of BAL with ω = 40%, and saves 10% of data annotation. On CIFAR10, BAL-SNN with ω = 20% defeats ActiveFT with ω = 40%, the performance of BAL-SNN with ω = 30% is close to that of confidence with ω = 40%, saves 10% of data annotation, and the performance of BAL-SNN with ω = 40% is 0.86% higher than that of BAL and 1.35% higher than that of Random. On CIFAR10DVS, the performance of BAL-SNN with ω = 30% exceeds the performance of Confidence, ActiveFT, Random, and BAL with ω = 40% by 1.26%, 1.44%, 1.68%, and 0.99% respectively, and saves at least 10% of data annotation.

[0186] Table 2

[0187]

[0188] (V) Small label budget: When the budget is small, querying typical examples is more beneficial for training, and when the budget is large, querying atypical or non-representative examples is more beneficial for training. To further verify the effectiveness of the BAL-NNR method of the present invention, experiments were conducted under a small label budget. The results are shown in Table 3, Figure 7 As shown in the figure, the performance comparison between different methods under different labeling ratios ω. On DVS128Gesture, BAL-NNR with ω = 10% is 10.07%, 9.73%, 9.38%, and 3.82% higher than the methods of selecting data based on confidence, ActiveFT, Random, and BAL respectively. On CIFAR10, BAL-NNR with ω = 8% is 10.11% higher than the second-best method and 17.24% higher than the ActiveFT method. On CIFAR10DVS, with ω = 10%, our method is 15.78%, 10.08%, 17.48%, and 1.42% higher than the methods of selecting data based on confidence, ActiveFT, Random, and BAL respectively.

[0189] Table 3

[0190]

[0191] (6) Comparison of active learning methods for the latest spiking neural networks: Compare the BAL-NNR of the present invention with the active learning method (SAL) of the current state-of-the-art spiking neural networks on CIFAR10. For the convenience of comparison, the SNN-ResNet18 model is used in both cases, and T = 12. The comparison results are as Figure 8 shown. Compared with the SAL method, the accuracy of the BAL-NNR method is more excellent in most cases. When the amount of labeled data is 1W, the accuracy of BAL-NNR is 2.63% higher than that of SAL.

[0192] (7) Convergence and effectiveness of the loss function: On CIFAR10, DVS128Gesture, and CIFAR10DVS, use the BAL-NNR model to test the loss curve. Set 200 epochs, for a total of 1400 epochs. As Figure 9 shown, at the beginning of each round of active learning training, due to the small amount of training data, the loss value is very high. As more and more data is added, the loss value gradually decreases. It can be seen that in the early stage of active learning training, the training loss has a large downward trend, and in the later stage of active learning, the loss value changes slowly. This indicates that the data selected by the active learning strategy in the early stage of the active learning algorithm is very effective.

[0193] In addition, in the first 100 epochs of each round, due to insufficient training samples, the loss curve does not converge completely stably. As new unlabeled samples are labeled and added, the loss increases sharply at the first epoch of each round. Then in the later stage, the loss curve gradually converges. Although all methods show the same general trend, the loss of the present invention is smaller in each epoch, and the loss function fluctuates less and is more stable.

[0194] (8) Visualization analysis: Use the t-SNE dimensionality reduction technique to implement the sample distribution diagrams of the first iteration and the last iteration of CIFAR10 as Figure 10 shown. As Figure 10 (a) shows, the feature separation between different categories of samples is not good enough at the beginning of the iteration. As Figure 10 (b) shows, after the first iteration, valuable outlier samples are significantly selected and marked with a yellow square. As Figure 10 (c) shows, after multiple iterations, the trained SNN model has better sample classification ability.

[0195] To further analyze the high similarity between each category, the feature maps of the SNN model without nuclear norm constraint algorithm and the algorithm with nuclear norm constraint are visualized in Figure 11 、 Figure 12 . As Figure 11 、 Figure 12As shown, the first row represents the example images of each category. The second and third rows show the feature representations before and after being directly encoded into pulses, and the graphs in the last four rows represent the feature representations after each block of the SNN model. It can be seen that, like traditional artificial neural network models, as the number of network layers increases, the features of the SNN become more and more abstract. In addition, to improve the model efficiency, T = 6 is set.

[0196] Comparison Figure 11 and Figure 12 It can be found from the second and third rows of and that since SNNs rely on time-series pulse signals for information transmission, after using the training algorithm with nuclear norm constraint, the network can better optimize the generation method of input pulses, making the pulse sequence generated by the encoding layer more accurate and representative. This means that the L NNR loss function can guide the network to better capture key features at the input end and reduce unnecessary noise. Comparison Figure 11 and Figure 12 It can be found from the fourth, fifth, and sixth rows of and that the improved loss function reduces the over-excitation of the network in the intermediate layer or the response to irrelevant features, making the neurons more focused on the features related to the target, reducing the information loss on shallow features and the redundancy on deep features. At the same time, it is found that the model of the present invention can more stably extract effective features for different input samples, indicating that using the L NNR loss function has better generalization performance for different input samples. Thus, it avoids the network falling into local minima and improves the generalization and accuracy of the network.

[0197] (IX) Ablation Experiment

[0198] An ablation experiment is conducted on the CIFAR10 dataset.

[0199] ① Figure 13 (a) shows the comparison of the metrics for dividing batches on CIFAR10. Two sub-pools use LDCD and CDD as sorting metrics respectively. The data therein are the average accuracies calculated for three runs.

[0200] The present invention proposes a standard for sorting the sub-pool , that is, LDCD. Experiments prove that the LDCD sorting has better effect than the CDD sorting.

[0201] ② Sampling Comparison

[0202] Three different sampling methods are compared, including (i) random sampling, (ii) a sampler based on K-means clustering, which uses K-means clustering features and samples the data closest to the cluster center. (iii) The sampler based on local density clustering in the present invention, which uses the method of local density clustering to sample through LDCD(δ) sorting. As Figure 13(As shown in (b), it can be seen that the sampler proposed by the present invention based on local density clustering has the best performance.)

[0203] To verify the effectiveness of introducing the penalty term nuclear norm regularization term in the cross-entropy loss function, the present invention conducted ablation experiments on the CIFAR10 dataset, and other parameter settings are shown in Table 1. As Figure 13 (As shown in (c), L CE represents using only the cross-entropy as the loss function, and L NNR represents the new loss function that introduces the penalty term nuclear norm in the cross-entropy loss function. From Figure 13 (c), it can be seen that using L NNR effectively improves the accuracy of model training. When ω is less than 25%, using the improved L NNR has a more obvious advantage than using L CE . This shows that the new loss function L NNR proposed by the present invention can make the model training more advantageous.)

[0204] To further verify the effectiveness of the loss function proposed by the present invention, the test loss values of the model using only the cross-entropy as the loss function and the loss function model with nuclear norm constraint are compared. As Figure 14 shown, experiments were conducted on the CIFAR10 dataset using ResNet19, and it was found that when using the L CE loss function, the loss function of each round has a slightly oscillating upward trend after 100 epochs. When using the L NNR loss function, the loss function of each round is lower than that of L CE , and the loss value of each round shows an oscillating downward trend. Generally speaking, the L NNR loss function proposed by the present invention has significant advantages over the L CE loss function in terms of stability, generalization ability, and finding the global minimum.)

[0205] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.)

Claims

1. A gesture recognition method based on balanced active learning and nuclear norm regularization model, characterized in that: The following steps are involved: Step 1: Construct a nuclear norm regularization model F based on a spiking neural network NNR , the nuclear norm regularized model F NNR An input layer, a hidden layer and an output layer connected in sequence are provided, wherein the hidden layer is provided with a feature extraction module, an adaptive average pooling layer and a first fully connected layer connected in sequence; Step 2: Use the pre-self-supervised task A and the active learning task B to regularize the nuclear norm model F NNR Perform training, optimize model parameters, and obtain the trained nuclear norm regularization model F NNR ; Step 3: The acquisition module acquires gesture image data a in real time and passes it to the preprocessing module; Step 4: the preprocessing module performs a preprocessing operation on the gesture image data a to obtain standard graphic data b and transmits it to the encoding module; Step 5: The encoding module encodes the standard graphic data b to obtain the encoded data c, and passes it to the trained nuclear norm regularization model F NNR ; Step 6: The trained nuclear norm regularization model F NNR The input layer obtains the encoded data c, performs a convolution operation on it, obtains the convolution data d, and passes it to the feature extraction module; Step 7: The feature extraction module performs a feature extraction operation on the convolution data d to obtain feature data e, and passes it through the adaptive average pooling layer; Step 8: The adaptive average pooling layer performs an average pooling operation on the feature data e to obtain pooled data f, and passes it to the first fully connected layer; Step 9: The first fully connected layer performs a fully connected operation on the pooled data f to obtain fully connected data g, and passes it to the output layer; Step 10: The output layer performs a full connection operation on the fully connected data g and outputs a gesture recognition result h.

2. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 1, characterized in that: In step 1, the feature extraction module is provided with 8 feature extraction blocks, the 8 feature extraction blocks have the same structure and are connected head to tail in sequence; Each of the feature extraction blocks is provided with a first convolutional layer, a first batch of normalization layers, a first pulse neuron layer, a second convolutional layer, a second batch of normalization layers, a residual connection layer and a second pulse neuron layer which are connected in sequence.

3. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 2, characterized in that: In step 2, the pre-self-supervision task A includes the following steps: Step A1: The acquisition module obtains and encodes an unlabeled data set X with N samples, and uses the data set X to regularize the nuclear norm model F NNR Conduct training; Step A2: Regularize the model F in the nuclear norm NNR The local density clustering distance δ of each data point in the data set X is calculated, and the data set X is sorted in ascending order according to the local density clustering distance δ to obtain an ordered data set.

4. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 2, characterized in that: In step 2, the active learning task B includes the following steps: Step B1: Calculate the nuclear norm regularization model F NNR The loss value LOSS after the rth training, and the balance factor μ is set according to the loss value LOSS, and then the corresponding sub-pool is generated by the balance factor μ Step B2: According to the posterior probability of the rth training, K samples are selected from the adaptive sub-pool for labeling; Step B3: Repeat steps B1-B2. When the number of labeled samples reaches the label budget, the training is completed.

5. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 3 or 4, characterized in that: The nuclear norm regularization model F NNR The LIF neuron model is used for training, and the forward iteration formula of the LIF neuron model is as follows: Among them, l represents the number of neurons in the l-th layer of the spiking neural network, k τ represents the attenuation factor, represents the membrane potential of the ith neuron in the nth layer at time t, represents the output of the i-th neuron in the l-th layer of the spiking neural network at time t, represents the presynaptic input, t represents the time constant, f() is the activation function; V th is the firing threshold. When the neuron membrane potential u exceeds a given threshold V th When , the neuron fires a pulse and resets the membrane potential to zero; represents the synaptic weight, is the bias, j represents the identifier of any data point; From formula (2), we can get Therefore, the back propagation calculation gradient formula of the LIF neuron model is as follows: in, is the second factor, which represents the partial derivative of the output of the LIF neuron with respect to the membrane potential; The first factor represents the partial derivative of the loss value loss with respect to the output of the previous layer of neurons; is the third factor, which represents the partial derivative of the presynaptic input to the synaptic weight; L NNR Indicates the loss value LOSS, Represents the labeled data p y,Θ Corresponding to the probability of label y, p y,Θ represents labeled data, T is the length of the pulse sequence, Denotes that the partial derivatives are calculated.

6. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 5, characterized in that: The nuclear norm regularization model F NNR The output layer calculation formula is as follows: The hidden layer calculation formula is as follows:

7. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 6, characterized in that: The hidden layer calculates and updates the neuron membrane potential through formulas (1)-(3), each feature extraction block outputs features to the output layer, and finally the firing rate of each neuron is output through the output layer, and the expression is as follows: in, Indicates the corresponding label y i The probability of x i Represents the i-th data point, i.e., the fully connected data g; The output layer calculates the probability distribution values ​​corresponding to various gesture category labels, and outputs the gesture category label corresponding to the maximum value of the probability distribution as the gesture recognition result h.

8. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 3, characterized in that: In step A3, according to the output result of the first fully connected layer, the calculation process of calculating the local density clustering distance δ of each data point in the data set X is as follows: The dataset X of N points is represented as Distance threshold d c The calculation expression is as follows: Among them, D i Represents point x i The distance between point x and its nearest neighbor i The density ρ i The calculation expression is as follows: Among them, d ij Represents x i and x j The Euclidean distance between For a given data set Point x i The local density clustering distance δ i The calculation expression is as follows: in, is a mathematical symbol representing any It means that formula (10) is satisfied for any data point j.

9. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 4, characterized in that: In step B1, the nuclear norm regularization model F NNR The loss value LOSS calculation formula is as follows: Where h = 4, λ represents the hyperparameter, L CE is the cross entropy loss, || || * The mathematical symbol for calculating the nuclear norm, Indicates the calculated output value The nuclear norm of ; Θ represents the rotation angle; Represents the probability of corresponding to label yi.

10. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 4, characterized in that: In step B1, the ordered data set X S Generate sub-pool The formula is as follows: in, μ is the balancing factor, L is the label pool in the rth training, N is the total number of images in the dataset X, R is the total number of training times, and the ordered dataset X S The kth sample in is sorted and represented as x k ; The number of samples in each sub-pool is: Among them, the minimum value of μ is At least K samples are marked for each training; In the first training, the sampler φ is trained from the sorted unlabeled pool X S Select the first K data from the , and in the rth (r>1) training, the sampler φ selects the K data with the smallest maximum posterior probability, of Calculated, therefore, the newly labeled data for in, is the nuclear norm regularization model F NNR The output after the (r-1)th training, is the rth sub-pool, and K is the number of data that need to be labeled in each cycle.

Images