Gesture recognition method based on balanced active learning and nuclear norm regularization model

Abstract

A gesture recognition method based on balanced active learning and a nuclear norm regularization model is characterized by: 1. Constructing a nuclear norm regularization model based on a spiking neural network; 2. Training the nuclear norm regularization model using a pre-trained self-supervised task and an active learning task; 3. Acquiring gesture image data using an acquisition module; 4. Preprocessing the gesture image data using a preprocessing module; 5. Encoding the standard graphic data b using an encoding module; 6. Convolving the encoded data using the input layer of the trained nuclear norm regularization model; 7. Extracting features from the convolutional data using a feature extraction module; 8. Performing average pooling on the feature data using an adaptive average pooling layer; 9. Performing a fully connected operation on the pooled data using a first fully connected layer to obtain fully connected data; 10. Performing a fully connected operation on the fully connected data using an output layer and outputting the gesture recognition result h. The effect is improved: the model's gesture recognition accuracy is increased.

Description

Technical Field

[0001] This invention relates to the field of spiking neural network technology, and in particular to a gesture recognition method based on balanced active learning and nuclear norm regularization model. Background Technology

[0002] In recent years, active learning has successfully addressed the high cost of labeled data in the field of Artificial Neural Networks (ANNs). It can be broadly categorized into two types: uncertainty-based active learning and class distribution-based active learning. While these methods offer insights into solving the high cost of labeled data, solutions based on class imbalance may cause the model to confuse data points near the decision boundary, while uncertainty-based strategies may sample overlapping parts of the dataset, resulting in selected data that does not represent the true data distribution of the overall dataset. Furthermore, training Spiking Neural Networks (SNNs) requires a large number of labeled Dynamic Vision Sensor (DVS) datasets, but acquiring such data is extremely difficult in the SNN field, and large-scale dynamic datasets are scarce. More importantly, because ANNs and SNNs differ in feature representation and information transmission, these methods are not suitable for SNNs. Regrettably, only two studies on active learning in the SNN field have been published, both based on uncertainty-based strategies. Therefore, the SNN field urgently needs to develop more novel active learning strategies to address the difficulties of data labeling and acquisition. Therefore, this paper aims to propose a new active learning strategy suitable for SNNs to help the model select more valuable samples for training, achieve optimal performance within a limited label budget, and thus solve the problems of data labeling and data acquisition difficulties.

[0003] Furthermore, the biologically inspired binary spiking activation mechanism of SNNs makes them more energy-efficient, but it also gives directly trained deep spiking neural networks characteristics such as discontinuity, non-invariance, temporal dynamics, and time dependence. This leads to a mismatch between the gradient descent method used in directly trained SNNs and the loss pattern, resulting in low generalization. Currently, many researchers are studying the discontinuity, temporal dynamics, and time dependence of SNNs. Although substantial progress has been made, the poor generalization problem caused by the discrete and non-differentiable nature of SNNs remains a major obstacle to the development of spiking neural networks. Therefore, it is necessary to design new loss functions to alleviate the problems of gradient mismatch and low model generalization.

[0004] Disadvantages of existing technologies: Spiking neural networks are often used in gesture recognition tasks. However, the loss pattern mismatch and low model generalization in directly trained spiking neural networks can lead to low gesture recognition accuracy. In addition, training spiking neural network (SNN) models requires a large amount of labeled dynamic datasets, which is costly to obtain labeled data. Summary of the Invention

[0005] This invention provides a gesture recognition method based on balanced active learning and nuclear norm regularization model, which can effectively improve the accuracy of gesture recognition results and reduce model training costs.

[0006] To achieve the above objectives, this invention provides a gesture recognition method based on balanced active learning and a nuclear norm regularization model, the key of which is:

[0007] Step 1: Construct a nuclear norm regularization model F based on a spiking neural network NNR The nuclear norm regularization model F NNR The system comprises an input layer, a hidden layer, and an output layer connected in sequence. The hidden layer comprises a feature extraction module, an adaptive average pooling layer, and a first fully connected layer connected in sequence.

[0008] Step 2: The nuclear norm regularization model F is trained using a pre-self-supervised task A and an active learning task B. NNR The model is trained and its parameters are optimized to obtain a well-trained nuclear norm regularized model F. NNR ;

[0009] Step 3: The acquisition module acquires gesture image data 'a' in real time and transmits it to the preprocessing module;

[0010] Step 4: The preprocessing module performs preprocessing operations on the gesture image data a to obtain standard graphic data b, which is then passed to the encoding module;

[0011] Step 5: The encoding module encodes the standard graphic data b to obtain encoded data c, and then passes it to the trained nuclear norm regularization model F. NNR ;

[0012] Step 6: The trained nuclear norm regularization model F NNR The input layer obtains the encoded data c, performs a convolution operation on it to obtain convolutional data d, and passes it to the feature extraction module;

[0013] Step 7: The feature extraction module performs feature extraction on the convolutional data d to obtain feature data e, and passes it through the adaptive average pooling layer;

[0014] Step 8: The adaptive average pooling layer performs average pooling on the feature data e to obtain pooled data f, and then passes it to the first fully connected layer;

[0015] Step 9: The first fully connected layer performs a full connection operation on the pooled data f to obtain fully connected data g, and then passes it to the output layer;

[0016] Step 10: The output layer performs a full connection operation on the fully connected data g and outputs the gesture recognition result h.

[0017] The preprocessing operation involves adjusting the spatial resolution of the gesture image data a to 48×48, which avoids excessive memory consumption caused by large original data and improves the model's computation speed.

[0018] Through the above design, firstly, local density clustering distance is introduced in the pre-self-supervised task to measure different data and solve the problem of data overlap; secondly, a balanced active learning strategy based on spiking neural networks is used to solve the problem of data confusion; finally, a training algorithm with nuclear norm regularization is introduced to effectively improve the generalization performance and accuracy of the model.

[0019] Preferably, in step 1, the feature extraction module is provided with 8 feature extraction blocks, and the 8 feature extraction blocks have the same structure and are connected end to end in sequence;

[0020] Each feature extraction block is provided with a first convolutional layer, a first batch of normalized layers, a first spiking neuron layer, a second convolutional layer, a second batch of normalized layers, a residual connection layer, and a second spiking neuron layer connected in sequence.

[0021] The feature extraction module is used to extract data features from the input samples; the adaptive average pooling layer is used to reduce the overall number of features by calculating sufficient statistics of features locally, thereby preventing overfitting and reducing computational load; and by taking the average value, errors are reduced, while maintaining invariance to rotation, translation, scaling, etc., thus reducing the increase in variance of the estimated value caused by the limited neighborhood size during the image feature extraction process.

[0022] Preferably, in step 2, the pre-self-supervised task A includes the following steps:

[0023] Step A1: The acquisition module obtains and encodes an unlabeled dataset X with N samples, and uses the dataset X to regularize the nuclear norm regularization model F. NNR Conduct training;

[0024] Step A2: In the nuclear norm regularization model F NNR The local density clustering distance δ for each data point in the dataset X is calculated, and the dataset X is sorted in ascending order based on the local density clustering distance δ to obtain an ordered dataset.

[0025] Pre-supervision tasks are mainly used to cluster data, enabling the model to extract features more effectively.

[0026] Preferably, in step 2, the active learning task B includes the following steps:

[0027] Step B1: Calculate the nuclear norm regularization model F NNR The loss value LOSS after the r-th training is used to set a balance factor μ, and then the corresponding sub-pool is generated using the balance factor μ.

[0028] Step B2: Select K samples from the adaptive sub-pool for labeling based on the posterior probability of the r-th training.

[0029] Step B3: Repeat steps B1-B2 until the number of labeled samples reaches the label budget, at which point training is complete.

[0030] As a preferred embodiment: the nuclear norm regularization model F NNR The LIF neuron model is used for training, and the forward iterative formula of the LIF neuron model is as follows:

[0031]

[0032] Where l represents the number of neurons in the l-th layer of the spiking neural network, and k τ Indicates the attenuation factor. This represents the membrane potential of the i-th neuron in the n-th layer at time t. This represents the output of the i-th neuron in the l-th layer of the spiking neural network at time t. V represents the presynaptic input, t represents the time constant, and f() is the activation function; th It is the emission threshold, when the neuron membrane potential u exceeds a given threshold V. th At this time, the neuron emits a pulse and resets the membrane potential to zero; Indicates synaptic weight, It is the bias, where j represents the identifier of any data point;

[0033] From formula (2), we can obtain Therefore, the formula for calculating the gradient during backpropagation in the LIF neuron model is as follows:

[0034]

[0035] in, The second factor represents the partial derivative of the LIF neuron's output with respect to the membrane potential; The first factor represents the partial derivative of the loss value with respect to the output of the previous layer neuron; The third factor represents the partial derivative of the presynaptic input with respect to the synaptic weights; L NNR Indicates the loss value LOSS. This represents labeled data p. y,Θ p corresponds to the probability of label y. y,Θ This represents labeled data, where T is the length of the pulse sequence. This indicates the calculation of partial derivatives.

[0036] As a preferred embodiment: the nuclear norm regularization model F NNR The formula for calculating the output layer is as follows:

[0037]

[0038] The formula for calculating the hidden layer is as follows:

[0039]

[0040] Preferably, the hidden layer calculates and updates the neuronal membrane potential using formulas (1)-(3), each feature extraction block outputs features to the output layer, and finally the emission rate of each neuron is output through the output layer, as shown in the following expression:

[0041]

[0042] in, This corresponds to the label y. i The probability of x; i Let g represent the i-th data point, i.e., the fully connected data.

[0043] The output layer calculates the probability distribution values ​​corresponding to various gesture category labels, and outputs the gesture category label corresponding to the maximum probability distribution value as the gesture recognition result h.

[0044] Preferably, in step A3, the calculation process for the local density clustering distance δ of each data point in the dataset X, based on the output of the first fully connected layer, is as follows:

[0045] A dataset X consisting of N points is represented as follows: Distance threshold d c The calculation expression is as follows:

[0046]

[0047] Among them, D i Representing point x i The distance to its nearest neighbor, point x i density ρ i The calculation expression is as follows:

[0048]

[0049] Where, d ij x represents iand x j The Euclidean distance between them;

[0050] For a given dataset Point x i Local density clustering distance δ i The calculation expression is as follows:

[0051]

[0052] in, Mathematical symbols represent anything. This means that for any data point j, formula (10) is satisfied.

[0053] Preferably: in step B1, the nuclear norm regularization model F NNR The formula for calculating the loss value (LOSS) is as follows:

[0054]

[0055] Where h = 4, λ represents the hyperparameter, and L CE For cross-entropy loss, ||| * Mathematical symbols used to calculate the nuclear norm. Indicates the calculated output value The nuclear norm; Θ represents the rotation angle;

[0056]

[0057] This corresponds to the label y. i The probability of.

[0058] To improve the generalization performance of the model, a penalty term was added at each time step to constrain the output distribution at each time step.

[0059] Preferably: in step B1, the ordered dataset X is... S Generating sub-pools The formula is as follows:

[0060]

[0061] in, μ is the balance factor, L is the label pool in the r-th training iteration, N is the total number of images in dataset X, R is the total number of training iterations, and X is the ordered dataset. S The k-th sample in the sorted data is represented as x. k ;

[0062] The number of samples in each sub-pool is:

[0063]

[0064] Where, the minimum value of μ is Each training session must label at least K samples;

[0065] In the first training iteration, the sampler φ starts from the sorted unlabeled pool X. S The first K data points are selected. In the r-th (r>1) training iteration, the sampler φ selects the K data points with the smallest maximum posterior probability. of Therefore, the newly labeled data is calculated. for

[0066]

[0067] in, For the nuclear norm regularization model F NNR The output after the (r-1)th training iteration, It is the r-th sub-pool, and K is the number of data items that need to be marked in each loop.

[0068] The beneficial effects of this invention are:

[0069] 1. Local density clustering distance (LDCD) is introduced as an effective measure of the amount of information carried by data, providing important reference data for selecting data that contributes more to model training. By selecting a subset of data, the performance can be similar to that of using the entire dataset with a subset of data, reducing the model's dependence on a large amount of training data and reducing the training cost of the gesture recognition model.

[0070] 2. A balanced active learning strategy based on spiking neural networks is proposed. This strategy takes into account the diversity and uncertainty of data under different label budgets by setting different balance factors, and selects the data with the most information for model training.

[0071] 3. To address the issues of loss pattern mismatch and low model generalization in directly trained spiking neural networks, this invention introduces a penalty term kernel norm into the cross-entropy loss function for training, which effectively enhances the model's generalization ability and improves the model's gesture recognition accuracy. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0073] Figure 2 This is a schematic diagram of the training process of the present invention;

[0074] Figure 3 This is a schematic diagram of the active learning task in this invention;

[0075] Figure 4A flowchart illustrating the workflow of the nuclear norm regularization training algorithm;

[0076] Figure 5 This is a comparison chart of the test results for medium-label budget in the examples;

[0077] Figure 6 This is a performance comparison chart of various methods at the same marking ratio in the examples;

[0078] Figure 7 This is a comparison chart of a small number of label budget test results in the examples;

[0079] Figure 8 This is a comparison chart of the accuracy of BAL-NNR and SAL test results in the example;

[0080] Figure 9 The diagram shows the influence of the BAL-NNR model on the loss curves of CIFAR10, DVS128 Gesture tests and CIFAR10DVS tests in the example.

[0081] Figure 10 A t-SNE visualization of the sample distribution on the CIFAR10 dataset;

[0082] Figure 11 The image shows the feature map of the SNN model without nuclear norm constraints in the example.

[0083] Figure 12 The feature map of the SNN model visualized in the example is an algorithm with added nuclear norm constraint.

[0084] Figure 13 This is a comparison chart of the LDCD sorting and CDD sorting effects on the datasets CIFAR10, DVS128 Gesture, and CIFAR10DVS in the example.

[0085] Figure 14 This is a comparison chart of the test loss values ​​of the model using cross-entropy as the loss function and the model using the nuclear norm constraint as the loss function in the embodiment.

[0086] Figure 15 For the nuclear norm regularization model F NNR A structural diagram. Detailed Implementation

[0087] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples. The following embodiments or drawings are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0088] like Figure 1 As shown: A gesture recognition method based on balanced active learning and nuclear norm regularization model, comprising the following steps:

[0089] Step 1: Construct a nuclear norm regularization model F based on a spiking neural network NNR The nuclear norm regularization model F NNR The system comprises an input layer, a hidden layer, and an output layer connected in sequence. The hidden layer comprises a feature extraction module, an adaptive average pooling layer, and a first fully connected layer connected in sequence.

[0090] Step 2: The nuclear norm regularization model F is trained using a pre-self-supervised task A and an active learning task B. NNR The model is trained and its parameters are optimized to obtain a well-trained nuclear norm regularized model F. NNR ;

[0091] Step 3: The acquisition module acquires gesture image data 'a' in real time and transmits it to the preprocessing module;

[0092] Step 4: The preprocessing module performs preprocessing operations on the gesture image data a to obtain standard graphic data b, which is then passed to the encoding module;

[0093] Step 5: The encoding module encodes the standard graphic data b to obtain encoded data c, and then passes it to the trained nuclear norm regularization model F. NNR ;

[0094] Step 6: The trained nuclear norm regularization model F NNR The input layer obtains the encoded data c, performs a convolution operation on it to obtain convolutional data d, and passes it to the feature extraction module;

[0095] Step 7: The feature extraction module performs feature extraction on the convolutional data d to obtain feature data e, and passes it through the adaptive average pooling layer;

[0096] Step 8: The adaptive average pooling layer performs average pooling on the feature data e to obtain pooled data f, and then passes it to the first fully connected layer;

[0097] Step 9: The first fully connected layer performs a full connection operation on the pooled data f to obtain fully connected data g, and then passes it to the output layer;

[0098] Step 10: The output layer performs a full connection operation on the fully connected data g and outputs the gesture recognition result h.

[0099] like Figure 15 As shown in (a), the input layer is a convolutional layer, the output layer is a fully connected layer, and the feature extraction module is provided with 8 feature extraction blocks. The 8 feature extraction blocks have the same structure and are connected end to end in sequence.

[0100] like Figure 15As shown in (b), each feature extraction block is provided with a first convolutional layer, a first batch normalization layer, a first spiking neuron layer, a second convolutional layer, a second batch normalization layer, a residual connection layer, and a second spiking neuron layer connected in sequence. Figure 15 In this context, Conv represents a convolutional layer, BN represents a batch normalization layer, LIF1 spiking neuron layer, and ⊕ represents a residual connection layer.

[0101] The nuclear norm regularization model F NNR The specific network parameters are shown in the table below:

[0102]

[0103] Where conv1 represents the input layer, Block1-8 represent eight sequentially connected feature extraction blocks, AdaptiveAvgPool represents the adaptive average pooling layer, FC1 represents the first fully connected layer, and FC2 represents the output layer.

[0104] like Figure 2 As shown: In step 2, the pre-self-supervised task A includes the following steps:

[0105] Step A1: The acquisition module obtains and encodes an unlabeled dataset X with N samples, and uses the dataset X to regularize the nuclear norm regularization model F. NNR Conduct training;

[0106] Step A2: In the nuclear norm regularization model F NNR The local density clustering distance δ for each data point in the dataset X is calculated, and the dataset X is sorted in ascending order based on the local density clustering distance δ to obtain an ordered dataset.

[0107] In step 2, the active learning task B includes the following steps:

[0108] Step B1: Calculate the nuclear norm regularization model F NNR The loss value LOSS after the r-th training is used to set a balance factor μ, and then the corresponding sub-pool is generated using the balance factor μ.

[0109] Step B2: Select K samples from the adaptive sub-pool for labeling based on the posterior probability of the r-th training.

[0110] Step B3: Repeat steps B1-B2 until the number of labeled samples reaches the label budget, at which point training is complete.

[0111] The nuclear norm regularization model F NNRThe LIF neuron model is used for training, and the forward iterative formula of the LIF neuron model is as follows:

[0112]

[0113] Where l represents the number of neurons in the l-th layer of the spiking neural network, and k τ Indicates the attenuation factor. This represents the membrane potential of the i-th neuron in the n-th layer at time t. This represents the output of the i-th neuron in the l-th layer of the spiking neural network at time t. V represents the presynaptic input, t represents the time constant, and f() is the activation function; th It is the emission threshold, when the neuron membrane potential u exceeds a given threshold V. th At this time, the neuron emits a pulse and resets the membrane potential to zero; Indicates synaptic weight, It is the bias, where j represents the identifier of any data point;

[0114] From formula (2), we can obtain Therefore, the formula for calculating the gradient during backpropagation in the LIF neuron model is as follows:

[0115]

[0116] in, The second factor represents the partial derivative of the LIF neuron's output with respect to the membrane potential; The first factor represents the partial derivative of the loss value with respect to the output of the previous layer neuron; The third factor represents the partial derivative of the presynaptic input with respect to the synaptic weights; L NNR Indicates the loss value LOSS. This represents labeled data p. y,Θ p corresponds to the probability of label y. y,Θ This represents labeled data, where T is the length of the pulse sequence. This indicates the calculation of partial derivatives.

[0117] The nuclear norm regularization model F NNR The formula for calculating the output layer is as follows:

[0118]

[0119] The formula for calculating the hidden layer is as follows:

[0120]

[0121] The second factor This refers to Dirac's delta function, a non-differentiable step function, which is typically approximated by a differentiable curve, i.e., gradient substitution. Various gradient substitution shapes exist, such as triangular, exponential, and rectangular curves. This embodiment selects a rectangular gradient substitution. Mathematically, it can be described as:

[0122]

[0123] Where 'a' is a hyperparameter that determines the width of h(·), and the gradient can be expressed in the interval as follows: a = 1.

[0124] The third factor The derivation is as follows:

[0125]

[0126] The hidden layer calculates and updates the neuronal membrane potential using formulas (1)-(3). Each feature extraction block outputs features to the output layer. Finally, the emission rate of each neuron is output through the output layer, as shown in the following expression:

[0127]

[0128] in, This corresponds to the label y. i The probability of x; i Let g represent the i-th data point, i.e., the fully connected data.

[0129] The output layer calculates the probability distribution values ​​corresponding to various gesture category labels, and outputs the gesture category label corresponding to the maximum probability distribution value as the gesture recognition result h.

[0130] In step A3, the calculation process for the local density clustering distance δ of each data point in the dataset X, based on the output of the first fully connected layer, is as follows:

[0131] A dataset X consisting of N points is represented as follows: Distance threshold d c The calculation expression is as follows:

[0132]

[0133] Among them, D i Representing point x i The distance to its nearest neighbor, point x i density ρ i The calculation expression is as follows:

[0134]

[0135] Where, d ij x represents i and x j The Euclidean distance between them;

[0136] The above formula only requires the use of radius d c The local density of each point is calculated using its neighborhood points. This significantly reduces the computational load and the influence of the main cluster on the density of smaller clusters, making the local density calculated using this formula more reasonable.

[0137] For a given dataset Point x i Local density clustering distance δ i The calculation expression is as follows:

[0138]

[0139] in, Mathematical symbols represent anything. This means that for any data point j, formula (10) is satisfied.

[0140] Points with larger δ values ​​carry more information and are therefore the most difficult to distinguish. Therefore, the dataset X is finally sorted in ascending order of δ to obtain data X. s .

[0141] In step B1, the nuclear norm regularization model F NNR The formula for calculating the loss value (LOSS) is as follows:

[0142]

[0143] Where h = 4, λ represents the hyperparameter, and L CE For cross-entropy loss, ||| * Mathematical symbols used to calculate the nuclear norm. Indicates the calculated output value The nuclear norm; Θ represents the rotation angle;

[0144]

[0145] This corresponds to the label y. i The probability of.

[0146] In step B1, the ordered dataset X is... S Generating sub-pools The formula is as follows:

[0147]

[0148] in, μ is the balance factor, L is the labeled pool in the r-th training, N is the total number of images in the dataset X, R is the total number of training times, and the ordered dataset X S The k-th sample in is denoted as x after sorting k ;

[0149] The number of samples in each sub-pool is:

[0150]

[0151] Among them, the minimum value of μ is At least K samples are labeled each time of training;

[0152] In the first training, the sampler φ selects the first K data from the sorted unlabeled pool X S In the r-th (r>1) training, the sampler φ selects K data with the smallest maximum posterior probability, which is calculated by of Therefore, the newly labeled data is

[0153]

[0154] Among them, is the output of the nuclear norm regularization model F NNR after the (r - 1)-th training, is the r-th sub-pool, and K is the number of data to be labeled in each cycle.

[0155] The present invention balances the uncertainty and diversity of sampled data through the balance factor μ. The present invention allows adjacent sub-pools to have an overlap (μ < 1) or an interval (μ > 1), as shown in Figure 2 (b). Therefore, the length of each sub-pool can dynamically meet the requirements of different annotation budgets. The balance factor μ is selected based on the maximum performance of the main task model F trained on n , and the expression is as follows:

[0156]

[0157] Among them is trained on , which is quite small, only 15% in ordinary experiments. Therefore, the balance between diversity and uncertainty can be easily achieved without too much computational effort.

[0158] Next, the corresponding active learning strategy is analyzed according to different balance factors μ. As can be seen from Equation (13), when 2 < i < I, ends at the One sample, and Beginning at the One sample.

[0159] Case 1: μ > 1. In this case, we can obtain Therefore, a total tag sub-pool exists. and There is overlap. Since some samples may have already been labeled in previous cycles, it is necessary to... Delete As shown in equation (13).

[0160] Case 2: When μ < 1, there exists an interval and and Therefore, equation (13) can be simplified to

[0161]

[0162] Case 3: μ = 1. In this case, we have Furthermore, there is no overlap or gap between nearby sub-pools, indicating that... For the sorted dataset X S The sub-pools are then uniformly divided. Therefore, equation (13) degenerates into uniformly divided sub-pools:

[0163]

[0164] Where I is the number of loops, and N represents the number of images. Dataset X S The k-th sample after sorting is represented as x. k .

[0165] Next, various experiments were conducted on different label budgets and from different perspectives to demonstrate the effectiveness of the gesture recognition method based on balanced active learning and nuclear norm regularization model proposed in this invention. The method was also compared with random sample selection methods, confidence-based active learning, ActiveFT, the current state-of-the-art active learning model in the field of spiking neural networks, and active learning baselines based on class balance.

[0166] (I) Experimental Data: The experiments were conducted on both static and dynamic datasets, including:

[0167] 1. Static dataset

[0168] 1) The CIFAR-10 dataset contains 60,000 color images, each with a size of 32*32*3 pixels. The training set has 50,000 samples, and the test set has 10,000 samples. CIFAR-10 has 10 categories, including airplanes, dogs, birds, cars, etc., with a wide range between categories.

[0169] 2. Dynamic datasets

[0170] 1) The DVS128Gesture dataset contains 11 gesture categories from 28 subjects captured by a DVS128 camera, consisting of 1176 training samples and 288 test samples, with captured images of size 128*128.

[0171] 2) The CIFAR10-DVS dataset was created by converting 10,000 frame-based images from the CIFAR10 dataset into event-stream-based images using a dynamic vision sensor (DVS). This resulted in the CIFAR10-DVS dataset, which is of moderate difficulty and contains rich temporal information. Since the original CIFAR10-DVS dataset contains a large number of events in a single event stream and has a resolution of 128×128, directly inputting it into an SNN without preprocessing would result in significant memory consumption and prolonged training time. Therefore, the spatial resolution of the CIFAR10-DVS dataset was adjusted to 48×48, and the dataset was divided into 9,000 training samples and 1,000 test samples.

[0172] (II) Comparison Method: The BAL-NNR of this invention is compared with the following active learning methods:

[0173] 1) Confidence level: The highest probability score is used to predict the uncertainty of a sample.

[0174] 2) Random sampling: This is the simplest baseline.

[0175] 3) State-of-the-art active learning model for spiking neural networks: Xie et al. used the loss prediction module ActiveLossNet to extract features of deep SNNs and selected valuable samples for training.

[0176] 4) ActiveFT: This method ensures class balance by selecting a sufficiently representative subset of data that is similar to the distribution of the entire unlabeled pool.

[0177] 5) BAL: Balanced active learning strategy in the field of artificial neural networks.

[0178] (III) Experimental Configuration: The initial sample set was labeled using a warm-start technique, and ResNet19 was used as the backbone. The experimental platform used was an Intel(R) Core(TM) i9-14900K@6.0GHz, NVIDIA GeForce RTX 3090 24G*2, and 64-bit Ubuntu 20.04. In each cycle, the model was trained starting with the weights from the most recent cycle. The average accuracy was calculated over three runs. Other detailed configurations are shown in Table 1.

[0179] Table 1 Experimental parameters

[0180]

[0181]

[0182] 1) Hyperparameter: The membrane potential time constant τ in the experiment. m Set to 0.5, neural firing threshold V th Set to 0.5. Resting potential V reset Set it to 0.0.

[0183] 2) Labeled data ratio: The standard for evaluating the performance of active learning methods is through labeling data at a fixed ratio. Accuracy of time In the following experiments, the medium-label budget model undergoes 7 rounds of active learning iterations. In each iteration, 10%, 15%, 20%, 25%, 30%, 35%, and 40% of the data from the unlabeled dataset are sequentially added to the training set as input data for the next iteration. The small-label budget model undergoes 5 rounds of active learning iterations. In each iteration, 2%, 4%, 6%, 8%, and 10% of the data from the unlabeled dataset are sequentially added to the training set as input data for the next iteration. The SNN model is retrained in each iteration.

[0184] (iv) Medium-sized label budget: such as Figure 5 As shown, when executed on CIFAR10, DVS128Gesture, and CIFAR10DVS, where the sorting metric for two of the subpools uses LDCD, the adaptive subpool shows a significant advantage.

[0185] As shown in Table 2, Figure 6As shown, the effectiveness of the invention is tested under a medium label budget. The performance of various methods under different label ratios ω is compared. The results show that BAL-NNR outperforms other methods under a medium label budget, especially for dynamic datasets. On DVS128 Gesture, BAL-SNN with ω=35% outperforms Random by 2.08% and is 0.11% better than BAL with ω=40%, saving 10% of data labeling. On CIFAR10, BAL-SNN with ω=20% outperforms ActiveFT with ω=40%, and BAL-SNN with ω=30% performs close to confidence with ω=40%, saving 10% of data labeling. At ω=40%, BAL-SNN improves performance by 0.86% compared to BAL and by 1.35% compared to Random. On CIFAR10DVS, BAL-SNN with ω=30% outperforms Confidence, ActiveFT, Random and BAL by 1.26%, 1.44%, 1.68% and 0.99% respectively with ω=40%, saving at least 10% of data annotation.

[0186] Table 2

[0187]

[0188] (v) Limited Label Budget: When the budget is limited, querying typical examples is more beneficial for training; when the budget is large, querying atypical or non-representative examples is more beneficial for training. To further verify the effectiveness of the BAL-NNR method of this invention, experiments were conducted with a limited label budget. The results are shown in Table 3. Figure 7 As shown, performance comparisons are performed between different methods at different labeling ratios ω. On the DVS128 Gesture, BAL-NNR at ω=10% is 10.07%, 9.73%, 9.38%, and 3.82% higher than methods based on confidence, ActiveFT, Random, and BAL data selection, respectively. On CIFAR10, BAL-NNR at ω=8% is 10.11% higher than the second-best method and 17.24% higher than the ActiveFT method. On the CIFAR10DVS, at ω=10%, our method is 15.78%, 10.08%, 17.48%, and 1.42% higher than methods based on confidence, ActiveFT, Random, and BAL data selection, respectively.

[0189] Table 3

[0190]

[0191] (VI) Comparison of the latest active learning methods for spiking neural networks: The BAL-NNR of this invention is compared with the current state-of-the-art active learning method (SAL) for spiking neural networks on CIFAR10. For ease of comparison, both use the SNN-ResNet18 model, with T=12. The comparison results are as follows: Figure 8 As shown, the BAL-NNR method generally outperforms the SAL method in terms of accuracy. When the amount of labeled data is 10,000, the accuracy of BAL-NNR is 2.63% higher than that of SAL.

[0192] (vii) Convergence and Efficiency of the Loss Function: The loss curves were tested using the BAL-NNR model on CIFAR10, DVS128Gesture, and CIFAR10DVS. 200 epochs were set for a total of 1400 epochs. For example... Figure 9 As shown, at the beginning of each round of active learning training, the loss value is very high due to the limited training data. As more and more data is added, the loss value gradually decreases. It can be seen that in the early stages of active learning training, the training loss shows a significant decreasing trend, while in the later stages, the loss value changes slowly. This indicates that the data selection strategy for active learning is very effective in the early stages of the algorithm.

[0193] Furthermore, in the first 100 epochs of each round, the loss curve does not fully converge due to insufficient training samples. As new unlabeled samples are labeled and added, the loss increases sharply in the first epoch of each round. Then, in later stages, the loss curve gradually converges. Although all methods show the same overall trend, the present invention exhibits smaller loss and less fluctuation in the loss function at each epoch, resulting in greater stability.

[0194] (viii) Visualization Analysis: The sample distribution diagrams for the first and last iterations of CIFAR10 are generated using t-SNE dimensionality reduction technology, as shown below. Figure 10 As shown. Figure 10 As shown in (a), the feature separation between different categories of samples is not good at the beginning of the iteration. Figure 10 As shown in (b), after the first iteration, valuable outliers were clearly selected and marked with yellow boxes. Figure 10 As shown in (c), after multiple iterations, the trained SNN model has a better sample classification ability.

[0195] To further analyze the high similarity between each category, in Figure 11 , Figure 12 The text visualizes the feature maps of the SNN model with and without kernel norm constraints. For example... Figure 11 , Figure 12As shown, the first row represents example images for each category. The second and third rows show the feature representations before and after direct encoding into impulses, and the last four rows represent the feature representations after each block of the SNN model. It can be seen that, similar to traditional artificial neural network models, the features of the SNN become increasingly abstract as the number of network layers increases. Furthermore, to improve model efficiency, T=6 is set.

[0196] contrast Figure 11 and Figure 12 The second and third lines reveal that because SNNs rely on time-series pulse signals for information transmission, using a training algorithm with kernel norm constraints allows the network to better optimize the generation of input pulses, making the pulse sequences generated by the coding layer more accurate and representative. This means that L NNR The loss function can guide the network to better capture key features at the input and reduce unnecessary noise. (Comparison) Figure 11 and Figure 12 Lines four, five, and six reveal that the improved loss function reduces over-firing in intermediate layers or responses to irrelevant features, allowing neurons to focus more on target-related features and reducing information loss in shallow features and redundancy in deep features. Furthermore, it was found that the model of this invention can extract effective features more stably with different input samples, indicating that using L... NNR The loss function exhibits better generalization performance across different input samples. This helps prevent the network from getting trapped in local minima, thus improving the network's generalization ability and accuracy.

[0197] (ix) Ablation test

[0198] Ablation experiments were conducted on the CIFAR10 dataset.

[0199] ① Figure 13 (a) Comparison of batch segmentation metrics on CIFAR10. The two sub-pools use LDCD and CDD as sorting metrics, respectively. The data represents the average precision after three runs.

[0200] This invention proposes a pair pool The sorting standard is LDCD. Experiments have shown that LDCD sorting performs better than CDD sorting.

[0201] ②Sampling comparison

[0202] Three different sampling methods are compared: (i) random sampling; (ii) a K-means clustering-based sampler that uses K-means clustering features and samples data closest to the cluster centers; and (iii) a local density clustering-based sampler in this invention, which uses local density clustering to sample by LDCD(δ) sorting. Figure 13As shown in (b), the sampler based on local density clustering proposed in this invention has the best performance.

[0203] To verify the effectiveness of introducing a penalty term and a nuclear norm regularization term into the cross-entropy loss function, this invention conducted ablation experiments on the CIFAR10 dataset. Other parameter settings are shown in Table 1. Figure 13 As shown in (c), L CE This indicates that only cross-entropy is used as the loss function, L NNR This represents a new loss function that introduces a penalty term (nuclear norm) into the cross-entropy loss function. From Figure 13 (c) It can be seen that: using L NNR This effectively improved the accuracy of model training. When ω is less than 25%, the improved L... NNR Compared to using L CE The advantages are more obvious. This illustrates that the new loss function L proposed in this invention has more obvious advantages. NNR This can make model training more advantageous.

[0204] To further verify the effectiveness of the loss function proposed in this invention, the test loss values ​​of a model using only cross-entropy as the loss function and a model using a loss function constrained by the nuclear norm are compared. Figure 14 As shown, experiments were conducted using ResNet19 on the CIFAR10 dataset, and it was found that when using L... CE When using the loss function, the loss value in each epoch generally shows a slight upward trend with fluctuations after 100 epochs. When using L... NNR When calculating the loss function, the loss function in each round is lower than L. CE Furthermore, the loss value in each round exhibits a fluctuating downward trend. In summary, the L proposed in this invention... NNR loss function compared to L CE The loss function has significant advantages in terms of stability, generalization ability, and finding the global minimum.

[0205] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A gesture recognition method based on balanced active learning and nuclear norm regularization model, characterized in that, Includes the following steps: Step 1: Construct a nuclear norm regularization model F based on a spiking neural network NNR The nuclear norm regularization model F NNR The system comprises an input layer, a hidden layer, and an output layer connected in sequence. The hidden layer comprises a feature extraction module, an adaptive average pooling layer, and a first fully connected layer connected in sequence. Step 2: The nuclear norm regularization model F is trained using a pre-self-supervised task A and an active learning task B. NNR The model is trained and its parameters are optimized to obtain a well-trained nuclear norm regularized model F. NNR ; Step 3: The acquisition module acquires gesture image data 'a' in real time and transmits it to the preprocessing module; Step 4: The preprocessing module performs preprocessing operations on the gesture image data a to obtain standard graphic data b, which is then passed to the encoding module; Step 5: The encoding module encodes the standard graphic data b to obtain encoded data c, and then passes it to the trained nuclear norm regularization model F. NNR ; Step 6: The trained nuclear norm regularization model F NNR The input layer obtains the encoded data c, performs a convolution operation on it to obtain convolutional data d, and passes it to the feature extraction module; Step 7: The feature extraction module performs feature extraction on the convolutional data d to obtain feature data e, and passes it through the adaptive average pooling layer; Step 8: The adaptive average pooling layer performs average pooling on the feature data e to obtain pooled data f, and then passes it to the first fully connected layer; Step 9: The first fully connected layer performs a full connection operation on the pooled data f to obtain fully connected data g, and then passes it to the output layer; Step 10: The output layer performs a full connection operation on the fully connected data g and outputs the gesture recognition result h; In step 2, the active learning task B includes the following steps: Step B1: Calculate the nuclear norm regularization model F NNR The loss value LOSS after the r-th training is used to set a balance factor μ, and then the corresponding sub-pool is generated using the balance factor μ. R is the total number of training iterations; Step B2: Based on the posterior probability of the r-th training, select K samples from the sub-pool for labeling; Step B3: Repeat steps B1-B2 until the number of labeled samples reaches the label budget, at which point training is complete. In step B1, the nuclear norm regularization model F NNR The formula for calculating the loss value (LOSS) is as follows: Where h = 4, λ represents the hyperparameter, and L CE For cross-entropy loss, ||| * Mathematical symbols used to calculate the nuclear norm. Indicates the calculated output value The nuclear norm; θ represents the rotation angle; This corresponds to the label y. i The probability, This represents labeled data p. y,Θ p corresponds to the probability of label y. y,Θ This represents labeled data, where T is the length of the pulse sequence.

2. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 1, characterized in that: In step 1, the feature extraction module is provided with 8 feature extraction blocks, and the 8 feature extraction blocks have the same structure and are connected end to end in sequence; Each feature extraction block is provided with a first convolutional layer, a first batch of normalized layers, a first spiking neuron layer, a second convolutional layer, a second batch of normalized layers, a residual connection layer, and a second spiking neuron layer connected in sequence.

3. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 2, characterized in that: In step 2, the pre-self-supervised task A includes the following steps: Step A1: The acquisition module obtains and encodes an unlabeled dataset X with N samples, and uses the dataset X to regularize the nuclear norm regularization model F. NNR Conduct training; Step A2: In the nuclear norm regularization model F NNR The local density clustering distance δ for each data point in the dataset X is calculated, and the dataset X is sorted in ascending order based on the local density clustering distance δ to obtain an ordered dataset. Ordered dataset X S The k-th sample in the sorted data is represented as x. k .

4. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 3, characterized in that: The nuclear norm regularization model F NNR The LIF neuron model is used for training, and the forward iterative formula of the LIF neuron model is as follows: Where l represents the number of neurons in the l-th layer of the spiking neural network, and k τ Indicates the attenuation factor. This represents the membrane potential of the i-th neuron in the n-th layer at time t. This represents the output of the i-th neuron in the l-th layer of the spiking neural network at time t. V represents the presynaptic input, t represents the time constant, and f() is the activation function; th It is the emission threshold, when the neuron membrane potential u exceeds a given threshold V. th At this time, the neuron emits a pulse and resets the membrane potential to zero; Indicates synaptic weight, It is the bias, where j represents the identifier of any data point; From formula (2), we can obtain Therefore, the formula for calculating the gradient during backpropagation in the LIF neuron model is as follows: in, The second factor represents the partial derivative of the LIF neuron's output with respect to the membrane potential; The first factor represents the partial derivative of the loss value with respect to the output of the previous layer neuron; The third factor represents the partial derivative of the presynaptic input with respect to the synaptic weights; L NNR This indicates the loss value (LOSS). This represents labeled data p. y,Θ p corresponds to the probability of label y. y,Θ This represents labeled data, where T is the length of the pulse sequence. This indicates the calculation of partial derivatives.

5. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 4, characterized in that: The nuclear norm regularization model F NNR The formula for calculating the output layer is as follows: The formula for calculating the hidden layer is as follows:

6. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 5, characterized in that: The hidden layer calculates and updates the neuronal membrane potential using formulas (1)-(3). Each feature extraction block outputs features to the output layer. Finally, the emission rate of each neuron is output through the output layer, as shown in the following expression: in, This corresponds to the label y. i The probability of x; i Let g represent the i-th data point, i.e., the fully connected data. The output layer calculates the probability distribution values ​​corresponding to various gesture category labels, and outputs the gesture category label corresponding to the maximum probability distribution value as the gesture recognition result h.

7. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 3, characterized in that: In step A3, the calculation process for the local density clustering distance δ of each data point in the dataset X, based on the output of the first fully connected layer, is as follows: A dataset X consisting of N points is represented as follows: Distance threshold d c The calculation expression is as follows: Among them, D i Representing point x i The distance to its nearest neighbor, point x i density ρ i The calculation expression is as follows: Where, d ij x represents i and x j The Euclidean distance between them; For a given dataset Point x i Local density clustering distance δ i The calculation expression is as follows: in, Mathematical symbols represent anything. This means that for any data point j, formula (10) is satisfied.

8. The gesture recognition method based on balanced active learning and nuclear norm regularization model according to claim 1, characterized in that: In step B1, the ordered dataset X is... S Generating sub-pools The formula is as follows: in, μ is the balance factor, L is the label pool in the r-th training iteration, N is the total number of images in dataset X, R is the total number of training iterations, and X is the ordered dataset. S The k-th sample in the sorted data is represented as x. k ; The number of samples in each sub-pool is: Where, the minimum value of μ is Each training session must label at least K samples; In the first training iteration, the sampler φ starts from the sorted unlabeled pool X. S Select the first K data points. In the r-th training iteration, r>1, the sampler φ selects the K data points with the smallest maximum posterior probability. of Therefore, the newly labeled data is calculated. for in, For the nuclear norm regularization model F NNR The output after the (r-1)th training iteration, It is the r-th sub-pool, and K is the number of data items that need to be marked in each loop.

Images