A gesture recognition method, system, device and computer readable storage medium

Abstract

This application discloses a gesture recognition method, system, device, and computer-readable storage medium, relating to the field of artificial intelligence technology. The method involves training a gesture recognition model using a first training view and a second training view. The gesture recognition model is built based on a spiking neural network model. The method extracts a first spatial feature sequence and a first spatial prediction vector from the first training view generated by the gesture recognition model. It also extracts a second spatial feature sequence and a second spatial prediction vector from the second training view generated by the gesture recognition model. Based on the differences between the first spatial feature sequence and the second spatial prediction vector, and the differences between the second spatial feature sequence and the first spatial prediction vector, a temporal contrast loss value is generated. The gesture recognition model is adjusted based on the temporal contrast loss value to obtain a trained gesture recognition model. A gesture view is then acquired. Finally, the gesture recognition model is applied to recognize the gesture view to obtain the gesture recognition result. This improves the accuracy of gesture recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a gesture recognition method, system, device, and computer-readable storage medium. Background Technology

[0002] Currently, in the control of wearable devices and other devices, spiking neural networks (SNNs) are theoretically only one-tenth the energy consumption of artificial neural networks (ANNs) due to their event-driven sparse computing paradigm. Therefore, gesture recognition models built with spiking neural networks can be used to recognize user gestures, and then the devices can be controlled based on the recognition results.

[0003] Considering that neural network models all need to be trained before use, the gesture recognition model can be trained using the contrastive learning approach of ANNs. However, contrastive learning is designed for continuously activated ANNs, which typically compresses the time-series pulse sequence into a single embedding vector through global mean pooling before calculating the loss. This does not take into account the essential characteristic of SNNs that convey information through time-series pulse sequences, and completely discards the dynamic activation information of SNNs across time steps. As a result, the gesture recognition model has significantly lower accuracy in static gesture recognition than ANNs of the same size, and even lower accuracy in dynamic gesture recognition tasks.

[0004] In conclusion, improving the accuracy of spiking neural networks in gesture recognition is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a gesture recognition method that can, to some extent, solve the technical problem of how to improve the accuracy of spiking neural networks in gesture recognition. This application also provides a gesture recognition system, an electronic device, and a computer-readable storage medium.

[0006] To achieve the above objectives, this application provides the following technical solution: Firstly, a gesture recognition method is provided, including: Obtain the first training view and the second training view; The gesture recognition model is trained using the first training view and the second training view. The gesture recognition model is built based on a spiking neural network model. Extract the first spatial feature sequence and the first spatial prediction vector of the first training view generated by the gesture recognition model; Extract the second spatial feature sequence and the second spatial prediction vector of the second training view generated by the gesture recognition model; Based on the differences between the first spatial feature sequence and the second spatial prediction vector, as well as the differences between the second spatial feature sequence and the first spatial prediction vector, a temporal contrast loss value is generated. Based on the time-series comparison loss value, the total loss value of the gesture recognition model is generated, and the gesture recognition model is adjusted according to the total loss value to obtain the trained gesture recognition model. Obtain the view of the gesture to be recognized; The gesture recognition model is applied to recognize the gesture view to obtain the gesture recognition result.

[0007] On the other hand, based on the differences between the first spatial feature sequence and the second spatial prediction vector, and the differences between the second spatial feature sequence and the first spatial prediction vector, a temporal contrastive loss value is generated, including: The temporal contrast loss value is generated by using the temporal contrast loss generation formula based on the difference between the first spatial feature sequence and the second spatial prediction vector, as well as the difference between the second spatial feature sequence and the first spatial prediction vector. The formula for generating time-series contrast loss includes: ; in, Indicates the time-series comparison loss value; D Indicates negative cosine similarity; This represents the value at time step t in the first spatial feature sequence; This represents the value at time step t in the first spatial prediction vector; This represents the value at time step t in the second spatial feature sequence; sg represents the value at time step t in the second spatial prediction vector; sg represents the gradient stopping operation; T represents the total number of time steps.

[0008] On the other hand, based on the temporal comparison loss value, the total loss value of the gesture recognition model is generated, including: Extract the feature dimension data output from the projection head of the gesture recognition model; Add a lower bound constraint to the standard deviation of the feature dimension data to generate a variance penalty term; Constrain the linear correlation between different feature dimensions in the feature dimension data and generate a covariance penalty term; Based on the temporal comparison loss value, the variance penalty term, and the covariance penalty term, the total loss value of the gesture recognition model is generated.

[0009] On the other hand, a lower bound constraint is added to the standard deviation of the feature dimension data to generate a variance penalty term, including: By generating a formula for the variance penalty term, a lower bound constraint is added to the standard deviation of the feature dimension data to generate the variance penalty term. The formulas for generating the variance penalty term include: ; in, This represents the value of the variance penalty term; Indicates the target variance threshold; The penalty coefficient represents the variance; Z represents the feature dimension data. d represents the standard deviation of the j-th feature dimension on the batch samples; d represents the embedding dimension; max represents taking the maximum value.

[0010] On the other hand, constraints are applied to the linear correlation between different feature dimensions in the feature dimension data to generate a covariance penalty term, including: By generating a covariance penalty term formula, the linear correlation between different feature dimensions in the feature dimension data is constrained, and a covariance penalty term is generated. The formulas for generating the covariance penalty term include: ; in, This represents the value of the covariance penalty term; The penalty coefficient for covariance is represented by Z; Z represents the feature dimension data. Denotes the covariance matrix of Z; d represents the value at the i-th row and j-th column of the covariance matrix; d represents the embedding dimension.

[0011] On the other hand, based on the temporal contrast loss value, the variance penalty term, and the covariance penalty term, a total loss value for the gesture recognition model is generated, including: Generate the global contrastive loss value for the gesture recognition model; The global contrast loss value, the temporal contrast loss value, the variance penalty term, and the covariance penalty term are weighted and calculated to generate the total loss value of the gesture recognition model.

[0012] On the other hand, it also includes: Obtain the unconstrained parameters set for the LIF layer in the gesture recognition model; The Sigmoid function is applied to map unconstrained parameters to obtain learnable parameters whose values ​​are within a set range; The learnable parameter is set as the membrane time constant of the LIF layer.

[0013] Secondly, a gesture recognition system is provided, including: The training view acquisition module is used to acquire the first training view and the second training view. The model training module is used to train the gesture recognition model using the first training view and the second training view. The gesture recognition model is built based on a spiking neural network model. The first spatial data extraction module is used to extract the first spatial feature sequence and the first spatial prediction vector of the first training view generated by the gesture recognition model. The second spatial data extraction module is used to extract the second spatial feature sequence and the second spatial prediction vector of the second training view generated by the gesture recognition model. The temporal contrast loss value generation module is used to generate temporal contrast loss values ​​based on the differences between the first spatial feature sequence and the second spatial prediction vector, as well as the differences between the second spatial feature sequence and the first spatial prediction vector. The total loss value generation module is used to generate the total loss value of the gesture recognition model based on the time-series comparison loss value, and adjust the gesture recognition model according to the total loss value to obtain the trained gesture recognition model. The gesture view acquisition module is used to acquire the gesture view to be recognized. The gesture recognition module is used to apply a gesture recognition model to recognize the gesture view and obtain the gesture recognition result.

[0014] Thirdly, an electronic device is provided, comprising: Memory, used to store computer programs; A processor for implementing the steps of any of the above-described gesture recognition methods when executing the computer program.

[0015] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the gesture recognition methods described above.

[0016] This application provides a gesture recognition method, which involves: acquiring a first training view and a second training view; training a gesture recognition model based on a spiking neural network model using the first and second training views; extracting a first spatial feature sequence and a first spatial prediction vector from the first training view generated by the gesture recognition model; extracting a second spatial feature sequence and a second spatial prediction vector from the second training view generated by the gesture recognition model; generating a temporal contrast loss value based on the differences between the first spatial feature sequence and the second spatial prediction vector, as well as the differences between the second spatial feature sequence and the first spatial prediction vector; generating a total loss value for the gesture recognition model based on the temporal contrast loss value, and adjusting the gesture recognition model according to the total loss value to obtain a trained gesture recognition model; acquiring a gesture view to be recognized; and applying the gesture recognition model to recognize the gesture view to obtain a gesture recognition result. In this application, the spatial feature sequence and corresponding spatial prediction vector of the gesture recognition model are extracted during the training process. Since the spatial feature sequence and spatial prediction vector represent the temporal pulse sequence of the spiking neural network transmitting view information, generating a temporal contrast loss value based on the difference between the spatial feature sequence and the spatial prediction vector is equivalent to incorporating the local features of each time step of the spiking neural network into the self-supervised signal system, and applying cross-view loss to the gesture recognition model at the time step level. Figure 1 By leveraging the temporal computation advantages of spiking neural networks to enhance the training accuracy of gesture recognition models through consistency constraints, the accuracy of gesture recognition models in recognizing gestures can be improved. This application also provides a gesture recognition system, electronic device, and computer-readable storage medium that solve the corresponding technical problems. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a gesture recognition method provided in an embodiment of this application; Figure 2 The backbone network structure diagram for pulse feature extraction; Figure 3 This is a schematic diagram illustrating the training of a gesture recognition model. Figure 4 This is a schematic diagram of the structure of a gesture recognition system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 6 This is another structural schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Please see Figure 1 , Figure 1 This is a flowchart of a gesture recognition method provided in an embodiment of this application.

[0021] This application provides a gesture recognition method that may include the following steps: Step S101: Obtain the first training view and the second training view.

[0022] Step S102: The gesture recognition model is trained using the first training view and the second training view. The gesture recognition model is built based on the spiking neural network model.

[0023] In practical applications, the gesture recognition model is built based on the spiking neural network model. Considering that the gesture recognition model needs to be trained, it is necessary to obtain the first training view and the second training view. Then, the gesture recognition model is trained using the first training view and the second training view. The basic framework used for training can be the SNN-SimSiam-T negative sample contrastive learning framework, which is to apply the SimSiam negative sample contrastive learning method to the spiking neural network and introduce time dimension processing capability.

[0024] In an exemplary embodiment, the backbone network of the gesture recognition model can employ Spiking ResNet-18, implemented based on the SpikingJelly framework. For small input sizes such as 32×32, the first MaxPool layer can be removed to avoid premature spatial compression. The network consists of an initial convolutional layer and four residual layers. Each residual block is arranged in the order of "batch normalization-convolution-LIF (Leaky Integrate-and-Fire) neurons." Taking 32×32 as an example, the backbone network structure is as follows: Figure 2 As shown, it includes a sequentially connected input layer, a conv (convolution) layer 1 with a stride of 1, a layer 2 with a stride of 2, a layer 3 with a stride of 2, a layer 4 with a stride of 2, and a global mean pooling layer.

[0025] In an exemplary embodiment, the first training view and the second training view may be generated based on a static image or an event stream. Taking static images as an example, the static images can be basic image classification benchmark datasets such as CIFAR-10. The normalized static image frames can be repeated T times in the time dimension, such as four times, to form a temporal tensor with the shape (T×C×H×W), where C represents the number of channels, H represents the height of the image, and W represents the width of the image. Then, the temporal tensor is augmented using the same process as SimSiam. For example, random cropping is performed with a scale of (0.2, 1.0), random horizontal flipping is performed with a probability of p=0.5, random color jitter is performed with brightness / contrast / saturation of 0.4 each, hue of 0.1 and a probability of 0.8, random grayscale is performed with a probability of p=0.2, and Gaussian blur is performed with a kernel size of 3, sigma (standard deviation range) of [0.1, 2.0] and a probability of 0.5. Two different augmented views are generated for the same temporal tensor, which are used as the first training view and the second training view, respectively. Taking event streams as an example, the original event sequence of each sample can be divided into two channels according to positive and negative polarities, and evenly divided into T=16 time windows on the time axis. Within each window, the events of the corresponding channel are binarized and statistically analyzed to obtain a polarity frame tensor of shape (T×2×H×W). Then, event stream-specific data augmentation is performed, such as adding frame order perturbation to obtain the first training view, for example, randomly selecting 2 frames with a probability of 0.3 to swap their temporal positions, or adding frame masks to obtain the second training view, for example, randomly setting all polarity channels of 1 frame to zero with a probability of 0.2 to simulate the random changes in gesture execution speed and the temporary data loss phenomenon that may occur when the event camera changes rapidly in lighting.

[0026] Step S103: Extract the first spatial feature sequence and the first spatial prediction vector of the first training view generated by the gesture recognition model.

[0027] Step S104: Extract the second spatial feature sequence and the second spatial prediction vector of the second training view generated by the gesture recognition model.

[0028] In practical applications, in order to incorporate the local features of each time step of the spiking neural network into the self-supervised signal system, it is necessary to extract the first spatial feature sequence and the first spatial prediction vector of the first training view generated by the gesture recognition model, and extract the second spatial feature sequence and the second spatial prediction vector of the second training view generated by the gesture recognition model.

[0029] In the exemplary embodiment, the principles for extracting the first spatial feature sequence and the second spatial feature sequence are the same, as are the principles for extracting the first spatial prediction vector and the second spatial prediction vector. For example, from the output stage of the backbone network layer 4, instead of performing time-dimensional mean pooling, the independent spatial features of each time step are retained, and global spatial mean pooling is performed on each step to obtain T temporal embedding spatial sequences. The temporal embedding spatial sequence is processed using a two-layer MLP (Multilayer Perceptron) temporal projection head with shared parameters, such as a 512-256-256 temporal projection head. This yields a low-dimensional temporal spatial embedding as a spatial feature sequence. A temporal prediction head then maps this spatial feature sequence to obtain a spatial prediction vector. It should be noted that when the gesture recognition model processes the first training view, it extracts the first spatial feature sequence and the first spatial prediction vector. When the gesture recognition model processes the second training view, it extracts the second spatial feature sequence and the second spatial prediction vector.

[0030] Step S105: Generate a time-series contrast loss value based on the difference between the first spatial feature sequence and the second spatial prediction vector, as well as the difference between the second spatial feature sequence and the first spatial prediction vector.

[0031] In practical applications, after extracting the corresponding spatial feature data, time-series comparison loss values ​​can be generated based on the differences between the first spatial feature sequence and the second spatial prediction vector, as well as the differences between the second spatial feature sequence and the first spatial prediction vector.

[0032] In an exemplary embodiment, the temporal contrastive loss can be obtained by taking the arithmetic mean of the local losses at T time steps, and a synchronous pairing method is adopted, that is, pairing the t-th step of the first training view with the t-th step of the second training view, without introducing negative samples across time steps. This is because the SNN activation states of adjacent time steps are highly correlated due to the continuity of the leaked integral. Treating features of adjacent steps as negative samples would generate a large number of false negative samples, which would interfere with the encoder's correct modeling of temporal continuity. Based on this, the temporal contrastive loss value can be generated by the temporal contrastive loss generation formula, according to the difference between the first spatial feature sequence and the second spatial prediction vector, and the difference between the second spatial feature sequence and the first spatial prediction vector; the temporal contrastive loss generation formula includes: ; in, Indicates the time-series comparison loss value; D Negative cosine similarity is defined as follows: ; This represents the value at time step t in the first spatial feature sequence; This represents the value at time step t in the first spatial prediction vector; This represents the value at time step t in the second spatial feature sequence; Let represent the value at time step t in the second spatial prediction vector; sg represents the gradient stopping operation; and T represents the total number of time steps. In this way, the temporal contrastive loss applies cross-viewpoints at the time step level. Figure 1 Consistency constraints enable the encoder to learn deep semantic features that are invariant to temporal transformations. This has been verified to contribute 0.4 percentage points to the accuracy improvement on CIFAR-10.

[0033] Step S106: Based on the temporal comparison loss value, generate the total loss value of the gesture recognition model, and adjust the gesture recognition model according to the total loss value to obtain the trained gesture recognition model.

[0034] In practical applications, after generating the temporal comparison loss value, the total loss value of the gesture recognition model can be generated based on the temporal comparison loss value, and the gesture recognition model can be adjusted according to the total loss value. The adjustment process can be flexibly set according to the application scenario to obtain a trained gesture recognition model.

[0035] In an exemplary embodiment, considering that the output of an SNN neuron is a binary impulse, i.e., the output is 0 or 1, compared to the floating-point continuous activation of an ANN, the statistical characteristics of binary activation determine that the impulse feature has a low variance tendency. The embedding vector is more likely to tend towards a low variance distribution, resulting in dimensionality collapse, i.e., multiple feature dimensions degenerate into constants, losing discriminative ability, leading to extremely poor feature quality of the pre-trained gesture recognition model. To avoid this phenomenon, in the process of generating the total loss value of the gesture recognition model based on the temporal comparison loss value, the feature dimensions of the projection head output of the gesture recognition model can be extracted. The data is processed as follows: A lower bound constraint is added to the standard deviation of the feature dimension data, generating a variance penalty term. This penalty is applied only when the standard deviation of a feature dimension falls below a threshold, constraining each feature dimension to carry sufficient information variation and preventing some feature dimensions from degenerating into constant outputs. A covariance penalty term is generated to constrain the linear correlation between different feature dimensions, further restricting their linear correlation and ensuring each feature dimension carries independent discriminative information. Based on the time-series comparison loss value, variance penalty term, and covariance penalty term, the total loss value of the gesture recognition model is generated. In this way, by explicitly constraining the lower bound of variance and applying covariance decorrelation penalties to the statistical characteristics of pulse binary features, the risk of dimensionality collapse can be effectively suppressed from a statistical perspective, ensuring the diversity and discriminativeness of the feature space. Verification shows that this can contribute a 1.1 percentage point improvement in accuracy.

[0036] In specific application scenarios, when adding a lower bound constraint to the standard deviation of feature dimension data to generate a variance penalty term, the variance penalty term generation formula can be used to add a lower bound constraint to the standard deviation of feature dimension data and generate a variance penalty term. The variance penalty term generation formula includes: ; in, This represents the value of the variance penalty term; This represents the target variance threshold, and its value can be 1.0, etc. The penalty coefficient represents the variance, and its value can be 25, 0, etc.; Z represents the feature dimension data. N represents the batch size; d represents the standard deviation of the j-th feature dimension on the in-batch samples; d represents the embedding dimension, for example, d is 2048; max represents taking the maximum value.

[0037] In specific application scenarios, when constraining the linear correlation between different feature dimensions in feature-dimensional data and generating a covariance penalty term, the covariance penalty term generation formula can be used to constrain the linear correlation between different feature dimensions in feature-dimensional data and generate a covariance penalty term. The covariance penalty term generation formula includes: ; in, This represents the value of the covariance penalty term; The value represents the penalty coefficient for covariance, which can be 1.0, etc.; Z represents the feature dimension data. Let Z be the covariance matrix. ; represents the value at the i-th row and j-th column in the covariance matrix; d represents the embedding dimension. In this way, the covariance penalty term can penalize the sum of squares of all off-diagonal covariance elements, driving the covariance matrix towards a diagonal structure, thus making the linear correlation between different feature dimensions approach zero.

[0038] In specific application scenarios, during the process of generating the total loss value of the gesture recognition model based on the temporal contrastive loss value, variance penalty term, and covariance penalty term, the global contrastive loss value of the gesture recognition model can also be generated. The total loss value of the gesture recognition model is generated by weighting the global contrastive loss value, the temporal contrastive loss value, the variance penalty term, and the covariance penalty term. ,For example ,in, wait.

[0039] It should be noted that in generating the global contrastive loss value of the gesture recognition model, the first spatiotemporal feature sequence and the first spatiotemporal prediction vector of the first training view generated by the gesture recognition model can be extracted, and the second spatiotemporal feature sequence and the second spatiotemporal prediction vector of the second training view generated by the gesture recognition model can be extracted. Based on the differences between the first spatiotemporal feature sequence and the second spatiotemporal prediction vector, and the differences between the second spatiotemporal feature sequence and the first spatiotemporal prediction vector, the global contrastive loss value is generated. The formula for generating the global contrastive loss is as follows: ,in, Indicates the time-series comparison loss value; D z1 represents the negative cosine similarity; p1 represents the first spatiotemporal feature sequence; z2 represents the first spatiotemporal prediction vector; p2 represents the second spatiotemporal feature sequence; and sg represents the gradient stopping operation.

[0040] It should also be noted that the principles for extracting the first and second spatiotemporal feature sequences are the same, as are the principles for extracting the first and second spatiotemporal prediction vectors. For example, a Spiking ResNet-18 encoder can be applied to process the training view to obtain backbone output features, with dimensions such as 512. These backbone output features are then processed through three MLP projection heads, successively reducing the dimensions from 512 to 2048, 2048, and 2048, to obtain the spatiotemporal feature sequences. Each fully connected layer of the three MLP projection heads is followed by batch normalization, and the batch normalization of the first two MLP projection heads is followed by ReLU (Rectified Linear Array). The unit (corrected linear unit) is activated, and the last layer of the MLP projection head only undergoes batch normalization with no activation. Finally, the spatiotemporal feature sequence is processed by the bottleneck structure prediction head to obtain the spatiotemporal prediction vector. The bottleneck structure prediction head processes the dimension from 2048 to 512 and then 2048 successively. The intermediate layers of the bottleneck structure prediction head undergo batch normalization and ReLU activation, while the output layer has no activation. It should be noted that when the gesture recognition model processes the first training view, it extracts the first spatiotemporal feature sequence and the first spatiotemporal prediction vector; when the gesture recognition model processes the second training view, it extracts the second spatiotemporal feature sequence and the second spatiotemporal prediction vector.

[0041] In practical applications, considering that SNNs use a globally fixed membrane time constant, and that neurons at different levels of SNNs need to capture features at different time scales—for example, shallow neurons process rapidly changing features such as local edges and motion directions, requiring a shorter time memory window, while deep neurons integrate slowly changing features such as global semantics and motion trajectories, requiring a longer time memory window—a fixed and uniform membrane time constant cannot meet these hierarchical temporal requirements. This results in gesture recognition models being unable to simultaneously adapt to dynamic patterns with different temporal spans, such as rapid waving and slow rotation, leading to low accuracy in recognizing gestures with long temporal dependencies. To avoid this situation, this application can also obtain unconstrained parameters set for the LIF layer in the gesture recognition model. For example, the gesture recognition model has a total of 16 independent... The parameters consist of two ResBlocks each from layer 1 to layer 4, with each block containing two LIF layers. The total number of parameters is only 16 scalars, and their impact on the total number of parameters in the model is negligible. The Sigmoid function is applied to map the unconstrained parameters, resulting in learnable parameters whose values ​​fall within a set range. Assuming the set interval is [0.5, 10.0], then It can be , The Sigmoid function is represented by a lower bound of 0.5, corresponding to a decay coefficient of decay = exp(-1 / 0.5) ≈ 0.135, suitable for shallow feature extraction requiring rapid response to instantaneous input. An upper bound of 10.0 corresponds to decay = exp(-1 / 10) ≈ 0.905, suitable for deep temporal modeling requiring long-term accumulation. The learnable parameter is set as the membrane time constant of the LIF layer. In this way, the membrane time constant of each LIF neuron layer is designed as a parameter that can be optimized through backpropagation, allowing neurons at different levels to automatically learn a temporal memory window suitable for their function, significantly improving the representation ability of neuromorphic event stream data. Verification shows that after training, each layer... The spontaneous formation of a hierarchical distribution with shallower receptive fields (1.5~2.5) and deeper receptive fields (3.0~5.0) validates the design intuition that neurons at different levels should have different temporal receptive fields. The results were validated using the DVS128 Gesture dataset, achieving a 1.9 percentage point improvement in accuracy and narrowing the gap between the accuracy of the self-supervised model and the upper bound of the supervised model to 0.2 percentage points.

[0042] It should be noted that the training process of the gesture recognition model can be flexibly combined according to the above embodiments. For example, the training process can combine all the above embodiments, and the overall process is as follows: Figure 3 As shown, the two Spiking ResNet-18s share parameters, and the two timing projection heads also share parameters.

[0043] In practical applications, gesture recognition models can be lightweighted to achieve an optimal balance between accuracy and energy efficiency, reducing the energy consumption of CPUs, GPUs, or neuromorphic chips. During lightweighting, channel-level structured pruning can be performed first. For example, the L1 norm distribution of channel weights naturally formed during pre-training can guide importance assessment, pruning the bottom 30% of redundant channels and correspondingly deleting the input dimensions of subsequent layers. Next, adaptive firing threshold optimization can be performed. For instance, after pruning, the LIF neuron threshold can be dynamically adjusted based on the actual average firing rate of each layer to maintain a balanced firing rate across the entire network. Then, soft constraints on the impulse firing rate can be applied, such as introducing a firing rate penalty term into the fine-tuning loss to control the average firing rate of the entire network at a target value of around 0.15, further reducing inference energy consumption. Finally, transfer learning can be performed. For example, after pruning, 50 rounds of end-to-end fine-tuning can be performed using a very small learning rate (e.g., 1e-4) to correct the accuracy loss caused by structural adjustments. Assuming the gesture model is trained using the SGD (Stochastic Gradient Descent) optimizer with an initial learning rate of 0.03, cosine annealing scheduling, and a total of 800 training epochs, and the neuromorphic data batch size is adjusted to 64 with an initial learning rate of 0.0075, and a total of 200 training epochs, gradient clipping is added to prevent explosion; in the linear evaluation phase, all encoder parameters are frozen, and a single-layer linear classifier is trained on the output features, using Top-1 accuracy as the evaluation metric for the quality of the pre-trained representation; verified, this embodiment can control the accuracy loss to within 2.7 percentage points under 2.0x parameter compression, and theoretically reduce the inference energy consumption to about 6.6% of that of an ANN of the same size, saving about 15 times the energy, making it particularly suitable for low-power deployment at the edge.

[0044] Step S107: Obtain the gesture view to be recognized.

[0045] Step S108: Apply the gesture recognition model to recognize the gesture view and obtain the gesture recognition result.

[0046] In practical applications, after training the gesture recognition model, the gesture view to be recognized can be obtained. The gesture view can be a static gesture view or a dynamic gesture view. Then, the gesture recognition model is applied to recognize the gesture view to obtain the gesture recognition result, which can be a gesture action, gesture command, etc.

[0047] This application provides a gesture recognition method, which involves: acquiring a first training view and a second training view; training a gesture recognition model based on a spiking neural network model using the first and second training views; extracting a first spatial feature sequence and a first spatial prediction vector from the first training view generated by the gesture recognition model; extracting a second spatial feature sequence and a second spatial prediction vector from the second training view generated by the gesture recognition model; generating a temporal contrast loss value based on the differences between the first spatial feature sequence and the second spatial prediction vector, as well as the differences between the second spatial feature sequence and the first spatial prediction vector; generating a total loss value for the gesture recognition model based on the temporal contrast loss value, and adjusting the gesture recognition model according to the total loss value to obtain a trained gesture recognition model; acquiring a gesture view to be recognized; and applying the gesture recognition model to recognize the gesture view to obtain a gesture recognition result. In this application, the spatial feature sequence and corresponding spatial prediction vector of the gesture recognition model are extracted during the training process. Since the spatial feature sequence and spatial prediction vector represent the temporal pulse sequence of the spiking neural network transmitting view information, generating a temporal contrast loss value based on the difference between the spatial feature sequence and the spatial prediction vector is equivalent to incorporating the local features of each time step of the spiking neural network into the self-supervised signal system, and applying cross-view loss to the gesture recognition model at the time step level. Figure 1 Consistency constraints are used to fully leverage the temporal computation advantages of spiking neural networks to enhance the training accuracy of gesture recognition models, thereby improving the recognition accuracy of gesture recognition models.

[0048] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a gesture recognition system provided in an embodiment of this application.

[0049] This application provides a gesture recognition system that may include: Training view acquisition module 101 is used to acquire a first training view and a second training view; Model training module 102 is used to train the gesture recognition model using the first training view and the second training view. The gesture recognition model is built based on the spiking neural network model. The first spatial data extraction module 103 is used to extract the first spatial feature sequence and the first spatial prediction vector of the first training view generated by the gesture recognition model. The second spatial data extraction module 104 is used to extract the second spatial feature sequence and the second spatial prediction vector of the second training view generated by the gesture recognition model; and the first spatial data extraction module and the second spatial data extraction module share parameters, and in some application scenarios, they can be the same module. The temporal contrast loss value generation module 105 is used to generate temporal contrast loss values ​​based on the differences between the first spatial feature sequence and the second spatial prediction vector, as well as the differences between the second spatial feature sequence and the first spatial prediction vector. The total loss value generation module 106 is used to generate the total loss value of the gesture recognition model based on the temporal comparison loss value, and adjust the gesture recognition model according to the total loss value to obtain the trained gesture recognition model. Gesture view acquisition module 107 is used to acquire the gesture view to be recognized; The gesture recognition module 108 is used to apply a gesture recognition model to recognize the gesture view and obtain the gesture recognition result.

[0050] This application provides a gesture recognition system, in which a temporal comparison loss value generation module can be used to: generate a temporal comparison loss value based on the difference between the first spatial feature sequence and the second spatial prediction vector, and the difference between the second spatial feature sequence and the first spatial prediction vector, using a temporal comparison loss generation formula; The formula for generating time-series contrast loss includes: ; in, Indicates the time-series comparison loss value; D Indicates negative cosine similarity; This represents the value at time step t in the first spatial feature sequence; This represents the value at time step t in the first spatial prediction vector; This represents the value at time step t in the second spatial feature sequence; sg represents the value at time step t in the second spatial prediction vector; sg represents the gradient stopping operation; T represents the total number of time steps.

[0051] This application provides a gesture recognition system, in which a total loss value generation module may include: The feature dimension extraction unit is used to extract the feature dimension data output by the projection head of the gesture recognition model; The variance penalty term generation unit is used to add a lower bound constraint to the standard deviation of the feature dimension data and generate a variance penalty term. The covariance penalty term generation unit is used to constrain the linear correlation between different feature dimensions in the feature dimension data and generate a covariance penalty term. The total loss value generation unit is used to generate the total loss value of the gesture recognition model based on the temporal comparison loss value, variance penalty term, and covariance penalty term.

[0052] This application provides a gesture recognition system in which a variance penalty term generation unit is used to: add a lower bound constraint to the standard deviation of the feature dimension data through a variance penalty term generation formula to generate a variance penalty term. The formulas for generating the variance penalty term include: ; in, This represents the value of the variance penalty term; Indicates the target variance threshold; The penalty coefficient represents the variance; Z represents the feature dimension data. d represents the standard deviation of the j-th feature dimension on the batch samples; d represents the embedding dimension; max represents taking the maximum value.

[0053] This application provides a gesture recognition system in which a covariance penalty term generation unit is used to: constrain the linear correlation between different feature dimensions in feature dimension data by using a covariance penalty term generation formula to generate a covariance penalty term; The formulas for generating the covariance penalty term include: ; in, This represents the value of the covariance penalty term; The penalty coefficient for covariance is represented by Z; Z represents the feature dimension data. Denotes the covariance matrix of Z; d represents the value at the i-th row and j-th column of the covariance matrix; d represents the embedding dimension.

[0054] This application provides a gesture recognition system, including a total loss value generation unit, which is used to: generate a global contrastive loss value for a gesture recognition model; and perform weighted calculations on the global contrastive loss value, the temporal contrastive loss value, the variance penalty term, and the covariance penalty term to generate the total loss value of the gesture recognition model.

[0055] The gesture recognition system provided in this application embodiment may further include: The unconstrained parameter acquisition module is used to acquire the unconstrained parameters set for the LIF layer in the gesture recognition model; The mapping module is used to apply the Sigmoid function to map unconstrained parameters to obtain learnable parameters whose values ​​are within a set range; The membrane time constant setting module is used to set the learnable parameter as the membrane time constant of the LIF layer.

[0056] This application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the gesture recognition method provided in the embodiments of this application. Please refer to... Figure 5 , Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0057] An electronic device provided in this application includes a memory 201 and a processor 202. The memory 201 stores a computer program, and the processor 202 executes the computer program to implement the steps of the gesture recognition method described in any of the above embodiments.

[0058] Please see Figure 6 Another electronic device provided in this application embodiment may further include: an input port 203 connected to the processor 202 for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside; and a communication module 205 connected to the processor 202 for enabling communication between the electronic device and the outside. The display unit 204 may be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes, but is not limited to, Mobile High-Definition Link (MHL), Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), wireless connection: Wireless Fidelity (WiFi), Bluetooth communication technology, Bluetooth Low Energy communication technology, and communication technology based on IEEE 802.11s.

[0059] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the gesture recognition method described in any of the above embodiments.

[0060] The computer-readable storage media involved in this application include random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (compact disc read-only memory), or any other form of storage media known in the art.

[0061] This application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the gesture recognition method described in any of the above embodiments.

[0062] For descriptions of relevant parts of the gesture recognition system, electronic device, and computer-readable storage medium provided in the embodiments of this application, please refer to the detailed descriptions of the corresponding parts in the gesture recognition method provided in the embodiments of this application, which will not be repeated here. Furthermore, parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0063] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture recognition method, characterized in that, include: Obtain the first training view and the second training view; The gesture recognition model is trained using the first training view and the second training view. The gesture recognition model is built based on a spiking neural network model. Extract the first spatial feature sequence and the first spatial prediction vector of the first training view generated by the gesture recognition model; Extract the second spatial feature sequence and the second spatial prediction vector of the second training view generated by the gesture recognition model; Based on the differences between the first spatial feature sequence and the second spatial prediction vector, as well as the differences between the second spatial feature sequence and the first spatial prediction vector, a temporal contrast loss value is generated. Based on the time-series comparison loss value, the total loss value of the gesture recognition model is generated, and the gesture recognition model is adjusted according to the total loss value to obtain the trained gesture recognition model. Obtain the view of the gesture to be recognized; The gesture recognition model is applied to recognize the gesture view to obtain the gesture recognition result.

2. The method according to claim 1, characterized in that, Based on the differences between the first spatial feature sequence and the second spatial prediction vector, and the differences between the second spatial feature sequence and the first spatial prediction vector, a temporal contrastive loss value is generated, including: The temporal contrast loss value is generated by using the temporal contrast loss generation formula based on the difference between the first spatial feature sequence and the second spatial prediction vector, as well as the difference between the second spatial feature sequence and the first spatial prediction vector. The formula for generating time-series contrast loss includes: ； in, Indicates the time-series comparison loss value; D Indicates negative cosine similarity; This represents the value at time step t in the first spatial feature sequence; This represents the value at time step t in the first spatial prediction vector; This represents the value at time step t in the second spatial feature sequence; sg represents the value at time step t in the second spatial prediction vector; sg represents the gradient stopping operation; T represents the total number of time steps.

3. The method according to claim 1, characterized in that, Based on the temporal comparison loss value, the total loss value of the gesture recognition model is generated, including: Extract the feature dimension data output from the projection head of the gesture recognition model; Add a lower bound constraint to the standard deviation of the feature dimension data to generate a variance penalty term; Constrain the linear correlation between different feature dimensions in the feature dimension data and generate a covariance penalty term; Based on the temporal comparison loss value, the variance penalty term, and the covariance penalty term, the total loss value of the gesture recognition model is generated.

4. The method according to claim 3, characterized in that, Add a lower bound constraint to the standard deviation of the feature dimension data to generate a variance penalty term, including: By generating a formula for the variance penalty term, a lower bound constraint is added to the standard deviation of the feature dimension data to generate the variance penalty term. The formulas for generating the variance penalty term include: ； in, This represents the value of the variance penalty term; Indicates the target variance threshold; The penalty coefficient represents the variance; Z represents the feature dimension data. d represents the standard deviation of the j-th feature dimension on the batch samples; d represents the embedding dimension; max represents taking the maximum value.

5. The method according to claim 3, characterized in that, Constraining the linear correlation between different feature dimensions in the feature dimension data generates a covariance penalty term, including: By generating a covariance penalty term formula, the linear correlation between different feature dimensions in the feature dimension data is constrained, and a covariance penalty term is generated. The formulas for generating the covariance penalty term include: ； in, This represents the value of the covariance penalty term; The penalty coefficient for covariance is represented by Z; Z represents the feature dimension data. Denotes the covariance matrix of Z; d represents the value at the i-th row and j-th column of the covariance matrix; d represents the embedding dimension.

6. The method according to claim 3, characterized in that, Based on the temporal contrast loss value, the variance penalty term, and the covariance penalty term, the total loss value of the gesture recognition model is generated, including: Generate the global contrastive loss value for the gesture recognition model; The global contrast loss value, the temporal contrast loss value, the variance penalty term, and the covariance penalty term are weighted and calculated to generate the total loss value of the gesture recognition model.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: Obtain the unconstrained parameters set for the LIF layer in the gesture recognition model; The Sigmoid function is applied to map unconstrained parameters to obtain learnable parameters whose values ​​are within a set range; The learnable parameter is set as the membrane time constant of the LIF layer.

8. A gesture recognition system, characterized in that, include: The training view acquisition module is used to acquire the first training view and the second training view. The model training module is used to train the gesture recognition model using the first training view and the second training view. The gesture recognition model is built based on a spiking neural network model. The first spatial data extraction module is used to extract the first spatial feature sequence and the first spatial prediction vector of the first training view generated by the gesture recognition model. The second spatial data extraction module is used to extract the second spatial feature sequence and the second spatial prediction vector of the second training view generated by the gesture recognition model. The temporal contrast loss value generation module is used to generate temporal contrast loss values ​​based on the differences between the first spatial feature sequence and the second spatial prediction vector, as well as the differences between the second spatial feature sequence and the first spatial prediction vector. The total loss value generation module is used to generate the total loss value of the gesture recognition model based on the time-series comparison loss value, and adjust the gesture recognition model according to the total loss value to obtain the trained gesture recognition model. The gesture view acquisition module is used to acquire the gesture view to be recognized. The gesture recognition module is used to apply a gesture recognition model to recognize the gesture view and obtain the gesture recognition result.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the gesture recognition method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the gesture recognition method as described in any one of claims 1 to 7.

Images