A gaze estimation method based on hybrid pulse neural network and joint sparsity optimization

Abstract

The application discloses a gaze estimation method based on a hybrid pulse neural network and joint sparsity optimization, and belongs to the field of computer vision and human-computer interaction. The asymmetric hybrid network architecture with a front-end lightness and a rear-end self-adaptation is adopted, and the full-connection calculation of the front-end redundancy is removed. The structure effectively realizes the lightness of the network parameters while guaranteeing the modeling ability of the time sequence characteristics, effectively reduces the total energy consumption of the single-step system, and improves the deployment cost-effectiveness in the edge computing scene. The pulse sparsity joint optimization strategy is introduced, the MFR regularization penalty term is added in the loss function, and the accumulation speed of the membrane potential of the hidden layer neurons is effectively inhibited through the alternative gradient back propagation. The method reduces the overall pulse firing quantity at the physical level, reduces the redundant pulse firing irrelevant to the prediction accuracy, effectively reduces the synaptic operation number (SOPs) in the reasoning stage, and maintains good feature expression robustness under high sparsity.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and human-computer interaction, and specifically relates to a gaze estimation method based on hybrid spiking neural networks and joint sparsity optimization. Background Technology

[0002] With the widespread adoption of head-mounted devices, near-eye gaze estimation plays a crucial role in applications such as virtual reality. Traditional methods are limited by frame rate and power consumption, while event cameras (such as dynamic vision sensors) provide a new hardware foundation for gaze estimation due to their high temporal resolution and low power consumption. Existing event-driven gaze estimation models based on spiking neural networks (SNNs), while pursuing high prediction accuracy, suffer from certain computational resource waste in their network structure and optimization objectives. This is mainly reflected in the following two aspects: First, there is redundancy in matrix computation in the spatial feature extraction layer. Traditional methods typically use structurally complex Adaptive Leaky-Integrate-and-Fire Neurons (ALIF) in the front-end convolutional layers responsible for extracting local edge features from images. ALIF neurons contain additional fully connected layers to dynamically calculate the membrane potential leakage constant and adaptive threshold based on the current input. Applying such neurons with fully connected computations to high-resolution convolutional feature maps results in the network generating a large number of dense matrix multiplication-addition operations (Dense MACs) in each frame, significantly increasing the number of static parameters and the computational power consumption of the underlying hardware.

[0003] Second, deep neural networks lack explicit constraints on pulse firing. The dynamic energy consumption of spiking neural networks is directly linked to the number of pulses fired by neurons (i.e., the number of synaptic operands, SOPs). Existing methods optimize loss functions only for prediction errors (such as pupil distance and gaze angle), without imposing regularization constraints on the average pulse firing rate (MFR) within the network. In the absence of explicit constraints, the network tends to transmit feature information by firing excessive pulses, leading to a large amount of invalid computation in deep networks. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned computational and energy consumption bottlenecks by improving the network hybrid architecture and end-to-end loss function, and to propose a gaze estimation method based on hybrid spiking neural networks and joint sparsity optimization.

[0005] The technical solution of this invention is as follows: A gaze estimation method based on hybrid spiking neural networks and joint sparsity optimization, comprising the following steps: Step 1: Select the near-eye fixation estimation dataset EV-Eye.

[0006] A near-eye gaze estimation dataset was constructed. This dataset includes event stream data captured by event cameras and corresponding real gaze labels, where the real gaze labels include pupil coordinates and gaze direction. The dataset also covers data from different participants under various eye-tracking tasks (smooth tracking, random saccades, and fixed gaze, etc.). The data in the near-eye gaze estimation dataset was divided into training and testing sets.

[0007] Step 2: Construct event-driven input representations.

[0008] The event stream data from the near-eye fixation estimation dataset in step 1 is extracted, and the event stream data is aggregated into a two-dimensional dual-channel tensor feature map using a fixed number of events per frame. , as the input to the hybrid spiking neural network, where Represents a time step.

[0009] Step 3: Construct a hybrid spiking neural network To address the characteristics of event-driven gaze estimation tasks, which feature relatively simple spatial feature extraction but complex temporal dynamic dependencies, an asymmetric hybrid spiking neural network is constructed, consisting of a front-end spatial feature extraction module and a back-end temporal feature decoding module. The front-end and back-end use different types of neurons to achieve structural asymmetry.

[0010] (1) Constructing a spatial feature extraction module At the network front end, a spatial feature extraction module is constructed for feature encoding. This module consists of four concatenated convolutional blocks. The first two convolutional blocks are composed of a cascaded two-dimensional convolutional layer (Conv2D), a batch normalization layer (BatchNorm2D), a max pooling layer (MaxPool2D), and a parametric leaky-integrate-and-fire (PLIF) layer. Then, in the... The time step, the first The forward propagation process of the convolutional blocks can be represented by formula (1.1): in, ; The latter two sets of convolutional blocks consist of a cascaded layer of two-dimensional convolutional layers (Conv2D), batch normalization layers (BatchNorm2D), and parametric spiking neurons (Parametric Leaky-Integrate-and-Fire, PLIF). This is then applied to the first... The time step, the first The forward propagation process of the convolutional blocks can be represented by formula (1.2): in, ; In formulas (1.1) and (1.2), This indicates that the feature tensor at the current time step is the output of the previous convolutional block and passed to the current group. This is the initial input received by the spatial feature extraction module. After spatial feature extraction through four sets of convolutional blocks, the feature map is obtained. The dimensions are gradually reduced, and finally transformed into a one-dimensional spatiotemporal feature vector through random deactivation layers and flattening operations. And it serves as the input to the temporal feature decoding module.

[0011] (2) Constructing a temporal feature decoding module In the network backend, a temporal feature decoding module is constructed for feature decoding. This module consists of four recurrent blocks. Each recurrent block comprises a linear layer, a batch normalization layer (BatchNorm1D), and an adaptive lean-integrate-and-fire neuron (ALIF) layer with long short-term memory capabilities. It utilizes feature concatenation across time steps to handle temporal dynamic dependencies. The input of the group loop block is Then in the first The time step, the first The forward propagation process of the group of circular blocks can be represented by formula (1.3): in, ; In formula (1.3), This indicates a feature concatenation operation. This indicates that the group was at the previous time step The output discrete pulse state. This is the initial input received by the temporal feature decoding module. After temporal feature decoding through 4 sets of convolutional blocks, the spatiotemporal feature vector is obtained. The features are gradually aggregated into deep eye-tracking trajectory features, and finally mapped through a linear layer and parametric spiking neurons (PLIF) with the reset mechanism disabled, into a five-dimensional prediction vector composed of pupil coordinates and gaze direction. That is, the current time step. The gaze estimation results, which include three-dimensional pupil coordinates. spherical coordinates of the gaze direction in two dimensions .

[0012] Step 4: Construct the joint loss function.

[0013] Building upon the regression loss, the network's average impulse rate (MFR) is introduced as a regularization penalty term into the total loss function to construct a joint loss function. The total loss function is defined as: in, The mean square error (MSE) is used to predict pupil coordinates. To predict the cosine distance loss in the gaze direction. This is a hyperparameter used to control the intensity of the regularization penalty term.

[0014] Step 5: Network training and result output.

[0015] During the model training phase, the training set partitioned in step 1 is input into the hybrid spiking neural network. Backpropagation is performed using surrogate gradients, where the gradient generated by the regularization penalty term in the joint loss function is propagated back along the computational graph to the membrane potential of each lightweight neuron. When updating weights, the optimizer automatically reduces the connection weights of feature channels that contribute less to prediction, making it difficult for the membrane potential of the corresponding lightweight neuron to exceed the firing threshold. After iterative training of the network on the training set until convergence, the test set data partitioned in step 1 is input as the test data into the trained hybrid spiking neural network, outputting the final gaze estimation result.

[0016] The beneficial effects of this invention are: (1) Reduce the number of static parameters and front-end computing power consumption. This invention adopts an asymmetric hybrid network architecture with a lightweight front-end and an adaptive back-end, eliminating redundant fully connected computations at the front end. While ensuring the ability to model temporal features, this structure effectively achieves lightweighting of network parameters, effectively reduces the total energy consumption of a single system step, and improves the deployment cost-effectiveness in edge computing scenarios.

[0017] (2) Reducing redundant pulses and invalid computations. This invention introduces a pulse sparsity joint optimization strategy, adding an MFR regularization penalty term to the loss function, which effectively suppresses the accumulation rate of membrane potentials in hidden layer neurons by replacing gradient backpropagation. This method reduces the overall number of pulses fired at the physical level, reduces redundant pulse firings that are irrelevant to prediction accuracy, effectively reduces the number of synaptic operands (SOPs) in the inference phase, and maintains good feature representation robustness under high sparsity. Attached Figure Description

[0018] Figure 1 This is a partial image display of the dataset described in this embodiment of the invention.

[0019] Figure 2This is the hybrid pulse neural network structure of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0021] To address the event-driven gaze estimation problem, this embodiment uses the EV-Eye gaze estimation dataset as the data source. In terms of data partitioning, the event frame sequence is divided into data segments of 1000 frames in length, and these segments are randomly divided into training and test sets at a ratio of 70% and 30%, respectively. A partial image from the dataset is shown below. Figure 1 As shown. In terms of data preprocessing, the event stream data input to the network is framed with a fixed number of events, with every 1750 events aggregated into a two-dimensional, two-channel tensor. Therefore, at any time step... The input to the network can be represented as a feature map. ,in and These represent the image height and width, and channel dimensions of the feature map, respectively. The positive and negative polarities of the corresponding events.

[0022] In terms of model construction, the hybrid spiking neural network front-end in this embodiment includes a spatial feature extraction module, which consists of four concatenated sets of convolutional blocks. The first two sets of convolutional blocks consist of a concatenated 2D convolutional layer (Conv2D), a batch normalization layer (BatchNorm2D), a max pooling layer (MaxPool2D), and a parametric spiking neuron (ParametricLeaky-Integrate-and-Fire, PLIF) layer. The latter two sets of convolutional blocks also consist of a concatenated 2D convolutional layer (Conv2D), a batch normalization layer (BatchNorm2D), and a parametric spiking neuron (ParametricLeaky-Integrate-and-Fire, PLIF) layer. The network back-end includes a temporal feature decoding module, which consists of four recurrent blocks. Each recurrent block consists of a linear layer, a batch normalization layer (BatchNorm1D), and an adaptive lean-integrate-and-fire neuron (ALIF) layer with long short-term memory capabilities. It utilizes feature concatenation across time steps to handle temporal dynamic dependencies. The hybrid spiking neural network structure, composed of the front-end and back-end, is as follows: Figure 2 As shown.

[0023] In terms of data processing, the input network consists of a two-dimensional dual-channel tensor feature map. After processing by the spatial feature extraction module, the feature map is gradually reduced in dimensionality and then transformed into a one-dimensional spatiotemporal feature vector through a random deactivation layer and a flattening operation. This data is then used as input to the temporal feature decoding module. After processing by the temporal feature decoding module, the spatiotemporal feature vector is obtained. The features are gradually aggregated into deep eye-tracking trajectory features, and finally mapped to a five-dimensional prediction vector through a linear layer and a parameterized spiking neuron (PLIF) with the reset mechanism disabled. As the network output, it represents the current time step. The gaze estimation results, which include three-dimensional pupil coordinates. spherical coordinates of the gaze direction in two dimensions .

[0024] For model training, this embodiment employs the Truncated Backpropagation Time Tribulation (BPTT) algorithm with a truncation step size of T=8, combined with the Forward Propagation Time Tribulation (FPTT) algorithm to stabilize gradient updates for slow dynamic variables. The optimizer uses the Adam algorithm, with an initial learning rate set to... This is combined with the ReduceLROnPlateau learning rate decay strategy. The weight hyperparameter of the regularization penalty term in the joint loss function is set to... By applying gradient penalties to neurons with high pulse firing frequencies, the network is encouraged to learn sparser feature representations. The maximum number of training epochs is set to 200, and the batch size is 80. During training, the average angular error of the model on the training set is recorded, and an optimal checkpoint backtracking strategy is used. During the testing phase, the weight states with the lowest training set error are loaded and finally evaluated on the test set.

[0025] To verify the effectiveness of this invention, a comparative experiment was designed in this embodiment, comparing the proposed method based on hybrid spiking neural networks and joint sparsity optimization (PLIF-ALIF architecture). The results were compared and evaluated with the traditional fully adaptive method (ALIF-ALIF architecture) without sparsity optimization. The experimental results are shown in Table 1.

[0026] Table 1: Performance Comparison of Traditional Methods and Invention Method

[0027] Experimental data show that, compared to traditional methods with higher computational costs, the method of this invention achieves a 19.9% ​​reduction in static parameters (from 2.31 M to 1.85 M), while maintaining a similar angular error (6.87°) and exhibiting superior pupil error (2.47 mm). Furthermore, the average pulse firing rate of this invention is reduced from 0.048 to 0.022, the number of deep synaptic operands is decreased by 62.9% (from 1.05 M to 0.39 M), and the total energy consumption of the single-step system is reduced from 206.61 μJ to 38.61 μJ. This verifies that the asymmetric hybrid architecture and joint optimization strategy proposed in this invention can effectively remove computational redundancy in spatial feature extraction while maintaining gaze estimation accuracy and effectively reducing the dynamic power consumption of the system.

Claims

1. A gaze estimation method based on hybrid spiking neural networks and joint sparsity optimization, characterized in that, The method includes the following steps: Step 1: Select the near-eye fixation estimation dataset EV-Eye; A near-eye gaze estimation dataset is constructed. The near-eye gaze estimation dataset includes event stream data collected by event cameras and corresponding real gaze labels, where the real gaze labels include pupil coordinates and gaze direction. The near-eye gaze estimation dataset covers data from different participants under different eye-tracking tasks. The data in the near-eye gaze estimation dataset is divided into training set and test set. Step 2: Construct event-driven input representations; The event stream data from the near-eye fixation estimation dataset in step 1 is extracted, and the event stream data is aggregated into a two-dimensional dual-channel tensor feature map using a fixed number of events per frame. , as the input to the hybrid spiking neural network, where Represents a time step; Step 3: Construct a hybrid spiking neural network An asymmetric hybrid spiking neural network is constructed, consisting of a front-end spatial feature extraction module and a back-end temporal feature decoding module, wherein different types of neurons are used in the front-end and back-end to achieve structural asymmetry; (1) Constructing a spatial feature extraction module At the network front end, a spatial feature extraction module is constructed for feature encoding. This module consists of four concatenated convolutional blocks; the first two concatenated blocks are composed of a 2D convolutional layer (Conv2D), a batch normalization layer (BatchNorm2D), a max pooling layer (MaxPool2D), and a parameterized spiking neuron (PLIF) layer. Then, in the... The time step, the first The forward propagation process of the convolutional blocks can be represented by formula (1.1): in, ; The last two sets of convolutional blocks consist of a cascaded two-dimensional convolutional layer Conv2D, a batch normalization layer BatchNorm2D, and a parameterized spiking neuron PLIF layer. In the th... The time step, the first The forward propagation process of the convolutional blocks can be represented by formula (1.2): in, ; In formulas (1.1) and (1.2), This indicates that the feature tensor at the current time step is the output of the previous convolutional block and passed to the current group. The initial input received by the spatial feature extraction module; after spatial feature extraction through four sets of convolutional blocks, the feature map... The dimensions are gradually reduced, and finally transformed into a one-dimensional spatiotemporal feature vector through random deactivation layers and flattening operations. And serve as input to the temporal feature decoding module; (2) Constructing a temporal feature decoding module In the network backend, a temporal feature decoding module is constructed for feature decoding. This module consists of four recurrent blocks; each recurrent block comprises a linear layer (Linear), a batch normalization layer (BatchNorm1D), and an adaptive neuron ALIF layer with long short-term memory capability. It utilizes feature concatenation across time steps to handle temporal dynamic dependencies. The input of the group loop block is Then in the first The time step, the first The forward propagation process of the group of circular blocks can be represented by formula (1.3): in, ; In formula (1.3), This indicates the feature concatenation operation. This indicates that the group was at the previous time step Output discrete pulse state The initial input received by the temporal feature decoding module; after temporal feature decoding through 4 sets of convolutional blocks, the spatiotemporal feature vector is obtained. The features are gradually aggregated into deep eye-tracking trajectory features, and finally mapped to a five-dimensional prediction vector composed of pupil coordinates and gaze direction through a linear layer and a parameterized spiking neuron PLIF with the reset mechanism disabled. That is, the current time step. The gaze estimation results, which include three-dimensional pupil coordinates. spherical coordinates of the gaze direction in two dimensions Step 4: Construct the joint loss function; Based on the regression loss, the network's average burst rate (MFR) is introduced as a regularization penalty term into the total loss function to construct a joint loss function; the total loss function is defined as: in, To predict the mean square error (MSE) of pupil coordinates, To predict the cosine distance loss in the gaze direction, This is a hyperparameter used to control the intensity of the regularization penalty term; Step 5: Network training and result output; During the model training phase, the training set partitioned in step 1 is input into the hybrid spiking neural network. Backpropagation is performed using alternative gradients. The gradient generated by the regularization penalty term in the joint loss function is backpropagated along the computation graph to the membrane potential of each lightweight neuron. When updating the weights, the optimizer can automatically reduce the connection weights of feature channels that contribute less to the prediction, making it difficult for the membrane potential of the corresponding lightweight neuron to exceed the firing threshold. After iteratively training the network on the training set until convergence, the test set data partitioned in step 1 is input as the test data into the trained hybrid spiking neural network, and the final gaze estimation result is output.

Images