Event camera based eye localization method and system

Abstract

The application discloses a human eye positioning method and system based on an event camera, and relates to the field of human eye positioning. The human eye is photographed by using the event camera, and then an event data set is obtained; the event data is divided to obtain a training set, a verification set and a test set; parameters in a training stage are acquired and initialized; the training set and the verification set are preprocessed; a pulse neural network model is built; training parameters are set and the pulse neural network model is trained; the trained pulse neural network model is saved; logs are saved; parameters in an inference stage are acquired; the test set is acquired and preprocessed; the trained model is acquired; the preprocessed test set is inferred by using the pulse neural network model; the upper left corner coordinates, the width and height of the prediction box, the category and the category confidence obtained by inference are saved; the position of the detected human eye is output, and the category and the corresponding confidence are framed and displayed. The application can quickly and accurately position the human eye.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human eye positioning, in particular to a human eye positioning method and system based on an event camera. BACKGROUND

[0002] With the development of society, the demand for human eye positioning is increasing. Human eye positioning is to position the user's pupil through camera shooting, and then judge the user's gaze area. At present, human eye positioning has a wide range of applications in human-computer interaction, virtual reality, medicine and safety fields, including tracking the user's line of sight to better understand the user's interactive behavior and intention and improve the efficiency of human-computer interaction; in virtual reality, to realize more realistic and smooth scene perspective conversion and more realistic visual experience.

[0003] The existing human eye positioning technology is based on the image frame of a traditional camera to determine the eye position, which has the disadvantages of low time resolution, high data volume and high data redundancy. SUMMARY

[0004] The present application aims to solve the problems of the prior art, and provides a human eye positioning method and system based on an event camera to solve the problems of low time resolution, high data volume and high data redundancy of the existing human eye positioning technology.

[0005] To solve the above problems, the present application provides a human eye positioning system based on an event camera, comprising:

[0006] An event camera is used to obtain event data and send the event data to a data set module;

[0007] A data set module is used to receive the event data sent by the event camera, obtain an event data set composed of event data, coordinates of human eyes and detection categories, and send the event data set to a data preprocessing module;

[0008] A data preprocessing module is used to receive the data set sent by the data set module and read out discrete event data, pre-process the event data to obtain pre-processed event data, and send the pre-processed event data to a pulse neural network model module; the method for pre-processing the event data is to re-encode the event data using a voxel cube method to obtain a dense representation of the event data;

[0009] A pulse neural network model module is used to receive the pre-processed event data sent by the data preprocessing module and extract features of different scales to obtain feature maps of different scales and send the feature maps of different scales to a detection module;

[0010] The detection module receives several feature maps of different scales sent by the spiking neural network model module; performs loss function calculations through the SSD detection head; and obtains the predicted eye position bounding box, display category, and confidence level of the eye position.

[0011] The spiking neural network model module further includes:

[0012] The feature extraction module consists of N stages. The first stage receives preprocessed event data from the data preprocessing module, extracts features from the event data to obtain a first-level feature map, and sends the first-level feature map to the next stage. The second to N-1 stages receive feature maps output from the previous stage, extract features from the feature maps to obtain new feature maps, and send them to the next stage and the detection module. The Nth stage receives feature maps from the N-1th stage, extracts features to obtain new feature maps, and sends the new feature maps to the attachment module and the detection module.

[0013] The additional module contains three convolutional blocks, Extras, which are used to receive the feature map sent by the Nth stage. Feature extraction and enhanced feature extraction are performed sequentially through Extras1 to Extras3 to obtain higher-scale feature maps. The feature maps of different scales output by Extras1 to Extras3 are sent to the final detection module.

[0014] A second aspect of the present invention provides a human eye localization method based on an event camera, comprising the following steps:

[0015] Step 1: Use an event camera to capture images of human eyes, obtain the captured event data, and label the coordinates and category information of the human eyes in the event data to obtain the event dataset;

[0016] Step 2: Divide the event data in the event dataset into training set, validation set and test set;

[0017] Step 3: Obtain the training phase parameters from the configuration file and initialize them;

[0018] Step 3.1: Read the parameters for the training phase from the configuration file; the parameters include the training set path, validation set path, detection category, batch size of samples read in each training session, cumulative sampling time, time step, size of time bin and input, learning rate, weight decay, number of training epochs, storage address for logs and training results model, aspect ratio of candidate boxes, intersection-over-union ratio (IoU), and threshold for redundancy removal (NMS).

[0019] Step 3.2: Determine whether the paths for the training set and validation set exist based on the parameters in the training phase of the configuration file. If they do not exist, stop running and report an error; if they exist, continue.

[0020] Step 4: Preprocess the training set to obtain the preprocessed training set;

[0021] Step 4.1: Obtain the training set;

[0022] Step 4.2: Preprocess the training set; the preprocessing method is to re-encode the event data in the training set from sparse representation to dense representation using the Voxelcubes method.

[0023] Step 5: Preprocess the validation set to obtain the preprocessed validation set;

[0024] Step 6: Build the CA-SpikingRepVGG neural network model;

[0025] Step 6.1: Build the SpikingRepVGG block, a multi-branch pulse module;

[0026] The spiking multi-branch module has a multi-branch structure, consisting of three branches. The first branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after a 3x3 convolutional layer. The second branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after a 1x1 convolutional layer. The third branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after the identity mapping (identity). The max pooling outputs of the three branches are summed, and the sum is then processed by a CA module and a ReLU activation function.

[0027] Step 6.2: Construct several states (Stages) sequentially based on the pulse multi-branch module;

[0028] Each Stage is composed of several SpikingRepVGG blocks connected in series.

[0029] Step 6.3: Construct a feature extraction module based on several stages;

[0030] Step 6.4: Construct an additional module, which contains three convolutional blocks, each of which contains a BN layer, a 1x1 convolution, and a PLIF module;

[0031] Step 6.5: Add an additional module after the feature extraction module;

[0032] Step 6.6: Add an SSD detection head after the additional module to obtain the final spiking neural network model CA-SpikingRepVGG.

[0033] Step 7: Set the training parameters and train the spiking neural network model using the training and validation sets;

[0034] Step 7.1: Set the spiking neural network model to training mode;

[0035] Step 7.2: Based on the training phase parameters in the configuration file in Step 3, load the preprocessed training set and validation set, and perform iterative training;

[0036] Step 7.3: Determine if the current training loop count is a multiple of 5. If not, proceed to step 7.4; if so, proceed to step 7.5.

[0037] Step 7.4: Train using the training set according to the parameters in Step 3. If the current training loop cannot be completed, an error feedback is issued, and the training phase parameters in the configuration file are modified according to the log. Then, Step 7.4 is executed again for training. If the current training loop is completed, the smoothed L1 loss function is calculated and the loss function of the training set is written to the log. Then, the learnable parameters are backpropagated in the direction of the decrease in the loss function of the training set to continue training.

[0038] The formula for calculating the loss function is as follows:

[0039]

[0040] In the formula, x represents the numerical difference between the predicted bounding box and the ground truth bounding box, smoothness L1 (x) is the loss function to be determined;

[0041] Step 7.5: Train using the validation set based on the training phase parameters from Step 3. If the current training loop cannot be completed, issue an error feedback, modify the training phase parameters in the configuration file according to the log, and then re-execute Step 7.5 for training. If the current training loop is completed, calculate the smoothed L1 loss function and write the loss function of the validation set into the log. Then, backpropagate the learnable parameters in the direction of the decrease in the loss function of the validation set to continue training.

[0042] Step 7.6: Determine if the current training loop count has reached the maximum value. If yes, proceed to step 7.7; otherwise, proceed to step 7.3.

[0043] Step 7.7: Check the running status of the spiking neural network model in real time through the visualization log, and determine whether the parameters in Step 3 meet the condition that the loss of both the training set and the validation set gradually decreases and the accuracy of the training set gradually increases. If so, save the best-performing model in the training rounds as the final spiking neural network model; otherwise, adjust the parameters in Step 3 and execute Step 7.1 to start training again.

[0044] Step 8: Save the trained spiking neural network model;

[0045] Step 8.1: Save the logs according to the directory where the configuration file is stored;

[0046] Step 8.2: Save the trained spiking neural network model according to the storage directory of the configuration file;

[0047] Step 9: Obtain the parameters from the inference phase configuration file;

[0048] Step 9.1: Read the parameters for the inference stage from the configuration file. The parameters include: test set path, model type and storage address of pre-trained model, batch size of samples read in each training session, cumulative sampling time, time step, size of timebox and input, and storage location of inference results.

[0049] Step 9.2: Determine whether the parameters for the inference stage in the configuration file are read correctly. If correct, proceed with the inference process; otherwise, terminate the program and display an error message.

[0050] Step 10: Obtain the test set and preprocess it to obtain the preprocessed test set;

[0051] Step 11: Obtain the trained model saved in Step 8;

[0052] Step 11.1: Locate the path to the trained spiking neural network model based on the parameters in the configuration file, and read the trained spiking neural network model;

[0053] Step 11.2: Determine whether a trained spiking neural network model has been obtained based on the position. If not, stop and prompt an error. If a trained spiking neural network model has been obtained, proceed to step 12 to perform inference.

[0054] Step 12: Use the trained spiking neural network model to perform inference on the preprocessed test set;

[0055] Step 12.1: Read the parameters for the inference phase from the configuration file;

[0056] Step 12.2: Load the trained spiking neural network model and the preprocessed test set according to the parameters of the inference stage in the configuration file;

[0057] Step 12.3: Set the spiking neural network model to inference mode and perform forward propagation;

[0058] Step 12.4: After forward propagation is completed, the predicted bounding box and the corresponding top-left corner coordinates (x, y), width and height of the predicted bounding box, class, and class confidence are obtained.

[0059] Step 13: Save the coordinates of the top left corner, the width and height of the predicted bounding box, the category, and the category confidence score obtained from the inference;

[0060] Step 14: Output the detected human eye positions, select them with a rectangle, and display the category and corresponding confidence level.

[0061] Compared with the prior art, the technical solution adopted in this invention has the following technical effects:

[0062] This invention proposes a human eye localization method and system based on an event camera. By using a high temporal resolution event camera to capture images of the human eye for near-eye imaging, the captured event data has high temporal resolution and can quickly and accurately locate the human eye. Attached Figure Description

[0063] Figure 1 This is a system block diagram of a human eye positioning system based on an event camera according to an embodiment of the present invention;

[0064] Figure 2 This is a flowchart of a human eye localization method based on an event camera in an embodiment of the invention;

[0065] Figure 3 This is a schematic diagram of event data in an embodiment of the invention;

[0066] Figure 4 This is a flowchart of the training phase in an embodiment of the invention;

[0067] Figure 5 This is a block diagram of the pulse multi-branch module structure in an embodiment of the invention;

[0068] Figure 6 This is a block diagram of the spiking neural network model in an embodiment of the invention;

[0069] Figure 7 This is a flowchart of the reasoning stage in an embodiment of the invention;

[0070] Figure 8 This is a diagram showing the eye positioning result in an embodiment of the invention. Detailed Implementation

[0071] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0072] This embodiment provides a human eye localization system based on an event camera, such as... Figure 1 As shown, it includes:

[0073] The event camera is used to acquire event data and send it to the dataset module.

[0074] The dataset module receives event data sent by the event camera; obtains an event dataset consisting of event data, human eye coordinates, and detection categories; and sends the event dataset to the data preprocessing module.

[0075] The data preprocessing module is used to receive the dataset sent by the dataset module and read the discrete event data; preprocess the event data to obtain preprocessed event data and send the preprocessed event data to the spiking neural network model module;

[0076] The method for preprocessing event data is as follows: the event data is re-encoded using the voxel cube method to obtain a dense representation of the event data;

[0077] The spiking neural network model module is used to receive preprocessed event data sent by the data preprocessing module, extract features at different scales, obtain feature maps at different scales, and send the feature maps at different scales to the detection module.

[0078] The detection module receives several feature maps of different scales sent by the spiking neural network model module; performs loss function calculations through the SSD detection head; and obtains the predicted eye position bounding box, display category, and confidence level of the eye position.

[0079] Furthermore, the spiking neural network model module further includes:

[0080] The feature extraction module consists of N stages. The first stage receives preprocessed event data from the data preprocessing module, extracts features from the event data to obtain a first-level feature map, and sends the first-level feature map to the next stage. The second to N-1 stages receive feature maps output from the previous stage, extract features from the feature maps to obtain new feature maps, and send them to the next stage and the detection module. The Nth stage receives feature maps from the N-1th stage, extracts features to obtain new feature maps, and sends the new feature maps to the attachment module and the detection module.

[0081] In this embodiment, the feature extraction module includes 5 stages, namely Stage 0 to Stage 4. The preprocessed event data is input through Stage 0 and then processed sequentially through Stage 0 to Stage 4 for feature extraction. The feature map output by each stage is used as the input of the next stage. Finally, feature maps of different scales output from Stage 1 to Stage 4 are obtained and these feature maps are sent to the detection module. The feature map output by Stage 4 is also sent to the auxiliary module.

[0082] The additional module contains three convolutional blocks, Extras, which are used to receive the feature map sent by the Nth stage. Feature extraction and enhanced feature extraction are performed sequentially through Extras1 to Extras3 to obtain higher-scale feature maps. The feature maps of different scales output by Extras1 to Extras3 are sent to the final detection module.

[0083] like Figure 2 As shown, the human eye localization method based on an event camera in this embodiment includes the following steps:

[0084] Step 1: Use an event camera to capture images of human eyes, obtain the captured event data, and label the coordinates and category information of the human eyes in the event data to obtain the event dataset;

[0085] In this embodiment, an event camera is used to capture images of the human eye, and the captured event data is saved. Visualization software is used to generate image data from the event data, and the image data is manually labeled, including the coordinates of the human eye and the detection category. This results in an event dataset composed of the event data, the coordinates of the human eye, and the detection category. Figure 3 As shown; in this embodiment, only human eye detection is performed, so the detection category is: 1 for detected and 0 for not detected, and no display is made for not detected;

[0086] Step 2: Divide the event data in the event dataset into training set, validation set and test set;

[0087] In this embodiment, the collected event dataset is divided into training set, validation set and test set according to a 6:2:2 ratio.

[0088] like Figure 4 As shown, the training phase specifically includes the following steps:

[0089] Step 3: Obtain the training phase parameters from the configuration file and initialize them;

[0090] Step 3.1: Read the parameters for the training phase from the configuration file; the parameters include the training set path, validation set path, detection category, batch size of samples read in each training session, cumulative sampling time, time step, size of time bin and input, learning rate, weight decay, number of training epochs, storage address for logs and training results model, aspect ratio of candidate boxes, intersection-over-union ratio (IoU), and threshold for redundancy removal (NMS).

[0091] In this embodiment, only human eye detection is performed, so the detection category is set to 1;

[0092] Step 3.2: Determine whether the paths for the training set and validation set exist based on the parameters in the training phase of the configuration file. If they do not exist, stop running and report an error; if they exist, continue.

[0093] Step 4: Preprocess the training set to obtain the preprocessed training set;

[0094] Step 4.1: Obtain the training set;

[0095] Step 4.2: Preprocess the training set;

[0096] The preprocessing method in this embodiment is as follows: the event data in the training set is re-encoded using the voxel cube method, which encodes the event data from sparse representation to dense representation;

[0097] Step 5: Preprocess the validation set to obtain the preprocessed validation set;

[0098] Step 6: Build the CA-SpikingRepVGG neural network model;

[0099] Step 6.1: Build the SpikingRepVGG block, a multi-branch pulse module;

[0100] The spiking multi-branch module has a multi-branch structure, consisting of three branches. The first branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after a 3x3 convolutional layer. The second branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after a 1x1 convolutional layer. The third branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after the identity mapping (identity). The max pooling outputs of the three branches are summed, and the sum is then processed by a CA module and a ReLU activation function.

[0101] In this embodiment, as followsFigure 5 As shown, the preprocessed event data undergoes 3x3 convolution, 1x1 convolution, and identity mapping to obtain features at different scales. These features are then passed through batch normalization (BN) layers to regularize the model and accelerate convergence. For each branch, a spiking neuron module (PFIL module) is added to introduce neuron models to different branches, increasing the model's feature extraction and representation capabilities. Max pooling is also introduced to reduce computation while preserving important feature information. The features obtained from max pooling at different scales are summed and then passed through a CA module. The CA module embeds positional information into the channels to obtain attention information for each feature point, improving the model's feature extraction capability. Finally, a ReLU activation function is applied to obtain a new feature. This activation function introduces non-linearity into the model while preventing overfitting.

[0102] Step 6.2: Construct several states (Stages) sequentially based on the pulse multi-branch module;

[0103] Each Stage is composed of several SpikingRepVGG blocks connected in series.

[0104] In this embodiment, Stage 0 consists of one pulse multi-branch module; Stage 1 consists of two pulse multi-branch modules connected in series; Stage 2 consists of four pulse multi-branch modules connected in series; Stage 3 consists of fourteen pulse multi-branch modules connected in series; and Stage 4 consists of one pulse multi-branch module. The preprocessed event data is processed through Stage 0 to obtain a feature map, and then sequentially passed through Stage 1, Stage 2, Stage 3, and Stage 4 for feature extraction, extracting higher semantic information. Four feature maps of different depths are obtained after passing through Stage 1, Stage 2, Stage 3, and Stage 4 respectively, facilitating multi-scale input to the detection head.

[0105] Step 6.3: Construct a feature extraction module based on several stages;

[0106] Step 6.4: Construct an additional module, which contains three convolutional blocks, each of which contains a BN layer, a 1x1 convolution, and a PLIF module;

[0107] In this embodiment, three convolutional blocks are added: Extras1, Extras2, and Extras3. The feature map output from Stage4 is further extracted using these three convolutional blocks. Each convolutional block outputs a feature map, which is then input to the detection head at multiple scales.

[0108] Step 6.5: Add an additional module after the feature extraction module;

[0109] Step 6.6: Add an SSD detection head after the additional module to obtain the final spiking neural network model CA-SpikingRepVGG.

[0110] In this embodiment, the SSD detection head receives feature maps from Stages 1 to 4 and feature maps from the three convolutional blocks to obtain category and location bounding box detection results. The final spiking neural network model is as follows: Figure 6 As shown.

[0111] Step 7: Set the training parameters and train the spiking neural network model using the training and validation sets;

[0112] Step 7.1: Set the spiking neural network model to training mode;

[0113] Step 7.2: Based on the training phase parameters in the configuration file in Step 3, load the preprocessed training set and validation set, and perform iterative training;

[0114] Step 7.3: Determine if the current training loop count is a multiple of 5. If not, proceed to step 7.4; if so, proceed to step 7.5.

[0115] Step 7.4: Train using the training set according to the parameters in Step 3. If the current training loop cannot be completed, an error feedback is issued, and the training phase parameters in the configuration file are modified according to the log. Then, Step 7.4 is executed again for training. If the current training loop is completed, the smoothed L1 loss function is calculated and the loss function of the training set is written to the log. Then, the learnable parameters are backpropagated in the direction of the decrease in the loss function of the training set to continue training.

[0116] The formula for calculating the loss function is as follows:

[0117]

[0118] In the formula, x represents the numerical difference between the predicted bounding box and the ground truth bounding box, smoothness L1 (x) is the loss function to be determined.

[0119] Step 7.5: Train using the validation set based on the training phase parameters from Step 3. If the current training loop cannot be completed, issue an error feedback, modify the training phase parameters in the configuration file according to the log, and then re-execute Step 7.5 for training. If the current training loop is completed, calculate the smoothed L1 loss function and write the loss function of the validation set into the log. Then, backpropagate the learnable parameters in the direction of the decrease in the loss function of the validation set to continue training.

[0120] Step 7.6: Determine if the current training loop count has reached the maximum value. If yes, proceed to step 7.7; otherwise, proceed to step 7.3.

[0121] Step 7.7: Check the running status of the spiking neural network model in real time through the visualization log, and determine whether the parameters in Step 3 meet the condition that the loss of both the training set and the validation set gradually decreases and the accuracy of the training set gradually increases. If so, save the best-performing model in the training rounds as the final spiking neural network model; otherwise, adjust the parameters in Step 3 and execute Step 7.1 to start training again.

[0122] Step 8: Save the trained spiking neural network model;

[0123] Step 8.1: Save the logs according to the directory where the configuration file is stored;

[0124] Step 8.2: Save the trained spiking neural network model according to the storage directory of the configuration file;

[0125] like Figure 7 As shown, the reasoning stage includes the following steps:

[0126] Step 9: Obtain the parameters from the inference phase configuration file;

[0127] Step 9.1: Read the parameters for the inference stage from the configuration file. The parameters include: test set path, model type and storage address of pre-trained model, batch size of samples read in each training session, cumulative sampling time, time step, size of timebox and input, and storage location of inference results.

[0128] Step 9.2: Determine whether the parameters for the inference stage in the configuration file are read correctly. If correct, proceed with the inference process; otherwise, terminate the program and display an error message.

[0129] Step 10: Obtain the test set and preprocess it to obtain the preprocessed test set;

[0130] Step 11: Obtain the trained model saved in Step 8;

[0131] Step 11.1: Locate the path to the trained spiking neural network model based on the parameters in the configuration file, and read the trained spiking neural network model;

[0132] Step 11.2: Determine whether a trained spiking neural network model has been obtained based on the position. If not, stop and prompt an error. If a trained spiking neural network model has been obtained, proceed to step 12 to perform inference.

[0133] Step 12: Use the trained spiking neural network model to perform inference on the preprocessed test set;

[0134] Step 12.1: Read the parameters for the inference phase from the configuration file;

[0135] Step 12.2: Load the trained spiking neural network model and the preprocessed test set according to the parameters of the inference stage in the configuration file;

[0136] Step 12.3: Set the spiking neural network model to inference mode and perform forward propagation;

[0137] Step 12.4: After forward propagation is completed, the predicted bounding box and the corresponding top-left corner coordinates (x, y), width and height of the predicted bounding box, class, and class confidence are obtained.

[0138] Step 13: Save the coordinates of the top left corner, the width and height of the predicted bounding box, the category, and the category confidence score obtained from the inference;

[0139] Step 14: Output the detected human eye positions, select them with a rectangle, and display the category and corresponding confidence level.

[0140] In this embodiment, as follows Figure 8 The position of the human eye is indicated by a rectangular box.

[0141] Mean average precision (mAP) and recall were selected as evaluation metrics. Specifically, mAP and recall were chosen using IoU mAP ([0.5:0.05:0.95]) and [IoU=0.50:0.95|area=all|maxDets=1], respectively. This model achieved an mAP of 70.8% and a recall of 76.0%, demonstrating excellent detection performance.

Claims

1. A human eye positioning system based on an event camera, characterized in that, include: The event camera is used to acquire event data and send it to the dataset module. The dataset module is used to receive event data sent by the event camera; The event dataset, consisting of event data, eye coordinates, and detection categories, is obtained and sent to the data preprocessing module. The data preprocessing module is used to receive the dataset sent by the dataset module and read the discrete event data; preprocess the event data to obtain preprocessed event data and send the preprocessed event data to the spiking neural network model module; the method of preprocessing the event data is: to re-encode the event data using the voxel cube method to obtain a dense representation of the event data; The spiking neural network model module is used to receive preprocessed event data sent by the data preprocessing module, extract features at different scales, obtain feature maps at different scales, and send the feature maps at different scales to the detection module. The detection module is used to receive several feature maps of different scales sent by the spiking neural network model module; Loss function calculation is performed using the SSD detection head; Obtain the predicted bounding box for the human eye position, the display category, and the confidence level of the human eye position; The spiking neural network model module further includes: The feature extraction module consists of N stages. The first stage receives preprocessed event data from the data preprocessing module, extracts features from the event data to obtain a first-level feature map, and sends the first-level feature map to the next stage. The second to N-1 stages receive feature maps output from the previous stage, extract features from the feature maps to obtain new feature maps, and send them to the next stage and the detection module. The Nth stage receives feature maps from the N-1th stage, extracts features to obtain new feature maps, and sends the new feature maps to the attachment module and the detection module. Each Stage is composed of several SpikingRepVGG blocks connected in series. The spiking multi-branch module has a multi-branch structure, consisting of three branches. The first branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after a 3x3 convolutional layer. The second branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after a 1x1 convolutional layer. The third branch adds a batch normalization (BN) layer, a spiking neuron model (PFIL), and a max pooling function sequentially after the identity mapping (identity). The max pooling outputs of the three branches are summed, and the sum is then processed by a CA module and a ReLU activation function. The additional module contains three convolutional blocks, Extras, which are used to receive the feature map sent by the Nth stage. Feature extraction and enhanced feature extraction are performed sequentially through Extras1 to Extras3 to obtain higher-scale feature maps. The feature maps of different scales output by Extras1 to Extras3 are sent to the final detection module. Each convolutional block contains a BN layer, a 1x1 convolution, and a PLIF module.

2. A human eye localization method based on an event camera, implemented using the human eye localization system based on an event camera as described in claim 1, is characterized in that... Includes the following steps: Step 1: Use an event camera to capture images of human eyes, obtain the captured event data, and label the coordinates and category information of the human eyes in the event data to obtain the event dataset; Step 2: Divide the event data in the event dataset into training set, validation set and test set; Step 3: Obtain the training phase parameters from the configuration file and initialize them; Step 4: Preprocess the training set to obtain the preprocessed training set; Step 5: Preprocess the validation set to obtain the preprocessed validation set; Step 6: Build the CA-SpikingRepVGG neural network model; Step 7: Set the training parameters and train the spiking neural network model using the training and validation sets; Step 8: Save the trained spiking neural network model; Step 8.1: Save the logs according to the directory where the configuration file is stored; Step 8.2: Save the trained spiking neural network model according to the storage directory of the configuration file; Step 9: Obtain the parameters from the inference phase configuration file; Step 10: Obtain the test set and preprocess it to obtain the preprocessed test set; Step 11: Obtain the trained model saved in Step 8; Step 12: Use the trained spiking neural network model to perform inference on the preprocessed test set; Step 13: Save the coordinates of the top left corner, the width and height of the predicted bounding box, the category, and the category confidence score obtained from the inference; Step 14: Output the detected human eye positions, select them with a rectangle, and display the category and corresponding confidence level.

3. The human eye localization method based on an event camera according to claim 2, characterized in that, Step 3 includes the following specific steps: Step 3.1: Read the parameters for the training phase from the configuration file; the parameters include the training set path, validation set path, detection category, batch size of samples read in each training session, cumulative sampling time, time step, size of time bin and input, learning rate, weight decay, number of training epochs, storage address for logs and training results model, aspect ratio of candidate boxes, intersection-over-union ratio (IoU), and threshold for redundancy removal (NMS). Step 3.2: Determine whether the paths for the training set and validation set exist based on the parameters in the training phase of the configuration file. If they do not exist, stop running and report an error; if they exist, continue.

4. The human eye localization method based on an event camera according to claim 2, characterized in that, Step 4 includes the following specific steps: Step 4.1: Obtain the training set; Step 4.2: Preprocess the training set; the preprocessing method is to re-encode the event data in the training set from sparse representation to dense representation using the voxel cube method.

5. The human eye localization method based on an event camera according to claim 2, characterized in that, Step 6 includes the following specific steps: Step 6.1: Build the SpikingRepVGG block, a multi-branch pulse module; Step 6.2: Construct several states (Stages) sequentially based on the pulse multi-branch module; Step 6.3: Construct a feature extraction module based on several stages; Step 6.4: Construct an additional module containing three convolutional blocks; Step 6.5: Add an additional module after the feature extraction module; Step 6.6: Add an SSD detection head after the additional module to obtain the final spiking neural network model CA-SpikingRepVGG.

6. The human eye localization method based on an event camera according to claim 2, characterized in that, Step 7 includes the following specific steps: Step 7.1: Set the spiking neural network model to training mode; Step 7.2: Based on the training phase parameters in the configuration file in Step 3, load the preprocessed training set and validation set, and perform iterative training; Step 7.3: Determine if the current training loop count is a multiple of 5. If not, proceed to step 7.4; if so, proceed to step 7.

5. Step 7.4: Train using the training set according to the parameters in Step 3. If the current training loop cannot be completed, an error feedback is issued, and the training phase parameters in the configuration file are modified according to the log. Then, Step 7.4 is executed again for training. If the current training loop is completed, the smoothed L1 loss function is calculated and the loss function of the training set is written to the log. Then, the learnable parameters are backpropagated in the direction of the decrease in the loss function of the training set to continue training. The formula for calculating the loss function is as follows: ； In the formula, The numerical difference between the predicted bounding box and the ground truth bounding box. The desired loss function; Step 7.5: Train using the validation set based on the training phase parameters from Step 3. If the current training loop cannot be completed, issue an error feedback, modify the training phase parameters in the configuration file according to the log, and then re-execute Step 7.5 for training. If the current training loop is completed, calculate the smoothed L1 loss function and write the loss function of the validation set into the log. Then, backpropagate the learnable parameters in the direction of the decrease in the loss function of the validation set to continue training. Step 7.6: Determine if the current training loop count has reached the maximum value. If yes, proceed to step 7.7; otherwise, proceed to step 7.

3. Step 7.7: Check the running status of the spiking neural network model in real time through the visualization log, and determine whether the parameters in Step 3 meet the condition that the loss of both the training set and the validation set gradually decreases and the accuracy of the training set gradually increases. If so, save the best-performing model in the training rounds as the final spiking neural network model; otherwise, adjust the parameters in Step 3 and execute Step 7.1 to start training again.

7. The human eye localization method based on an event camera according to claim 2, characterized in that, Step 9 includes the following specific steps: Step 9.1: Read the parameters for the inference stage from the configuration file. The parameters include: test set path, model type and storage address of pre-trained model, batch size of samples read in each training session, cumulative sampling time, time step, size of timebox and input, and storage location of inference results. Step 9.2: Determine whether the parameters for the inference stage in the configuration file are read correctly. If correct, proceed with the inference process; otherwise, terminate the program and display an error message.

8. The human eye localization method based on an event camera according to claim 2, characterized in that, Step 11 includes the following specific steps: Step 11.1: Locate the path to the trained spiking neural network model based on the parameters in the configuration file, and read the trained spiking neural network model; Step 11.2: Determine whether a trained spiking neural network model has been obtained based on the position. If not, stop and prompt an error. If a trained spiking neural network model has been obtained, proceed to step 12 to perform inference.

9. The human eye localization method based on an event camera according to claim 2, characterized in that, Step 12 includes the following specific steps: Step 12.1: Read the parameters for the inference phase from the configuration file; Step 12.2: Load the trained spiking neural network model and the preprocessed test set according to the parameters of the inference stage in the configuration file; Step 12.3: Set the spiking neural network model to inference mode and perform forward propagation; Step 12.4: After forward propagation is completed, the predicted bounding box and the corresponding top-left corner coordinates (x, y), width and height of the predicted bounding box, class, and class confidence are obtained.

Images