Eye movement tracking method and device, display equipment, storage medium and program product
Through pulse neural network and pulse vision sensor, the optical flow signal is converted into pulse signals and directly input into the eye movement tracking model, solving the problems of high power consumption and delay in the existing technology, realizing high-precision and low-energy eye movement tracking, improving User interaction experience and system reliability.
Patent Information
- Application Number
- CN202410953120.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-15
AI Technical Summary
The existing image-based eye tracking scheme requires high resolution and high frame rate image sensors and powerful image processing capabilities, resulting in high power consumption, large delay, and susceptible to ambient light interference, reducing user interaction experience and track accuracy.
The pulse neural network is used to track eye movements based on pulse signals, convert the optical flow signal into pulse signals through pulse vision sensors, and directly input the eye movement tracking model for processing, avoiding complex image processing algorithms, simulating the working principle of biological nervous system, and improving Time resolution and energy consumption reduction.
Real-time eye tracking with higher accuracy and low power consumption is achieved, reducing system latency and power consumption, improving user interaction experience and tracking accuracy, suitable for portable virtual reality and augmented reality devices.
Smart Images

Figure CN118965095A_ABST
Abstract
Description
Background Art
[0002] An eye tracking system that relies on image vision sensors to collect eye images requires the image vision sensors to have high resolution and high frame rate, and the eye tracking system to have powerful image processing capabilities. It can obtain useful eye movement information from the images by capturing and analyzing eye images in real time. This method will increase the power consumption of the system and may also increase the system latency, thereby reducing the user's interactive experience.
[0003] In addition, the imaging processing in the eye tracking solution may be affected by ambient light or other external interference, which may affect the accuracy and reliability of eye tracking.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The purpose of the present disclosure is to provide an eye tracking method, an eye tracking device, a display device, a computer-readable storage medium and a computer program product, which can at least to some extent improve the problem of poor interactive experience for users caused by eye tracking solutions in related technologies.
[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0007] According to one aspect of the present disclosure, there is provided an eye tracking method, comprising: in response to a received pulse signal, inputting the pulse signal into an eye tracking model, the eye tracking model outputting eye tracking information corresponding to the pulse signal, the pulse signal being generated by converting a collected optical flow signal, the optical flow signal comprising optical flow information reflecting eye movement, and the eye tracking model being generated based on a pulse neural network.
[0008] In some embodiments, before inputting the pulse signal into the eye tracking model in response to the received pulse signal, the method further includes: constructing the eye tracking model.
[0009] In some embodiments, constructing the eye tracking model includes: training an artificial neural network using an eye tracking dataset to obtain an initial model for eye tracking; and converting the initial model into the eye tracking model based on a mapping relationship between the artificial neural network and the pulse neural network.
[0010] In some embodiments, an eye tracking dataset is used to train an artificial neural network to obtain an initial model for eye tracking, including: using a feature extraction network to encode the eye tracking dataset frame by frame to obtain high-dimensional semantic features that describe eye features in the time domain, and inputting the high-dimensional semantic features into a first fully connected layer and a second fully connected layer respectively, so that the first fully connected layer outputs a predicted yaw angle of the eye, and the second fully connected layer outputs a predicted pitch angle of the eye; based on the relationship between the predicted yaw angle and the corresponding yaw angle label, and the relationship between the predicted pitch angle and the corresponding pitch angle label, updating the model parameters of the feature extraction network, the first fully connected layer, and the second fully connected layer until the training is completed to obtain the initial model, wherein the eye tracking information is determined based on the predicted yaw angle and the predicted pitch angle in the time domain output by the initial model.
[0011] In some embodiments, based on the relationship between the predicted yaw angle and the corresponding yaw angle label, and the relationship between the predicted pitch angle and the corresponding pitch angle label, the model parameters of the feature extraction network, the first fully connected layer, and the second fully connected layer are updated until the training is completed, and also include: constructing a loss function based on a linear combination of a cross entropy loss function and a mean square error; performing back propagation in the time domain on the feature extraction network, the first fully connected layer, and the second fully connected layer based on the loss function to update the model parameters based on back propagation.
[0012] In some embodiments, based on the mapping relationship between the artificial neural network and the pulse neural network, the initial model is converted into the eye tracking model, including: the mapping relationship includes a feature mapping relationship between the dimension of the high-dimensional vector learned by the initial model and the emission frequency of the pulse signal, and based on the feature mapping relationship, the model parameters of the initial model are mapped to the pulse neural network; the parallel first fully connected layer and the second fully connected layer are converted into synaptic connections to obtain multiple related neurons; and the eye tracking model is determined based on the multiple related neurons.
[0013] In some embodiments, in response to the received pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye tracking information corresponding to the pulse signal, including: in response to the pulse signal, determining a pulse pattern and a timing characteristic of the pulse signal within a specified time window, stimulating corresponding neurons in the eye tracking model based on the pulse pattern and the timing characteristic, and generating an output pulse;
[0014] The excited neurons transmit the output pulses to other neurons through synaptic connections until the eye tracking model outputs the yaw angle and pitch angle of the eye corresponding to the optical flow information within the time window as the eye tracking information.
[0015] In some embodiments, the method further includes: generating adjustment instructions for the virtual environment in which the user is located based on the eye tracking information, wherein the adjustment instructions include at least one of a display angle, display content, and interaction information of the virtual environment.
[0016] According to another aspect of the present disclosure, an eye tracking device is provided, including: a tracking module, for responding to a received pulse signal, inputting the pulse signal into an eye tracking model, and the eye tracking model outputs eye tracking information corresponding to the pulse signal, the pulse signal is generated by converting a collected optical flow signal, the optical flow signal includes optical flow information reflecting eye movement, and the eye tracking model is generated based on a pulse neural network.
[0017] According to another aspect of the present disclosure, a head-mounted display device is provided, comprising: a pulse vision sensor for generating a pulse signal; a processor for processing the pulse signal and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned eye tracking methods by executing the executable instructions.
[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned eye tracking methods is implemented.
[0019] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned eye tracking methods is implemented.
[0020] The eye tracking solution provided by the embodiments of the present disclosure converts the collected optical flow signals into pulse signals. The pulse signals can better capture timing information and rapid changes, which helps to accurately capture eye movement behavior. The pulse signals are used as input and processed by an eye tracking model. The eye tracking model is generated based on a pulse neural network. The neurons of the eye tracking model are stimulated according to the pattern and time characteristics of the input pulses, and then transmit and process information in the network. This processing method does not require the use of complex image processing algorithms to extract useful eye movement information from the captured images, and is closer to the working principles of the biological nervous system. The high temporal resolution and low energy consumption characteristics of the eye tracking model also help to achieve real-time eye movement tracking with higher accuracy and lower power consumption.
[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0023] Figure 1 A schematic diagram showing a head mounted display device in an embodiment of the present disclosure;
[0024] Figure 2 A schematic flow chart showing an eye tracking method in an embodiment of the present disclosure is shown;
[0025] Figure 3 A schematic flow chart showing another eye tracking method in an embodiment of the present disclosure;
[0026] Figure 4 A schematic flow chart of another eye tracking method in an embodiment of the present disclosure is shown;
[0027] Figure 5 A schematic flow chart of another eye tracking method in an embodiment of the present disclosure is shown;
[0028] Figure 6 A schematic diagram showing an eye tracking device according to an embodiment of the present disclosure;
[0029] Figure 7 A schematic diagram of a display device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0031] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0032] Eye tracking technology is a technology that can detect and track the position and movement of the user's eyes. It is of great significance in the application of head-mounted displays. By accurately and quickly tracking the user's eye movements, the system can provide a more natural and intuitive interactive experience, creating a more immersive and realistic virtual or augmented reality environment for users.
[0033] The current implementation of eye tracking technology mainly relies on image-based visual sensors, such as CCD or CMOS image sensors (CIS), which determine the position and movement of the eyeball by capturing and analyzing the image of the eyeball. As a common implementation method, the visual sensor captures the image of the user's eyeball, and then analyzes these images through image processing algorithms to identify the position and movement of the eyeball.
[0034] However, existing image-based eye tracking solutions have some significant disadvantages. First, image-based solutions usually require image sensors with high resolution and high frame rate, as well as powerful image processing capabilities to capture and analyze images of the eyeballs in real time. This not only increases the power consumption of the system, but may also increase the latency of the system, thereby reducing the user's interactive experience. Secondly, image-based eye tracking solutions usually require complex image processing algorithms to extract useful eye movement information from the captured images, which may increase the computational burden of the system, further increasing power consumption and latency. Finally, due to the need for imaging processing, existing image-based eye tracking solutions may be affected by ambient light or other external interference, thereby affecting the accuracy and reliability of eye tracking.
[0035] To facilitate understanding, several terms involved in the present disclosure are first explained below.
[0036] VR (Virtual Reality): Virtual reality is a technology that uses computer technology to simulate a three-dimensional virtual world, providing users with simulated experiences in vision, hearing, touch and other aspects.
[0037] AR (Augmented Reality): Augmented reality is a technology that integrates virtual information such as computer-generated images and sounds into the real world to enhance the user's perception of the real world.
[0038] CIS (CMOS Image Sensor): CMOS image sensor is a commonly used image sensor technology that can convert optical signals into electrical signals and capture and process images.
[0039] HMD (Head-Mounted Display): A head-mounted display device is a device that wears the display directly on the head or in the form of glasses and can provide users with virtual images or images integrated with the real world.
[0040] Pulse Vision Sensor: Used to output pulse stream signals based on light changes.
[0041] Spiking neural network: It is an artificial neural network model that simulates the working mode of neurons in biological nervous system. Different from traditional artificial neural network models (such as feedforward neural network and recurrent neural network), the information transmission between neurons in spiking neural network is realized through the pulse signals emitted by neurons.
[0042] Figure 1 It is a schematic diagram of the structure of a head-mounted display device provided by an exemplary embodiment of the present application.
[0043] like Figure 1 As shown, the head mounted display device includes a device body 102 and a pulse vision sensor 104. A computing unit is also provided in the device body 102, and the computing unit is coupled to the pulse vision sensor 104 via interfaces such as MIPI (Mobile Industry Processor Interface) / LVDS (Low-Voltage Differential Signaling). The eye tracking model is calculated on the computing unit.
[0044] Among them, the pulse vision sensor is responsible for sensing the movement of the eyeball and encoding the visual signal photon stream into a bit stream; the bit stream representing the visual signal will then enter the computing unit, where the bit stream serves as the input of the pulse neural network and the output is the result of eye tracking.
[0045] For example, the pulse vision sensor converts the captured light signal directly into a pulse signal without the need for imaging. This design can significantly reduce the system's perception delay, processing delay, and computational burden. The pulse vision sensor can be placed on the inside of a head-mounted display device (HMD), close to the user's eyes, to capture the user's eye movement information in real time. After that, the pulse vision sensor transmits the data to the computing unit (processor) through an interface such as MIPI / LVDS, and the computing unit runs the pulse neural network to obtain the results.
[0046] Among them, head-mounted display devices include but are not limited to virtual reality headsets, augmented reality helmets, mixed reality headsets and smart glasses.
[0047] The setting of the pulse vision sensor can transform the signal processing from the image domain to the pulse domain, that is, use the pulse neural network to process the perceived data and obtain the eye tracking results. Compared with the solution that requires complex image processing algorithms to analyze the captured images to achieve eye tracking, since complex image processing algorithms mean higher computational costs and delays, the use of pulse neural networks to directly analyze the pulse signals captured by the pulse vision sensor is conducive to reducing the computing burden and power consumption of the device and improving the operating efficiency of the device.
[0048] The eye tracking method in this example implementation will be described in more detail below with reference to the accompanying drawings and embodiments.
[0049] like Figure 2 As shown, an eye tracking method according to an embodiment of the present disclosure includes:
[0050] Step S202, in response to the received pulse signal, input the pulse signal into an eye tracking model, and the eye tracking model outputs eye tracking information corresponding to the pulse signal. The pulse signal is converted and generated according to the collected optical flow signal, and the optical flow signal includes optical flow information reflecting eye movement. The eye tracking model is generated based on a pulse neural network.
[0051] In some embodiments, a pulse vision sensor may be used to collect optical flow signals and convert the optical flow signals into pulse signals, that is, the light intensity variation process of each pixel collected within a period of time is represented as a pulse signal stream.
[0052] In some embodiments, optical flow includes but is not limited to the instantaneous speed of pixel movement of the eye on the observation imaging plane of the pulse vision sensor. The correspondence between the previous frame and the current frame is determined by utilizing the changes in pixels in the time domain in the acquired image sequence and the correlation between adjacent frames to calculate the movement information of the eyes between adjacent frames.
[0053] In some embodiments, the optical flow signal is a waveform or sequence for transmission, and the optical flow information is the above-mentioned optical flow content extracted from the optical flow signal.
[0054] In visual perception, optical flow refers to the change in light pattern caused by the movement of an object in the field of vision. This change can be perceived by the visual system and used to understand the movement of the object and its own position in space. The optical flow signal reflecting eye movement refers to the signal generated by the change in optical flow in the surrounding environment caused by eye movement. When the eyeball moves, the optical flow in the surrounding environment will change. This change can be captured by the sensor and converted into a pulse signal. The pulse signal is used to reflect information such as the change in light intensity, the direction and speed of the light pattern. This information is used to reflect the movement state of the eyeball and the change in visual focus. Therefore, the optical flow signal reflecting eye movement is actually a signal that describes the change in optical flow caused by eye movement, which can be used to analyze eye movement behavior, identify fixation points, and track line of sight movement.
[0055] In some embodiments, a pulse vision sensor is used to convert the collected optical flow signal reflecting eye movement into a pulse signal. In this process, the pulse vision sensor receives the light signal from the eye and converts it into a pulse signal. Such conversion can better reflect the dynamic changes in the eye movement process and provide more accurate and detailed data for subsequent eye movement tracking.
[0056] The eye tracking model is generated based on a spiking neural network (SNN). The spiking neural network has characteristics that are more similar to biological nervous systems and can effectively process time correlation and the order of events. In this model, neurons are excited according to the pattern and time characteristics of the input pulse signal, and the eye tracking process is achieved through connections and information transmission in the network.
[0057] In addition, eye tracking information includes but is not limited to the position of the eyeball, movement speed, gaze angle, etc.
[0058] In this embodiment, the collected optical flow signals are converted into pulse signals, so that the imaging process is not required. The pulse signals can better capture timing information and rapid changes, which is helpful to accurately capture eye movement behavior. The pulse signals are used as input and processed by the eye tracking model. The eye tracking model is generated based on a pulse neural network. The neurons of the eye tracking model are stimulated according to the pattern and time characteristics of the input pulses, and then transmit and process information in the network. This processing method does not require the use of complex image processing algorithms to extract useful eye movement information from the captured images, and is closer to the working principle of the biological nervous system. The high temporal resolution and low energy consumption characteristics of the eye tracking model also help to achieve real-time eye movement tracking with higher accuracy and lower power consumption.
[0059] Furthermore, by setting up a pulse vision sensor, the pulse vision sensor converts the collected optical flow signal into a pulse signal. Compared with the traditional image-based vision sensor, the pulse vision sensor is a sensor based on the brain-like working principle. It can directly convert the photon flow into a pulse flow, thereby avoiding the need for traditional image sensors to perform exposure imaging steps. The pulse vision sensor directly generates a pulse signal, which is suitable for subsequent pulse neural network processing algorithms, greatly reducing the perception delay, and also achieving the capture of light signals at a higher frame rate.
[0060] In some embodiments, before inputting the pulse signal into the eye tracking model in response to the received pulse signal, the method further includes: constructing the eye tracking model.
[0061] like Figure 3 As shown, in some embodiments, building an eye tracking model includes:
[0062] Step S302: training an artificial neural network using the eye tracking data set to obtain an initial model for eye tracking.
[0063] Among them, the eye tracking dataset includes eye movement data and corresponding markers such as gaze point and gaze duration.
[0064] The artificial neural network structure can be a deep neural network (DNN), a convolutional neural network (CNN) or a recurrent neural network (RNN), etc.
[0065] The eye tracking dataset is used for supervised learning. The neural network parameters are continuously adjusted through the back-propagation algorithm so that the model can accurately predict eye movement behavior. The trained initial model is verified and tested, and its performance on the eye tracking dataset is evaluated, including accuracy, recall rate, precision and other indicators to obtain the initial model.
[0066] Step S304: based on the mapping relationship between the artificial neural network and the spiking neural network, convert the initial model into an eye tracking model.
[0067] Among them, in view of the characteristics of spiking neural networks, including pulse signal processing, timing characteristics and synaptic weights, it is necessary to understand the differences and mapping relationships between spiking neural networks and traditional artificial neural networks. Based on the understanding of the mapping relationship, the initial model is converted accordingly. This process may involve steps such as adjusting weights and thresholds, processing timing information, etc., to ensure that similar eye tracking functions can be achieved in SNN. By adjusting the structure and parameters of the artificial neural network to better adapt to the processing and timing characteristics of pulse signals, the conversion process can also include operations such as adjusting the activation function and changing the way neurons are connected.
[0068] By representing features in SNN as a series of pulses whose firing frequency or time encoding is proportional to the size of the original feature value, the original visual information is effectively converted into a pulse pattern that can be processed in the SNN architecture.
[0069] In this embodiment, an initial model for the eye tracking task is obtained through training of an artificial neural network. The model can accurately predict eye movement behavior information such as gaze point and gaze duration based on the input eye movement data. The converted eye tracking model has better pulse signal processing capabilities and can more effectively extract eye movement behavior information from pulse signals, including timing processing and pulse signal transmission, thereby improving the performance of the eye tracking model in the pulse neural network, and ultimately achieving the conversion from the initial model based on the artificial neural network to the eye tracking model adapted to the pulse neural network, thereby improving the eye tracking model's processing capability and accuracy for pulse signals.
[0070] like Figure 4 As shown, in some embodiments, an artificial neural network is trained using an eye tracking dataset to obtain an initial model for eye tracking, including:
[0071] Step S402: Encode the eye tracking data set frame by frame using a feature extraction network to obtain high-dimensional semantic features that describe eye features in the time domain.
[0072] Among them, a feature extraction network is used to process each frame of the eye tracking dataset to extract eye features in the time domain, which usually include information such as the position and size of the pupil, the skin color around the eyes, and the eye movement trajectory. Then, these eye features are encoded into high-dimensional semantic features, and some classic deep learning models such as CNN (convolutional neural network), RNN (recurrent neural network) or ResNet (residual network) can be used.
[0073] For example, a common scenario of using ResNet-50 as a feature extraction network is to use a pre-trained ResNet-50 model to extract high-dimensional features. ResNet-50 learns abstract feature representations in images through multi-layer convolution and residual modules, which can capture rich information and semantic content of images.
[0074] In addition, high-dimensional semantic features can include eye structure features, gaze point features, eye movement path features, eye movement speed features, and eye expression features. High-dimensional semantic features can be organized into sequences in the time domain, describing the changes and evolution of eye features over time.
[0075] Step S404, inputting the high-dimensional semantic features into the first fully connected layer and the second fully connected layer respectively, so that the first fully connected layer outputs the predicted yaw angle of the eye, and the second fully connected layer outputs the predicted pitch angle of the eye.
[0076] Among them, two completely independent fully connected layers are used to predict the yaw angle and pitch angle of the eye respectively.
[0077] For example, the fully connected layer usually includes multiple neuron nodes, and the mapping from features to outputs is realized by learning weights and bias parameters. The obtained high-dimensional semantic features are respectively input into the first fully connected layer and the second fully connected layer for linear transformation and nonlinear activation. The first fully connected layer is responsible for outputting the predicted yaw angle of the eye, that is, the rotation angle of the eye in the horizontal direction; the second fully connected layer is responsible for outputting the predicted pitch angle of the eye, that is, the rotation angle of the eye in the vertical direction. These two angles represent the rotation posture of the eye in space and can reflect the direction of sight.
[0078] Step S406, based on the relationship between the predicted yaw angle and the corresponding yaw angle label, and the relationship between the predicted pitch angle and the corresponding pitch angle label, update the model parameters of the feature extraction network, the first fully connected layer, and the second fully connected layer until the training is completed to obtain an initial model, wherein the eye tracking information is determined based on the predicted yaw angle and the predicted pitch angle in the time domain output by the initial model.
[0079] Among them, according to the difference between the predicted yaw angle and the actual label, and the difference between the predicted pitch angle and the actual label, the error of the model is calculated through the loss function, and the model parameters of the feature extraction network, the first fully connected layer, and the second fully connected layer are continuously updated using optimization algorithms such as gradient descent to make the predicted value close to the actual value. The training process will be repeated for multiple iterations until the model converges, that is, the error is reduced to a smaller threshold, and the initial eye tracking model is obtained after the update is completed.
[0080] In this embodiment, the eye tracking data is encoded through a feature extraction network, and the obtained high-dimensional semantic features can more comprehensively describe the eye features and improve the accuracy and richness. By respectively inputting the predicted values of the yaw angle and the pitch angle, a model that can accurately predict the eye posture is trained, making the eye tracking information more accurate and reliable. By updating the model parameters according to the difference between the actual label and the predicted value, the self-optimization and learning ability of the model is realized. The final initial model can accurately determine the eye tracking information based on the predicted yaw angle and pitch angle in the time domain.
[0081] In some embodiments, based on the relationship between the predicted yaw angle and the corresponding yaw angle label, and the relationship between the predicted pitch angle and the corresponding pitch angle label, the model parameters of the feature extraction network, the first fully connected layer, and the second fully connected layer are updated until the training is completed, further comprising:
[0082] The loss function is constructed based on a linear combination of the cross entropy loss function and the mean squared error.
[0083] Among them, the cross entropy loss function is defined as:
[0084] H(y,p)=-∑ i y i logp i (1)
[0085] Among them, i is the sample number, y i is the true label (0 or 1) of the i-th sample belonging to a certain category, p i It is the predicted probability value of the model that the i-th sample belongs to a certain category.
[0086] The mean square error is defined as:
[0087]
[0088] Where N is the number of samples, y is the true label of the i-th sample, and p is the model's predicted value for the i-th sample.
[0089] Its linear combination is defined as:
[0090] Loss(y,p)=H(y,p)+β·MSE(y,p) (3)
[0091] Among them, β is the weight value.
[0092] Based on the loss function, the feature extraction network, the first fully connected layer, and the second fully connected layer are back-propagated in the time domain to update the model parameters based on back-propagation.
[0093] Among them, the back propagation process includes: the input data passes through the feature extraction network, fully connected layer and other modules to obtain the prediction result, the cross entropy loss and mean square error of the prediction result and the true label are calculated, the gradient of the loss function to the model parameters is calculated, and the parameters of the feature extraction network and the fully connected layer are updated according to the gradient information. The above steps are repeated until the loss function converges or the set stop condition is reached.
[0094] In this embodiment, by constructing a loss function based on a linear combination of the cross entropy loss function and the mean square error, and updating the parameters of the feature extraction network and the fully connected layer through back propagation, the model performance can be improved to a certain extent, the generalization ability can be enhanced, and the model parameters can be effectively optimized to better adapt to the needs of complex tasks.
[0095] In some embodiments, based on the mapping relationship between the artificial neural network and the pulse neural network, converting the initial model into an eye tracking model includes:
[0096] The mapping relationship includes a feature mapping relationship between the dimension of the high-dimensional vector learned by the initial model and the emission frequency of the pulse signal, and the model parameters of the initial model are mapped to the pulse neural network based on the feature mapping relationship.
[0097] The parallel first fully connected layer and the second fully connected layer are converted into synaptic connections to obtain multiple related neurons.
[0098] Among them, by exploring the correlation and conversion function between the two, a feature mapping relationship between the dimension of the high-dimensional vector learned by the initial model and the emission frequency of the pulse signal is established.
[0099] Furthermore, the model parameters learned by the initial model are mapped to the spiking neural network through a mapping relationship. Specifically, the weights and bias parameters are converted into a form suitable for the spiking neural network to ensure that the model parameters can be effectively applied in the spiking neural network.
[0100] Furthermore, based on the model parameters obtained by mapping, a pulse neural network structure is designed to ensure that multiple related neurons can effectively capture the feature representations learned by the initial model. These neurons can correspond to different aspects of the features in the initial model according to different mapping relationships, thereby comprehensively considering multiple aspects of information.
[0101] Determine an eye tracking model based on multiple correlated neurons.
[0102] Among them, based on multiple related neurons, an eye tracking model can be further determined. This model can be a classifier, a regressor or other neural network structure suitable for the task, and the feature information extracted by these related neurons is used to predict the eye tracking task.
[0103] Exemplarily, the following steps may be used to convert an artificial neural network including a feature extraction network and two parallel fully connected layers into a spiking neural network:
[0104] First, the input features are converted into pulse signals so that the pulse signals can be directly received.
[0105] Secondly, select a spiking neuron model suitable for eye tracking information processing. These neurons need to be able to receive pulse input and generate pulse output to capture the timing information in the eye movement data. You can consider using a spiking neuron model, such as the LIF model (Leaky Integrate-and-Fire) of spiking neurons.
[0106] Again, the two parallel fully connected layers are converted into synaptic connections in a spiking neural network. The connection between each neuron needs to transmit information according to the pulse signal and generate output pulses at the appropriate time to realize the processing and learning of eye tracking information.
[0107] Finally, based on the features and patterns learned by the network, information such as the target position and attention distribution in the eye movement data is inferred, thereby realizing the processing and prediction of the eye tracking task.
[0108] In this embodiment, by establishing a feature mapping relationship between high-dimensional vectors and pulse signals, effective conversion from learned features to pulse signal frequencies is achieved, and more meaningful feature information is extracted. By utilizing multiple related neurons, the feature information of the input data can be more comprehensively captured, thereby improving the expressiveness and generalization capabilities of the model, making the eye tracking model more accurate and robust. By determining the eye tracking model, a more suitable model structure can be designed according to specific task requirements, thereby improving the efficiency and accuracy of task completion.
[0109] like Figure 5 As shown, in some embodiments, in response to the received pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye tracking information corresponding to the pulse signal, including:
[0110] Step S502 , in response to the pulse signal, determining the pulse mode and timing characteristics of the pulse signal within a specified time window.
[0111] Among them, for the input pulse signal, the system will analyze and process it according to the pulse mode and timing characteristics within the specified time window, and determine the representation method of different pulse modes and the timing characteristics of the pulse signal on the time axis by identifying the frequency, amplitude, duration and other characteristics of the pulse signal.
[0112] Step S504, stimulating corresponding neurons in the eye tracking model based on the pulse pattern and timing characteristics, and generating output pulses.
[0113] Among them, the neurons can be neurons in the fully connected layer. According to the pulse pattern and timing characteristics, the system will stimulate the corresponding neurons in the eye tracking model. Each neuron responds and generates output pulses at the appropriate time according to the received pulse signal. These pulses contain information about the eye movement data.
[0114] Step S506, the excited neurons transmit output pulses to other neurons through synaptic connections until the eye tracking model outputs the yaw angle and pitch angle of the eye corresponding to the optical flow information within the time window as eye tracking information.
[0115] Among them, eye tracking information can be generated based on the yaw angle and pitch angle output from multiple time windows.
[0116] In addition, the excited neurons transmit output pulses to other neurons through synaptic connections, forming a network structure for information transmission. These synaptic connections will effectively transmit information to the next layer of neurons according to the timing characteristics and weights of the pulse signals until the eye tracking model outputs the required eye yaw and pitch angles.
[0117] In this embodiment, since the pulse neural network is naturally suitable for processing timing information, the eye tracking model can accurately capture the timing characteristics of the pulse signal, thereby more accurately characterizing the timing changes of the eye movement data, and then making full use of the timing characteristics of the pulse signal and the synaptic connections between neurons to achieve efficient and accurate processing of the eye movement data.
[0118] In some embodiments, it also includes: generating adjustment instructions for the virtual environment in which the user is located based on the eye tracking information, and the adjustment instructions include at least one of the display angle, display content and interaction information of the virtual environment.
[0119] Exemplarily, the gaze point and the scanning path are determined according to the user's eye tracking information, and the display angle of the virtual environment is adjusted to make it easier for the user to observe the area of interest.
[0120] Determine the user's attention focus based on the user's eye tracking information, update the content displayed in the virtual environment, and highlight the information that the user is paying attention to.
[0121] Determine eye movement behavior based on the user's eye tracking information, adjust interactive elements in the virtual environment, and provide a more personalized interactive experience.
[0122] In this embodiment, after obtaining the yaw and pitch angles of the eyes, this information can be used to achieve a more natural interaction and immersive experience. According to the user's line of sight, the AR / VR system can dynamically adjust the content in the virtual environment to provide targeted information and interaction. For example, in a virtual reality game, the plot and character reactions of the game can change according to the direction of the player's line of sight, increasing the attractiveness and immersion of the game. In educational or training simulations, the system can provide relevant information or guidance based on the user's gaze point to help users learn more effectively. In addition, these eye movement data can also be used to optimize the user interface, making the interface design more in line with the user's natural line of sight flow, thereby improving the user experience.
[0123] The eye tracking solution and head-mounted display device using the eye tracking solution disclosed in the present invention directly convert the collected photon stream into a pulse stream by adopting a pulse vision sensor, thereby reducing the system perception delay and making the eye tracking more real-time and accurate. In addition, the pulse vision sensor can respond to the changes of the light signal in real time and provide a very high frame rate, so that the eye tracking system can capture more details of the user's eye movements, thereby improving the accuracy and reliability of the system. Furthermore, the pulse domain processing algorithm greatly reduces the power consumption and delay of the system, making the eye tracking technology more suitable for portable VR and AR devices.
[0124] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0125] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0126] Refer to the following Figure 6 hereinafter is a description of the eye tracking device 600 according to this embodiment of the present disclosure. Figure 6 The eye tracking device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0127] The eye tracking device 600 is in the form of a hardware module. The components of the eye tracking device 600 may include but are not limited to: a tracking module 602, which is used to respond to the received pulse signal, input the pulse signal into the eye tracking model, and the eye tracking model outputs eye tracking information corresponding to the pulse signal, the pulse signal is converted and generated according to the collected optical flow signal, the optical flow signal includes optical flow information reflecting eye movement, and the eye tracking model is generated based on a pulse neural network.
[0128] Refer to the following Figure 7 A head mounted display device 700 according to this embodiment of the present disclosure is described. Figure 7 The head mounted display device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0129] like Figure 7As shown, the head mounted display device 700 is in the form of a general purpose computing device. The components of the head mounted display device 700 may include, but are not limited to: the at least one processing unit 710, the at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).
[0130] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can perform the following steps: Figure 2 Step S202 shown in , and other steps defined in the eye tracking method of the present disclosure.
[0131] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .
[0132] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0133] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0134] The head mounted display device 700 may also communicate with one or more external devices 760 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the display device, and / or communicate with any device that enables the head mounted display device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. In addition, the head mounted display device 700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 750. As shown, the network adapter 750 communicates with other modules of the head mounted display device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the display device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0135] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0136] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.
[0137] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0138] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0139] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0140] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0141] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0142] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0143] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0144] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. An eye tracking method, characterized in that: include: In response to the received pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye tracking information corresponding to the pulse signal. The pulse signal is generated by converting the collected optical flow signal, and the optical flow signal includes optical flow information reflecting eye movement. The eye tracking model is generated based on a pulse neural network.
2. The eye tracking method according to claim 1, characterized in that: Before inputting the pulse signal into the eye tracking model in response to the received pulse signal, the method further includes: The eye tracking model is constructed.
3. The eye tracking method according to claim 2, characterized in that: Constructing the eye tracking model, including: The artificial neural network is trained using the eye tracking dataset to obtain an initial model for eye tracking; Based on the mapping relationship between the artificial neural network and the spiking neural network, the initial model is converted into the eye tracking model.
4. The eye tracking method according to claim 3, characterized in that: The artificial neural network is trained using the eye tracking dataset to obtain the initial model for eye tracking, including: Using a feature extraction network to encode the eye tracking dataset frame by frame, and obtaining high-dimensional semantic features describing eye features in the time domain; Inputting the high-dimensional semantic features into a first fully connected layer and a second fully connected layer respectively, so that the first fully connected layer outputs a predicted yaw angle of the eye, and the second fully connected layer outputs a predicted pitch angle of the eye; Based on the relationship between the predicted yaw angle and the corresponding yaw angle label, and the relationship between the predicted pitch angle and the corresponding pitch angle label, the model parameters of the feature extraction network, the first fully connected layer, and the second fully connected layer are updated until the training is completed to obtain the initial model. Wherein, the eye tracking information is determined based on the predicted yaw angle and the predicted pitch angle in the time domain output by the initial model.
5. The eye tracking method according to claim 4, characterized in that: Based on the relationship between the predicted yaw angle and the corresponding yaw angle label, and the relationship between the predicted pitch angle and the corresponding pitch angle label, updating the model parameters of the feature extraction network, the first fully connected layer, and the second fully connected layer until the training is completed, further comprising: Construct the loss function based on the linear combination of the cross entropy loss function and the mean square error; Based on the loss function, back-propagation is performed on the feature extraction network, the first fully connected layer, and the second fully connected layer in the time domain to update the model parameters based on back-propagation.
6. The eye tracking method according to claim 3, characterized in that: Based on the mapping relationship between the artificial neural network and the spiking neural network, the initial model is converted into the eye tracking model, including: The mapping relationship includes a feature mapping relationship between the dimension of the high-dimensional vector learned by the initial model and the emission frequency of the pulse signal, and the model parameters of the initial model are mapped to the pulse neural network based on the feature mapping relationship; Convert the parallel first fully connected layer and the second fully connected layer into synaptic connections to obtain multiple related neurons; The eye tracking model is determined based on the plurality of correlated neurons.
7. The eye tracking method according to claim 1, characterized in that: In response to the received pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye tracking information corresponding to the pulse signal, including: In response to the pulse signal, determining a pulse pattern and timing characteristics of the pulse signal within a specified time window; Exciting corresponding neurons in the eye tracking model based on the pulse pattern and the timing characteristics, and generating output pulses; The excited neurons transmit the output pulses to other neurons through synaptic connections until the eye tracking model outputs the yaw angle and pitch angle of the eye corresponding to the optical flow information within the time window as the eye tracking information.
8. The eye tracking method according to any one of claims 1 to 7, characterized in that: Also includes: An adjustment instruction for the virtual environment in which the user is located is generated based on the eye tracking information, and the adjustment instruction includes at least one of a display angle, display content, and interaction information of the virtual environment.
9. An eye tracking device, characterized in that: include: A tracking module is used to respond to a received pulse signal and input the pulse signal into an eye tracking model, and the eye tracking model outputs eye tracking information corresponding to the pulse signal. The pulse signal is generated by converting the collected optical flow signal, and the optical flow signal includes optical flow information reflecting eye movement. The eye tracking model is generated based on a pulse neural network.
10. A head mounted display device, characterized in that: include: A pulse vision sensor for generating a pulse signal; A processor, used for processing the pulse signal; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the eye tracking method according to any one of claims 1 to 8 by executing the executable instructions.
11. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the eye tracking method according to any one of claims 1 to 8 is implemented.
12. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the eye tracking method according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Self-adaptive eyeball tracking method and device based on deep learning
CN120279591A
