Binocular eye movement gaze estimation method

Abstract

The application belongs to the technical field of deep learning vision, and particularly relates to a binocular eye movement visual line estimation method. The application is an improvement on the basis of the existing appearance-based eye movement visual line estimation method, and comprises adding a cascade network structure and introducing an attention mechanism, and introducing prior visual line estimation as input. The application first constructs a binocular eye movement visual line estimation network model for extracting high-level semantic features of left eyes and right eyes from left eye and right eye pictures; extracts semantic features from the introduced prior visual line; and then fuses the semantic features to obtain a final visual line estimation result. The application introduces prior visual line estimation, so that the network is more prone to convergence. The cascade network structure is added and the attention mechanism is introduced, so that the algorithm can have better generalization and stability, and the algorithm performance is improved. The visual line is represented by using a quaternion, so that the prediction result is more smooth.

Description

Technical Field

[0001] This invention belongs to the field of deep learning vision technology, specifically relating to a binocular eye-tracking gaze estimation method. Background Technology

[0002] Gaze estimation methods can be broadly categorized into two types: geometry-based methods and appearance-based methods. Geometry-based methods detect key features of the eye (such as the corners of the eyes and pupil positions) and then calculate the gaze based on these features. Appearance-based methods, on the other hand, directly learn a model that maps the appearance to the gaze. A common approach is to input a single eye image or two images (left and right eyes) into a deep neural network for regression to obtain the gaze estimate. Currently, gaze is represented by the angles projected from the pitch (vertical direction) and yaw (horizontal direction). Loss is calculated as the difference between the two angles, but the spatial representation of angle differences lacks good continuity and is not ideal for directional representation.

[0003] This invention improves the appearance-based line-of-sight estimation algorithm by adding prior line-of-sight estimation as input, adding a cascaded network structure, and introducing an attention mechanism, thereby enhancing the algorithm's performance. Summary of the Invention

[0004] The purpose of this invention is to provide a binocular eye movement line estimation method that provides a smooth operation and accurate prediction results.

[0005] The binocular eye-tracking gaze estimation method proposed in this invention is an improvement on existing appearance-based eye-tracking gaze estimation methods. It enhances estimation performance by adding a cascaded network structure, introducing an attention mechanism, and incorporating prior gaze estimation as input. This invention first constructs a binocular eye-tracking gaze estimation network model, which includes: subnetworks... Subnetwork Subnetwork Subnetwork Among them, sub-network High-level semantic features for extracting left and right eye images from left and right eye images; sub-network Used to extract semantic features from the introduced prior gaze; subnetwork Used for subnetworks High-level semantic features and subnetworks extracted from the left and right eyes The extracted prior gaze semantic features are fused to obtain fused semantic features; subnetwork Used for subnetworks The extracted high-level semantic features of the left and right eyes and the sub-network The fused semantic features are processed to obtain the final gaze estimation result. During the training phase, regression estimation based on the fused semantic features can be added to predict the gaze, thereby improving the expressive power of the fused semantic features.

[0006] The sub-network Typically, it can be built using a convolutional neural network; it can also be built using a fully connected network, and in other cases, it can be built using other neural networks.

[0007] The sub-network Typically, it can be built using a convolutional neural network; it can also be built using a fully connected network, and in other cases, it can be built using other neural networks.

[0008] The sub-network Typically, it can be built using a convolutional neural network; it can also be built using a fully connected network, and in other cases, it can be built using other neural networks.

[0009] The sub-network Typically, it can be built using a convolutional neural network; it can also be built using a fully connected network, and in other cases, it can be built using other neural networks.

[0010] The specific steps for eye-tracking gaze estimation are as follows:

[0011] (1) Take a picture of the left eye with a camera. left Right eye image I right Input to subnetwork The high-level semantic features F of the left eye were extracted separately. left Right eye high-level semantic features F right ;

[0012] (2) The prior line of sight G reg Input to subnetwork In subnetwork Each layer produces corresponding feature outputs, which serve as subnetworks. The corresponding network layer input; where:

[0013] The prior line of sight G reg It is during the training process that G is used reg =G reg +N (where N is the noise added by the annotation information), and generated during the process of repeatedly training the same set of data n times; specifically, the current output result is used as the input for the next iteration of training, and the current output result is called the prior line of sight G. reg ;

[0014] Prior line of sight G before input reg Perform position encoding in the following form:

[0015] γ(G reg )=(sin(2 0 πG reg ),cos(2 0 πG reg ),…,sin(2 n-1 πG reg ),cos(2 n-1 πG reg ))

[0016] This is used to improve the overall generalization ability of the network. n is generally 7, but can also be adjusted to a reasonable empirical value based on the actual training situation.

[0017] (3) The high-level semantic features F of the left eye output in step (1) left Right eye high-level semantic features F right Input to subnetwork The prior line-of-sight features output from the network layer in step (2) are input into the sub-network. The corresponding network layer is used to obtain the fused semantic features F. f ;

[0018] (4) During training, semantic features F are fused. f The regression produces auxiliary line-of-sight result G. reg By analyzing the auxiliary line of sight results G, reg Supervised training to improve fusion F f Semantic feature representation ability, constraint F f Semantic features provide a higher-level semantic expression for more effective gaze outcomes. During inference, regression will not be performed to generate auxiliary gaze outcomes G. reg ;

[0019] (5) The high-level semantic features F generated in step (1) for the left eye left Right eye high-level semantic features F right The fused semantic features F generated in step (4) f Input to subnetwork Generate the final line-of-sight estimation result G reg .

[0020] Among them, F left F right F f An attention relationship (Query, Key, Value) is formed, and the final gaze estimation result G is obtained by searching the gaze space. reg :

[0021] G reg=softmax(F left F right )F f

[0022] In this invention, line of sight G reg The following representation method is used:

[0023] Currently, the line of sight is represented by the angles projected from the pitch (vertical direction) and yaw (horizontal direction). Loss is calculated by the difference between the two angles, but the representation of the angle difference in space does not have good continuity and is not particularly good for directional representation.

[0024] In this invention, the line of sight is represented by a quaternion, and the line of sight is represented by a modulus l. mod Error with line of sight l diff To represent errors, making the line-of-sight representation more continuous:

[0025] l mod =||M od (G reg )-1|| 2

[0026]

[0027] Among them, M od Find the modulus function, l mod : Magnitude loss, The inverse of the line of sight, l diff Line of sight error.

[0028] The beneficial effects of this invention are as follows:

[0029] By adding prior line-of-sight estimation as input, the network converges more easily; the quaternion line-of-sight representation method provides more accurate prediction results.

[0030] Adding a cascaded network structure and introducing an attention mechanism enables the algorithm to generalize better and be more stable, resulting in a smoother running process. Attached Figure Description

[0031] Figure 1 This is a flowchart of the present invention.

[0032] Figure 2 For the present invention The output of each layer of the subnetwork is fused into The sub-network input is fused as shown in the diagram. Among them, I left Left eye image; I right Right eye image; F left :Left eye semantic features; F right : Right eye semantic features; F lr : Semantic features of the left eye, Fleft Right eye semantic features F right Merge; F f : Semantic features fused from left and right eyes with prior gaze; G, reg : Auxiliary eye-tracking gaze obtained by regressing the semantic features of the left and right eyes and prior gaze; G reg Final line of sight estimation.

[0033] Figure 3 This is a structural diagram of the system of the present invention.

[0034] Figure 4 This is a structural diagram of the hardware layer in this invention.

[0035] Figure 5 This is a structural diagram of the operating system layer in this invention.

[0036] Figure 6 This is a structural diagram of the application layer in this invention.

[0037] Figure 7 This is a flowchart illustrating the use of the present invention in the system.

[0038] The diagram is labeled as follows: 1. Hardware layer, 2. Driver layer, 3. System layer, 4. Application layer; 5. Camera, 6. Processor, 7. Display; 8. Camera reading, 9. Algorithm processing, 10. Gaze direction feedback; 20. Clock management, 21. Interrupt handling, 22. Device driver, 23. Process management, 24. Memory management, 25. Device management, 26. Non-kernel functions. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] First, construct the application framework, including: hardware layer 1, driver layer 2, system layer 3, and application layer 4, such as... Figure 3 As shown.

[0041] Hardware layer 1 is located at the bottom layer, and its structure is as follows: Figure 4 As shown, the system includes a camera 5, a processor 6, and a display 7. The camera 5 is used to read image and depth data; typically, a grayscale or RGB camera can be used. The camera 5 transmits data to the processor 6 for processing via an I / O bus. The processor 6 performs calculations and can be a standard CPU (such as an ARM or x86 processor) or a high-performance computing processor (such as a TPU or GPU). The processor 6 is connected to the display 7 via a video bus to show the calculation results.

[0042] System layer 3 runs on processor 6 and controls hardware layer 1 through driver layer 2, which uses a driver. Application layer 4, running above system layer 3, implements the various processes and functions for calling the gaze estimation method. The driver layer mainly includes a camera driver module, a GPIO port driver module, and an LCD display driver module, used to control camera 5, processor 6, and display 7, respectively.

[0043] Application layer 4 includes camera reading module 8, algorithm processing module 9, and gaze direction feedback module 10, such as... Figure 6 As shown, the output of the camera reading module 8 is connected to the input of the algorithm processing module 9, and the output of the algorithm processing module 9 is connected to the input of the line-of-sight direction feedback 10.

[0044] Camera reading module 8 is the gaze estimation data input module. It calls the lower-level driver layer to obtain image data from the real environment and then passes the data to the algorithm processing module 9.

[0045] The algorithm processing module 9 processes the acquired image data. First, it performs face detection to determine whether a face exists. Then, based on the detection results, it extracts the corresponding left and right eye regions and calculates the gaze estimate using prior information algorithms. The gaze direction is then fed back and the result is displayed on the display hardware.

[0046] References:

[0047] [1]Zhang,

[0048] [2] Zhang,

[0049] [3]Cheng,Y.,Lu,F.,and Zhang,X.(2018).Appearance-based gaze estimationviaevaluation-guided asymmetric regression.In The European Conference onComputerVision(ECCV).

[0050] [4]Park,S.,Spurr,A.,and Hilliges,O.(2018).Deep Pictorial GazeEstimation.In European Conference on Computer Vision(ECCV),pages 741–757.

[0051] [5]Seonwook Park,Xucong Zhang,Andreas Bulling,OtmarHilliges(2018).Learning tofind eye region landmarks for remote gaze estimation inunconstrained settings.ACMSymposium on Eye Tracking Research and Applications(ETRA)

[0052] [6]Yu,Y.,Liu,G.,and Odobez,J.-M.(2018).Deep multitask gaze estimationwitha constrained landmark-gaze model.European Conference on Computer VisionWorkshop(ECCVW).

[0053] [7]Zhang,X.,Sugano,Y.,Fritz,M.,and Bulling,A.(2016).It’s Written AllOverYour Face:Full-Face Appearance-Based Gaze Estimation.IEEE Conference onComputerVision and Pattern Recognition Workshops(CVPRW).

[0054] [8]Zhu,W.and Deng,H.(2017).Monocular free-head 3d gaze tracking withdeep learningand geometry constraints.In The IEEE International Conference onComputer Vision(ICCV).

Claims

1. A binocular eye-tracking gaze estimation method, characterized in that, It is an improvement on existing appearance-based eye-tracking gaze estimation methods by adding a cascaded network structure and introducing an attention mechanism, and by incorporating prior gaze estimation as input to improve estimation performance. First, a binocular eye-tracking gaze estimation network model is constructed. This model includes: subnetworks Subnetwork Subnetwork Subnetwork Among them, sub-network High-level semantic features for extracting left and right eye images from left and right eye images; sub-network Used to extract semantic features from the introduced prior gaze; subnetwork Used for subnetworks High-level semantic features and subnetworks extracted from the left and right eyes The extracted prior gaze semantic features are fused to obtain fused semantic features; subnetwork Used for subnetworks The extracted high-level semantic features of the left and right eyes and the sub-network The fused semantic features are processed to obtain the final gaze estimation result. During the training phase, regression estimation based on the fused semantic features can be added to predict the gaze, improving the expressive power of the fused semantic features. The specific steps for eye-tracking gaze estimation are as follows: (1) Take a picture of the left eye with a camera. left Right eye image I right Input to subnetwork The high-level semantic features F of the left eye were extracted separately. left Right eye high-level semantic features F right ; (2) The prior line of sight G reg Input to subnetwork In subnetwork Each layer produces corresponding feature outputs, which serve as subnetworks. The corresponding network layer input; where: The prior line of sight G reg During the training process, G reg =G reg +N, which is generated during the process of repeatedly training the same set of data n times, where N is the noise added by the labeling information; specifically, the current output result is used as the input for the next iteration of training; Prior line of sight G before input reg Perform position encoding in the following form: γ(G reg )=(sin(2 0 πG reg ),cos(2 0 πG reg ),…,sin(2 n-1 πG reg ),cos(2 n-1 πG reg )) Used to enhance the overall generalization capability of the network; (3) The high-level semantic features F of the left eye output in step (1) left Right eye high-level semantic features F right Input to subnetwork The prior line-of-sight features output from the network layer in step (2) are input into the sub-network. The corresponding network layer is used to obtain the fused semantic features F. f ; (4) During training, semantic features F are fused. f The regression produces the auxiliary line of sight result G` reg By analyzing the auxiliary line of sight result G` reg Supervised training to improve the fusion of semantic features F f Expressive ability, constrained semantic features F f To achieve a more effective high-level semantic expression of gaze results, regression will not be performed during inference to generate auxiliary gaze results. (5) The high-level semantic features F generated in step (1) for the left eye left Right eye high-level semantic features F right The fused semantic features F generated in step (4) f Input to subnetwork Generate the final line-of-sight estimation result G reg ; Among them, the high-level semantic feature F of the left eye left Right eye high-level semantic features F right , F fusion semantic features f An attention relationship is formed, and the final gaze estimation result G is obtained by searching in the gaze space. reg : G reg =softmax(F left ,F right )F f 。 2. The binocular eye-tracking gaze estimation method according to claim 1, characterized in that, The sub-network Subnetwork Subnetwork Subnetwork They can be constructed from either convolutional neural networks or fully connected networks.

3. The binocular eye-tracking gaze estimation method according to claim 1, characterized in that, The line of sight G mentioned in step (5) reg The following representation method is used: The direction of the gaze is represented by quaternions, and the loss of the gaze magnitude is represented by l. mod ), and line-of-sight error l diff To represent errors, making the line-of-sight representation more continuous: l mod =||M od (G reg )-1|| 2 Among them, M od To find the modulus function, l mod : is the magnitude loss, : represents the inverse of the line of sight, l diff This is due to line-of-sight error.

Images