Light field display sight tracking method based on deep learning and light field display method

Abstract

The invention relates to a light field display sight tracking method based on deep learning and a light field display method. The light field display sight tracking method comprises the following steps: capturing current head and eye images of a user in real time by using a camera; performing face region detection on the head image by using a face detection model, and returning a face bounding box to obtain a face image; performing face feature point detection on the face image by using the face feature point detection model to obtain face feature point information, and obtaining head posture information by using an EPnP algorithm; preprocessing the eye image; the preprocessed eye image is transmitted into a pre-trained sight prediction model, the head posture information serves as a parameter to be input into a full connection layer of the model, and the sight prediction model outputs a predicted sight vector. The method can show good accuracy and robustness under complex conditions, so that a light field display system can monitor the watching position of a user in real time and correspondingly update the displayed content, and a better display effect is achieved.

Description

Technical Field

[0001] The present invention relates to the field of display technology, and in particular to a light field display sight tracking method and a light field display method based on deep learning. Background Art

[0002] Light field display uses a microlens array to capture and reproduce the spatial information of a three-dimensional scene or object, thereby providing viewers with a stereoscopic visual experience with a real sense of depth without the need for additional equipment. This technology is mainly achieved by configuring an array of many tiny lenses in front of the display panel. Each microlens can capture image fragments at a specific angle and reorganize these fragments into an image array containing multiple perspectives. When these images are reprojected through the microlens array, they can reconstruct the light distribution under multiple perspectives in space, thereby simulating the real three-dimensional effect of objects or scenes. This method can provide all-round parallax information and continuously changing viewing angles, making the presented color three-dimensional images more realistic and natural. At present, research on light field display mainly focuses on large depth of field, high frame rate, adaptive rendering, high resolution, and wide field of view.

[0003] Gaze tracking refers to the use of computer vision, optical detection and other means to detect the gaze-related conditions of the subject, such as head posture, eye position, and relative position of eye rotation, and perform relevant analysis on the information obtained to obtain the subject's current gaze position in order to achieve gaze tracking and judgment operations.

[0004] Combining light field display with gaze tracking can improve the resolution, depth of field, field of view and other parameters of the light field display system, and realize human-computer interaction in light field display. The gaze tracking technology based on light field display requires the use of a two-dimensional plane-based gaze tracking system to obtain the user's relevant head posture and eye information, thereby deriving the corresponding gaze direction, and then switching different light field image sources according to the user's gaze position.

[0005] In the paper "Cai H, Hui Y, Zhou X, et al. Robust gaze estimation via normalized iris center-eye corner vector [C]. International Conference on Intelligent Robotics and Applications. Springer International Publishing, 2016.", Cai H et al. proposed a feature-based gaze tracking algorithm that predicts the gaze direction of the human eye by using the features to establish a regression equation. The method first extracts key eye feature points from the image, and then uses these feature points to establish a regression equation to predict the gaze direction of the human eye. This method accurately locates multiple feature points in the left and right eye regions and combines the position differences of these points to construct a model that can accurately reflect the gaze direction. This detection method first uses the target detection algorithm to detect the face area, and then performs subsequent feature point detection. Based on the position of these feature points, the eye area is further tracked.

[0006] The invention patent with publication number CN115236871A discloses a "desktop light field display system and method based on human eye tracking and bidirectional backlight", in which the rendering device includes two acquisition modules, the first acquisition module is used to acquire the image data currently required to be rendered, and the second acquisition module is used to acquire the human eye position coordinates, and the second acquisition module can acquire the human eye position information through two depth cameras with depth acquisition function, including the depth information, vertical space coordinates and horizontal space coordinates of the human eye. Then, the spatial position coordinates of the convergence of the human eye are calculated according to the angle between the position information of the human eye and the holographic functional screen, and its rendering device then renders according to the convergence coordinates of the human eye, and the above operations are repeated to achieve high refresh rate and high resolution three-dimensional image display. However, the system and method only use the spatial position information of the human eye, and do not use the feature information of the human eye image. Therefore, when the head posture changes, the accuracy of its line of sight gaze point prediction is low.

[0007] The above sight detection method can be used routinely, but it cannot be applied to larger deflection angles. The main reason is that the human head will rotate and tilt appropriately, and the head posture will change, resulting in face deflection. Since feature point detection is based on facial features, the face area must be confirmed first, and the change of the face area will seriously affect the subsequent feature point detection-recognition operation, thus greatly affecting the recognition success rate and recognition rate. At the same time, the accuracy of these methods is low in complex scenes such as occlusion and light exposure. Summary of the invention

[0008] In view of the above-mentioned problems existing in the existing gaze detection methods, the present invention provides a light field display gaze tracking method based on deep learning and a light field display method based on the method.

[0009] The technical solution adopted by the present invention is as follows:

[0010] A light field display sight tracking method based on deep learning, the method comprising the following steps:

[0011] Use the camera to capture the user's current image in real time;

[0012] Use the face detection model to detect the face area of ​​the user's current image, and return the face bounding box to obtain the face image;

[0013] Performing facial feature point detection on the facial image using a facial feature point detection model to obtain facial feature point information, obtaining head posture information using an EPnP algorithm based on the facial feature point information, and cropping the facial image according to the facial feature point information to obtain an eye image;

[0014] Preprocessing the eye image to obtain a preprocessed eye image;

[0015] The preprocessed eye image is input into a pretrained gaze prediction model and the head posture information is input as a parameter into a fully connected layer in the gaze prediction model. The gaze prediction model adopts a VGG16 network, and on the basis of the original VGG16 network, the stride of all maximum pooling layers is modified to 1, and the head posture information is spliced ​​into a flattened feature vector to form a new feature vector, and the spliced ​​feature vector is then input into a fully connected layer for classification. The gaze prediction model extracts key eye features of the eye image through a series of convolutional layers, including pupil position, eyeball contour, and iris texture, and then inputs the key eye features into a fully connected layer, predicts the user's gaze vector through a regression method, and outputs the predicted gaze vector.

[0016] The above-mentioned deep learning-based light field display gaze tracking method introduces a deep neural network model to detect the gaze information of the human eye. At the same time, it combines traditional visual recognition technology to form a complete real-time human eye detection and gaze prediction solution. Compared with traditional visual methods, this solution shows extremely high accuracy and real-time performance in complex scenes and occlusion conditions. It can show good accuracy and robustness under complex conditions, so that the light field display system can monitor the user's gaze position in real time and update the displayed content accordingly to achieve better display effects.

[0017] At the same time, the present invention also proposes a light field display method based on the above light field display sight tracking method, the method comprising the following steps:

[0018] Step 1: construct a visual perception model, where the visual perception model is used to adjust the primitive image in the light field acquisition stage according to the sight vector;

[0019] Step 2: Determine whether the face detection model detects a face or an eye within the light field imaging observation range. If so, execute step 3; otherwise, the light field display screen directly displays the initial primitive image.

[0020] Step 3: Activate binocular cameras to retrieve depth information in real time, measure the distance of the human eye in real time, and obtain the sight vector of the human eye according to the above-mentioned light field display sight tracking method, and send the sight vector of the human eye to the visual perception model. The visual perception model controls the light field display screen to switch to the primitive image corresponding to the current depth of field acquisition range and the distance of the human eye.

[0021] The above light field display method is based on the above light field display sight tracking method, which can detect the user's sight direction and the distance from the human eye in real time, and then adjust the focus of the display so that the area the user is looking at remains clear, while other areas are blurred or gradually out of focus, so as to achieve the depth of field effect in the simulated real world, that is, to display different focal points at objects at different distances. At the same time, the user can interact by directly looking at a specific area on the screen. For example, in a virtual environment displayed by a light field, the user can select, move or operate a virtual object by looking at different positions without using external devices such as a mouse, handle or touch screen, and dynamically adjust the displayed content. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a light field display sight tracking method based on deep learning according to an embodiment of the present invention;

[0023] Figure 2 This is the VGG16 network structure diagram;

[0024] Figure 3 This is a flowchart for preprocessing images in MPIIFaceGaze;

[0025] Figure 4 is a flow chart of a light field display method according to an embodiment of the present invention;

[0026] Figure 5 This is the imaging light path diagram of the visual perception model. DETAILED DESCRIPTION

[0027] The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0028] like Figure 1As shown, this embodiment provides a light field display line of sight tracking method based on deep learning. The method can be roughly divided into two steps, namely an image capturing step and a line of sight prediction step.

[0029] In the first image capture step, a camera is first used to capture a high-resolution image of the user in real time. These cameras are usually installed in devices (such as binocular cameras) and can capture real-time images of the user in real time. After capturing a high-resolution image, a face detection model such as OpenCV's Haar cascade classifier is used to detect the face area of ​​the captured image, and the bounding box of the face is returned to obtain a face image. The "leave one person" strategy is adopted in image capture, that is, only the most important face information is selected. After face detection, the corresponding face feature point detection is performed. This process uses a pre-trained face feature point detection model to detect face feature points on the face image to obtain face feature point information. The commonly used face feature point detection model is the pre-trained Dlib-68-point face feature point model, which covers commonly used recognition areas of human faces such as the corners of the eyes, mouth, and nose. Combined with the face feature point detection performed in the previous step, the position of the eyes can be found in the detected face area according to the face feature point information, and then the face image is cropped to obtain the eye image. The obtained eye image is then preprocessed accordingly. The preprocessing process includes image grayscale processing, image normalization processing, image denoising processing and other processes. At the same time, the eye image is adjusted to a horizontal state and scaled to a uniform specified size to improve the prediction accuracy of the model, and finally the preprocessed eye image is obtained.

[0030] At the same time, based on the detected facial feature point information, the EPnP (Efficient Perspective-n-Point) algorithm is used to obtain the head posture information to obtain the user's current head posture information. The EPnP algorithm is an algorithm for camera posture estimation. It can calculate the spatial position information of the camera based on a known set of three-dimensional space points and the projection points of these points in the two-dimensional image, which includes the position and direction of the camera.

[0031] In the second gaze prediction step, the preprocessed eye image is input into the pretrained gaze prediction model, and the model will extract the key eye features of the eye image through a series of convolutional layers. These key eye features include pupil position, eyeball contour, iris texture, etc. Finally, the gaze prediction model inputs the extracted key eye features into the fully connected layer and inputs the head posture information as a parameter into the fully connected layer, predicts the user's gaze vector through the regression method, and outputs the predicted gaze vector as the result of gaze tracking.

[0032] In this step, a deep neural network is used to map eye feature information and head posture information to the line of sight of the human eye. At the same time, by inverting the left and right eye images, a single model can process the left and right eyes. This embodiment uses an improved VGG16 network as the line of sight prediction model. The original VGG16 network is a neural network with a depth of 16 layers, including 13 convolutional layers and 3 fully connected layers. Its specific basic network structure is as follows: Figure 2 As shown. It uses a smaller 3×3 convolution kernel and applies the ReLU activation function after each convolution layer to introduce nonlinear elements, thereby enhancing the model's ability to process complex image features. This network design enables the system to learn higher-level image features. Its convolution unit is composed of multiple convolutional layers alternating with pooling layers, working together to reduce the size of the feature map. All convolutional layers use the same number of filters to maintain consistency and coherence in feature extraction. The subsequent pooling layer performs spatial downsampling, which helps to reduce the number of parameters of the network and expand the receptive field. Finally, the fully connected layer constitutes the classification module, which is responsible for converting the previously extracted convolutional features into predicted probabilities for each category. Based on the original VGG16 network, this embodiment improves it by modifying its pooling layer stride and passing the head posture information as a parameter into the fully connected layer. Specifically, the stride of all maximum pooling layers is reduced from 2 to 1 to retain more spatial information; the head posture information (a three-dimensional vector) is concatenated into the flattened feature vector to form a new feature vector, which is then input into the fully connected layer for classification.

[0033] During the training phase of the neural network, the role of the loss function is to quantitatively evaluate the gap between the model's prediction results and the actual target. A decrease in the loss function value usually means improved model performance and increased stability. It measures the degree of deviation between the model's predicted output f(x) and the actual observed value y. The smaller the value, the more stable and reliable the model's prediction performance. This embodiment uses the mean squared error (MSE) as the loss function to evaluate the difference between the predicted value and the true value. The formula of the MSE loss function is as follows:

[0034]

[0035] Among them, y pred,i is the predicted value of the i-th quantity in the data set, y true,i is the true value of the i-th quantity in the data set, N is the amount of data in the data set, and MSE is the mean square error.

[0036] When using the MSE loss function for optimization, the gradient descent algorithm is used. The optimization process specifically includes the following steps:

[0037] Initialize model parameters;

[0038] According to the parameters of the current model, the predicted value is calculated, and the loss is calculated according to the formula of the MSE loss function;

[0039] Compute gradients: Calculate the gradient of the loss function with respect to the model parameters to determine the contribution of each parameter to the total loss;

[0040] Update parameters: According to the calculated gradient, update the model parameters with the preset learning rate, and repeat the above steps to gradually reduce the value of the MSE loss function.

[0041] The optimization efficiency of deep neural network training depends largely on the timely adjustment of the learning rate. Using the learning rate scheduler, the learning rate can be flexibly adjusted at different stages of training to optimize the learning process of the model. StepLR is a common scheduling method, so the StepLR learning rate scheduler can be used to adjust the learning rate. The StepLR learning rate scheduler multiplies the learning rate by a factor γ every certain amount of processes, that is, after step_size. The formula is as follows:

[0042] 1. new =lr current ×γ (2)

[0043] Among them, lr new is the updated learning rate, lr current is the current learning rate.

[0044] The dataset used for training the gaze prediction model is MPIIFaceGaze, which contains 15 subjects and eye images of each subject. These eye images consist of left eye images and right eye images. The dataset contains eye images of each eye and the gaze direction of each eye. MPIIFaceGaze also contains full-face images of the subjects and provides additional facial feature point annotations. The gaze features of eye images are trained and learned through image data containing human eye information collected from multiple angles, directions and under different lighting conditions, so as to achieve real-time and accurate detection of gaze by the gaze prediction model.

[0045] like Figure 3 As shown in Figure 1, during model training, the images in the MPIIFaceGaze dataset need to be preprocessed, including image normalization, flipping, and histogram equalization. During image normalization, the image is adjusted according to the center point, target point, and head posture information. The center point of normalization is set to the center of the eyeball, and the target point refers to the inner and outer corners of the eye.

[0046] After normalization, the image needs to be flipped, which is in the loop that processes each image. The image is flipped according to the left or right eye. The which_eye variable indicates whether the image is currently being processed. If it is a left-eye image (which_eye == "left"), no flipping is performed. If it is a right-eye image (which_eye! = "left"), a flipping operation is performed. This is because the right-eye image needs to be flipped horizontally to be consistent with the left-eye image. The line of sight direction flipping operation is also performed. Since the line of sight direction needs to be consistent with the image flipping, the line of sight vector is adjusted accordingly. At the same time, the head rotation flipping operation is also required. Similar to the line of sight direction, the head rotation also needs to be consistent with the image flipping. The line of sight starting point flipping operation is also required to reverse the X coordinate of the line of sight starting point to reflect the horizontal flipping of the image. This is because after the image is flipped, the left and right coordinates need to be adjusted accordingly.

[0047] The above preprocessing operation ensures that both the left eye and right eye images are processed in the same way and can be analyzed in a unified manner. Finally, the image histogram equalization is performed to improve the image contrast and the accuracy of subsequent processing.

[0048] Two important auxiliary functions are also used in model training: gazeto3d function converts 2D gaze direction into a vector in 3D space, and angular function calculates the angle between two vectors. The expression of gazeto3d function is as follows:

[0049]

[0050] Among them, gaze g t[0] is the x-coordinate of the gaze vector, gaze g t[1] is the y coordinate of the gaze vector, gaze g t[2] is the z coordinate of the gaze vector, gaze[0] is the pitch angle, and gaze[1] is the yaw angle.

[0051] The test function angular is used to calculate the angle between two three-dimensional vectors (the predicted gaze vector gaze and the label gaze vector label). This angle is used to measure the difference between the gaze direction predicted by the model and the actual direction.

[0052] Calculate the inner product of the predicted gaze vector gaze and the label gaze vector label:

[0053]

[0054] Among them, total is the inner product, gaze iis the predicted sight vector coordinate, label i is the label sight vector coordinate.

[0055] Normalize the inner product (to avoid calculation errors):

[0056]

[0057] Among them, ||gaze|| is the modulus of the predicted gaze vector, and ||label|| is the modulus of the label gaze vector.

[0058] Calculate the angle (in radians) between the predicted gaze vector gaze and the label gaze vector label:

[0059]

[0060] Convert the angle θ from radians to degrees. The converted angle is:

[0061]

[0062] The light field display gaze tracking method proposed in this embodiment introduces a deep neural network model to detect the gaze information of the human eye, and combines it with traditional visual recognition technology to form a complete real-time human eye detection and gaze prediction solution. Compared with traditional visual methods, this solution shows extremely high accuracy and real-time performance in complex scenes and occlusion conditions, and can show good accuracy and robustness under complex conditions, so that the light field display system can monitor the user's gaze position in real time and update the displayed content accordingly to achieve better display effects.

[0063] Among the various gaze tracking methods, different methods have significant differences in terms of average rotation angle difference, recognition accuracy and deployment difficulty. The average angle difference is the mean of the angle between the gaze direction predicted by the model of the entire test set and the true label. Although the traditional OpenCV-based method is easy to deploy and requires a smaller data set, its average rotation angle difference is between 6-8 degrees, and the recognition accuracy is only 70-80%, which cannot meet the needs of high-precision applications. In contrast, the facial feature point-based method has improved both the average rotation angle difference and the recognition accuracy, which are 3-5 degrees and 85-90% respectively, which can usually meet most application requirements and can be implemented in real time on modern hardware. The gaze prediction model of the present invention shows more outstanding performance and further improves the accuracy. The average rotation angle difference is reduced to 2-3 degrees, and the recognition accuracy reaches 90%, which is very suitable for applications requiring high precision. The model needs to be deployed on a CUDA device, which has high requirements for hardware.

[0064] Another embodiment further proposes a light field display method based on the light field display sight tracking method of the above embodiment, which maps these coordinates to the virtual environment according to the predicted sight vector, that is, obtains the coordinates related to the sight tracking area and uses them to switch the light field image. Figure 4 As shown, the light field display method includes the following steps:

[0065] Step 1: construct a visual perception model, which is used to adjust the primitive image in the light field acquisition stage according to the sight vector;

[0066] Step 2: Determine whether the face detection model detects a face or an eye within the light field imaging observation range. If so, execute step 3; otherwise, the light field display screen directly displays the uncalibrated initial primitive image;

[0067] Step 3: Activate binocular cameras to retrieve depth information in real time, measure the distance of the human eye in real time, and obtain the line of sight vector of the human eye according to the light field display line of sight tracking method of the aforementioned embodiment, and send the line of sight vector of the human eye to the visual perception model. The visual perception model controls the light field display screen to switch to the primitive image corresponding to the current depth of field acquisition range and the human eye distance, that is, the primitive image corresponding to the optimal viewing angle and distance, thereby changing the displayed primitive image within different visual perception ranges to achieve the best viewing effect of the light field display.

[0068] In the acquisition phase of light field display, when the acquisition equipment parameters are fixed, the light field image collected can only clearly record the three-dimensional scene within the fixed depth of field, and can be reconstructed into a clear three-dimensional image in the reconstruction phase. Figure 5 The focal plane in the image is the surface where the lens object distance is located, while the imaging plane is the surface where the lens image distance is located. Figure 5 As shown in the figure, three points A, B, and C are located in different depths of field. Object point A located on the focal plane can be clearly imaged as a light point A' on the image plane through the lens. However, object points B and C are located outside the focal plane (defocused plane). After being imaged by the lens, light points B' and C' are formed. They cannot be clearly imaged as light spots, but appear as diffuse spots on the imaging plane. Since the spatial positions of points B and C are both on the defocused plane (point B is in front of the focal plane and point C is behind the focal plane), they are far away from the focal plane of the lens. Finally, points B and C form diffraction spots with diameters of ωA and ωB respectively on the imaging plane.

[0069] The minimum angular distance between two luminous points that the human eye can distinguish in space is called the limiting resolution angle, and its reciprocal is called the angular resolution of the human eye. The angular resolution range of the human eye is ±5°. When this range is exceeded, the angular resolution will be significantly reduced. In the area exceeding ±20° on the retina, the human eye loses its focusing ability and can only perceive the intensity of light, which will cause retinal blur. Therefore, during the light field display acquisition stage, as long as the spatial positions of point B and point C are within a certain range of the focusing plane, the imaging effect of point B and point C on the imaging plane is still good after the light field display is reconstructed. When the viewing distance and angle are fixed, the optimal depth of field acquisition range in the light field acquisition stage can be obtained by performing reverse light analysis based on the visual performance of the human eye and the reconstructed diffraction spot diameter.

[0070] Similarly, when an image occupies the entire angle range visible to the human eye, the best visual effect will be produced. When the acquisition and reconstruction parameters of the light field display are fixed, the optimal viewing distance is determined. At the same time, a viewing distance range can be obtained based on this. When the viewing distance of the human eye is within this range, the viewer can obtain a light field display imaging effect that is less affected by the visual characteristics of the human eye; when the viewing distance of the human eye is outside this range, the imaging effect of the light field display will be poor. At this time, the light field primitive image obtained by the original optimal depth of field acquisition range will no longer be applicable, but the depth of field acquisition range needs to be adjusted to obtain a new light field primitive image.

[0071] Through the above method, a visual perception model can be constructed to adjust the primitive image in the light field acquisition stage according to the tracked sight vector to optimize the light field display effect.

[0072] When using the sight prediction model to switch the primitive images of the light field display, the real-time detection frame rate can reach more than 25 frames through algorithm optimization and hardware acceleration, meeting the real-time requirements. The application of the sight prediction model improves the distance range and spatial resolution of viewing the light field display, providing a better stereoscopic display effect. In terms of the evaluation indicators of the light field display effect, such as the Structural Similarity Index (SSIM) and the Peak Signal to Noise Ratio (PSNR), there are good data results.

[0073] The light field display method of this embodiment applies the above-mentioned light field display sight tracking method to the light field display scene, and combines it with an advanced visual perception model to switch the primitive images of the light field display according to the position of the human eye and the direction of the sight, so as to achieve a better light field display effect and provide users with a more immersive and interactive three-dimensional visual experience.

[0074] The light field display method of this embodiment can detect the user's line of sight and the distance from the human eye in real time, and then adjust the focus of the display so that the area the user is looking at remains clear, while other areas are blurred or gradually out of focus, so as to achieve a depth of field effect simulating the real world, that is, displaying different focal points at objects at different distances. At the same time, the user can perform interactive operations by directly looking at a specific area on the screen. For example, in a virtual environment displayed by a light field, the user can select, move or operate a virtual object by looking at different positions without using external devices such as a mouse, handle or touch screen, and dynamically adjust the displayed content.

[0075] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A light field display sight tracking method based on deep learning, characterized in that: The following steps are involved: Use the camera to capture the user's current image in real time; Use the face detection model to detect the face area of ​​the user's current image, and return the face bounding box to obtain the face image; Performing facial feature point detection on the facial image using a facial feature point detection model to obtain facial feature point information, obtaining head posture information using an EPnP algorithm based on the facial feature point information, and cropping the facial image according to the facial feature point information to obtain an eye image; Preprocessing the eye image to obtain a preprocessed eye image; The preprocessed eye image is input into a pretrained gaze prediction model and the head posture information is input as a parameter into a fully connected layer in the gaze prediction model. The gaze prediction model adopts a VGG16 network, and on the basis of the original VGG16 network, the stride of all maximum pooling layers is modified to 1, and the head posture information is spliced ​​into a flattened feature vector to form a new feature vector, and the spliced ​​feature vector is then input into a fully connected layer for classification. The gaze prediction model extracts key eye features of the eye image through a series of convolutional layers, including pupil position, eyeball contour, and iris texture, and then inputs the key eye features into a fully connected layer, predicts the user's gaze vector through a regression method, and outputs the predicted gaze vector.

2. The light field display sight tracking method based on deep learning according to claim 1, characterized in that: The sight line prediction model adopts the MSE loss function during the training process.

3. The light field display sight tracking method based on deep learning according to claim 2, characterized in that: The sight line prediction model is optimized using a gradient descent algorithm, and the optimization process includes the following steps: Initialize model parameters; According to the parameters of the current model, the predicted value is calculated, and the loss is calculated according to the MSE loss function; Calculate the gradient of the MSE loss function with respect to the model parameters to determine the contribution of each parameter to the total loss; According to the calculated gradient, the model parameters are updated with the preset learning rate, and the above steps are repeated to gradually reduce the value of the MSE loss function.

4. The light field display sight tracking method based on deep learning according to claim 3, characterized in that: The learning rate is adjusted using the StepLR learning rate scheduler.

5. The light field display sight tracking method based on deep learning according to any one of claims 1 to 4, characterized in that: The data set used for training the gaze prediction model is MPIIFaceGaze, and the images in MPIIFaceGaze are preprocessed, including image normalization, flipping and histogram equalization, wherein the flipping includes image flipping, gaze direction flipping, head rotation flipping and gaze starting point flipping.

6. The light field display sight tracking method based on deep learning according to any one of claims 1 to 4, characterized in that: The face detection model uses the Haar cascade classifier of OpenCV.

7. The light field display sight tracking method based on deep learning according to any one of claims 1 to 4, characterized in that: The facial feature point detection model uses a pre-trained Dlib-68-point facial feature point model.

8. The light field display sight tracking method based on deep learning according to any one of claims 1 to 4, characterized in that: The process of preprocessing the eye image includes grayscale processing, image normalization processing and image denoising processing on the eye image, and adjusting the eye image to a horizontal state and scaling it to a specified size.

9. The light field display sight tracking method based on deep learning according to any one of claims 1 to 4, characterized in that: The camera is a camera installed in a binocular camera.

10. A light field display method, characterized in that: The following steps are involved: Step 1: construct a visual perception model, where the visual perception model is used to adjust the primitive image in the light field acquisition stage according to the sight vector; Step 2: Determine whether the face detection model detects a face or an eye within the light field imaging observation range. If so, execute step 3; otherwise, the light field display screen directly displays the initial primitive image. Step 3: Activate binocular cameras to retrieve depth information in real time, measure the distance of the human eye in real time, and obtain the sight vector of the human eye according to the light field display sight tracking method according to any one of claims 1 to 9, and send the sight vector of the human eye to the visual perception model, and the visual perception model controls the light field display screen to switch to a primitive image corresponding to the current depth of field acquisition range and the distance of the human eye.

Images