Human eye tracking method and device, storage medium and naked eye three-dimensional display system

Abstract

The embodiment of the invention provides a human eye tracking method and device, a storage medium and a naked eye three-dimensional display system. The human eye tracking method comprises the steps that coordinate values of k human face key points corresponding to front m-1 frames of target images and the mth frame of target image are acquired, the target images are images including the human face of a target object, m is a positive integer larger than 1, and k is a positive integer larger than 1; based on the mth frame of target image, determining coordinate initial values of k face key points corresponding to the mth frame of target image; based on the coordinate values of the k face key points corresponding to the first m-1 frames of target images, processing the coordinate initial values of the k face key points corresponding to the mth frame of target image to obtain the coordinate values of the k face key points corresponding to the mth frame of target image; and according to the coordinate values of the k face key points corresponding to the mth frame of target image, determining the three-dimensional coordinate values of the human eyes of the target object corresponding to the mth frame of target image.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to, but are not limited to, the field of computer vision technology, and in particular to a human eye tracking method and device, a storage medium, and a naked eye three-dimensional display system. Background Art

[0002] With the continuous development of display technology, naked-eye three-dimensional (3D) display technology has become the focus of people's attention and is increasingly used in electronic products such as mobile phones or tablets. Its principle is mainly to use the left and right eyes of a person to receive the left eye image and the right eye image with a certain parallax respectively. When the two parallax images are received by the left and right eyes of a person respectively, the image information is superimposed and fused through the brain to construct a 3D visual display effect. Viewers can directly watch 3D images on naked-eye 3D display products without wearing additional auxiliary equipment such as 3D glasses or helmets. Therefore, it has become the main development direction of new display technology and has been widely used in film and television entertainment, advertising media, exhibitions, engineering design, education and teaching and other fields.

[0003] In the field of naked-eye 3D display technology, eye tracking technology based on computer vision is usually required. However, the three-dimensional coordinates of the human eye obtained by the current eye tracking technology have low accuracy, which can easily cause naked-eye 3D display products to have visible image jitters, affecting the viewing experience. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for tracking human eyes, the method comprising: obtaining coordinate values ​​of k human face key points corresponding to each of the first m-1 target image frames and an m-th target image frame, wherein the target image is an image including a face of a target object, m is a positive integer greater than 1, and k is a positive integer greater than 1; based on the m-th target image frame, determining initial coordinate values ​​of the k human face key points corresponding to the m-th target image frame; based on the coordinate values ​​of the k human face key points corresponding to each of the first m-1 target images, processing the initial coordinate values ​​of the k human face key points corresponding to the m-th target image frame to obtain the coordinate values ​​of the k human face key points corresponding to the m-th target image frame; determining the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-th target image frame according to the coordinate values ​​of the k human face key points corresponding to the m-th target image frame.

[0006] In some exemplary embodiments, the facial key points include at least: any one or more of the key points corresponding to the left eye of the target object, the key points corresponding to the right eye of the target object, the key points corresponding to the nose of the target object, the key points corresponding to the left corner of the target object's mouth, and the key points corresponding to the right corner of the target object's mouth.

[0007] In some exemplary embodiments, the processing of the initial coordinate values ​​of the k facial key points corresponding to the m-th frame of the target image based on the coordinate values ​​of the k facial key points corresponding to the first m-1 frames of the target image to obtain the coordinate values ​​of the k facial key points corresponding to the m-th frame of the target image includes:

[0008] Based on the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image and the coordinate values ​​of the k facial key points corresponding to the first m-1 frames target image, determine the average coordinate value of the k facial key points corresponding to the m-th frame target image;

[0009] Calculate the difference between the initial coordinate value of the k facial key points corresponding to the m-th frame target image and the average coordinate value of the k facial key points to obtain the coordinate deviation value of the k facial key points corresponding to the m-th frame target image;

[0010] Determine a first weight and a second weight based on a coordinate deviation value of a first face key point and a coordinate deviation value of a second face key point among coordinate deviation values ​​of k face key points corresponding to the target image of the mth frame;

[0011] Determine the coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the average coordinate value of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the coordinate deviation value of the first facial key points, and the coordinate deviation value of the second facial key points.

[0012] In some exemplary embodiments, determining the first weight and the second weight based on the coordinate deviation value of the first face key point and the coordinate deviation value of the second face key point among the coordinate deviation values ​​of the k face key points corresponding to the m-th frame target image includes:

[0013] From the coordinate deviation values ​​of the k facial key points corresponding to the m-th frame target image, obtain the coordinate deviation value of the left eye key point as the coordinate deviation value of the first facial key point, and obtain the coordinate deviation value of the right eye key point as the coordinate deviation value of the second facial key point;

[0014] Based on the coordinate deviation value of the first facial key point, according to the formula Determine the first weight; wherein W(0) represents the first weight, Dx(0) represents the x-coordinate deviation value in the coordinate deviation values ​​of the first facial key point, and Dy(0) represents the y-coordinate deviation value in the coordinate deviation values ​​of the first facial key point;

[0015] Based on the coordinate deviation value of the second facial key point, according to the formula Determine the second weight; wherein W(1) represents the second weight, Dx(1) represents the x-coordinate deviation value in the coordinate deviation value of the second facial key point, and Dy(1) represents the y-coordinate deviation value in the coordinate deviation value of the second facial key point.

[0016] In some exemplary embodiments, determining the coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the coordinate average of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the coordinate deviation value of the first facial key point, and the coordinate deviation value of the second facial key point includes:

[0017] Based on the x-coordinate average value of the coordinate average values ​​of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the x-coordinate deviation value of the coordinate deviation value of the first facial key point, and the x-coordinate deviation value of the coordinate deviation value of the second facial key point, according to the formula Determine the x coordinate value of the coordinate values ​​of the k facial key points corresponding to the m-th frame target image;

[0018] Based on the y-coordinate average value of the coordinate average values ​​of the k facial key points corresponding to the m-th frame target image, the first weight and the second weight, the y-coordinate deviation value of the coordinate deviation value of the first facial key point and the y-coordinate deviation value of the coordinate deviation value of the second facial key point, according to the formula Determine the y coordinate value of the coordinate values ​​of the k facial key points corresponding to the m-th frame target image;

[0019] Wherein, Lx(i)′ represents the x-coordinate value of the coordinate value of the i-th face key point among the k face key points corresponding to the m-th frame target image, and Ly(i)′ represents the y-coordinate value of the coordinate value of the i-th face key point among the k face key points corresponding to the m-th frame target image. It represents the average x-coordinate of the average coordinates of the i-th face key point among the k face key points corresponding to the target image of the m-th frame. represents the y-coordinate average value of the coordinate average values ​​of the i-th face key point among the k face key points corresponding to the m-th frame target image, W(0) represents the first weight, W(1) represents the second weight, Dx(0) represents the x-coordinate deviation value in the coordinate deviation values ​​of the first face key point, Dy(0) represents the y-coordinate deviation value in the coordinate deviation values ​​of the first face key point, Dx(1) represents the x-coordinate deviation value in the coordinate deviation values ​​of the second face key point, Dy(1) represents the y-coordinate deviation value in the coordinate deviation values ​​of the second face key point, i=1,...,k.

[0020] In some exemplary embodiments, determining the three-dimensional coordinate values ​​of the eyes of the target object corresponding to the m-th frame of the target image according to the coordinate values ​​of k facial key points corresponding to the m-th frame of the target image includes:

[0021] Using a human eye coordinate estimation algorithm, based on the coordinate values ​​of k facial key points corresponding to the m-th frame target image, estimate the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image;

[0022] Obtaining the z coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to each of the first m-1 frames of target images;

[0023] The three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image is determined based on the estimated three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image and the z coordinate value in the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the first m-1 frames of target images.

[0024] In some exemplary embodiments, determining the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame of the target image according to the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame of the target image and the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to each of the first m-1 frames of the target image comprises:

[0025] Based on the z coordinate estimated value in the three-dimensional coordinate estimated value of the human eye of the target object corresponding to the m-th frame target image and the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the previous m-1 frames of target images, according to the formula Calculate the z coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image;

[0026] Based on the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, the z coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and the x coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, according to the formula Determine an x ​​coordinate value in the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-th frame target image;

[0027] Based on the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, the z coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and the y coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, according to the formula Determine the y coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image;

[0028] Wherein, z′ represents the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, represents the sum of the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image and the z coordinate values ​​in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the previous m-1 frames of target images, x′ represents the x coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, z(m) represents the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, x(m) represents the x coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, y′ represents the y coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and y(m) represents the y coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image.

[0029] In some exemplary embodiments, determining the initial coordinate values ​​of k facial key points corresponding to the m-th frame of the target image based on the m-th frame of the target image includes:

[0030] Segment the image region where the face of the target object is located from the m-th frame target image to obtain a target face region map corresponding to the m-th frame target image;

[0031] Performing denoising on the target face region map corresponding to the target image of the m-th frame to obtain a denoised target face region map corresponding to the target image of the m-th frame;

[0032] Perform facial key point detection on the denoised target face area map corresponding to the m-th frame target image to obtain the initial coordinate values ​​of k facial key points corresponding to the m-th frame target image.

[0033] In some exemplary embodiments, segmenting the image region where the face of the target object is located from the m-th frame of the target image to obtain a target face region map corresponding to the m-th frame of the target image includes:

[0034] Get the face detection frame corresponding to the target image of the mth frame;

[0035] Based on the face detection frame corresponding to the m-th frame target image, the image area where the face of the target object is located is segmented from the m-th frame target image to obtain a target face area map corresponding to the m-th frame target image.

[0036] In some exemplary embodiments, the step of obtaining a face detection frame corresponding to the m-th frame of the target image includes:

[0037] Perform face detection on the m-th target image to obtain a face detection frame corresponding to the m-th target image; or obtain a face detection frame corresponding to the m-1-th target image as the face detection frame corresponding to the m-th target image.

[0038] In some exemplary embodiments, the denoising process is performed on the target face region map corresponding to the m-th frame target image to obtain the denoised target face region map corresponding to the m-th frame target image, including:

[0039] Determine whether the resolution of the target face region map corresponding to the m-th frame target image meets a preset image resolution;

[0040] If not satisfied, the target face region map corresponding to the m-th frame target image is normalized at a preset image resolution to obtain a normalized target face region map corresponding to the m-th frame target image; the normalized target face region map corresponding to the m-th frame target image is denoised to obtain a denoised target face region map corresponding to the m-th frame target image;

[0041] Alternatively, if the condition is satisfied, a denoising process is performed on the target face region map corresponding to the m-th frame target image to obtain a denoised target face region map corresponding to the m-th frame target image.

[0042] In some exemplary embodiments, the denoising process is performed on the target face region map corresponding to the m-th frame target image to obtain the denoised target face region map corresponding to the m-th frame target image, including:

[0043] A denoising process is performed on the target face region map corresponding to the m-th frame target image by using any one of a Gaussian filter and a mean filter to obtain a denoised target face region map corresponding to the m-th frame target image.

[0044] In a second aspect, an embodiment of the present disclosure provides an eye tracking device, comprising: a processor and a memory storing a computer program that can be run on the processor, wherein the processor is configured to implement the steps of the eye tracking method described in any of the above embodiments when executing the program.

[0045] In a third aspect, an embodiment of the present disclosure provides a non-volatile computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the steps of the human eye tracking method described in any one of the above embodiments.

[0046] In a fourth aspect, an embodiment of the present disclosure provides a naked-eye three-dimensional display system, the system comprising: a naked-eye three-dimensional display device, an image acquisition device, a human eye tracking device, and a processing device, wherein:

[0047] The image acquisition device is arranged on the naked-eye 3D display device, and is arranged to acquire a target image, wherein the target image is an image including a face of a target object, and the target object is an object viewing a display screen of the naked-eye 3D display device;

[0048] The eye tracking device is configured to perform the steps of the eye tracking method according to any one of claims 1 to 12;

[0049] The processing device is configured to process the image to be displayed according to the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image output by the human eye tracking device to obtain a naked-eye three-dimensional synthetic image; and send the naked-eye three-dimensional synthetic image to the naked-eye three-dimensional display device;

[0050] The naked-eye 3D display device is configured to display the naked-eye 3D synthetic image.

[0051] The human eye tracking method and device, storage medium, and naked-eye 3D display system provided by the exemplary embodiments of the present disclosure, after obtaining the coordinate values ​​of k human face key points corresponding to the m-th frame target image and the first m-1 frame target image, determine the initial coordinate values ​​of the k human face key points corresponding to the m-th frame target image through the m-th frame target image, and then perform low-latency human face key point denoising processing on the initial coordinate values ​​of the k human face key points corresponding to the m-th frame target image through the coordinate values ​​of the k human face key points corresponding to the first m-1 frame target image, so that the noise of the obtained coordinate values ​​of the k human face key points corresponding to the m-th frame target image can be reduced. Next, the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-th frame target image are determined through the accurate coordinate values ​​of the k human face key points corresponding to the m-th frame target image, so that the accuracy of the obtained three-dimensional coordinates of the human eye can be improved, and the accuracy and timeliness of human eye tracking can be maintained. Therefore, when applied to the naked-eye 3D display system, since the accuracy of the three-dimensional coordinates of the human eye is improved, the naked-eye 3D display product can avoid the visible picture jitter of the human eye, thereby improving the naked-eye 3D display effect.

[0052] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. Other advantages of the present disclosure can be realized and obtained by the schemes described in the description and the drawings.

[0053] Other aspects will be apparent upon reading and understanding the drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide an understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure. The shapes and sizes of the components in the accompanying drawings do not reflect the actual proportions and are only intended to illustrate the contents of the present disclosure.

[0055] Figure 1 is a schematic diagram of a framework of a naked-eye 3D display system in an exemplary embodiment of the present disclosure;

[0056] Figure 2 A flowchart of a human eye tracking method in an exemplary embodiment of the present disclosure is provided;

[0057] Figure 3 is a schematic diagram of k facial key points of a target object's face in an exemplary embodiment of the present disclosure;

[0058] Figure 4 Another flowchart of the human eye tracking method in the exemplary embodiment of the present disclosure is shown;

[0059] Figure 5 is a schematic structural diagram of a human eye tracking device in an exemplary embodiment of the present disclosure;

[0060] Figure 6 It is a schematic structural diagram of a naked-eye three-dimensional display system in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] Multiple embodiments are described herein, but the description is exemplary rather than restrictive, and there may be more embodiments and implementations within the scope of the embodiments described herein. Although many possible feature combinations are shown in the drawings and discussed in the exemplary embodiments, many other combinations of the disclosed features are possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0062] When describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be appreciated by those of ordinary skill in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation on the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can easily understand that these orders can be changed and still remain within the spirit and scope of the disclosed embodiments.

[0063] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0064] The following describes some of the terms or technologies involved in the embodiments of the present disclosure:

[0065] Naked-eye 3D display: means that viewers can watch 3D images without wearing glasses, helmets or any other visual aids.

[0066] Eye tracking: refers to the real-time detection of the 3D coordinates of the human eye.

[0067] De-jittering: refers to eliminating or reducing the coordinate noise output by eye tracking.

[0068] Filter: refers to a signal processing method that removes signal noise.

[0069] Image acquisition: refers to the use of image acquisition devices such as cameras or webcams to capture images containing the viewer's face.

[0070] Face detection: It means locating the image area where the face is located given an image containing a face.

[0071] Face key point detection: refers to locating the key areas of the face, such as eyebrows, eyes, nose, mouth, facial contour, etc., from a given detected face area map.

[0072] Face Quality Detection: refers to the standardized evaluation of the input face area map after face detection, removing low-quality images, and sending images that meet the preset quality standards to the next processing flow to improve processing efficiency. It is generally based on the principle of feature extraction. It uses neural networks to learn from massive data to obtain the features that face quality detection focuses on (such as light, blur, angle, occlusion, expression, noise, etc.) and make quality judgments, and finally give a quality score value.

[0073] In the embodiments of the present disclosure, the "coordinates of key points of the face" used refer to two-dimensional coordinates, including the x-coordinates of the key points of the face and the y-coordinates of the key points of the face. The "three-dimensional coordinates of the eyes of the target object" used include the x-coordinates of the eyes of the target object, the y-coordinates of the eyes of the target object, and the z-coordinates of the eyes of the target object.

[0074] The embodiments of the present disclosure provide a naked-eye 3D display system. Figure 1 FIG. 1 is a schematic diagram of a naked-eye 3D display system in an exemplary embodiment of the present disclosure. Figure 1As shown, the naked-eye 3D display system may include: a naked-eye 3D display 11, a camera 12 and a processor 13; wherein the naked-eye 3D display 11 is configured to display a naked-eye 3D synthetic image so that the viewer can view an immersive and realistic 3D display effect; the camera 12 is configured to collect images containing the viewer's face in real time, and the processor 13 is configured to collect images containing the viewer's face in real time through a camera installed on the naked-eye 3D display, detect facial key points using a deep learning model, analyze the viewer's eye 3D coordinates based on the facial key points, and achieve eye tracking; the acquired 3D video source is rearranged according to the obtained eye 3D coordinates, and the naked-eye 3D left and right view pixel arrangement is adjusted in real time to obtain a naked-eye 3D synthetic image (including naked-eye 3D left and right views), so that the viewer can see high-quality 3D image effects in a wide range.

[0075] However, during the research process, the inventors of the present disclosure discovered that due to the image noise of the image containing the viewer's face collected by the camera and the error of the deep learning model, the three-dimensional coordinates of the human eye obtained by the eye tracking technology often have large jitter noise and low accuracy, which can easily cause the naked eye 3D display to have visible picture jitter, affecting the viewing experience.

[0076] The disclosed embodiments provide a human eye tracking method, which can improve the accuracy of the obtained three-dimensional coordinates of the human eye by reducing the coordinate noise of the key points of the human face in the human eye tracking technology, and can maintain the timeliness of the human eye tracking. Therefore, when applied to a naked-eye 3D display system, the naked-eye 3D display product can avoid the occurrence of visible picture jitter in the naked-eye 3D display product due to the improved accuracy of the human eye tracking, thereby improving the naked-eye 3D display effect.

[0077] Figure 2 FIG. 1 is a flow chart of a human eye tracking method in an exemplary embodiment of the present disclosure. Figure 2 As shown, the human eye tracking method may include the following steps:

[0078] Step 21: Obtain the coordinate values ​​of k facial key points corresponding to the m-th frame target image and the previous m-1 frames target image, respectively, wherein m is a positive integer greater than 1, k is a positive integer greater than 1, and the target image is an image including the face of the target object;

[0079] Step 22: Based on the m-th frame target image, determine the initial coordinate values ​​of k facial key points corresponding to the m-th frame target image;

[0080] Step 23: Based on the coordinate values ​​of the k facial key points corresponding to the first m-1 frames of target images, the initial coordinate values ​​of the k facial key points corresponding to the m-th frame of target image are processed to obtain the coordinate values ​​of the k facial key points corresponding to the m-th frame of target image;

[0081] Step 24: According to the coordinate values ​​of the k facial key points corresponding to the m-th frame target image, determine the three-dimensional coordinate values ​​of the human eyes of the target object corresponding to the m-th frame target image.

[0082] Thus, in the eye tracking method in the embodiment of the present disclosure, after obtaining the coordinate values ​​of the k facial key points corresponding to the m-th frame target image and the first m-1 frame target image, the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image are determined by the m-th frame target image, and then the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image are subjected to low-latency facial key point denoising processing, so that the noise of the obtained coordinate values ​​of the k facial key points corresponding to the m-th frame target image can be reduced. Next, the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-th frame target image are determined by the accurate coordinate values ​​of the k facial key points corresponding to the m-th frame target image, so that the accuracy of the obtained three-dimensional coordinates of the human eye can be improved, and the accuracy and timeliness of the eye tracking can be maintained. Therefore, when applied to the naked-eye 3D display system, since the accuracy of the three-dimensional coordinates of the human eye is improved, the naked-eye 3D display product can avoid the visible picture jitter of the human eye, thereby improving the naked-eye 3D display effect.

[0083] In some exemplary embodiments, the face of the target object may refer to the face of the viewer who is looking at the naked-eye 3D display device. The target image may be an image including the face of the target object captured when the viewer is watching the display screen of the naked-eye 3D display device. Exemplarily, step S1 may include: when the viewer is watching the display screen of the naked-eye 3D display device, an image capture device (such as a camera) is used to capture an image of the target object looking at the naked-eye 3D display screen, and obtain a target image including the face of the target object.

[0084] In some exemplary embodiments, the k facial key points may include at least one or more of: a key point corresponding to the left eye of the target object, a key point corresponding to the right eye of the target object, a key point corresponding to the nose of the target object, a key point corresponding to the left corner of the mouth of the target object, and a key point corresponding to the right corner of the mouth of the target object. Figure 3 As shown, the key points corresponding to the left eye of the target object may include: the center point P1 of the left eye of the target object, the key points corresponding to the right eye of the target object may include: the center point P2 of the right eye of the target object, the key points corresponding to the nose of the target object may include: the center point P3 of the tip of the nose of the target object, the key points corresponding to the left corner of the mouth of the target object may include: the center point P4 of the left corner of the mouth of the target object, and the key points corresponding to the right corner of the mouth of the target object may include: the center point P5 of the right corner of the mouth of the target object.

[0085] Of course, in addition to the key points exemplified above, other key points that can ultimately determine the three-dimensional coordinates of the human eye may also be used, such as the center point of the left eye corner of the target object, the center point of the right eye corner of the target object, the contour point of the left eye of the target object, the contour point of the right eye of the target object, the center point of the chin of the target object, the center point of the left eyebrow of the target object, the center point of the right eyebrow of the target object, etc. Here, the embodiments of the present disclosure do not limit this.

[0086] In some exemplary embodiments, step 23 may include:

[0087] Step 231: based on the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image and the coordinate values ​​of the k facial key points corresponding to the first m-1 frames target image, determine the average coordinate values ​​of the k facial key points corresponding to the m-th frame target image;

[0088] Step 232: Calculate the difference between the initial coordinate value of the k facial key points corresponding to the m-th frame target image and the average coordinate value of the k facial key points to obtain the coordinate deviation value of the k facial key points corresponding to the m-th frame target image;

[0089] Step 233: determining a first weight and a second weight based on the coordinate deviation value of the first face key point and the coordinate deviation value of the second face key point among the coordinate deviation values ​​of the k face key points corresponding to the target image of the mth frame;

[0090] Step 234: Determine the coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the coordinate average value of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the coordinate deviation value of the first facial key point and the coordinate deviation value of the second facial key point.

[0091] In some exemplary embodiments, the initial values ​​of the coordinates of the facial key points (i.e., the coordinate values ​​of the facial key points before denoising) may include the initial values ​​of the x-coordinates of the facial key points and the initial values ​​of the y-coordinates of the facial key points. The coordinate deviation values ​​of the facial key points may include: the x-coordinate deviation values ​​of the facial key points and the y-coordinate deviation values ​​of the facial key points. The coordinate average values ​​of the facial key points may include: the x-coordinate average values ​​of the facial key points and the y-coordinate average values ​​of the facial key points. The coordinate values ​​of the facial key points (i.e., the coordinate values ​​of the facial key points after denoising) may include: the x-coordinate values ​​of the facial key points and the y-coordinate values ​​of the facial key points.

[0092] In some exemplary embodiments, taking the case where k facial key points may include at least: a key point corresponding to the left eye of the target object, a key point corresponding to the right eye of the target object, a key point corresponding to the nose of the target object, a key point corresponding to the left corner of the mouth of the target object, and a key point corresponding to the right corner of the mouth of the target object, step 231 may include:

[0093] Step 2311: Based on the initial x-coordinate values ​​of the coordinates of the k facial key points corresponding to the m-th frame target image and the x-coordinate values ​​of the coordinates of the k facial key points corresponding to the first m-1 frames target image, according to the formula Determine the average x-coordinate value of the coordinates of the k facial key points corresponding to the target image of the mth frame.

[0094] Step 2312: Based on the initial y coordinate values ​​of the coordinates of the k facial key points corresponding to the m-th frame target image and the y coordinate values ​​of the coordinates of the k facial key points corresponding to the first m-1 frames target image, according to the formula Determine the average y coordinate of the average coordinates of the k facial key points corresponding to the m-th frame target image.

[0095] in, It represents the average x-coordinate of the average coordinates of the i-th face key point among the k face key points corresponding to the target image of the m-th frame. represents the average value of the y coordinate of the coordinates of the i-th face key point among the k face key points corresponding to the m-th frame target image, Lx(n,i) represents the x coordinate of the coordinates of the i-th face key point among the k face key points corresponding to the n-th frame target image, represents the sum of the x-coordinate value of the coordinates of the ith face key point among the k face key points corresponding to the m-th frame target image and the x-coordinate value of the coordinates of the ith face key point among the k face key points corresponding to the first m-1 frames target image, Ly(n,i) represents the y-coordinate value of the coordinates of the ith face key point among the k face key points corresponding to the n-th frame target image, It represents the sum of the initial y-coordinate value of the coordinates of the i-th face key point among the k face key points corresponding to the m-th frame target image and the y-coordinate value of the coordinates of the i-th face key point among the k face key points corresponding to the previous m-1 frames target image, n=1,...,m, i=1,...,k. For example, k=1,...,5. Here, when n is m, Lx(m,i) represents the initial x-coordinate value of the coordinates of the i-th face key point among the k face key points corresponding to the m-th frame target image, and Ly(m,i) represents the initial y-coordinate value of the coordinates of the i-th face key point among the k face key points corresponding to the m-th frame target image. When n takes the value of 1, …, m-1 respectively, Lx(1,i), …, Lx(m-1,i) respectively represent the x-coordinate value in the coordinate values ​​of the i-th face key point among the k face key points corresponding to the target image of the 1st frame, …, the x-coordinate value in the coordinate values ​​of the i-th face key point among the k face key points corresponding to the target image of the m-1th frame; Ly(1,i), …, Ly(m-1,i) represent the y-coordinate value in the coordinate values ​​of the i-th face key point among the k face key points corresponding to the target image of the m-1th frame, …, the y-coordinate value in the coordinate values ​​of the i-th face key point among the k face key points corresponding to the target image of the m-1th frame.

[0096] In some exemplary embodiments, taking the case where k facial key points may include at least: a key point corresponding to the left eye of the target object, a key point corresponding to the right eye of the target object, a key point corresponding to the nose of the target object, a key point corresponding to the left corner of the mouth of the target object, and a key point corresponding to the right corner of the mouth of the target object, step 233 may include:

[0097] Step 2331: obtaining the coordinate deviation value of the left eye key point from the coordinate deviation values ​​of the k facial key points corresponding to the m-th frame target image as the coordinate deviation value of the first facial key point, and obtaining the coordinate deviation value of the right eye key point as the coordinate deviation value of the second facial key point;

[0098] Step 2332: Based on the coordinate deviation value of the first facial key point, according to the formula Determine the first weight; wherein W(0) represents the first weight, Dx(0) represents the x-coordinate deviation value of the coordinate deviation value of the first facial key point (i.e., the x-coordinate deviation value of the left eye key point corresponding to the m-th frame target image), and Dy(0) represents the y-coordinate deviation value of the coordinate deviation value of the first facial key point (i.e., the y-coordinate deviation value of the left eye key point corresponding to the m-th frame target image);

[0099] Step 2333: Based on the coordinate deviation value of the second facial key point, according to the formula Determine the second weight; wherein W(1) represents the second weight, Dx(1) represents the x-coordinate deviation value in the coordinate deviation value of the second facial key point (i.e., the x-coordinate deviation value of the right eye key point corresponding to the m-th frame target image), and Dy(1) represents the y-coordinate deviation value in the coordinate deviation value of the second facial key point (i.e., the y-coordinate deviation value of the right eye key point corresponding to the m-th frame target image).

[0100] In some exemplary embodiments, step 234 may include:

[0101] Step 2341: Based on the x-coordinate average value of the coordinate average values ​​of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the x-coordinate deviation value of the coordinate deviation value of the first facial key point, and the x-coordinate deviation value of the coordinate deviation value of the second facial key point, according to the formula Determine the x coordinate value of the coordinate values ​​of the k facial key points corresponding to the m-th frame target image;

[0102] Step 2342: Based on the average y coordinate value of the coordinate average values ​​of the k facial key points corresponding to the m-th frame target image, the first weight and the second weight, the y coordinate deviation value of the coordinate deviation value of the first facial key point and the y coordinate deviation value of the coordinate deviation value of the second facial key point, according to the formula Determine the y coordinate value of the coordinate values ​​of the k facial key points corresponding to the m-th frame target image;

[0103] Wherein, Lx(i)′ represents the x-coordinate value of the coordinate value of the i-th face key point among the k face key points corresponding to the m-th frame target image, and Ly(i)′ represents the y-coordinate value of the coordinate value of the i-th face key point among the k face key points corresponding to the m-th frame target image. It represents the average x-coordinate of the average coordinates of the i-th face key point among the k face key points corresponding to the target image of the m-th frame. represents the average value of the y coordinate of the coordinate average values ​​of the i-th face key point among the k face key points corresponding to the m-th frame target image, W(0) represents the first weight, W(1) represents the second weight, Dx(0) represents the x coordinate deviation value of the coordinate deviation value of the first face key point, Dy(0) represents the y coordinate deviation value of the coordinate deviation value of the first face key point, Dx(1) represents the x coordinate deviation value of the coordinate deviation value of the second face key point, Dy(1) represents the y coordinate deviation value of the coordinate deviation value of the second face key point, i=1,…,k. For example, k=1,…,5.

[0104] In some exemplary embodiments, step 24 may include:

[0105] Step 241: using a human eye coordinate estimation algorithm, based on the coordinate values ​​of k facial key points corresponding to the m-th frame target image, estimate the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image;

[0106] Step 242: obtaining the z coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to each of the first m-1 frames of target images;

[0107] Step 243: Determine the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image based on the estimated three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image and the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the first m-1 frames of target images.

[0108] In some exemplary embodiments, the eye coordinate estimation algorithm may be based on an eye positioning algorithm based on pupil distance, or may use a PNP (Perspective-N-Points) algorithm. Of course, other algorithms that can convert two-dimensional coordinates into three-dimensional coordinates may also be used. This is not limited in the embodiments of the present disclosure.

[0109] In some exemplary embodiments, step 241 may include: using an eye positioning algorithm based on pupil distance, based on the coordinate values ​​of k facial key points corresponding to the m-th frame target image, estimating the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image.

[0110] In some other exemplary embodiments, step 241 may include: using a PNP algorithm to estimate the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image based on the coordinate values ​​of k facial key points corresponding to the m-th frame target image.

[0111] In some exemplary embodiments, step 243 may include:

[0112] Step 2431: Based on the z coordinate estimated value in the three-dimensional coordinate estimated value of the human eye of the target object corresponding to the m-th frame target image and the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to each of the previous m-1 frames of target images, according to the formula Calculate the z coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image;

[0113] Step 2432: Based on the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, the z coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and the x coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, according to the formula Determine the x coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image;

[0114] Step 2433: Based on the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, the z coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and the y coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, according to the formula Determine the y coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image;

[0115] Wherein, z′ represents the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, represents the sum of the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image and the z coordinate values ​​in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the previous m-1 frames of target images, x′ represents the x coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, z(m) represents the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, x(m) represents the x coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, y′ represents the y coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and y(m) represents the y coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image.

[0116] Here, n represents the frame number, and a larger frame number corresponds to a later frame; m represents the frame number of the current frame, n=1,…,m.

[0117] Thus, in the disclosed embodiment, the three-dimensional coordinate estimation value of the human eye of the target object is estimated based on the coordinate values ​​of the k human face key points corresponding to the m-th frame target image through the human eye coordinate estimation algorithm, and then the z coordinate estimation value in the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image is denoised by the z coordinate value of the human eye of the target object corresponding to each of the first m-1 frames of target images, so as to obtain the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image. Thus, the z coordinate noise in the three-dimensional coordinate value of the human eye can be reduced. Moreover, by determining the x coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image based on the z coordinate estimation value in the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image, the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, and the x coordinate estimation value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, it can be ensured that the x coordinate in the three-dimensional coordinate value of the human eye is not affected by the z coordinate error. In addition, the y coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image is determined by the z coordinate estimate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, and the y coordinate estimate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, so that the y coordinate in the three-dimensional coordinate value of the human eye can be ensured not to be affected by the z coordinate error. In this way, it is achieved that the z coordinate noise in the three-dimensional coordinate value of the human eye is reduced while ensuring that the x coordinate and y coordinate in the three-dimensional coordinate value of the human eye are not affected by the z coordinate error, so that the accuracy of the three-dimensional coordinate of the human eye of the target object can be further improved, thereby improving the naked-eye 3D display effect.

[0118] In some other exemplary embodiments, after step 24, the human eye tracking method may further include: step 25: filtering the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-th frame target image through a filter to obtain the three-dimensional coordinate output values ​​of the human eye of the target object corresponding to the m-th frame target image. In this way, jitter noise can be reduced.

[0119] Exemplarily, step 25 may include: using a mean filter to filter the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-th frame target image to obtain the three-dimensional coordinate output values ​​of the human eye of the target object corresponding to the m-th frame target image.

[0120] Here, when the window size of the mean filter is too large, although the jitter noise can be reduced, signal delay will be introduced, resulting in untimely update of the naked eye 3D picture. Therefore, those skilled in the art can determine the appropriate window size of the mean filter according to the actual scene, and the embodiments of the present disclosure do not limit this.

[0121] In some exemplary embodiments, step 22 may include:

[0122] Step 221: segmenting the image region where the face of the target object is located from the m-th frame target image, and obtaining a target face region map corresponding to the m-th frame target image;

[0123] Step 222: performing denoising processing on the target face region map corresponding to the m-th frame target image to obtain a denoised target face region map corresponding to the m-th frame target image;

[0124] Step 223: perform facial key point detection on the denoised target face region map corresponding to the m-th frame target image to obtain initial coordinate values ​​of k facial key points corresponding to the m-th frame target image.

[0125] Thus, in the eye tracking method in the disclosed embodiment, before performing face key point detection, the target face area map corresponding to the m-th frame target image is subjected to denoising to obtain the denoised target face area map corresponding to the m-th frame target image, and then face key point detection is performed on the denoised target face area map corresponding to the m-th frame target image to obtain the initial coordinate values ​​of the k face key points corresponding to the m-th frame target image. In this way, the face key point detection noise can be reduced, so that the accuracy of the coordinate values ​​of the k face key points corresponding to the m-th frame target image, which are subsequently determined based on the initial coordinate values ​​of the k face key points corresponding to the m-th frame target image, is further improved. Therefore, the accuracy of the three-dimensional coordinates of the target object's human eye can be further improved, thereby improving the naked-eye 3D display effect.

[0126] In some exemplary embodiments, step 221 may include:

[0127] Step 2211: Obtain the face detection frame corresponding to the m-th frame target image;

[0128] Step 2212: based on the face detection frame corresponding to the target image of the mth frame, segment the image area where the face of the target object is located from the target image of the mth frame, and obtain the target face area map corresponding to the target image of the mth frame.

[0129] In some exemplary embodiments, step 2211 may include: performing face detection on the m-th frame target image to obtain a face detection frame corresponding to the m-th frame target image.

[0130] In some other exemplary embodiments, step 2211 may include: obtaining a face detection frame corresponding to the target image of the m-1th frame as the face detection frame corresponding to the target image of the mth frame. In this way, the face tracking algorithm can be simplified, the processing speed can be improved, and delays can be avoided.

[0131] In some exemplary embodiments, the face detection frame may refer to a detection frame that can include the position information of the face of the target object in the target image. Exemplarily, the face detection frame may be a minimum rectangular frame that can include all the information of the face in the target image. For example, the face detection frame may include: the coordinates of the upper left corner of the image area where the face of the target object is located in the target image are marked as (x1, y1), and the coordinates of the lower right corner of the face area map in the target image are marked as (x2, y2).

[0132] In an exemplary embodiment, the number of target objects simultaneously looking at the naked-eye 3D display device may be one, or may be two or more. Correspondingly, the number of face detection frames may be one, or may be two or more. For example, when the number of target objects simultaneously looking at the naked-eye 3D display screen is two, the positional relationship between the face detection frames of the two target objects may be adjacent or far apart, and the area occupied by the two face detection frames in the target image may be the same, or may be different.

[0133] In an exemplary embodiment, when the number of target objects looking at the naked-eye 3D display device is one, only one viewer needs to be tracked. Therefore, it can be considered that the viewer moves slowly, and the face detection frame of the previous frame target image (such as the m-1th frame target image) can be used as the face detection frame of the current frame target image (such as the mth frame target image). In this way, the face tracking algorithm is simplified, the processing speed can be improved, and delays can be avoided.

[0134] In some exemplary embodiments, step 2211 may include: performing face detection on the target image using a trained face detection model to determine the target object.

[0135] In an exemplary embodiment, the face detection model may adopt a face detection model based on an adboost (adaptiveboosting, adaptive enhancement) algorithm, or may adopt a face detection model based on deep learning, or may adopt other face detection models. For example, a face detection model based on deep learning may adopt a Retinaface model or a Yolov7face model, etc. For example, other face detection models may include: models generated using target detection algorithms such as Fast Region Convolutional Neural Networks (Fast R-CNN), Faster Region Convolutional Neural Networks (Faster R-CNN), or Regionbased Fully Convolutional Network (R-FCN).

[0136] In some exemplary embodiments, step 222 may include: using a filter to perform denoising processing on the target face region map corresponding to the m-th frame target image to obtain a denoised target face region map corresponding to the m-th frame target image.

[0137] In some exemplary embodiments, the filter may be a Gaussian filter (eg, a Gaussian filter with a window of 3x3), or a mean filter. Here, the embodiments of the present disclosure do not limit this.

[0138] In some other exemplary embodiments, taking the filter including any one of a Gaussian filter and a mean filter as an example, step 222 may include: using any one of a Gaussian filter and a mean filter to denoise the target face area map corresponding to the m-th frame target image, and obtaining the denoised target face area map corresponding to the m-th frame target image.

[0139] In this way, before performing facial key point detection, the target face area map can be denoised by a filter so that the denoised target face area map can be used for facial key point detection later. In this way, the noise of facial key point detection of the deep learning model can be reduced.

[0140] In the process of facial key point detection, the image quality of the face region map will seriously affect the detection accuracy of the facial key points. The worse the image quality of the face region map, the lower the accuracy of facial key point detection. Therefore, before performing facial key point detection, it is possible to first determine whether the image quality of the face region map meets the image quality requirements.

[0141] In some exemplary embodiments, before step 222, the human eye tracking method may further include:

[0142] Step 31: performing face quality detection on the target face region map corresponding to the m-th frame target image to obtain a face quality detection result corresponding to the m-th frame target image;

[0143] Step 32: Determine whether the obtained face quality detection result is not less than a preset threshold;

[0144] Step 33: If it is determined that the obtained face quality detection result is not lower than the preset threshold, execute steps 222 to 223, and determine the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the target face area map corresponding to the m-th frame target image.

[0145] Step 34: If it is determined that the obtained face quality detection result is lower than the preset threshold, obtain the target face area map corresponding to other target images (such as the m-1th frame target image) before the mth frame whose face quality detection result is not lower than the preset threshold, as the target face area map corresponding to the mth frame target image.

[0146] Thus, when the face quality detection result corresponding to the m-th frame target image is not lower than the preset threshold, it indicates that the image quality of the target face region map corresponding to the current m-th frame target image to be processed is good, therefore, the target face region map corresponding to the m-th frame target image can be used to perform subsequent steps to determine the initial coordinate values ​​of the k face key points corresponding to the m-th frame target image, so as to determine the three-dimensional coordinates of the human eye of the target object with high accuracy. However, when the face quality detection result corresponding to the m-th frame target image is not lower than the preset threshold, it indicates that the image quality of the target face region map corresponding to the m-th frame target image to be processed is poor, and face key point detection noise is easily introduced, and it is not suitable for performing subsequent steps. At this time, the target face region map corresponding to other target images (such as the m-1-th frame target image) whose face quality detection results are not lower than the preset threshold and before the m-th frame can be used as the target face region map corresponding to the m-th frame target image to perform subsequent steps to determine the initial coordinate values ​​of the k face key points corresponding to the m-th frame target image, so as to determine the three-dimensional coordinates of the human eye of the target object with relatively high accuracy. In this way, it can be ensured that the image quality transmitted to the subsequent face key point denoising stage is relatively good, the face key point detection noise can be reduced, and thus the accuracy of the initial coordinate value of the face key point can be improved.

[0147] In some other exemplary embodiments, before step 223, the human eye tracking method may further include:

[0148] Step 41: performing face quality detection on the denoised target face region map corresponding to the m-th frame target image, and obtaining a face quality detection result corresponding to the denoised target face region map corresponding to the m-th frame target image;

[0149] Step 42: Determine whether the face quality detection result corresponding to the denoised target face region map corresponding to the m-th frame target image is not less than a preset threshold;

[0150] Step 43: If it is determined that the obtained face quality detection result is not lower than the preset threshold, execute step 223 to determine the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the denoised target face area map corresponding to the m-th frame target image.

[0151] Step 44: If it is determined that the obtained face quality detection result is lower than the preset threshold, obtain the target face area map corresponding to other target images (such as the m-1th frame target image) whose face quality detection results are not lower than the preset threshold and before the mth frame, as the target face area map corresponding to the mth frame target image, and execute steps 222 to 223.

[0152] In an exemplary embodiment, a CNN (Convolutional Neural Networks) deep learning model based on ResNet (Residual Neural Network) can be used to detect face quality. For example, two models can be used to detect facial key points and face quality respectively, or a single model can be used to detect facial key points and face quality at the same time. Here, the embodiments of the present disclosure do not limit this.

[0153] In this way, it can be ensured that the image quality of the target face area map transmitted to the subsequent face key point detection link is good, the face key point detection noise can be reduced, and thus the accuracy of the initial coordinate values ​​of the k face key points corresponding to the m-th frame target image can be improved.

[0154] In some exemplary embodiments, step 223 may include:

[0155] Step 2231: Determine whether the resolution of the target face region map corresponding to the m-th frame target image meets the preset image resolution;

[0156] Step 2232: if it is determined that the resolution of the target face region map corresponding to the target image of the m-th frame does not meet the preset image resolution, the target face region map corresponding to the target image of the m-th frame is normalized at the preset image resolution to obtain the normalized target face region map corresponding to the target image of the m-th frame; denoising is performed on the normalized target face region map corresponding to the target image of the m-th frame to obtain the denoised target face region map corresponding to the target image of the m-th frame;

[0157] Step 2233: If it is determined that the resolution of the target face area map corresponding to the m-th frame target image meets the preset image resolution, denoising is performed on the target face area map corresponding to the m-th frame target image to obtain the denoised target face area map corresponding to the m-th frame target image.

[0158] In some exemplary embodiments, the face detection frame is scaled with a preset image resolution as a standardized size, such as 128x128, and the resolution of the target face region map can be normalized to the standardized size. This is convenient for subsequent processing, for example, it is convenient to use a filter such as a smoothing filter to perform denoising on the face image in order to obtain a denoised target face region map. Of course, the preset image resolution can also be other sizes, which can be set by those skilled in the art according to subsequent processing. Here, the embodiments of the present disclosure do not limit this.

[0159] Figure 4 A second flow chart of the human eye tracking method is provided in the embodiment of the present disclosure, such as Figure 4 As shown, the human eye tracking method may include:

[0160] Step 401: Obtain the mth frame of the target image; m represents the current frame.

[0161] For example, during the image acquisition process, a frame of target image may be read from an image acquisition device (such as a camera), which is recorded as the mth frame of target image.

[0162] For example, the camera may be a monocular RGB (red, green, and blue) camera, or an infrared grayscale image camera. Here, the embodiments of the present disclosure do not limit this.

[0163] Step 402: Perform face detection on the m-th frame target image to obtain a face detection frame corresponding to the m-th frame target image.

[0164] In some exemplary embodiments, a face detection model is used to perform face detection on the face of the target object in the acquired m-th frame target image, and the position of the detected face can be marked with a rectangular frame (i.e., a face detection frame), and the coordinates of the upper left corner of the rectangular frame are marked as (x1, y1), and the lower right corner is marked as (x2, y2). For example, the face detection model can use an adboost-based face detector, or a deep learning-based face detection model (such as a Retinaface model or a Yolov7face model), etc.

[0165] In some exemplary embodiments, when only one viewer needs to be tracked, simple face tracking can be used, assuming that the viewer moves slowly, and using the face detection frame of the previous target image (i.e., the m-1th target image) as the face detection frame of the current target image (i.e., the mth target image). In this way, the algorithm processing speed can be accelerated.

[0166] Step 403: based on the face detection frame corresponding to the m-th frame target image, segment the image region where the face of the target object is located from the m-th frame target image to obtain a target face region map corresponding to the m-th frame target image.

[0167] Step 404: performing denoising processing on the target face region map corresponding to the m-th frame target image to obtain a denoised target face region map corresponding to the m-th frame target image.

[0168] In some exemplary embodiments, the target face region map corresponding to the m-th frame target image may be scaled, and the image resolution of the target face region map corresponding to the m-th frame target image may be normalized to a standard size, such as 128x128, and then a smoothing filter may be used to perform denoising on the target face region map corresponding to the m-th frame target image. In this way, a denoised target face region map corresponding to the m-th frame target image may be obtained. For example, the smoothing filter may be a Gaussian filter with a window of 3x3, or a mean filter may be used.

[0169] Step 405: performing face quality detection on the denoised target face region image corresponding to the m-th frame target image to obtain a face quality detection result;

[0170] Step 406: If it is determined that the obtained face quality detection result is not lower than the preset threshold, perform face key point detection on the denoised target face region map corresponding to the m-th frame target image to obtain the initial coordinate values ​​of k face key points corresponding to the m-th frame target image;

[0171] For example, Figure 3 As shown, k facial key points may include: the center point of the left eye of the target object, the center point of the right eye of the target object, the center point of the nose tip of the target object, the center point of the left corner of the mouth of the target object, and the center point of the right corner of the mouth of the target object. For example, facial key point detection and facial quality detection may use a CNN deep learning model based on ResNet, and two models may be used to detect key points and facial quality respectively, or a single model may be used to detect key points and facial quality simultaneously.

[0172] Step 407: Obtain the coordinate values ​​of k facial key points corresponding to the first m-1 frames of target images;

[0173] For example, by executing steps 401 to 407, the coordinate values ​​of k facial key points corresponding to each of the m-frame target images can be obtained, including: the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image and the coordinate values ​​of the k facial key points corresponding to each of the first m-1 frames target images. Wherein, it is recorded as {Lx(n,i),Ly(n,i)}, n=1,…,m, i=1,…,k. For example, k=1,…,5, where n represents the frame number, the larger the frame number, the later the frame, and m represents the current frame. i represents the current key point, and k=1,…,5 respectively correspond to the center point of the left eye of the target object, the center point of the right eye of the target object, the center point of the nose tip of the target object, the center point of the left corner of the mouth of the target object, and the center point of the right corner of the mouth of the target object, a total of 5 facial key points.

[0174] Step 408: Based on the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image and the coordinate values ​​of the k facial key points corresponding to the first m-1 frames target image, determine the average coordinate value of the k facial key points corresponding to the m-th frame target image, as shown in the following formula:

[0175]

[0176] in, It represents the average x-coordinate of the average coordinates of the i-th face key point among the k face key points corresponding to the target image of the m-th frame. represents the average value of the y coordinate of the coordinates of the i-th face key point among the k face key points corresponding to the m-th frame target image, Lx(n,i) represents the x coordinate of the coordinates of the i-th face key point among the k face key points corresponding to the n-th frame target image, represents the sum of the x-coordinate value of the coordinates of the ith face key point among the k face key points corresponding to the m-th frame target image and the x-coordinate value of the coordinates of the ith face key point among the k face key points corresponding to the first m-1 frames target image, Ly(n,i) represents the y-coordinate value of the coordinates of the ith face key point among the k face key points corresponding to the n-th frame target image, It represents the sum of the initial y-coordinate value of the coordinates of the i-th face key point among the k face key points corresponding to the m-th frame target image and the y-coordinate value of the coordinates of the i-th face key point among the k face key points corresponding to the previous m-1 frames target image, n=1,…,m, i=1,…,k. For example, k=1,…,5. Here, when n=m, Lx(m,i) represents the initial x-coordinate value of the coordinates of the k face key points corresponding to the m-th frame target image, and Ly(m,i) represents the initial y-coordinate value of the coordinates of the k face key points corresponding to the m-th frame target image.

[0177] Step 409: Calculate the difference between the initial coordinate value of the k facial key points corresponding to the m-th frame target image and the average coordinate value of the k facial key points to obtain the coordinate deviation value of the k facial key points corresponding to the m-th frame target image, as shown in the following formula;

[0178]

[0179] Wherein, Lx(m,i) represents the initial value of the x-coordinate of the initial values ​​of the coordinates of the i-th face key point among the k face key points corresponding to the m-th frame target image, and Ly(m,i) represents the initial value of the y-coordinate of the initial values ​​of the coordinates of the i-th face key point among the k face key points corresponding to the m-th frame target image. It represents the average x-coordinate of the average coordinates of the i-th face key point among the k face key points corresponding to the target image of the m-th frame. represents the average value of the y coordinate of the coordinate average of the i-th face key point among the k face key points corresponding to the m-th frame target image, Dx(i) represents the x coordinate deviation value of the i-th face key point among the k face key points corresponding to the m-th frame target image (i.e., the difference between the x coordinate estimate value of the i-th face key point and the x coordinate average value), Dy(i) represents the y coordinate deviation value of the i-th face key point among the k face key points corresponding to the m-th frame target image (i.e., the difference between the y coordinate estimate value of the i-th face key point and the y coordinate average value), i = 1, ..., k. For example, k = 1, ..., 5

[0180] Step 410: obtaining the coordinate deviation value of the left eye key point from the coordinate deviation values ​​of the k facial key points corresponding to the m-th frame target image as the coordinate deviation value of the first facial key point, and obtaining the coordinate deviation value of the right eye key point as the coordinate deviation value of the second facial key point;

[0181] Step 411: Determine a first weight based on the coordinate deviation value of the first facial key point, and determine a second weight based on the coordinate deviation value of the second facial key point, as shown in the following formula:

[0182]

[0183] Among them, W(0) represents the first weight, Dx(0) represents the x-coordinate deviation value in the coordinate deviation value of the first facial key point (such as the x-coordinate deviation value of the left eye key point corresponding to the m-th frame target image), Dy(0) represents the y-coordinate deviation value in the coordinate deviation value of the first facial key point (such as the y-coordinate deviation value of the left eye key point corresponding to the m-th frame target image), W(1) represents the second weight, Dx(1) represents the x-coordinate deviation value in the coordinate deviation value of the second facial key point (such as the x-coordinate deviation value of the right eye key point corresponding to the m-th frame target image), and Dy(1) represents the y-coordinate deviation value in the coordinate deviation value of the second facial key point (such as the y-coordinate deviation value of the right eye key point corresponding to the m-th frame target image).

[0184] The coordinate deviation value of the first facial key point may include: an x-coordinate deviation value of the left eye key point corresponding to the m-th frame target image and a y-coordinate deviation value of the left eye key point corresponding to the m-th frame target image.

[0185] The coordinate deviation value of the second facial key point includes: the x-coordinate deviation value of the right eye key point corresponding to the m-th frame target image and the y-coordinate deviation value of the right eye key point corresponding to the m-th frame target image.

[0186] Step 412: Determine the x-coordinate values ​​in the coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the x-coordinate average value in the coordinate average values ​​of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the x-coordinate deviation value in the coordinate deviation value of the first facial key point, and the x-coordinate deviation value in the coordinate deviation value of the second facial key point; and determine the y-coordinate values ​​in the coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the y-coordinate average value in the coordinate average values ​​of the k facial key points corresponding to the m-th frame target image, the first weight and the second weight, the y-coordinate deviation value in the coordinate deviation value of the first facial key point, and the y-coordinate deviation value in the coordinate deviation value of the second facial key point; as shown in the following formula:

[0187]

[0188] Wherein, Lx(i)′ represents the x-coordinate value of the coordinate value of the i-th face key point among the k face key points corresponding to the m-th frame target image, and Ly(i)′ represents the y-coordinate value of the coordinate value of the i-th face key point among the k face key points corresponding to the m-th frame target image. It represents the average x-coordinate of the average coordinates of the i-th face key point among the k face key points corresponding to the target image of the m-th frame. represents the y-coordinate average value of the coordinate average values ​​of the i-th face key point among the k face key points corresponding to the m-th frame target image, W(0) represents the first weight, W(1) represents the second weight, Dx(0) represents the x-coordinate deviation value of the coordinate deviation value of the first face key point (such as the x-coordinate deviation value of the left eye key point corresponding to the m-th frame target image), Dy(0) represents the y-coordinate deviation value of the coordinate deviation value of the first face key point (such as the y-coordinate deviation value of the left eye key point corresponding to the m-th frame target image), Dx(1) represents the x-coordinate deviation value of the coordinate deviation value of the second face key point (such as the x-coordinate deviation value of the right eye key point corresponding to the m-th frame target image), Dy(1) represents the y-coordinate deviation value of the coordinate deviation value of the second face key point (such as the y-coordinate deviation value of the right eye key point corresponding to the m-th frame target image), i=1,...,k, such as k=1,...,5.

[0189] Step 413: using a human eye coordinate estimation algorithm, based on the coordinate values ​​of the k facial key points corresponding to the m-th frame target image, estimate the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image;

[0190] Exemplarily, the eye coordinate estimation algorithm may adopt an eye positioning algorithm based on pupil distance, or may adopt a PNP algorithm, etc. Here, the embodiments of the present disclosure do not limit this.

[0191] Step 414: obtaining the z coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to each of the first m-1 frames of target images;

[0192] Step 415: According to the estimated 3D coordinate value of the human eye of the target object corresponding to the m-th frame target image and the z coordinate value in the 3D coordinate value of the human eye of the target object corresponding to each of the first m-1 frames target image, the 3D coordinate value of the human eye of the target object corresponding to the m-th frame target image is determined, as shown in the following formula:

[0193]

[0194] Wherein, z′ represents the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, Represents the sum of the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object and the z coordinate values ​​in the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the first m-1 frames of target images, x′ represents the x coordinate value in the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-th frame of target image, z(m) represents the z coordinate estimated value of the human eye of the target object corresponding to the m-th frame of target image, x(m) represents the x coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame of target image, y′ represents the y coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame of target image, and y(m) represents the y coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame of target image. Here, when n is m, z(m) represents the estimated z coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image; when n is 1, ..., m-1 respectively, z(1), ..., z(m-1) respectively represent the z coordinate values ​​in the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the 1st frame target image, ..., the z coordinate values ​​in the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-1th frame target image.

[0195] In this way, after obtaining the three-dimensional coordinate values ​​of the human eyes of the target object corresponding to the m-th frame target image, the three-dimensional coordinate values ​​of the human eyes of the target object corresponding to the m-th frame target image can be stored and output.

[0196] Thus, the eye tracking method provided in the disclosed embodiment can reduce the noise of the deep learning model face key point detection by performing face image denoising filtering before face key point detection; by performing low-latency face key point denoising filtering on the obtained face key points after face key point detection, the face key point noise can be further reduced; by performing Z coordinate denoising filtering on the obtained three-dimensional eye coordinates before outputting the three-dimensional eye coordinates, the Z coordinate noise can be further reduced, and the X and Y direction angles are not affected by the Z coordinate error. Thus, by designing a low-latency denoising filtering algorithm in multiple links, the technical effect of greatly reducing the jitter of the eye tracking coordinates is achieved, and the timeliness of the eye tracking is maintained, thereby solving the technical problem of the jitter of the naked eye 3D picture display. When the eye tracking method provided in the disclosed embodiment is applied to a naked eye 3D display system, high-quality naked eye 3D image display can be achieved.

[0197] The description of the above application scenario embodiment is similar to the description of the above human eye tracking method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the application scenario embodiment of the present disclosure, those skilled in the art should refer to the description in the method embodiment of the present disclosure for understanding, and no further description is given here.

[0198] Based on the same inventive concept, an embodiment of the present disclosure also provides an eye tracking device, comprising: a processor and a memory storing a computer program that can be run on the processor, wherein the processor is configured to implement the steps of the eye tracking method in one or more of the above-mentioned exemplary embodiments when executing the program.

[0199] In some exemplary embodiments, Figure 5 As shown, the eye tracking device 50 may include: at least one processor 51; and at least one memory 52 and a bus 53 connected to the processor 51; wherein the processor 51 and the memory 52 communicate with each other via the bus 53; the processor 51 is configured to call program instructions in the memory 52 to execute the steps of the eye tracking method in one or more of the above-mentioned exemplary embodiments.

[0200] In some exemplary embodiments, the processor may be, for example, a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, application-specific integrated circuits, and other chips with computing functions. A general-purpose processor may be a microprocessor (MPU) or the processor may be any conventional processor, etc. Here, the embodiments of the present disclosure do not limit this.

[0201] In some exemplary embodiments, the memory may include a non-permanent memory and / or a non-volatile memory in a computer-readable storage medium, for example, the memory may be a random access memory (RAM) a read-only memory (ROM) or a flash memory (Flash RAM), or any combination thereof, which is configured to store a program executed by a processor, and the memory may include at least one memory chip. Here, the embodiments of the present disclosure do not limit this.

[0202] In some exemplary embodiments, the bus may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. Figure 5 In the figure, various buses are all labeled as bus 53. Here, the embodiment of the present disclosure does not limit this.

[0203] In the implementation process, the processing performed by the eye tracking device can be completed by the hardware integrated logic circuit in the processor or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0204] Based on the same inventive concept, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the steps of the human eye tracking method in one or more of the above-mentioned exemplary embodiments.

[0205] In some exemplary embodiments, the computer-readable storage medium may include, but is not limited to, ROM / RAM, a magnetic disk or an optical disk, etc. Here, the embodiments of the present disclosure do not limit this.

[0206] Based on the same inventive concept, the embodiment of the present disclosure also provides a naked eye three-dimensional display system, such as Figure 6 As shown, the naked eye 3D display system may include: a naked eye 3D display device 61, an image acquisition device 62, an eye tracking device 63 and a processing device 64, wherein:

[0207] The image acquisition device 62 is provided on the naked-eye 3D display device 61 and is configured to acquire a target image, where the target image is an image including a face of a target object, and the target object is an object viewing a display screen of the naked-eye 3D display device 61;

[0208] The eye tracking device 63 is configured to execute the steps of the eye tracking method in one or more of the above exemplary embodiments; the three-dimensional coordinate value of the eye of the target object corresponding to the m-th frame target image is sent to the processing device 64;

[0209] The processing device 64 is configured to process the image to be displayed according to the three-dimensional coordinate value of the human eye of the target object corresponding to the target image of the mth frame to obtain a naked-eye three-dimensional synthetic image; and send the naked-eye three-dimensional synthetic image to the naked-eye three-dimensional display device 61;

[0210] The naked-eye 3D display device 61 is configured to display a naked-eye 3D synthetic image.

[0211] In some exemplary embodiments, the image acquisition device may use a monocular RGB camera, or may use an infrared grayscale image camera, etc. Here, the embodiments of the present disclosure do not limit this.

[0212] In some exemplary embodiments, the eye tracking device 63 may be a general-purpose central processing unit (CPU) or other general-purpose processors, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. As an example, the processing device 64 may include one or more CPUs.

[0213] In some exemplary embodiments, the processing device 64 may be a general-purpose central processing unit (CPU) or other general-purpose processors, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. As an example, the processing device 64 may include one or more CPUs.

[0214] In an exemplary embodiment, at least two modules / devices among the naked-eye 3D display device 61, the image acquisition device 62, the human eye tracking device 63, and the processing device 64 may be integrated into one device, or may be respectively arranged on different devices. Here, the embodiments of the present disclosure do not limit this.

[0215] The description of the above eye tracking device embodiment, non-transient computer-readable storage medium embodiment, or naked-eye 3D display system embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the eye tracking device embodiment, non-transient computer-readable storage medium embodiment, or naked-eye 3D display system embodiment of the present disclosure, those skilled in the art should refer to the description of the method embodiment of the present disclosure for understanding. No further details will be given here.

[0216] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0217] Although the embodiments disclosed in the present disclosure are as above, the above contents are only embodiments adopted for facilitating the understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the field to which the present disclosure belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A human eye tracking method, characterized in that: The method comprises: Obtain the coordinate values ​​of k facial key points corresponding to the first m-1 frames of target images and the mth frame of target image, wherein the target image is an image including the face of the target object, m is a positive integer greater than 1, and k is a positive integer greater than 1; Based on the m-th frame target image, determine the initial coordinate values ​​of k facial key points corresponding to the m-th frame target image; Based on the coordinate values ​​of the k facial key points corresponding to the first m-1 frames of target images, the initial coordinate values ​​of the k facial key points corresponding to the m-th frame of target image are processed to obtain the coordinate values ​​of the k facial key points corresponding to the m-th frame of target image; According to the coordinate values ​​of k facial key points corresponding to the m-th frame target image, the three-dimensional coordinate values ​​of the human eyes of the target object corresponding to the m-th frame target image are determined.

2. The method according to claim 1, characterized in that The facial key points include at least: any one or more of the key points corresponding to the left eye of the target object, the key points corresponding to the right eye of the target object, the key points corresponding to the nose of the target object, the key points corresponding to the left corner of the target object's mouth, and the key points corresponding to the right corner of the target object's mouth.

3. The method according to claim 2, characterized in that The method processes the initial coordinate values ​​of the k facial key points corresponding to the m-th frame of the target image based on the coordinate values ​​of the k facial key points corresponding to the first m-1 frames of the target image to obtain the coordinate values ​​of the k facial key points corresponding to the m-th frame of the target image, including: Based on the initial coordinate values ​​of the k facial key points corresponding to the m-th frame target image and the coordinate values ​​of the k facial key points corresponding to the first m-1 frames target image, determine the average coordinate value of the k facial key points corresponding to the m-th frame target image; Calculate the difference between the initial coordinate value of the k facial key points corresponding to the m-th frame target image and the average coordinate value of the k facial key points to obtain the coordinate deviation value of the k facial key points corresponding to the m-th frame target image; Determine a first weight and a second weight based on a coordinate deviation value of a first face key point and a coordinate deviation value of a second face key point among coordinate deviation values ​​of k face key points corresponding to the target image of the mth frame; Determine the coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the average coordinate value of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the coordinate deviation value of the first facial key points, and the coordinate deviation value of the second facial key points.

4. The method according to claim 3, characterized in that The method of determining the first weight and the second weight based on the coordinate deviation value of the first face key point and the coordinate deviation value of the second face key point among the coordinate deviation values ​​of the k face key points corresponding to the m-th frame target image comprises: From the coordinate deviation values ​​of the k facial key points corresponding to the m-th frame target image, obtain the coordinate deviation value of the left eye key point as the coordinate deviation value of the first facial key point, and obtain the coordinate deviation value of the right eye key point as the coordinate deviation value of the second facial key point; Based on the coordinate deviation value of the first facial key point, according to the formula Determine the first weight; wherein W(0) represents the first weight, Dx(0) represents the x-coordinate deviation value in the coordinate deviation values ​​of the first facial key point, and Dy(0) represents the y-coordinate deviation value in the coordinate deviation values ​​of the first facial key point; Based on the coordinate deviation value of the second facial key point, according to the formula Determine the second weight; wherein W(1) represents the second weight, Dx(1) represents the x-coordinate deviation value in the coordinate deviation value of the second facial key point, and Dy(1) represents the y-coordinate deviation value in the coordinate deviation value of the second facial key point.

5. The method according to claim 3 or 4, characterized in that: The step of determining the coordinate values ​​of the k facial key points corresponding to the m-th frame target image based on the average coordinate values ​​of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the coordinate deviation value of the first facial key point, and the coordinate deviation value of the second facial key point comprises: Based on the x-coordinate average value of the coordinate average values ​​of the k facial key points corresponding to the m-th frame target image, the first weight, the second weight, the x-coordinate deviation value of the coordinate deviation value of the first facial key point, and the x-coordinate deviation value of the coordinate deviation value of the second facial key point, according to the formula Determine the x coordinate value of the coordinate values ​​of the k facial key points corresponding to the m-th frame target image; Based on the y-coordinate average value of the coordinate average values ​​of the k facial key points corresponding to the m-th frame target image, the first weight and the second weight, the y-coordinate deviation value of the coordinate deviation value of the first facial key point and the y-coordinate deviation value of the coordinate deviation value of the second facial key point, according to the formula Determine the y coordinate value of the coordinate values ​​of the k facial key points corresponding to the m-th frame target image; Wherein, Lx(i)′ represents the x-coordinate value of the coordinate value of the i-th face key point among the k face key points corresponding to the m-th frame target image, and Ly(i)′ represents the y-coordinate value of the coordinate value of the i-th face key point among the k face key points corresponding to the m-th frame target image. It represents the average x-coordinate of the average coordinates of the i-th face key point among the k face key points corresponding to the target image of the m-th frame. represents the y-coordinate average value of the coordinate average values ​​of the i-th face key point among the k face key points corresponding to the m-th frame target image, W(0) represents the first weight, W(1) represents the second weight, Dx(0) represents the x-coordinate deviation value in the coordinate deviation values ​​of the first face key point, Dy(0) represents the y-coordinate deviation value in the coordinate deviation values ​​of the first face key point, Dx(1) represents the x-coordinate deviation value in the coordinate deviation values ​​of the second face key point, Dy(1) represents the y-coordinate deviation value in the coordinate deviation values ​​of the second face key point, i=1,...,k.

6. The method according to any one of claims 1 to 4, characterized in that The step of determining the three-dimensional coordinate values ​​of the eyes of the target object corresponding to the m-th frame target image according to the coordinate values ​​of k facial key points corresponding to the m-th frame target image comprises: Using a human eye coordinate estimation algorithm, based on the coordinate values ​​of k facial key points corresponding to the m-th frame target image, estimate the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image; Obtaining the z coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to each of the first m-1 frames of target images; The three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image is determined based on the estimated three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image and the z coordinate value in the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the first m-1 frames of target images.

7. The method according to claim 6, characterized in that The method of determining the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image according to the three-dimensional coordinate estimation value of the human eye of the target object corresponding to the m-th frame target image and the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to each of the first m-1 frames target images comprises: Based on the z coordinate estimated value in the three-dimensional coordinate estimated value of the human eye of the target object corresponding to the m-th frame target image and the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the previous m-1 frames of target images, according to the formula Calculate the z coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image; Based on the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, the z coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and the x coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, according to the formula Determine an x ​​coordinate value in the three-dimensional coordinate values ​​of the human eye of the target object corresponding to the m-th frame target image; Based on the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, the z coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and the y coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, according to the formula Determine the y coordinate value of the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image; Wherein, z′ represents the z coordinate value in the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image, represents the sum of the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image and the z coordinate values ​​in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the previous m-1 frames of target images, x′ represents the x coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, z(m) represents the z coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, x(m) represents the x coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, y′ represents the y coordinate value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image, and y(m) represents the y coordinate estimated value in the three-dimensional coordinate estimated values ​​of the human eye of the target object corresponding to the m-th frame target image.

8. The method according to any one of claims 1 to 4, characterized in that The step of determining initial coordinate values ​​of k facial key points corresponding to the m-th frame target image based on the m-th frame target image includes: Segment the image region where the face of the target object is located from the m-th frame target image to obtain a target face region map corresponding to the m-th frame target image; Performing denoising on the target face region map corresponding to the target image of the m-th frame to obtain a denoised target face region map corresponding to the target image of the m-th frame; Perform facial key point detection on the denoised target face area map corresponding to the m-th frame target image to obtain the initial coordinate values ​​of k facial key points corresponding to the m-th frame target image.

9. The method according to claim 8, characterized in that The step of segmenting the image region where the face of the target object is located from the m-th frame target image to obtain a target face region map corresponding to the m-th frame target image includes: Get the face detection frame corresponding to the target image of the mth frame; Based on the face detection frame corresponding to the m-th frame target image, the image area where the face of the target object is located is segmented from the m-th frame target image to obtain a target face area map corresponding to the m-th frame target image.

10. The method according to claim 9, characterized in that The step of obtaining a face detection frame corresponding to the target image of the mth frame includes: Perform face detection on the m-th target image to obtain a face detection frame corresponding to the m-th target image; or obtain a face detection frame corresponding to the m-1-th target image as the face detection frame corresponding to the m-th target image.

11. The method according to claim 8, characterized in that The denoising process is performed on the target face region map corresponding to the m-th frame target image to obtain the denoised target face region map corresponding to the m-th frame target image, including: Determine whether the resolution of the target face region map corresponding to the target image of the mth frame meets the preset image resolution; If not satisfied, the target face region map corresponding to the target image of the m-th frame is normalized at the preset image resolution to obtain the normalized target face region map corresponding to the target image of the m-th frame; the normalized target face region map corresponding to the target image of the m-th frame is denoised to obtain the denoised target face region map corresponding to the target image of the m-th frame; Alternatively, if the condition is satisfied, a denoising process is performed on the target face region map corresponding to the m-th frame target image to obtain a denoised target face region map corresponding to the m-th frame target image.

12. The method according to claim 8, characterized in that The denoising process is performed on the target face region map corresponding to the m-th frame target image to obtain the denoised target face region map corresponding to the m-th frame target image, including: A denoising process is performed on the target face region map corresponding to the m-th frame target image by using any one of a Gaussian filter and a mean filter to obtain a denoised target face region map corresponding to the m-th frame target image.

13. An eye tracking device, comprising: A processor and a memory storing a computer program executable on the processor, wherein the processor is configured to implement the steps of the human eye tracking method according to any one of claims 1 to 12 when executing the program.

14. A non-transitory computer-readable storage medium storing computer-executable instructions, wherein: The computer executable instructions are configured to execute the steps of the human eye tracking method according to any one of claims 1 to 12.

15. A naked eye three-dimensional display system, characterized in that: The system comprises: a naked eye three-dimensional display device, an image acquisition device, an eye tracking device and a processing device, wherein: The image acquisition device is arranged on the naked-eye 3D display device, and is arranged to acquire a target image, wherein the target image is an image including a face of a target object, and the target object is an object viewing a display screen of the naked-eye 3D display device; The eye tracking device is configured to perform the steps of the eye tracking method according to any one of claims 1 to 12; The processing device is configured to process the image to be displayed according to the three-dimensional coordinate value of the human eye of the target object corresponding to the m-th frame target image output by the human eye tracking device to obtain a naked-eye three-dimensional synthetic image; and send the naked-eye three-dimensional synthetic image to the naked-eye three-dimensional display device; The naked-eye 3D display device is configured to display the naked-eye 3D synthetic image.

Images