Line of sight estimation method and apparatus

Abstract

The method comprises the following steps: obtaining a target image, the target image presenting a face image of a to-be-tested user; performing face feature detection on the target image to obtain face feature information of the to-be-tested user; inputting the target image and the face feature information into a machine learning model to perform gaze estimation, and obtaining a gaze estimation result output by the machine learning model; the machine learning model has been trained based on sample images, sample face features and sample gaze information, the sample face features are face features presented in the sample images, and the sample gaze information is used to indicate a gaze presented in the sample images; and determining a gaze point position of the to-be-tested user in the target image according to the gaze estimation result. The model obtained through pre-training is used to process the image of the user to obtain the gaze estimation result, so that the operation complexity is reduced, and the gaze estimation efficiency and accuracy are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer vision, and particularly relate to a gaze estimation method and device. BACKGROUND

[0002] With the rapid development of science and technology, the application field of computer vision is more and more extensive, which can be applied not only in the field of human-computer interaction. For example, gaze estimation technology refers to estimating the gaze direction and viewpoint position of a user by collecting the image information of the user's eyes, so as to realize the control of a computer and the like. In recent years, with the development of new generation information technologies such as big data and artificial intelligence, higher requirements have been put forward for gaze estimation technology.

[0003] In the related art, the method of estimating the gaze direction of the eyes and then determining the gaze point position is generally to determine the gaze point position information of the eyeball by using the pupil-corneal reflection method, that is, to form multiple light spots in the eyeball of the user by using multiple light sources, to obtain the image of the user's eyes, and then to obtain the gaze point position information according to the light spots corresponding to all the light sources detected on the image of the eyes.

[0004] However, the existing method needs to use additional light sources and sensor devices, and the operation is complex, and when the user's eyeball movement amplitude is large or the user's head movement amplitude is large, part of the light sources cannot form light spots in the eyes, thereby causing the problem of being unable to perform gaze estimation or inaccurate gaze estimation. SUMMARY

[0005] Embodiments of the present disclosure provide a gaze estimation method and device to overcome the technical problems of inaccurate gaze estimation and complex device operation in the prior art.

[0006] In a first aspect, embodiments of the present disclosure provide a gaze estimation method, comprising:

[0007] obtaining a target image, the target image presenting a face image of a user to be measured;

[0008] performing face feature detection on the target image to obtain face feature information of the user to be measured;

[0009] inputting the target image and the face feature information into a machine learning model to perform gaze estimation, and obtaining a gaze estimation result output by the machine learning model; the machine learning model has been trained based on sample images, sample face features, and sample gaze information, the sample face features are face features presented in the sample images, and the sample gaze information is used to indicate the gaze presented in the sample images;

[0010] According to the line-of-sight estimation result, a viewpoint position of the to-be-tested user in the target image is determined.

[0011] In a second aspect, the embodiments of the present disclosure provide a line-of-sight estimation device, comprising:

[0012] An image acquisition module is configured to acquire a target image, wherein the target image presents a face image of a to-be-tested user.

[0013] A feature detection module is configured to perform face feature detection on the target image to obtain face feature information of the to-be-tested user.

[0014] A line-of-sight estimation module is configured to input the target image and the face feature information into a machine learning model to perform line-of-sight estimation, and obtain a line-of-sight estimation result output by the machine learning model, wherein the machine learning model has been trained based on sample images, sample face features, and sample line-of-sight information, the sample face features are face features presented in the sample images, and the sample line-of-sight information is used to indicate line-of-sight presented in the sample images.

[0015] A viewpoint determination module is configured to determine a viewpoint position of the to-be-tested user in the target image according to the line-of-sight estimation result.

[0016] In a third aspect, the embodiments of the present disclosure provide an electronic device, comprising at least one processor and a memory.

[0017] The memory stores computer-executable instructions.

[0018] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the line-of-sight estimation method as described in the first aspect and various possible designs of the first aspect.

[0019] In a fourth aspect, the embodiments of the present disclosure provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the line-of-sight estimation method as described in the first aspect and various possible designs of the first aspect is implemented.

[0020] In a fifth aspect, the embodiments of the present disclosure provide a computer program product, comprising a computer program, and when a processor executes the computer program, the line-of-sight estimation method as described in the first aspect and various possible designs of the first aspect is implemented.

[0021] The method provided by the embodiment of the present disclosure comprises the following steps: obtaining a target image of a user to be measured, the target image presenting a face image of the user to be measured; performing face feature detection on the target image to obtain face feature information of the user to be measured; inputting the target image and the face feature information into a machine learning model; the machine learning model has been trained based on sample images, sample face features and sample line-of-sight information, the sample face features are face features presented in the sample images, and the sample line-of-sight information is used to indicate the line of sight presented in the sample images; therefore, the line-of-sight estimation result of the user to be measured can be directly obtained through the machine learning model; and finally, the line-of-sight estimation result output by the machine learning model is used to determine the point-of-view position of the user to be measured in the target image. The line-of-sight estimation result of the user to be measured is obtained by processing the image of the user through the machine learning model obtained through pre-training, and the whole process does not need to additionally increase light sources, sensors and other devices, thereby reducing the operation complexity and greatly improving the line-of-sight estimation efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without any creative labor.

[0023] Figure 1 Application scenario of the line-of-sight estimation method provided by the embodiment of the present disclosure Figure 1 ;

[0024] Figure 2 Flowchart of the line-of-sight estimation method provided by the embodiment of the present disclosure Figure 1 ;

[0025] Figure 3 Application scenario of the line-of-sight estimation method provided by the embodiment of the present disclosure Figure 2 ;

[0026] Figure 4 Flowchart of the line-of-sight estimation method provided by the embodiment of the present disclosure Figure 2 ;

[0027] Figure 5 Application scenario of the line-of-sight estimation method provided by the embodiment of the present disclosure Figure 3 ;

[0028] Figure 6 Application scenario of the line-of-sight estimation method provided by the embodiment of the present disclosure Figure 4 ;

[0029] Figure 7 A flowchart of a training method of a face detection model provided by an embodiment of the present disclosure is shown in FIG. 1.

[0030] Figure 8 A flowchart of a training method of a machine learning model provided by an embodiment of the present disclosure is shown in FIG. 2.

[0031] Figure 9 A structural diagram of a gaze estimation device provided by an embodiment of the present disclosure is shown in FIG. 3.

[0032] Figure 10 A hardware structural diagram of an electronic device provided by an embodiment of the present disclosure is shown in FIG. 4. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some but not all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.

[0034] In related technologies, a method for determining a gaze point position by estimating an eye gaze direction generally includes: determining eye gaze point position information by using a pupil-corneal reflection (P-CR) method, that is, forming multiple light spots in a user's eyeball by using multiple light sources, and obtaining an image of the user's eye, then detecting light spots corresponding to all light sources on the image of the user's eye to obtain a P-CR vector, when the user's head and eyeball are stationary, the P-CR vector has a one-to-one corresponding relationship with a gaze point vector on a computer screen that the user faces, and the P-CR vector is mapped to the gaze point vector by a mapping function, that is, the gaze point position information is obtained by implementing gaze estimation. However, the existing method needs to use additional light sources and sensor devices, and the operation is complex. When the user's eyeball has a large motion amplitude or the user's head has a large motion amplitude, some light sources cannot form light spots in the eye. In the case that no light spot is formed or only part of the light spot is formed in the eye, an accurate P-CR vector cannot be obtained, and therefore gaze estimation cannot be performed or the gaze estimation is inaccurate. If a complete P-CR vector is needed when the user's eyeball has a large motion amplitude or the user's head has a large motion amplitude, a technician needs to manually adjust the position of the light source so that all light sources can form light spots in the eye. In general, the user's head and eyeball are in a state of constant motion, and therefore the position of the light source needs to be adjusted frequently when the existing method is used for gaze estimation, which greatly increases the complexity of device operation, reduces the efficiency of gaze estimation, and also reduces the accuracy of gaze estimation.

[0035] To solve the problem, the technical concept of the embodiment of the present disclosure mainly includes: first, two models are trained by a deep learning method, which are a face detection model and a machine learning model. The face detection model is used for face detection on an image containing user facial information to extract user facial feature information, such as user facial rectangular region information, face key point data, visibility of each face key point, face horizontal rotation angle, face pitch angle, face rotation angle, and eye distance. The machine learning model is used for gaze estimation based on user facial feature information to obtain user left eye position information, right eye position information, and average gaze direction of both eyes. When gaze estimation of a user is needed, an image containing user facial information is collected by an image collection device, and then the image is input into the face detection model to directly obtain user facial feature information output by the face detection model. Then, the user facial feature information is input into the machine learning model as an input quantity to directly obtain gaze estimation results output by the machine learning model. Finally, the gaze point position of the user on a target plane is calculated according to the gaze estimation results, and then it is determined whether the user's gaze falls within a preset region in the target plane according to the gaze point position. The embodiment of the present disclosure processes the image of the user by two pre-trained models to obtain the gaze estimation results. The whole process does not need to add extra light sources, sensors, and other devices, reduces the operation complexity, and greatly improves the gaze estimation efficiency and accuracy.

[0036] Figure 1 Application scenario of the gaze estimation method provided by the embodiment of the present disclosure Figure 1 .

[0037] As shown in Figure 1 , the basic architecture of the application scenario provided by the embodiment mainly includes: an image collection device 101 and a server 102. The image collection device collects an image containing user facial information and sends the collected image to the server. The server processes the image to obtain the gaze estimation results of the user.

[0038] It should be noted that the image collection device in the embodiment can be a camera or other mobile devices with camera function, such as mobile phones, tablets, smart wearable devices, displays, etc. The embodiment is not limited specifically.

[0039] Figure 2 Flowchart of the gaze estimation method provided by the embodiment of the present disclosure Figure 1 The execution subject of the method provided in the embodiment can be the server in the embodiment shown in Figure 2 .

[0040] As shown in Figure 3 , the method provided in the embodiment can include the following steps.

[0041] S201, acquire a target image, the target image presenting a face image of a to-be-tested user.

[0042] In this step, an image capturing device such as a camera captures an image containing face information of the to-be-tested user to obtain the target image, and then actively sends the target image to the server. Alternatively, after the image capturing device such as a camera captures an image of the to-be-tested user, the image is stored in a local memory, and then the server sends a corresponding acquisition instruction to the image capturing device to acquire the target image of the to-be-tested user.

[0043] In some embodiments, as shown in FIG. 2A, the camera is located in a first direction of the to-be-tested user, the first direction being a direction in which the head of the to-be-tested user faces when in a normal state (i.e., the L direction indicated in FIG. 2A), so that the camera can capture an image containing the complete head and face of the to-be-tested user. Figure 3 Figure 3 In some embodiments, as shown in FIG. 2A, the camera is located in a first direction of the to-be-tested user, the first direction being a direction in which the head of the to-be-tested user faces when in a normal state (i.e., the L direction indicated in FIG. 2A), so that the camera can capture an image containing the complete head and face of the to-be-tested user.

[0044] It should be noted that the target image in this embodiment can be an image captured by the camera, or a frame of video recorded by the camera.

[0045] S202, perform face feature detection on the target image to obtain face feature information of the to-be-tested user.

[0046] In some embodiments, a face detection model can be used to detect the face feature information of the to-be-tested user in the target image. The target image of the to-be-tested user is input into the face detection model as an input quantity, and the face detection model performs face feature detection on the target image to obtain the face feature information of the to-be-tested user output by the face detection model.

[0047] In some embodiments, the face feature information includes face rectangular region information, face key point data, visibility of each face key point, face horizontal rotation angle, face pitch angle, face rotation angle, and interocular distance of the to-be-tested user.

[0048] S203, input the target image and the face feature information into a machine learning model to perform gaze estimation, and obtain a gaze estimation result output by the machine learning model.

[0049] ​The machine learning model has been trained based on sample images, sample facial features, and sample gaze information. Therefore, the machine learning model can estimate the gaze of the user under test based on the input target image and facial feature information. The sample facial features are the facial features presented in the sample image, and the sample gaze information is used to indicate the gaze presented in the sample image. It is used to estimate the gaze based on the facial feature information output by the face detection model. The detailed training process of the machine learning model will be described in detail in the relevant embodiments below, and will not be specifically described here.

[0050] Specifically, the facial rectangular region information, facial key point data, visibility of each facial key point, facial horizontal angle, facial pitch angle, facial rotation angle, and interocular distance of the user under test are input into the machine learning model as inputs to obtain the gaze estimation results such as the left eye position information, right eye position information, and average gaze direction of both eyes output by the machine learning model.

[0051] S204, Based on the gaze estimation result, determine the viewpoint position of the user to be tested in the target image.

[0052] In this step, the gaze estimation results include: the position information of the left and right eyes in three-dimensional space and the average gaze direction of both eyes. The three-dimensional space is a three-dimensional space established with the location of the target point as the origin.

[0053] It should be noted that the target point can be any point in the user's space.

[0054] For example, such as Figure 3 As shown, target plane 31 is located in the first direction (i.e., the L direction) of the user under test. A three-dimensional coordinate system is established with target point O as the origin. The average gaze direction of the user's eyes is represented as follows: Figure 4 The vector representing the average gaze direction of both eyes, as shown in the figure, output by the machine learning model in step S203, is denoted as G(g). x g y g z ), where g x Let g be the angle between the line of sight and the X-axis in the three-dimensional coordinate system. y Let g be the angle between the line of sight and the Y-axis in the three-dimensional coordinate system. z Let P be the angle between the line of sight and the Z-axis in the 3D coordinate system. The viewpoint is denoted as point P, and its coordinates are P(p). x p y p z Assume that in this embodiment, the left eye position information of the user being tested in the gaze estimation result output by the machine learning model is E. L (x l y l, z l ), the right eye position information is E R (x r , y r , z r ), then the binocular intermediate point coordinates are determined according to the left eye position information and the right eye position information, and the binocular intermediate point coordinates are taken as the eye coordinates of the to-be-measured user, denoted as E(e x , e y , e z ), wherein, According to the eye coordinates E(e x , e y , e z ) and the gaze vector G(g x , g y , g z ), the coordinates of the viewpoint P can be calculated according to the straight line equation in the space, wherein the straight line equation is:

[0055] In this embodiment, by obtaining a target image of the to-be-measured user, the target image presents a face image of the to-be-measured user, face feature detection is performed on the target image to obtain face feature information of the to-be-measured user; then the target image and the face feature information are input into a machine learning model, and since the machine learning model has been trained based on sample images, sample face features and sample gaze information, and the sample face features are face features presented in the sample images, the sample gaze information is used to indicate the gaze presented in the sample images, therefore, the machine learning model is used, therefore, the gaze estimation result of the to-be-measured user can be directly obtained through the machine learning model, and finally the viewpoint position of the to-be-measured user in the target image is determined according to the gaze estimation result output by the machine learning model. The machine learning model pre-trained in the present embodiment is used to process the image of the user to obtain the gaze estimation result, and the whole process does not need to additionally increase light sources, sensors and other devices, thereby reducing the operation complexity and greatly improving the gaze estimation efficiency and accuracy.

[0056] Figure 2 The flowchart of the gaze estimation method provided by the present embodiment Figure 2 , the present embodiment further describes the gaze estimation method based on the embodiment shown in Figure 4 .

[0057] As shown in Figure 3 , the method provided by the present embodiment comprises the following steps.

[0058] S401, obtaining a target image, the target image presents a face image of a to-be-measured user.

[0059] In this step, the camera or other image acquisition device acquires an image containing the facial information of the user to be tested to obtain the target image, and then actively sends the target image to the server. Alternatively, after the camera or other image acquisition device acquires the image of the user to be tested, the image is stored in the local memory, and then the server sends a corresponding acquisition instruction to the image acquisition device to acquire the target image of the user to be tested.

[0060] In some embodiments, as shown in FIG. 2, the camera is located in a first direction of the user to be tested, which is the direction facing the user to be tested when the head of the user to be tested is in a normal state (i.e., the L direction as indicated in FIG. 2), so that the camera can acquire an image containing the complete head and face of the user to be tested. Figure 3 Figure 5 In some embodiments, as shown in FIG. 2, the camera is located in a first direction of the user to be tested, which is the direction facing the user to be tested when the head of the user to be tested is in a normal state (i.e., the L direction as indicated in FIG. 2), so that the camera can acquire an image containing the complete head and face of the user to be tested.

[0061] It should be noted that the target image in this embodiment can be an image captured by the camera or a frame of video recorded by the camera.

[0062] S402, acquiring image information of the target image, the image information including width and height of the target image, space occupied by each row of pixels in the target image, color space of the target image, and direction information of the target image.

[0063] Specifically, after the server acquires the target image, the image information is extracted, including the width width, the height height, the space stride occupied by each row of pixels in the target image, the color space color space of the target image, and the direction information orientation of the target image.

[0064] The width and height of the target image are represented by pixels.

[0065] Specifically, the space stride occupied by each row of pixels in the target image is determined according to the row of pixels and the space occupied by each pixel, i.e., stride = number of bytes occupied by each pixel * width, if stride is not a multiple of 4, then stride = stride * width + (4 - stride mod 4). For example, there are 11 pixels in a row (i.e., width = 11), and for a 32-bit image (i.e., each pixel occupies 4 bytes), stride = 11 * 4 = 44. For example, there are 11 pixels in a row (i.e., width = 11), and for a 24-bit image (i.e., each pixel occupies 3 bytes), stride = 11 * 3 = 33, then stride is not a multiple of 4, in order to ensure byte alignment, stride = 33 + 3 = 36.

[0066] ​The orientation information of the target image is an angle of clockwise rotation or an angle of counterclockwise rotation of the target image.

[0067] S403, input the image information of the target image into the pre-trained face detection model to obtain face feature information of the to-be-tested user output by the face detection model.

[0068] The face feature information includes face rectangular region information, face key point data, visibility of each face key point, face horizontal rotation angle, face pitch angle, face rotation angle, and eye distance of the to-be-tested user.

[0069] Specifically, the target image and the corresponding image information are input into the face detection model as input quantities, the face detection model performs face feature detection, and then the face feature information is output.

[0070] For example, the face detection result output by the face detection model includes the following information:

[0071]

[0072]

[0073] The bef_ai_rect rect represents the face rectangular region of the to-be-tested user in the target image, the score is the confidence of the target image, the bef_ai_fpoint points_array

[106] is the face key point data, the visibility_array

[106] represents the visibility of each key point, the yaw is the face horizontal rotation angle, the pitch is the face pitch angle, the roll is the face rotation angle, the eye_dist is the eye distance, and the ID is the face ID.

[0074] Specifically, in this embodiment, the face key point data is a face 106 key point array, including eyebrow, eye, nose, mouth, ear, face contour, etc. visibility_array

[106] is the visibility of each face key point, which is used to indicate whether the key point is blocked. If the key point is not blocked, the corresponding data of the key point is 1.0, and if it is blocked, the corresponding data of the key point is 0.0. Yaw is the horizontal angle of the face, and the data is the real measurement of left negative and right positive, that is, assuming that the face of the user in the target image is turned to the left, yaw is negative, and the value of yaw is the angle of the face turned to the left (i.e. the angle between the direction of the face turned to the left and the direction of the face facing straight in the natural state of the user); if the face of the user in the target image is turned to the right, yaw is positive, and the value of yaw is the angle of the face turned to the right. Pitch is the pitch angle of the face, and the data is the real measurement of up negative and down positive. If the face of the user in the target image is looking up, pitch is negative, and the value of pitch is the angle of the face looking up (i.e. the angle between the direction of the face looking up and the direction of the face facing straight in the natural state of the user); if the face of the user in the target image is looking down, pitch is positive, and the value of pitch is the angle of the face looking down. Roll is the rotation angle of the face, and the data is the real measurement of left negative and right positive. If the head of the user in the target image is rotated to the left, roll is negative, and the value of pitch is the angle between the direction of the head rotated to the left and the vertical direction; if the head of the user in the target image is rotated to the right, roll is positive, and the value of pitch is the angle between the direction of the head rotated to the right and the vertical direction.

[0075] It should be noted that ID is the unique identifier of the user, and each detected face has a unique faceID. After the face tracking is lost and then detected again, a new faceID will be generated.

[0076] S404, inputting the target image and the face feature information into a pre-trained machine learning model to obtain a gaze estimation result output by the machine learning model.

[0077] Specifically, the target image and the face feature information are input into the machine learning model as input quantities to obtain a gaze estimation result output by the machine learning model, the gaze estimation result including position information of left eye and right eye in a three-dimensional space and an average gaze direction of both eyes, the three-dimensional space being a three-dimensional space established with a position of a target point as an origin.

[0078] For example, the gaze estimation result output by the machine learning model is the following information:

[0079]

[0080] wherein, leyepos[3] and reye_pos[3] are the positions of the left eye and the right eye in the three-dimensional space, leyegaze[3], reye_gaze[3] and mid_gaze[3] represent the gaze direction of the left eye, the gaze direction of the right eye and the average gaze direction of the two eyes, respectively.

[0081] S405, judging whether the line of sight of the user to be measured falls in the preset region on the target plane according to the line of sight estimation result.

[0082] Specifically, after obtaining the gaze direction of the user to be measured and the positions of the left eye and the right eye in the three-dimensional space, the position information of the view point of the line of sight of the user to be measured on the target plane can be determined, and according to the position information of the view point and the position information of the preset region in the three-dimensional space, it can be judged whether the view point falls in the preset region.

[0083] It should be noted that the target point (O point) can be any point in the space where the user to be measured is located. In the embodiment, in order to facilitate calculation, the target point (O point) can be a point on the target plane. The straight line where the horizontal coordinate axis (X axis) and the vertical coordinate (Y axis) of the three-dimensional space are located is any two mutually perpendicular straight lines on the target plane passing through the target point, and the straight line where the vertical coordinate axis (Z axis) of the three-dimensional space is located passes through the target point and is perpendicular to the straight line of the target plane.

[0084] For example, a three-dimensional coordinate system established with a point on the target plane as the origin is shown in Figure 5 The target plane is located in the first direction (i.e. L direction) of the user to be measured, and the average gaze direction of the two eyes of the user to be measured is represented as G direction shown in Figure 5 The view point of the line of sight on the target plane is denoted as P point.

[0085] In one or more possible cases of the embodiment, the determination of the position information of the view point of the line of sight of the user to be measured on the target plane according to the line of sight estimation result comprises: determining the position information of the intermediate point between the left eye and the right eye according to the position information of the left eye and the right eye in the three-dimensional space; determining the position information of the view point of the line of sight of the user to be measured on the target plane according to the position information of the intermediate point, the average gaze direction of the two eyes, the position information of the target plane in the three-dimensional space and the first formula.

[0086] Specifically, it is assumed that the position information of the left eye of the user to be measured in the line of sight estimation result output by the machine learning model in the embodiment is E L (x l , y l , z l ), and the position information of the right eye is E R (xr , y r , z r ), then the binocular midpoint coordinates are determined according to the left eye position information and the right eye position information, and the binocular midpoint coordinates are taken as the eye coordinates of the user to be measured, denoted as E(e x , e y , e z ), wherein, According to the binocular midpoint coordinates E(e x , e y , e z ), the line-of-sight vector G(g x , g y , g z ), the coordinates of the viewpoint P can be calculated according to the first formula, wherein the first formula is: wherein e x , e y and e z represent the horizontal coordinate, the vertical coordinate and the vertical coordinate of the midpoint in the three-dimensional space, respectively, g x , g y and g z represent the included angle between the average line-of-sight direction of the two eyes and the horizontal coordinate axis, the vertical coordinate axis and the vertical coordinate axis in the three-dimensional space, respectively, p x , p y and p z represent the horizontal coordinate, the vertical coordinate and the vertical coordinate of the viewpoint in the three-dimensional space, respectively.

[0087] Further, since the target point (O point) is a point on the target plane, and the straight lines on which the horizontal coordinate axis and the vertical coordinate axis of the three-dimensional space are located are any two mutually perpendicular straight lines on the target plane passing through the target point, and the straight line on which the vertical coordinate axis of the three-dimensional space is located passes through the target point and is perpendicular to the target plane; therefore, the position of the target plane can be represented as Z = 0, so the vertical coordinate p z = 0 of the viewpoint of the line-of-sight of the user to be measured on the target plane, and the first formula is:

[0088] In one or more possible cases of the present embodiment, after the viewpoint position coordinates are determined, it can be further judged whether the viewpoint falls within a preset region on the target plane, specifically, according to the viewpoint position information of the line-of-sight on the target plane, whether the viewpoint is within the preset region is judged, including:

[0089] When the horizontal, vertical, and ordinate coordinates of the viewpoint satisfy preset conditions, the viewpoint is determined to be within a preset region. The preset conditions include: the horizontal coordinate of the viewpoint is within a first range, the vertical coordinate of the viewpoint is within a second range, and the vertical coordinate of the viewpoint is within a third range. The starting and ending values ​​of the first, second, and third ranges are determined based on the coordinates of each edge point of the preset region. The starting and ending values ​​of the first range are the minimum and maximum values ​​of the horizontal coordinates of each edge point of the preset region, respectively; the starting and ending values ​​of the second range are the minimum and maximum values ​​of the vertical coordinates of each edge point of the preset region, respectively; and the starting and ending values ​​of the third range are the minimum and maximum values ​​of the vertical coordinates of each edge point of the preset region, respectively.

[0090] For example, such as Figure 6 As shown, the target area is a rectangular region enclosed by dashed lines in the target plane. The coordinates of the four vertices of this rectangular region are (x1, y1, 0), (x2, y1, 0), (x1, y2, 0), and (x2, y2, 0). The first range of the x-coordinate of the preset region is [x1, x2], the second range of the y-coordinate is [y1, y2], and the third range of the vertical coordinate is 0. Therefore, the x-coordinate p of the viewpoint position P is determined. x Does it belong to the first range [x1, x2]? (Using the ordinate p) y If the viewpoint belongs to the second range [y1, y2], then the viewpoint is determined to be within the preset area; otherwise, the viewpoint is determined to be outside the preset area.

[0091] In this embodiment, the user's image is processed by two pre-trained models to obtain the gaze estimation result. The entire process does not require additional light sources, sensors, or other equipment, which reduces the complexity of operation and greatly improves the efficiency and accuracy of gaze estimation.

[0092] It should be noted that the line-of-sight estimation method provided in this disclosure can be used in various application scenarios, such as online intelligent education scenarios and artificial intelligence control scenarios. The following will take the implementation of this solution in a specific application scenario as an example for detailed explanation.

[0093] Figure 4 Application scenarios of the line-of-sight estimation method provided in the embodiments of this disclosure Figure 6 The application scenario provided in this embodiment is an online education scenario. Users learn online by watching videos, texts and other content played on the display screen 61. The camera 62 located on one side of the display screen captures the user's target image in real time and sends the captured target image to the server. The server estimates the user's gaze based on the target image and determines whether the user's gaze falls on the display screen.

[0094] It can be understood that the camera can be a camera separately arranged on the display screen, and the position of the camera can be any position on one side of the display screen. In the embodiment, in order to ensure that the user face information contained in the target image collected is complete, as shown in Figure 6 , the camera 62 is placed directly above the display screen 61, so that the user's face is directly opposite the camera when watching the display screen.

[0095] It should be noted that some display screens have a camera in the middle of the upper edge, so the camera provided by the display screen can also be used to collect the target image of the user.

[0096] Specifically, the camera collects the target image of the user in front of the display screen in real time, and sends the target image to the server 63. After the server 63 obtains the target image, the image information is extracted, including the width width, the height height, the space stride occupied by each row of pixels in the target image, the color space color space of the target image, and the orientation information orientation of the target image. Then, the target image and the corresponding image information are input into the face detection model as input quantities. The face detection model performs face detection, and then outputs the face feature information, including the face rectangular region information of the user to be measured, the face key point data, the visibility of each face key point, the face horizontal corner, the face pitch angle, the face rotation angle, and the interocular distance. The target image and the face feature information are input into the machine learning model as input quantities, and the line of sight estimation result output by the machine learning model is obtained, the line of sight estimation result includes: the position information of the left eye and the right eye in the three-dimensional space and the average line of sight direction of the two eyes, and the three-dimensional space is a three-dimensional space established with the position of the target point as the origin. After obtaining the line of sight direction of the user to be measured and the position information of the left eye and the right eye in the three-dimensional space, the position information of the line of sight of the user to be measured on the target plane can be determined, and according to the position information of the line of sight and the position information of the display screen in the three-dimensional space, it can be judged whether the line of sight falls within the display screen.

[0097] In one possible case of the embodiment, the position of the camera and the display screen are on the same plane, that is, on the target plane, and the three-dimensional space is a three-dimensional coordinate system established with the position of the camera as the origin, as shown in Figure 7 .

[0098] Further, the total duration of the line of sight falling into the display screen during the entire online learning process of the user is counted, and the ratio of the total duration to the entire online learning duration is calculated. If the ratio is greater than a preset threshold, it means that the user has a high online learning efficiency. If the ratio is less than the preset threshold, it means that the user's line of sight often leaves the display screen during the entire online learning process, and the learning efficiency is low.

[0099] It should be noted that the target plane in this embodiment is the plane on which the display screen is located, and the preset area in the target plane is the area occupied by the display screen.

[0100] It should be noted that the parts not described in detail in this embodiment can refer to the detailed description in other method embodiments of the present application, which will not be repeated here.

[0101] In the online education application scenario in this embodiment, the line-of-sight estimation algorithm can accurately determine whether the user's line of sight falls on the screen, and further determine the user's learning efficiency by counting the total duration of the line of sight falling within the display screen.

[0102] Figure 7 The flowchart of the training method of the face detection model provided in the embodiment of the present disclosure is shown. In this embodiment, the training process of the face detection model used in the above method embodiments is described.

[0103] As shown in Figure 8 The training method of the face detection model provided in this embodiment can include the following steps.

[0104] S701, a first image set is obtained, the first image set includes a plurality of sample images, and the sample images present a face image of a user.

[0105] Specifically, the camera located at the direction facing the user's head in the normal state collects a large number of images containing user face information to obtain a first image set, which contains images of various poses such as user face horizontal left and right rotation, pitch, and left and right head tilt, and then the first image set is sent to the server.

[0106] S702, image information of each sample image in the first image set is extracted, and the image information includes the image width and height, the space occupied by each row of pixels in the image, the color space of the image, and the direction information of the image.

[0107] Specifically, after the server obtains the target image, the image information is extracted, including the width width of the target image, the height height, the space stride occupied by each row of pixels in the target image, the color space color space of the target image, and the direction information orientation of the target image.

[0108] The width and height of the target image are represented by pixels.

[0109] Specifically, the space stride occupied by each row of pixels in the target image is determined according to the space occupied by the row of pixels and each pixel, that is, stride = number of bytes occupied by each pixel * width, if the stride is not a multiple of 4, then stride = stride * width + (4 - stride mod 4). For example, there are 11 pixels in a row (i.e. width = 11), and for a 32-bit image (i.e. each pixel occupies 4 bytes), stride = 11 * 4 = 44. For example, there are 11 pixels in a row (i.e. width = 11), and for a 24-bit image (i.e. each pixel occupies 3 bytes), stride = 11 * 3 = 33, then stride is not a multiple of 4, in order to ensure byte alignment, then stride = 33 + 3 = 36.

[0110] The orientation of the target image is the angle of rotation of the bit image clockwise or counterclockwise.

[0111] S703, according to the input labeled data, label training labels on each sample image in the first image set, the training labels include sample facial features corresponding to each sample image, and the sample facial features include face rectangular region information of a user, face identification, face key point data, visibility of each face key point, face horizontal corner, face pitch angle, face rotation angle, interocular distance and image confidence.

[0112] Specifically, the server sends the images in the first image set to the display terminal for display, and the user inputs the image corresponding labeled data through the display terminal, such as the user using a rectangular frame to frame the user's face region in the image, inputting the user's face rotation direction in the image and the image rotation direction, etc. Then the server automatically identifies and labels the face key point data in the image according to the user's labeled face rectangular region to generate the image corresponding training labels.

[0113] S704, generating first training data according to the first image set, image information of each sample image in the first image set and training labels corresponding to each sample image.

[0114] S705, inputting the first training data into a pre-established first deep learning network for training to obtain a face detection model.

[0115] Specifically, the first image set, image information of each image in the first image set and training labels corresponding to each image can be used as first training data, which is input into a pre-established first deep learning network for training to obtain a face detection model.

[0116] In this embodiment, the face detection model is trained by generating the first training data, and the face detection model can be used to quickly and accurately detect the face of the user target image collected in real time to obtain the face key data.

[0117] Figure 8 The flowchart of the training method of the machine learning model provided in the embodiment of the present disclosure is provided, and the training process of the machine learning model used in the above method embodiment is described in this embodiment.

[0118] As Figure 6 shown, the training method of the face detection model provided in this embodiment can include the following steps.

[0119] S801, determining the sample gaze information of the user corresponding to the face identifier in each sample image in the first image set, the sample gaze information including: left eye position information, right eye position information, left eye gaze direction, right eye gaze direction and average gaze direction of both eyes of the user in three-dimensional space.

[0120] Specifically, a three-dimensional coordinate system is established with the position of the camera as the origin, and a target plane in the plane of the camera facing the user as the target plane, and the horizontal coordinate (X-axis) and vertical coordinate (Y-axis) of the three-dimensional coordinate system are each on a straight line as any two perpendicular straight lines on the target plane passing through the target point. The vertical coordinate (Z-axis) of the three-dimensional space is on a straight line passing through the target point and perpendicular to the target plane (for reference Figure 9 the position of the target plane, the camera and the three-dimensional coordinate system in the embodiment shown). When collecting each image in the first image set, the left eye position information, right eye position information, left eye gaze direction, right eye gaze direction and average gaze direction of both eyes of the user in the three-dimensional coordinate system corresponding to each image are calculated in time, so as to obtain the binocular gaze features of the user corresponding to each image in the first image set.

[0121] S802, generating second training data according to the first training data and the sample gaze information of the user corresponding to each face identifier.

[0122] S803, inputting the second training data into a pre-established second deep learning network for training to obtain the machine learning model.

[0123] Specifically, the first image set, image information corresponding to each image in the first image set, face key data corresponding to each image, and user binocular vision features corresponding to the face identification in each image are taken as second training data, and input to a second deep learning network established in advance for training, so as to obtain a machine learning model. The target image of the user collected in real time and the corresponding face key data are input to the machine learning model, and the machine learning model can quickly and accurately output the gaze estimation result of the user corresponding to the target image, including the left eye position information, the right eye position information, the left eye gaze direction, the right eye gaze direction, and the average gaze direction of the binoculars, etc.

[0124] It should be noted that, in the process of training the machine learning model, the coordinate system referenced by the eye position information, the gaze direction and other data in the second training data, and the reference coordinate system of the gaze estimation result obtained by using the machine learning model are the same three-dimensional coordinate system.

[0125] In this embodiment, the machine learning model is trained by generating the second training data, and the machine learning model can be used to quickly and accurately detect the gaze of the target image of the user collected in real time and the face key data obtained by the face detection model detecting the target image, and obtain the gaze estimation result.

[0126] Figure 9 The structure diagram of the gaze estimation device provided by the embodiment of the present disclosure is provided. The device provided by the embodiment can be deployed in a server to implement the corresponding method steps in the server.

[0127] As shown in Figure 10 The device provided by the embodiment of the present disclosure includes an image acquisition module 91, a feature detection module 92, a gaze estimation module 93, and a viewpoint determination module 94; wherein,

[0128] The image acquisition module is configured to acquire a target image, and the target image presents a face image of a user to be detected.

[0129] The feature detection module is configured to detect the face features of the target image to obtain the face feature information of the user to be detected.

[0130] The gaze estimation module is configured to input the target image and the face feature information into a machine learning model for gaze estimation to obtain the gaze estimation result output by the machine learning model. The machine learning model has been trained based on sample images, sample face features, and sample gaze information. The sample face features are the face features presented in the sample images, and the sample gaze information is used to indicate the gaze presented in the sample images.

[0131] a viewpoint determination module configured to determine a viewpoint position of the user to be measured in the target image according to the line-of-sight estimation result.

[0132] In one or more possible embodiments, the feature detection module is specifically configured to:

[0133] obtain image information of the target image, the image information comprising a width and a height of the target image, a space occupied by each row of pixels in the target image, a color space of the target image, and directional information of the target image;

[0134] input the image information of the target image into the pre-trained face detection model to obtain face feature information of the user to be measured output by the face detection model, the face feature information comprising face rectangular region information, face key point data, visibility of each face key point, a face horizontal rotation angle, a face pitch angle, a face rotation angle, and an interocular distance of the user to be measured.

[0135] In one or more possible embodiments, the line-of-sight estimation result comprises position information of the left eye and the right eye in a three-dimensional space and an average line-of-sight direction of the two eyes, the three-dimensional space being a three-dimensional space established with a position of a target point as an origin;

[0136] The viewpoint determination module is specifically configured to:

[0137] determine, according to the position information of the left eye and the right eye in the three-dimensional space, position information of a middle point between the left eye and the right eye;

[0138] determine, according to the position information of the middle point, the average line-of-sight direction of the two eyes, and a first formula, the viewpoint position of the user to be measured.

[0139] In one or more possible embodiments, the first formula is:

[0140]

[0141] wherein e x , e y , and e z represent horizontal coordinates, vertical coordinates, and vertical coordinates of the middle point in the three-dimensional space, g x , g y , and g z represent included angles between the average line-of-sight direction of the two eyes and horizontal coordinate axes, vertical coordinate axes, and vertical coordinate axes in the three-dimensional space, and p x , p y , and p z represent horizontal coordinates, vertical coordinates, and vertical coordinates of the viewpoint in the three-dimensional space.

[0142] In one or more possible embodiments, the target point is any point on the target plane, the straight line where each of the horizontal coordinate axis and the vertical coordinate axis of the three-dimensional space is located is any two mutually perpendicular straight lines on the target plane passing through the target point, and the straight line where the vertical coordinate axis of the three-dimensional space is located passes through the target point and is perpendicular to the target plane;

[0143] The first formula is:

[0144]

[0145] wherein e x , e y and e z represent the horizontal coordinate, the vertical coordinate and the vertical coordinate of the intermediate point in the three-dimensional space respectively, g x , g y and g z represent the included angle between the average line of sight direction of the two eyes and the horizontal coordinate axis, the included angle between the vertical coordinate axis and the vertical coordinate axis in the three-dimensional space respectively, p x and p y represent the horizontal coordinate and the vertical coordinate of the viewpoint in the three-dimensional space respectively, and the vertical coordinate of the viewpoint is 0.

[0146] In one or more possible embodiments, the line of sight estimation apparatus further comprises a position determination module 95 configured to determine whether the viewpoint of the user to be measured is in a preset area according to the position of the viewpoint of the user to be measured, the preset area being an area in a target plane.

[0147] In one or more possible embodiments, the position determination module is specifically configured to determine that the viewpoint is in the preset area when the horizontal coordinate, the vertical coordinate and the vertical coordinate of the viewpoint satisfy a preset condition, the preset condition comprising that the horizontal coordinate of the viewpoint is in a first range, the vertical coordinate of the viewpoint is in a second range and the vertical coordinate of the viewpoint is in a third range.

[0148] wherein the starting value and the end value of each of the first range, the second range and the third range are determined according to the coordinates of each edge point of the preset area, and the starting value and the end value of the first range are respectively the minimum value and the maximum value of the horizontal coordinate of each edge point of the preset area, the starting value and the end value of the second range are respectively the minimum value and the maximum value of the vertical coordinate of each edge point of the preset area, and the starting value and the end value of the third range are respectively the minimum value and the maximum value of the vertical coordinate of each edge point of the preset area.

[0149] In one or more possible embodiments, the line-of-sight estimation apparatus further comprises a model training module 96 configured to: obtain a first image set, the first image set comprising a plurality of sample images, the sample images presenting facial images of a user; generate first training data according to the first image set; and input the first training data into a pre-established first deep learning network for training to obtain a face detection model.

[0150] In one or more possible embodiments, the model training module is specifically configured to:

[0151] extract image information of each sample image in the first image set, the image information comprising image width and height, space occupied by each row of pixels in the image, color space of the image, and orientation information of the image;

[0152] label training labels on each sample image in the first image set according to inputted annotation data, the training labels comprising sample facial features corresponding to the each sample image, the sample facial features comprising facial rectangular region information of the user, face identification, face key point data, visibility of each face key point, facial horizontal turning angle, facial pitch angle, facial rotation angle, interocular distance, and image confidence;

[0153] generate first training data according to the first image set, image information of each sample image in the first image set, and the training labels corresponding to each sample image.

[0154] In one or more possible embodiments, the model training module is further configured to:

[0155] extract image information of each sample image in the first image set, the image information comprising image width and height, space occupied by each row of pixels in the image, color space of the image, and orientation information of the image;

[0156] label training labels on each sample image in the first image set according to inputted annotation data, the training labels comprising sample facial features corresponding to the each sample image, the sample facial features comprising facial rectangular region information of the user, face identification, face key point data, visibility of each face key point, facial horizontal turning angle, facial pitch angle, facial rotation angle, interocular distance, and image confidence;

[0157] generate first training data according to the first image set, image information of each sample image in the first image set, and the training labels corresponding to each sample image.

[0158] The apparatus provided in the embodiment can be used to execute the technical solutions of the method embodiments, and has similar implementation principles and technical effects, which will not be described here again in the embodiment.

[0159] Reference Figure 10 which shows a structural diagram of an electronic device 100 suitable for implementing embodiments of the present disclosure, which can be a terminal device or a server. Among them, the terminal device can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDA), tablet computers (PAD), portable media players (PMP), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 10 The electronic device shown is only an example and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0160] As Figure 10 shown, the electronic device 100 can include a processing device (such as a central processor, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are also stored. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0161] Generally, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 1008 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 1009. The communication devices 1009 can allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. Although ​ The electronic device 1000 is shown with various devices, but it should be understood that all of the devices shown are not required, and more or fewer devices can alternatively be implemented.

[0162] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

[0163] It should be noted that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, in which the computer readable program code is carried. Such a propagated data signal can take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer readable medium can be transmitted by any suitable medium, including, but not limited to, a wire, an optical fiber, an RF (radio frequency) or the like, or any suitable combination of the above.

[0164] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device.

[0165] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to execute the methods shown in the embodiments described above.

[0166] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0167] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0168] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself. For example, the first obtaining unit can also be described as a unit for obtaining at least two Internet protocol addresses.

[0169] The functions described in this specification can be performed at least in part by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0170] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0171] In a first aspect, according to one or more embodiments of the present disclosure, a line-of-sight estimation method is provided, comprising:

[0172] obtaining a target image, the target image presenting a face image of a user to be tested;

[0173] detecting a face feature of the target image to obtain face feature information of the user to be tested;

[0174] inputting the target image and the face feature information into a machine learning model to perform line-of-sight estimation, to obtain a line-of-sight estimation result output by the machine learning model; the machine learning model has been trained based on sample images, sample face features, and sample line-of-sight information, the sample face features being face features presented in the sample images, and the sample line-of-sight information being used to indicate a line-of-sight presented in the sample images;

[0175] determining a viewpoint position of the user to be tested in the target image according to the line-of-sight estimation result.

[0176] According to one or more embodiments of the present disclosure, the detecting a face feature of the target image to obtain face feature information of the user to be tested comprises:

[0177] obtaining image information of the target image, the image information including a width and a height of the target image, a space occupied by each row of pixels in the target image, a color space of the target image, and direction information of the target image;

[0178] inputting image information of the target image into the pre-trained face detection model to obtain face feature information of the to-be-tested user output by the face detection model, the face feature information including face rectangular region information, face key point data, visibility of each face key point, face horizontal rotation angle, face pitch angle, face rotation angle, and interocular distance of the to-be-tested user.

[0179] According to one or more embodiments of the present disclosure, the line-of-sight estimation result includes position information of the left eye and the right eye in a three-dimensional space and an average line-of-sight direction of the two eyes, the three-dimensional space being a three-dimensional space established with a position of a target point as an origin.

[0180] The determining of the viewpoint position of the to-be-tested user in the target image according to the line-of-sight estimation result includes:

[0181] The determining of the intermediate point position information between the left eye and the right eye according to the position information of the left eye and the right eye in the three-dimensional space includes:

[0182] The determining of the viewpoint position of the to-be-tested user according to the intermediate point position information, the average line-of-sight direction of the two eyes, and a first formula includes:

[0183] According to one or more embodiments of the present disclosure, the first formula is a spatial straight line equation.

[0184] According to one or more embodiments of the present disclosure, the method further includes:

[0185] The judging of whether the viewpoint of the to-be-tested user is in a preset region according to the viewpoint position of the to-be-tested user includes:

[0186] According to one or more embodiments of the present disclosure, the judging of whether the viewpoint of the to-be-tested user is in a preset region according to the viewpoint position of the to-be-tested user includes:

[0187] The determining of the viewpoint in the preset region when the horizontal coordinate, the vertical coordinate, and the vertical coordinate of the viewpoint satisfy a preset condition, the preset condition including that the horizontal coordinate of the viewpoint is in a first range, the vertical coordinate of the viewpoint is in a second range, and the vertical coordinate of the viewpoint is in a third range.

[0188] The starting value and the end value of the first range, the second range and the third range are determined according to the coordinates of the edge points of the preset area, and the starting value and the end value of the first range are the minimum value and the maximum value of the horizontal coordinates of the edge points of the preset area respectively, the starting value and the end value of the second range are the minimum value and the maximum value of the vertical coordinates of the edge points of the preset area respectively, and the starting value and the end value of the third range are the minimum value and the maximum value of the vertical coordinates of the edge points of the preset area respectively.

[0189] According to one or more embodiments of the present disclosure, the method further comprises:

[0190] obtaining a first image set comprising a plurality of sample images, the sample images presenting a facial image of a user;

[0191] generating first training data according to the first image set;

[0192] inputting the first training data into a pre-established first deep learning network for training to obtain a face detection model.

[0193] According to one or more embodiments of the present disclosure, the generating first training data according to the first image set comprises:

[0194] extracting image information of each sample image in the first image set, the image information comprising the width and height of the image, the space occupied by each row of pixels in the image, the color space of the image and the direction information of the image;

[0195] annotating training labels on each sample image in the first image set according to input annotation data, the training labels comprising sample facial features corresponding to each sample image, the sample facial features comprising facial rectangular region information of the user, a face identifier, face key point data, visibility of each face key point, a face horizontal turning angle, a face pitch angle, a face rotation angle, an interocular distance and an image confidence;

[0196] generating first training data according to the first image set, the image information of each sample image in the first image set and the training labels corresponding to each sample image.

[0197] According to one or more embodiments of the present disclosure, the method further comprises:

[0198] determining sample gaze information of the face identifier in each sample image in the first image set corresponding to the user, the sample gaze information comprising left eye position information, right eye position information, left eye gaze direction, right eye gaze direction and average gaze direction of both eyes of the user in a three-dimensional space;

[0199] generate second training data according to the first training data and sample gaze information of a user corresponding to each face label;

[0200] input the second training data into a second deep learning network established in advance for training to obtain a machine learning model.

[0201] In a second aspect, according to one or more embodiments of the present disclosure, a gaze estimation apparatus is provided, comprising:

[0202] an image acquisition module configured to acquire a target image, the target image presenting a face image of a user to be measured;

[0203] a feature detection module configured to perform face feature detection on the target image to obtain face feature information of the user to be measured;

[0204] a gaze estimation module configured to input the target image and the face feature information into a machine learning model for gaze estimation to obtain a gaze estimation result output by the machine learning model, the machine learning model having been trained based on sample images, sample face features and sample gaze information, the sample face features being face features presented in the sample images, and the sample gaze information being used to indicate gaze presented in the sample images;

[0205] a viewpoint determination module configured to determine a viewpoint position of the user to be measured in the target image according to the gaze estimation result.

[0206] According to one or more embodiments of the present disclosure, the feature detection module is specifically configured to:

[0207] acquire image information of the target image, the image information including width and height of the target image, space occupied by each row of pixels in the target image, color space of the target image and direction information of the target image;

[0208] input the image information of the target image into the face detection model trained in advance to obtain face feature information of the user to be measured output by the face detection model, the face feature information including face rectangular region information of the user to be measured, face key point data, visibility of each face key point, face horizontal rotation angle, face pitch angle, face rotation angle and interocular distance.

[0209] According to one or more embodiments of the present disclosure, the gaze estimation result includes position information of left eye and right eye in a three-dimensional space and average gaze direction of both eyes, the three-dimensional space being a three-dimensional space established with a target point as an origin.

[0210] The viewpoint determination module is specifically configured to:

[0211] According to the position information of the left eye and the right eye in the three-dimensional space, determine the intermediate point position information between the left eye and the right eye;

[0212] According to the intermediate point position information, the average line-of-sight direction of the two eyes and a first formula, determine the viewpoint position of the user to be measured.

[0213] According to one or more embodiments of the present disclosure, the line-of-sight estimation device further comprises a position determination module configured to determine whether the viewpoint of the user to be measured is within a preset area according to the viewpoint position of the user to be measured, the preset area being an area in a target plane.

[0214] In one or more possible embodiments, the position determination module is specifically configured to determine that the viewpoint is within the preset area when the horizontal coordinate, the vertical coordinate and the vertical coordinate of the viewpoint satisfy a preset condition, the preset condition comprising that the horizontal coordinate of the viewpoint is within a first range, the vertical coordinate of the viewpoint is within a second range and the vertical coordinate of the viewpoint is within a third range.

[0215] The starting value and the end value of the first range, the second range and the third range are determined according to the coordinates of the edge points of the preset area, and the starting value and the end value of the first range are the minimum value and the maximum value of the horizontal coordinates of the edge points of the preset area, respectively, the starting value and the end value of the second range are the minimum value and the maximum value of the vertical coordinates of the edge points of the preset area, respectively, and the starting value and the end value of the third range are the minimum value and the maximum value of the vertical coordinates of the edge points of the preset area, respectively.

[0216] According to one or more embodiments of the present disclosure, the line-of-sight estimation device further comprises a model training module configured to obtain a first image set, the first image set comprising a plurality of sample images, the sample images presenting facial images of users; generate first training data according to the first image set; input the first training data into a first deep learning network established in advance for training to obtain a face detection model.

[0217] According to one or more embodiments of the present disclosure, the model training module is specifically configured to:

[0218] extract image information of each sample image in the first image set, the image information comprising the width and height of the image, the space occupied by each row of pixels in the image, the color space of the image and the direction information of the image;

[0219] Based on the input annotation data, training labels are annotated on each sample image in the first image set. The training labels include the sample facial features corresponding to each sample image. The sample facial features include the user's facial rectangular region information, face identifier, facial key point data, visibility of each facial key point, facial horizontal angle, facial pitch angle, facial rotation angle, interpupillary distance, and image confidence.

[0220] First training data is generated based on the first image set, the image information of each sample image in the first image set, and the training label corresponding to each sample image.

[0221] According to one or more embodiments of this disclosure, the model training module is further configured to:

[0222] Extract image information from each sample image in the first image set. The image information includes the image width and height, the space occupied by each row of pixels in the image, the color space of the image, and the orientation information of the image.

[0223] Based on the input annotation data, training labels are annotated on each sample image in the first image set. The training labels include the sample facial features corresponding to each sample image. The sample facial features include the user's facial rectangular region information, face identifier, facial key point data, visibility of each facial key point, facial horizontal angle, facial pitch angle, facial rotation angle, interpupillary distance, and image confidence.

[0224] First training data is generated based on the first image set, the image information of each sample image in the first image set, and the training label corresponding to each sample image.

[0225] In a seventh aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0226] The memory stores computer-executed instructions;

[0227] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the line-of-sight estimation method as described in the first aspect and various possible designs of the first aspect.

[0228] Eighthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the line-of-sight estimation method as described in the first aspect and various possible designs of the first aspect.

[0229] In a ninth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the line-of-sight estimation method according to the first aspect and various possible designs of the first aspect.

[0230] The above description is merely illustrative of the exemplary embodiments of the present disclosure and the principles of the technology involved. It is understood that the disclosure of the present disclosure is not limited to the above-mentioned technical features of the specific combination, and should also cover other technical solutions formed by any combination of the above-mentioned technical features or equivalent features without departing from the above-mentioned disclosure concept. For example, the above-mentioned features are replaced with the technical features disclosed in the present disclosure (but not limited to) having similar functions to form a technical solution.

[0231] In addition, although each operation is described in a specific order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable subcombination.

[0232] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A line-of-sight estimation method, characterized in that, include: Acquire a target image, which presents a facial image of the user to be tested; Facial feature detection is performed on the target image to obtain the facial feature information of the user under test. The facial feature information includes the facial rectangular region information of the user under test, facial key point data, visibility of each facial key point, facial horizontal angle, facial pitch angle, facial rotation angle, and interocular distance. The target image and the facial feature information are input into a machine learning model for gaze estimation, and the gaze estimation result output by the machine learning model is obtained. The machine learning model has been trained based on sample images, sample facial features, and sample gaze information. The sample facial features are the facial features presented in the sample images, and the sample gaze information is used to indicate the gaze presented in the sample images. Based on the gaze estimation results, the viewpoint position of the user to be tested in the target image is determined.

2. The method according to claim 1, characterized in that, The step of performing facial feature detection on the target image to obtain the facial feature information of the user to be tested includes: The image information of the target image is obtained, including the width and height of the target image, the space occupied by each row of pixels in the target image, the color space of the target image, and the orientation information of the target image; The image information of the target image is input into a pre-trained face detection model to obtain the facial feature information of the user to be tested, which is output by the face detection model.

3. The method according to claim 1, characterized in that, The line-of-sight estimation results include: the position information of the left and right eyes in three-dimensional space and the average line-of-sight direction of both eyes, wherein the three-dimensional space is a three-dimensional space established with the location of the target point as the origin; Determining the viewpoint position of the user to be tested in the target image based on the gaze estimation result includes: Based on the position information of the left and right eyes in three-dimensional space, determine the position information of the midpoint between the left and right eyes; The viewpoint position of the user under test is determined based on the midpoint position information, the average gaze direction of both eyes, and the first formula.

4. The method according to claim 3, characterized in that, The first formula is the equation of a straight line in space.

5. The method according to any one of claims 1-4, characterized in that, Also includes: Based on the viewpoint position of the user under test, it is determined whether the viewpoint of the user under test is within a preset area, where the preset area is a region in the target plane.

6. The method according to claim 5, characterized in that, The step of determining whether the viewpoint of the user under test is within a preset area based on the viewpoint position of the user under test includes: When the horizontal, vertical, and ordinate coordinates of the viewpoint meet preset conditions, the viewpoint is determined to be within a preset area. The preset conditions include: the horizontal coordinate of the viewpoint is within a first range, the vertical coordinate of the viewpoint is within a second range, and the vertical coordinate of the viewpoint is within a third range. The starting and ending values ​​of the first, second, and third ranges are determined based on the coordinates of each edge point of the preset region. The starting and ending values ​​of the first range are the minimum and maximum values ​​of the horizontal coordinates of each edge point of the preset region, respectively. The starting and ending values ​​of the second range are the minimum and maximum values ​​of the vertical coordinates of each edge point of the preset region, respectively. The starting and ending values ​​of the third range are the minimum and maximum values ​​of the vertical coordinates of each edge point of the preset region, respectively.

7. The method according to any one of claims 1-4, characterized in that, Also includes: Obtain a first image set, which includes multiple sample images, the sample images of which present the user's facial images; Generate first training data based on the first image set; The first training data is input into a pre-established first deep learning network for training to obtain a face detection model.

8. The method according to claim 7, characterized in that, The step of generating the first training data based on the first image set includes: Extract image information from each sample image in the first image set. The image information includes the image width and height, the space occupied by each row of pixels in the image, the color space of the image, and the orientation information of the image. Based on the input annotation data, training labels are annotated on each sample image in the first image set. The training labels include the sample facial features corresponding to each sample image. The sample facial features include the user's facial rectangular region information, face identifier, facial key point data, visibility of each facial key point, facial horizontal angle, facial pitch angle, facial rotation angle, interpupillary distance, and image confidence. First training data is generated based on the first image set, the image information of each sample image in the first image set, and the training label corresponding to each sample image.

9. The method according to claim 8, characterized in that, Also includes: Determine the sample gaze information of the user corresponding to the face identifier in each sample image of the first image set. The sample gaze information includes: the user's left eye position information, right eye position information, left eye gaze direction, right eye gaze direction, and average gaze direction of both eyes in three-dimensional space. Based on the first training data and the sample gaze information of each user corresponding to each face identifier, second training data is generated; The second training data is input into a pre-established second deep learning network for training to obtain a machine learning model.

10. A line-of-sight estimation device, characterized in that, include: An image acquisition module is used to acquire a target image, which presents a facial image of the user to be tested. The feature detection module is used to perform facial feature detection on the target image to obtain the facial feature information of the user under test. The facial feature information includes the facial rectangular region information of the user under test, facial key point data, visibility of each facial key point, facial horizontal angle, facial pitch angle, facial rotation angle, and interocular distance. A gaze estimation module is used to input the target image and the facial feature information into a machine learning model for gaze estimation, and obtain the gaze estimation result output by the machine learning model; the machine learning model has been trained based on sample images, sample facial features and sample gaze information, wherein the sample facial features are the facial features presented in the sample images, and the sample gaze information is used to indicate the gaze presented in the sample images; The viewpoint determination module is used to determine the viewpoint position of the user to be tested in the target image based on the line-of-sight estimation result.

11. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the line-of-sight estimation method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the line-of-sight estimation method as described in any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the line-of-sight estimation method according to any one of claims 1-9.

Images