Gaze point positioning method and device, electronic device and storage medium

Abstract

The embodiment of the present application discloses a method and device for locating a gaze point, an electronic device, and a storage medium, the method comprising: obtaining a facial image of a target user and an eyeball radius of a target eyeball of the target user; determining first coordinate information and second coordinate information according to the facial image and the eyeball radius of the target eyeball; wherein the first coordinate information is used to indicate the coordinate position of the eyeball center of the target eyeball on the first coordinate system, and the second coordinate information is used to indicate the coordinate position of the pupil center of the target eyeball on the first coordinate system, and the first coordinate system is constructed by a two-dimensional plane where the display screen is located and a straight line perpendicular to the two-dimensional plane; determining the gaze point of the target user on the display screen according to the first coordinate information and the second coordinate information. It can be seen that the embodiment of the present application is conducive to realizing the positioning of the eyeball gaze point only through the facial image, without the need to use infrared equipment.

Description

Technical Field

[0001] The present application relates to the field of eye tracking technology, and in particular to a gaze point positioning method and device, an electronic device and a storage medium. Background Art

[0002] As the photography function of electronic devices becomes more and more powerful, eye tracking through the camera of electronic devices has gradually become a research hotspot. At present, in eye tracking technology, pupil corneal reflex (pupil-CR) eye tracking using corneal reflection points has been maturely applied. However, Pupil-CR eye tracking requires the use of infrared and other equipment to form corneal reflection points on the cornea, and cannot be applied to scenes without infrared equipment. Therefore, how to locate the eye gaze point without infrared equipment requires further research. Summary of the invention

[0003] The embodiments of the present application provide a gaze point positioning method and device, an electronic device and a storage medium, in the hope of realizing the positioning of the eye gaze point only through a face image.

[0004] In a first aspect, an embodiment of the present application provides a gaze point positioning method, characterized in that it is applied to an electronic device, wherein the electronic device includes a display screen; the method includes:

[0005] Acquire a face image of a target user and an eyeball radius of a target eyeball of the target user;

[0006] Determine first coordinate information and second coordinate information according to the face image and the eyeball radius of the target eyeball; wherein the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball in a first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball in the first coordinate system, and the first coordinate system is constructed by a two-dimensional plane where the display screen is located and a straight line perpendicular to the two-dimensional plane;

[0007] The gaze point of the target user on the display screen is determined according to the first coordinate information and the second coordinate information.

[0008] In a second aspect, an embodiment of the present application provides a gaze point positioning device, which is applied to an electronic device, wherein the electronic device includes a display screen; the device includes a processing unit, and the processing unit is used to:

[0009] Acquire a face image of a target user and an eyeball radius of a target eyeball of the target user;

[0010] Determine first coordinate information and second coordinate information according to the face image and the eyeball radius of the target eyeball; wherein the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball in a first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball in the first coordinate system, and the first coordinate system is constructed by a two-dimensional plane where the display screen is located and a straight line perpendicular to the two-dimensional plane;

[0011] The gaze point of the target user on the display screen is determined according to the first coordinate information and the second coordinate information.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface and a display screen, wherein the memory stores one or more programs, and the one or more programs are executed by the processor, and the one or more programs are used to execute instructions of the steps in the first aspect of the embodiment of the present application.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program is operable to enable a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0015] It can be seen that in the embodiment of the present application, first, the facial image of the target user and the eyeball radius of the target eyeball of the target user are obtained. Then, the first coordinate information and the second coordinate information are determined according to the facial image and the eyeball radius of the target eyeball. Finally, the gaze point of the target user on the display screen is determined according to the first coordinate information and the second coordinate information. Since the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball on the first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball on the first coordinate system, the gaze point of the target user on the display screen is located by the three-dimensional coordinates of the eyeball center of the target eyeball and the three-dimensional coordinates of the pupil center of the target eyeball, thereby realizing the positioning of the eyeball gaze point only through the facial image without the need for infrared equipment, which is conducive to expanding the application scope of eye tracking technology and simplifying the steps of eye tracking to improve the efficiency of gaze point positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a structural schematic diagram of a target user gazing at an electronic device provided in an embodiment of the present application;

[0018] Figure 2 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of the structure of pupil corneal reflection eye tracking provided in an embodiment of the present application;

[0020] Figure 4 This is a schematic diagram of the structure of another pupil-corneal reflection eye tracking provided in an embodiment of the present application;

[0021] Figure 5 It is a flowchart of a method for locating a gaze point provided in an embodiment of the present application;

[0022] Figure 6 This is a schematic diagram of a structure for locating a user's eyeball through a camera of an electronic device provided in an embodiment of the present application;

[0023] Figure 7 is a structural schematic diagram of a face image provided in an embodiment of the present application;

[0024] Figure 8 It is a structural schematic diagram of an eye positioning model provided in an embodiment of the present application;

[0025] Fig. 9 is a schematic structural diagram of a first coordinate system and a second coordinate system provided in an embodiment of the present application;

[0026] Fig.10 is a structural schematic diagram of the transformation between a first coordinate system and a second coordinate system provided in an embodiment of the present application;

[0027] Fig.11 This is a structural diagram of obtaining a first mapping relationship table provided in an embodiment of the present application;

[0028] Fig.12 This is a structural diagram of a principle of near-big-far-small provided by an embodiment of the present application;

[0029] Fig.13 It is a structural schematic diagram of another eye positioning model provided in an embodiment of the present application;

[0030] Fig.14 It is a structural schematic diagram of another eye positioning model provided in an embodiment of the present application;

[0031] Fig.15 It is a structural schematic diagram of another eye positioning model provided in an embodiment of the present application;

[0032] Fig.16 It is a block diagram of the functional units of a gaze point positioning device provided in an embodiment of the present application;

[0033] Fig.17 It is a structural schematic diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0035] Specifically, the electronic device of the present embodiment may be an electronic device with a display screen and a camera, may be various intelligent terminals, mobile terminals, handheld devices and vehicle-mounted devices with display screens and cameras, may be user equipment (UE), terminal device, session initiation protocol (SIP) phone, personal digital assistant (PDA), personal computer (PC), terminal device supporting 5G communication system and terminal device in the public land mobile network (PLMN) of future evolution, etc., may also be a mobile phone, a tablet computer (Pad), a computer with wireless transceiver function, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal device in industrial control, wireless terminal device in self-driving, wireless terminal device in remote medical, wireless terminal device in smart grid, wireless terminal device in transportation safety, wireless terminal device in smart city, etc. The invention relates to wireless terminal devices in a city or a smart home.

[0036] As a possible example, in the embodiment of the present application, the electronic device may be a wearable device. Among them, the wearable device may be called a wearable smart device, which is a general term for wearable devices that are intelligently designed and developed by applying wearable technology to daily wear, such as glasses, gloves, watches, clothing and shoes. In addition, the wearable device may be a portable device that is worn directly on the body or integrated into clothing or accessories, such as a smart watch, smart glasses, a smart bracelet with vital sign monitoring, or smart jewelry.

[0037] In addition, in the embodiment of the present application, the electronic device may include a display screen and a camera. The display screen may be used to display a face image of the current target user, etc., and may be a liquid crystal display (LCD), an organic light emitting diode display (OLED), a thin film diode (TFD) display, or a thin film transistor (TFT) display.

[0038] Specifically, the camera can be used to capture the face of the current target user, and can include an eye tracking module, a lens, and an image sensor, etc. Among them, the eye tracking module can be used to perform the gaze point positioning of the target user's eyeball in this embodiment; the lens can be composed of several lens structures, and the lens can be a plastic lens (plastic) or a glass lens (glass), etc.; the image sensor can be divided into a charge-coupled device (CCD) and a complementary metal oxide semiconductor (CMOS), etc.

[0039] For the convenience of explanation, the following takes the electronic device as an example of a mobile terminal to illustrate the scenario in which the target user looks at the electronic device in this embodiment. Figure 1 It should be noted that Figure 1 There is no particular limitation on the electronic device in this embodiment.

[0040] exist Figure 1 In the example scene 10, the target user's eyeball 110 and an electronic device 120 may be included. The electronic device 120 may include a display screen 121 and a camera 122, and the electronic device 120 may locate the gaze point position 130 of the target user's eyeball 110 on the display screen 121.

[0041] The following embodiment of the present application further introduces the hardware structure of the electronic device. Figure 2The electronic device 200 may include a processor 210, a communication module 220, an eye tracking module 230, a power management module 240, and a storage module 250. The processor 210 connects and controls the communication module 220, the eye tracking module 230, the power management module 240, and the storage module 250 in the form of a corresponding bus. At the same time, the processor 210 is the control center of the electronic device 200, and connects various parts of the electronic device 200 through various interfaces and lines.

[0042] Specifically, the processor 210 calls the stored data in the memory by running or executing the software programs and / or modules in the storage module 250 to execute various functions of the electronic device 200 and process data, and monitor the overall operation of the electronic device 200. Optionally, the processor 210 may include a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.

[0043] Specifically, the communication module 220 can implement functions such as the second-generation 2G mobile communication technology network, the third-generation 3G mobile communication technology network, the fourth-generation 4G mobile communication technology network and the fifth-generation 5G mobile communication technology network to perform reception and transmission of wireless mobile network data, and can provide 2.4GHz and 5GHz channel spectrum resources to perform reception and transmission of network data.

[0044] Specifically, the eye tracking module 230 may be used to perform processing operations such as gaze point positioning on the eyes of the target user in this embodiment.

[0045] Specifically, the power management module 240 may include a power management chip and may provide management functions such as power conversion, distribution, and detection for the cloud server 110 .

[0046] Specifically, the storage module 250 can be used to store software programs and / or modules, and can include a program storage area and a data storage area. The program storage area can be used to store an operating system or a software program required for at least one function, and the software program required for the at least one function can be used to perform the gaze point positioning function in the embodiment of the present application; the data storage area can be used to store a face graphic of a target user or an eyeball radius of a target eyeball of a target user, etc.

[0047] In addition, the electronic device of the embodiment of the present application may include a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as processing facial images of target users. Moreover, the embodiment of the present application does not specifically limit the specific structure of the execution subject of the method provided in the embodiment of the present application, as long as it can communicate according to the method provided in the embodiment of the present application by running a program that records the code of the method provided in the embodiment of the present application, for example, the execution subject of the method provided in the embodiment of the present application can be an electronic device, or a functional module in an electronic device that can call and execute a program.

[0048] Before introducing the gaze point positioning method provided by the embodiment of the present application in detail, the eye tracking technology involved in the embodiment of the present application is introduced. Among them, the eye tracking technology is the process of measuring the movement of the eyes. Among them, the most concerned event in the eye tracking technology is to determine the user's gaze point position or gaze point position. In other words, the eye tracking technology performs image processing through relevant instruments and equipment to locate the pupil position, and calculates the gaze point position or gaze point position of the user's eyes through relevant algorithms.

[0049] (1) Pupil Corneal Reflex (Pupil-CR) Eye Tracking

[0050] The core of video-based eye tracking technology is to capture a series of user's eye images through one or more cameras. Among them, the eye tracking camera used in electronic devices usually adopts the method of infrared camera. Then, the image processing algorithm in the eye tracking software is used to identify two key positions in each eye image sent by the eye tracking camera, namely the pupil center and the corneal reflection point. The corneal reflection point is the point where the light emitted by a fixed light source (such as an infrared device or an infrared illuminator) is reflected back on the cornea, such as Figure 3 As the user's eyes rotate from one side to the other, the position of the corneal reflection point remains unchanged, while the position of the pupil center changes, as shown in Figure 4 As shown. It can be seen that Pupil-CR eye tracking requires the use of infrared and other equipment to form corneal reflection points on the cornea, so it cannot be applied to scenes without infrared and other equipment.

[0051] (2) Gaze point prediction based on convolutional neural network

[0052] By annotating multiple human eye images and inputting the annotated multiple human eye images into the convolutional neural network, the network convolutional neural network device is trained. Finally, the eye gaze point position is predicted by the successfully trained convolutional neural network. However, the gaze point prediction based on the convolutional neural network requires a large number of eye images for training. At the same time, the annotation process of the eye images takes a lot of time, and the accuracy of the predicted eye gaze point position is low.

[0053] In view of the above-mentioned shortcomings, without using infrared equipment, the embodiment of the present application locates the eye gaze point of the target user through the face image of the target user.

[0054] Combined with the above description, the following will introduce the execution steps of the gaze point positioning method from the perspective of method examples. Figure 5 . Figure 5 1 is a flow chart of a method for locating a gaze point provided in an embodiment of the present application, which is applied to an electronic device, wherein the electronic device includes a display screen; the method includes:

[0055] S510: Acquire a face image of a target user and an eyeball radius of a target eyeball of the target user.

[0056] It should be noted that the target user may refer to a user who is currently looking at the display screen of the electronic device. In addition, the present embodiment approximates the eyes of the target user as an eyeball, so the target eyeball of the target user may be the left eye or the right eye of the target user. It is understandable that when the target eyeball is the left eye of the target user, the technical solution in the present embodiment can locate the gaze point of the left eye on the display screen of the electronic device. Similarly, the gaze point of the right eye on the display screen can also be located according to the technical solution in the present embodiment. In other words, the present embodiment can locate the gaze point of a single eye of the target user on the display screen, and can also locate the gaze point of both eyes of the target user on the display screen. For the convenience of description, the present embodiment will be specifically introduced through the example of a single eye, and the example of both eyes can be obtained in the same way.

[0057] Specifically, the electronic device may further include a camera, and the camera may be a front camera of the electronic device.

[0058] In a possible example, obtaining the facial image of the target user and the eyeball radius of the target eyeball of the target user may include the following operations: when the image of the target user is detected to exist within the framing range of the camera, obtaining the facial image of the target user and the eyeball radius of the target eyeball of the target user; or, when the image of the target user is detected to exist in the preview image, obtaining the facial image of the target user and the eyeball radius of the target eyeball of the target user; wherein the preview image may be used to represent the image currently presented on the display screen; or, when the target user is looking at the display screen, obtaining the facial image of the target user and the eyeball radius of the target eyeball of the target user. It is understandable that when the image of the target user is within the framing range of the camera, the facial image of the target user is obtained by the camera. Or, when the image of the target user is presented on the current screen of the display screen, the facial image of the target user is obtained by the camera; or, when the target user is looking at the display screen, the facial image of the target user is obtained by the camera, and there is no specific limitation on this.

[0059] It should be further explained that the above-mentioned image of the target user may include a full-body image of the target user, an upper-body image of the target user, a facial image of the target user, or an image of other parts of the target user. Therefore, the camera of the electronic device in this embodiment needs to track or capture the facial position of the target user by means of rotation, zooming, etc., and then obtain the facial image of the target user, so as to ensure that an accurate facial image is obtained. For example, when the full-body image of the target user is within the camera's framing range, the camera tracks or captures the facial position of the target user by means of rotation, zooming, etc., and then obtains the facial image of the target user.

[0060] Furthermore, the eyeball radius of the target eyeball of the target user may be a pre-stored empirical value or may be determined based on the face image of the target user. The following is an example of how to obtain the eyeball radius of the target eyeball.

[0061] In one possible example, obtaining the eyeball radius of the target eyeball of the target user may include the following operations: inputting the facial image of the target user into a preset age model to obtain a first age value, which can be used to represent the age of the target user; or, receiving the first age value input by the target user; and determining the eyeball radius of the target eyeball according to the first age value.

[0062] It should be noted that, since the size of a person's eyeball will no longer change after the age of 6-8, this embodiment considers determining the radius of a person's eyeball through the age range of the person. Secondly, this embodiment can obtain the age of the target user, that is, the first age value, by inputting the face image of the target user into a preset age model or by the target user inputting it himself. Among them, the preset age model can extract feature parameters from the corners of the eyes, cheeks, forehead and other parts in the face image, and compare the extracted feature parameters with the preset feature parameter library in the preset age model to obtain a comparison result, and finally determine the first age value through the comparison result.

[0063] S520: Determine first coordinate information and second coordinate information according to the face image and the eyeball radius of the target eyeball.

[0064] Among them, the first coordinate information can be used to represent the coordinate position of the eye center of the target eye in the first coordinate system, and the second coordinate information can be used to represent the coordinate position of the pupil center of the target eye in the first coordinate system. The first coordinate system is constructed by a two-dimensional plane where the display screen is located and a straight line perpendicular to the two-dimensional plane.

[0065] It should be noted that, since the target user's eye is approximated as an eyeball in this embodiment, the eyeball center of the target eyeball is the center of the approximate eyeball, and the pupil center of the target eyeball is located on the spherical surface of the approximate eyeball. At this time, the distance between the eyeball center of the target eyeball and the pupil center of the target eyeball is the eyeball radius of the target eyeball.

[0066] Furthermore, the electronic device also includes a camera, so the coordinate origin of the first coordinate system can be located at the position of the camera, and the positive direction of the Z axis of the first coordinate system is in the same direction as the optical axis of the camera.

[0067] For an example, see Figure 6First, when the image of the target user is detected within the viewing range of the camera 6102 of the electronic device 610, the face image of the target user and the eyeball radius of the target eyeball of the target user are obtained. In this embodiment, the target eyeball of the target user is approximated as a sphere 620, and the center 6201 of the sphere is used as the eyeball center of the target eyeball, and a point 6202 on the spherical surface of the sphere 620 is used as the pupil center of the target eyeball. At the same time, a three-dimensional coordinate system is constructed with the two-dimensional plane where the display screen 6101 is located as the XY axis and the straight line perpendicular to the display screen 6101 as the Z axis, and the coordinate origin of the three-dimensional coordinate system is located at the position where the camera 6102 is located, and the positive direction of the Z axis is in the same direction as the optical axis of the camera 6102. Then, the eyeball radius R of the target eyeball of the target user is determined by the face image. Finally, the coordinate positions of the center 6201 and the point 6202 on the three-dimensional coordinate system are determined according to the face image and the radius R of the target eyeball.

[0068] The present embodiment below provides an example of how to determine the first coordinate information and the second coordinate information based on the face image and the eyeball radius of the target eyeball.

[0069] In a possible example, determining the first coordinate information and the second coordinate information based on the facial image and the eyeball radius of the target eyeball may include the following operations: determining the third coordinate information, the fourth coordinate information and the first size information based on the facial image; determining the first coordinate information and the second coordinate information based on the third coordinate information, the fourth coordinate information, the first size information and the eyeball radius of the target eyeball; wherein the third coordinate information may be used to represent the coordinate position of the first image on the second coordinate system, the first image may be used to represent the imaging of the eyeball center of the target eyeball on the facial image, the second coordinate system may be constructed by a two-dimensional plane where the facial image is located, and the two-dimensional plane where the facial image is located is parallel to the two-dimensional plane where the display screen is located; the fourth coordinate information may be used to represent the coordinate position of the second image on the second coordinate system, the second image may be used to represent the imaging of the pupil center of the target eyeball on the facial image; the first size information may be used to represent the size of the eyeball contour of the target eyeball on the facial image in the second coordinate system.

[0070] It should be noted that, since the facial image includes the imaging of the eyeball center of the target eyeball (first image), the imaging of the pupil center of the target eyeball (second image), and the eyeball contour in the imaging of the target eyeball, this embodiment considers first determining the coordinate position of the first image on the two-dimensional coordinate system constructed by the two-dimensional plane where the facial image is located (third coordinate information), the coordinate position of the second image on the two-dimensional coordinate system (fourth coordinate information), and the size of the eyeball contour on the two-dimensional coordinate system (first size information). Finally, this embodiment determines the first coordinate information and the second coordinate information based on the third coordinate information, the fourth coordinate information, the first size information and the eyeball radius of the target eyeball. In addition, the coordinate origin of the second coordinate system can be the center point on the facial image of the target user, or a preset point on the facial image. At the same time, the size of the eyeball contour on the second coordinate system can be the area, circumference or radius of the eyeball contour on the second coordinate system.

[0071] For an example, see Figure 7 .exist Figure 7 In the figure, face image 710 represents the face image of the target user, image 720 represents the imaging of the target eyeball of the target user on face image 710, image 7201 represents the imaging of the eyeball center of the target eyeball on face image 710, image 7202 represents the imaging of the pupil center of the target eyeball on face image 710, and region 730 represents the eyeball contour of the target eyeball of the target user on face image 710. Then, the two-dimensional plane where face image 710 is located is taken as the xy axis to construct a two-dimensional coordinate system. The coordinate origin of the two-dimensional coordinate system is the center point on face image 710, and the two-dimensional plane where face image 710 is located is parallel to the two-dimensional plane where the display screen of the electronic device is located. Finally, the coordinate position of image 7201 on the two-dimensional coordinate system, the coordinate position of image 7202 on the two-dimensional coordinate system, and the size r of region 730 on the two-dimensional coordinate system are determined.

[0072] It can be seen that in the process of determining the first coordinate information and the second coordinate information according to the face image and the eyeball radius of the target eyeball, the second coordinate system is first established through the two-dimensional plane where the face image is located, and then the coordinate position of the first image in the second coordinate system, the coordinate position of the second image in the second coordinate system, and the size of the eyeball contour in the second coordinate system are determined, so that the coordinate position of the eyeball center of the target eyeball in the first coordinate system and the coordinate position of the pupil center of the target eyeball in the first coordinate system are determined through the first image, the second image and the eyeball contour, which is conducive to further realizing the positioning of the eyeball gaze point only through the RGB image.

[0073] The present embodiment below will provide an example to illustrate how to determine the third coordinate information, the fourth coordinate information and the first size information based on a face image.

[0074] In one possible example, determining the third coordinate information, the fourth coordinate information and the first size information based on the facial image may include the following operations: inputting the facial image into a pre-trained eye positioning model to obtain a first image, a second image and an eyeball contour; determining the coordinate position of the first image in the second coordinate system to obtain the third coordinate information; determining the coordinate position of the second image in the second coordinate system to obtain the fourth coordinate information; determining the size of the eyeball contour in the second coordinate system to obtain the first size information; wherein the eye positioning model may include a first convolutional neural network model and a second convolutional neural network model, the first convolutional neural network model may be used to extract the first image, the second image and the eyeball contour from the facial image, and the second convolutional neural network model may be used to determine whether the first image, the second image and the eyeball contour extracted by the first network model meet preset standards.

[0075] It should be noted that the existing methods for extracting eye features in face images may include pre-trained convolutional neural network (CNN) models, local binary pattern (LBP), histogram of oriented gradient (HOG), scale-invariant feature transform (SIFT), etc. Among them, the convolutional neural network can be trained by a preset image information library, and the corresponding eye features in the face image are extracted by the trained convolutional neural network model; LBP is to mark the difference between the center point pixel and its adjacent domain pixels by a preset threshold; HOG is a feature descriptor used for object detection, which constructs features by calculating and counting the gradient direction histogram of the local area of ​​the image; SIFT obtains features by finding the descriptors of the feature points in the image and their related sizes and directions, and performs image feature point matching.

[0076] Since the existing convolutional neural network usually only uses one network architecture model to directly extract the eye features in the face image, and there is no corresponding other network architecture to verify and analyze the extracted eye features, the accuracy of the extracted eye features is low. Therefore, this example considers using an eye positioning model with a dual network architecture. Among them, the eye positioning model of this example includes a first convolutional neural network model and a second convolutional neural network model. The first convolutional neural network model first extracts the first image, the second image and the eyeball contour from the face image, and then the first image, the second image and the eyeball contour extracted by the second convolutional neural network model meet the preset standard. If the preset standard is not met, it is necessary to re-extract by the first convolutional neural network model. In addition, the second convolutional neural network model determines whether the first image, the second image and the eyeball contour extracted by the first network model meet the preset standard. It can be determined by whether the probability value output by the classifier in the second convolutional neural network model meets the preset threshold.

[0077] It can be seen that extracting eye features in face images by adopting an eye localization model with a dual network architecture is beneficial to improving the accuracy of the extracted eye features.

[0078] The following embodiment will provide an example to illustrate the first convolutional neural network model and the second convolutional neural network model.

[0079] In a possible example, the first convolutional neural network model may include a first input layer, M first hidden layers and a first output layer, the M first hidden layers include at least one convolution layer, at least one pooling layer and at least one deconvolution layer; wherein, the convolution kernel size of the convolution layer in at least one convolution layer is k1*k1 pixels, the step size of the convolution layer in at least one convolution layer is s1, the activation function of the convolution layer in at least one convolution layer includes a rectified linear function unit ReUL function, the filter size of the pooling layer in at least one pooling layer is f1*f1, the step size of the pooling layer in at least one pooling layer is s2, the convolution kernel size of the deconvolution layer in at least one deconvolution layer is k1*k1 pixels, and the activation function of the convolution layer in at least one convolution layer includes a rectified linear function unit ReUL function. The step size of the deconvolution layer in the convolution layer is 1 / s1, k1 is an integer of [3,6], s1 is an integer of [1,3], f1 is an integer of [2,4], s2 is 1 or 2, and M is an integer greater than 1; and the second convolutional neural network model may include a second input layer, N second hidden layers, and a second output layer; wherein the convolution kernel size of each convolution layer in the N second hidden layers is k2*k2 pixels, the step size of each convolution layer in the N second hidden layers is s3, the activation function of each convolution layer in the N second hidden layers includes a leaky corrected linear function unit LeakyReUL function, k2 is 5 or 6, s3 is 1 or 2, and N is an integer greater than 1.

[0080] For an example, see Figure 8 .exist Figure 8 In the embodiment of the present invention, a face image with a size of 128*128*1 is first input into the first convolutional neural network model. The hidden layer of the first convolutional neural network model may include a first convolutional layer, a second convolutional layer, a first pooling layer, a third convolutional layer, a fourth convolutional layer, a second pooling layer, a first deconvolutional layer, a second deconvolutional layer, a third deconvolutional layer, a fourth deconvolutional layer, a fifth deconvolutional layer and a sixth deconvolutional layer connected in sequence; the convolution kernel size of the convolutional layer in the hidden layer is 4*4 pixels, the step size of the convolutional layer is 2, and the activation function of the convolutional layer includes a ReUL function; the filter size of the pooling layer is 4*4 pixels, the step size of the pooling layer is 2; the filter size of the deconvolution layer is 4*4 pixels, and the step size of the deconvolution layer is 1 / 2; the number of feature maps is 1, 32, 64, 128, 256, 512, 1024, 512, 256, 128, 64, 32, 1 respectively. Then, the eye features output by the first convolutional neural network model are input into the second convolutional neural network model, and the probability value is output through the classifier. Among them, the hidden layer of the second convolutional neural network model may include a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, and a tenth convolutional layer connected in sequence. The convolution kernel size of the convolutional layer in the hidden layer is 5*5 pixels, the step size of the convolutional layer is 1, and the activation function of the convolutional layer includes the LeakyReUL function. Finally, according to the probability value output by the classifier, it is determined whether to use the eye features output by the first convolutional neural network model as the first image, the second image, and the eye contour in the face image. If the probability value output by the classifier meets the preset threshold, the eye features output by the first convolutional neural network model are used as the first image, the second image, and the eye contour in the face image. Otherwise, the eye features are re-extracted from the face image by the first convolutional neural network model.

[0081] The present embodiment below will give an example of how to determine the first coordinate information and the second coordinate information according to the third coordinate information, the fourth coordinate information, the first size information and the eyeball radius of the target eyeball.

[0082] In one possible example, determining the first coordinate information and the second coordinate information based on the third coordinate information, the fourth coordinate information, the first size information and the eye radius of the target eyeball may include the following operations: converting the third coordinate information from the second coordinate system to the first coordinate system to obtain the fifth coordinate information; converting the fourth coordinate information from the second coordinate system to the first coordinate system to obtain the sixth coordinate information; determining the first coordinate information and the second coordinate information based on the fifth coordinate information, the sixth coordinate information, the first size information and the eye radius of the target eyeball.

[0083] It should be noted that the third coordinate information and the fourth coordinate information are two-dimensional coordinate information, while the fifth coordinate information and the sixth coordinate information are three-dimensional coordinate information. It can be seen that this example converts the third information from the second coordinate system to the first coordinate system, and converts the fourth information from the second coordinate system to the first coordinate system, thereby establishing an association relationship between the coordinate system where the first image is located and the coordinate system where the eye center of the target eye is located, and an association relationship between the coordinate system where the second image is located and the coordinate system where the pupil center of the target eye is located, thereby further realizing the positioning of the eye gaze point only through the RGB image according to the above association relationship.

[0084] Furthermore, if the face image of the target user is obtained by a camera, this example also needs to consider the focal length of the camera in the process of converting from the second coordinate system to the first coordinate system. Therefore, converting the third coordinate information from the second coordinate system to the first coordinate system to obtain the fifth coordinate information may include the following operations: converting the third coordinate information from the second coordinate system to the first coordinate system according to the focal length of the camera to obtain the fifth coordinate information. Similarly, converting the fourth coordinate information from the second coordinate system to the first coordinate system to obtain the sixth coordinate information may include the following operations: converting the fourth coordinate information from the second coordinate system to the first coordinate system according to the focal length of the camera to obtain the sixth coordinate information.

[0085] The following specifically describes how to convert the second coordinate system to the first coordinate system through the focal length of the camera.

[0086] It should be noted that the first coordinate system is a three-dimensional coordinate system constructed with the two-dimensional plane where the display screen of the electronic device is located as the XY axis and the straight line perpendicular to the two-dimensional plane as the Z axis. At the same time, the second coordinate system is a two-dimensional coordinate system constructed with the two-dimensional plane where the face image of the target user is located as the xy axis. Then, the following conditions are satisfied between the first coordinate system and the second coordinate system: 1) The two-dimensional plane where the display screen is located is parallel to the two-dimensional plane where the face image is located; 2) The X-axis of the first coordinate system is parallel to the x-axis of the second coordinate system, and the Y-axis of the first coordinate system is parallel to the y-axis of the second coordinate system; 3) The coordinate origin of the first coordinate system is 0 c The origin of the second coordinate system is 0 i The distance between is the focal length of the camera, such as Fig. 9 shown.

[0087] Combine the following Fig.10 , an example is given of how to convert the third coordinate information from the second coordinate system to the first coordinate system to obtain the fifth coordinate information according to the focal length of the camera.

[0088] exist Fig.10In the 3D coordinate system, the first coordinate system is composed of the X-axis, Y-axis and Z-axis, and the second coordinate system is composed of the X-axis and Y-axis. c The origin of the second coordinate system is 0 i The distance between them is the focal length f of the camera. Then, the third coordinate information is expressed as coordinates (x1, y1) in the second coordinate system, and the fifth coordinate information is expressed as coordinates (X1, Y1, Z1) in the third coordinate system. Finally, the conversion of coordinates (x1, y1) to coordinates (X1, Y1, Z1) satisfies the following formula:

[0089]

[0090] It should be noted that the sixth coordinate information can be combined with the above Fig. 9 and Fig.10 The same logic applies.

[0091] The present embodiment below will give an example of how to determine the first coordinate information and the second coordinate information according to the fifth coordinate information, the sixth coordinate information, the first size information and the eyeball radius of the target eyeball.

[0092] In a possible example, determining the first coordinate information and the second coordinate information according to the fifth coordinate information, the sixth coordinate information, the first size information, and the eyeball radius of the target eyeball may include the following operations: determining a first straight line according to the fifth coordinate information and a first preset mapping table; wherein the first preset mapping table may be used to represent a mapping relationship between the fifth coordinate information and the first straight line, and the first straight line is a straight line formed by the coordinate origin of the first coordinate system to the eyeball center of the target eyeball; determining a first distance value according to the first size information and the second preset mapping table; wherein the second preset mapping table may be used to represent a mapping relationship between the first size information and the first distance value; A mapping relationship is defined as follows: the first distance value can be used to represent the distance from the coordinate origin of the first coordinate system to the center of the target eyeball; the first coordinate information is determined according to the first straight line and the first distance value; the first sphere equation for modeling the target eyeball is determined according to the first coordinate information and the eyeball radius of the target eyeball; the second straight line is determined according to the sixth coordinate information and the third preset mapping table; wherein the third preset mapping table can be used to represent the mapping relationship between the sixth coordinate information and the second straight line, and the second straight line is formed from the coordinate origin of the first coordinate system to the pupil center of the target eyeball; the intersection of the second straight line on the first sphere equation is calculated to obtain the second coordinate information.

[0093] It should be noted that, since the first preset mapping table is used to represent the mapping relationship between the fifth coordinate information and the first orientation value, the first orientation value corresponding to the fifth coordinate information can be found from the first preset mapping table. Similarly, since the second preset mapping table is used to represent the mapping relationship between the first size information and the first distance value, the first distance value corresponding to the first size information can be found from the second preset mapping table. Since the third preset mapping table is used to represent the mapping relationship between the sixth coordinate information and the second straight line, the second straight line corresponding to the sixth coordinate information can be found from the third preset mapping table.

[0094] Combine the following Fig.11 An example of how to obtain the first preset mapping table in this example is given. It is known that a first coordinate system is constructed by taking the two-dimensional plane where the display screen of the electronic device is located as the XY axis and the straight line perpendicular to the two-dimensional plane as the Z axis. First, place a piece of white paper directly in front of the camera of the electronic device 1110, the white paper is parallel to the display screen of the electronic device, and ensure that the distance between the position of the camera and the position of the white paper is fixed. Secondly, place a tiny sphere 1120 on the white paper, and ensure that the spherical surface of the sphere 1120 is extremely small (equivalent to the center of the eyeball), and the sphere 1120 is within the field of view 1130 of the camera. Again, obtain an image of the sphere 1120 through the camera, and calibrate the position information of the sphere 1120 in the image on the first coordinate system, that is, (X1, Y1, Z1). Then, since the position of the sphere 1120 is fixed relative to the position of the white paper, and the position of the white paper is fixed relative to the position of the camera, the azimuth angle (α1, β1, γ1) of the sphere 1120 relative to the camera in the first coordinate system can be obtained, so that the straight line 1140 is calculated by (α1, β1, γ1), and the mapping relationship between (X1, Y1, Z1) and the straight line 1140 is established. Among them, α1 represents the angle between the straight line 1140 and the two-dimensional plane where the display screen is located (the plane where the XY axis of the first coordinate system is located), β1 represents the angle between the mapping of the straight line 1140 on the plane where the XY axis of the first coordinate system is located and the positive direction of the Y axis, γ1 represents the angle between the mapping of the straight line 1140 on the plane where the XY axis of the first coordinate system is located and the positive direction of the X axis, and the straight line 1140 represents the straight line from the coordinate origin in the first coordinate system to the sphere 1120. Finally, the position of the sphere 1120 on the white paper is continuously adjusted to obtain different azimuth angles, and the camera obtains multiple images of the sphere 1120 after the position adjustment, and then the multiple images are repeatedly labeled as above, so as to obtain the following mapping relationship:

[0095]

[0096] Wherein, n represents the number of times the sphere 1120 is adjusted. When n is large enough, (X i,Y i ,Z i ) can be effectively distributed in the image, thus ensuring that (X i ,Y i ,Z i ) and (α i ,β i ,γ i ) is the accuracy of the mapping relationship between them, and finally (α i ,β i ,γ i ) calculates the corresponding straight line and establishes (X i ,Y i ,Z i ) and the corresponding straight lines.

[0097] It should be noted that the process of obtaining the third preset mapping table in this example can be consistent with the technical solution described in the above “how to obtain the first preset mapping table”, which will not be repeated here.

[0098] Combine the following Fig.12 In this example, an example is given to illustrate how to obtain the second preset mapping table. Fig.12 As shown, since the sphere is larger when near and smaller when far away within the field of view of the camera, the size of the sphere is kept fixed and the distance from the sphere to the camera is continuously adjusted. The camera then obtains multiple images after the distance from the sphere to the camera is adjusted, thereby obtaining a mapping relationship between the size of the sphere and the distance from the sphere to the camera.

[0099] Combine the following Figure 6 , Fig.11 and Fig.13 In this example, an example is given to illustrate how to determine the first coordinate information according to the first straight line and the first distance value, and how to determine the first sphere equation for modeling the target eyeball according to the first coordinate information and the eyeball radius of the target eyeball. Fig.13In the example, it is known that the first coordinate system is constructed by taking the two-dimensional plane where the display screen of the electronic device is located as the XY axis and the straight line perpendicular to the two-dimensional plane as the Z axis. First, since the first preset mapping table stores the mapping relationship between the fifth coordinate information and the first straight line, the straight line 1310 is obtained. Among them, the straight line 1310 is calculated by (α, β, γ), α represents the angle between the straight line 1310 and the plane where the XY axis of the first coordinate system is located, β represents the angle between the mapping of the straight line 1310 on the plane where the XY axis of the first coordinate system is located and the positive direction of the Y axis, and γ represents the angle between the mapping of the straight line 1310 on the plane where the XY axis of the first coordinate system is located and the positive direction of the X axis. Secondly, since the second preset mapping table stores the mapping relationship between the first size information and the first distance value, the distance L from the coordinate origin of the first coordinate system to the eyeball center of the target eyeball is obtained. Then, the coordinate position (X0, Y0, Z0) of the eyeball center of the target eyeball in the first coordinate system can be calculated by the straight line 1310 and the distance L. Among them, the coordinate position (X0, Y0, Z0) can satisfy the following formula:

[0100]

[0101] Finally, the sphere equation is calculated by the coordinate position (X0, Y0, Z0) and the eyeball radius of the target eyeball:

[0102] (X-X0) 2 +(Y-Y0) 2 +(Z-Z0) 2 =R 2 .

[0103] Combine the following Fig.14 In this example, an example is given to illustrate how to calculate the intersection point of the second straight line on the first sphere equation to obtain the second coordinate information. Fig.14 In the figure, it is known that the first coordinate system is constructed by taking the two-dimensional plane where the display screen of the electronic device is located as the XY axis and the straight line perpendicular to the two-dimensional plane as the Z axis. Among them, the sphere 1410 represents the first sphere equation for modeling the target eyeball determined by the first coordinate information and the eyeball radius of the target eyeball, and the straight line 1420 represents the second straight line formed by the coordinate origin of the first coordinate system to the pupil center of the target eyeball. Finally, the intersection 1430 between the sphere 1410 and the straight line 1420 is calculated. Among them, the intersection 1430 represents the pupil center of the target eyeball. It should be noted that there may be two intersections between the sphere 1410 and the straight line 1420, so this example takes the intersection closest to the camera as the intersection 1430.

[0104] S530: Determine the gaze point of the target user on the display screen according to the first coordinate information and the second coordinate information.

[0105] The gaze point refers to the intersection of the visual axis of the target user gazing at the display screen and the two-dimensional plane where the display screen is located.

[0106] The present embodiment below provides an example of how to determine the gaze point position of the target user on the display screen according to the first coordinate information and the second coordinate information.

[0107] In a possible example, determining the gaze point position of the target user on the display screen based on the first coordinate information and the second coordinate information can include the following operations: constructing a straight line from the first coordinate information to the second coordinate information to obtain a third straight line; rotating the third straight line along a preset direction by a preset threshold to obtain a fourth straight line, the fourth straight line being used to represent the visual axis of the target user's gaze on the display screen; determining the intersection of the fourth straight line on the two-dimensional plane where the display screen is located to obtain the gaze point of the target user on the display screen.

[0108] It should be noted that since the straight line between the center of the target eyeball and the center of the pupil of the target eyeball has an angle of a certain preset threshold (greater than 5°) with the visual axis of the target user, this example ensures the accuracy of positioning the target user's gaze point on the display screen by rotating the third straight line along a preset direction by the angle of the preset threshold.

[0109] For example, in Fig.15 In the example, it is known that a first coordinate system is constructed by taking the two-dimensional plane where the display screen of the electronic device is located as the XY axis and the straight line perpendicular to the two-dimensional plane as the Z axis. Among them, straight line 1510 represents a third straight line formed by the first coordinate information to the second coordinate information; straight line 1520 represents a fourth straight line obtained by rotating the third straight line along a preset direction by a preset threshold. Finally, the intersection of straight line 1520 on the plane where the XY axis of the first coordinate system is located is calculated to obtain the gaze point 1530 of the target user on the display screen.

[0110] Specifically, the preset direction can be obtained by acquiring multiple face images of the target user within a preset time, and estimating the eye movement direction of the target user through the multiple face images. For example, when the user is looking at the display screen, the user's eyeballs may move along with the content on the display screen, so the user's eyeball movement direction is estimated by acquiring multiple face images within a certain period of time. If the user's eyeball turns to the right, the third straight line is rotated to the left by an angle greater than 5°.

[0111] It can be seen that in the embodiment of the present application, first, the facial image of the target user and the eyeball radius of the target eyeball of the target user are obtained. Then, the first coordinate information and the second coordinate information are determined according to the facial image and the eyeball radius of the target eyeball. Finally, the gaze point of the target user on the display screen is determined according to the first coordinate information and the second coordinate information. Since the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball on the first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball on the first coordinate system, the gaze point of the target user on the display screen is located by the three-dimensional coordinates of the eyeball center of the target eyeball and the three-dimensional coordinates of the pupil center of the target eyeball, thereby realizing the positioning of the eyeball gaze point only through the facial image without the need for infrared equipment, which is conducive to expanding the application scope of eye tracking technology and simplifying the steps of eye tracking to improve the efficiency of gaze point positioning.

[0112] The above mainly introduces the scheme of this embodiment from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0113] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division, and there may be other division methods in actual implementation.

[0114] In the case of an integrated unit, Fig.16 A functional unit block diagram of a gaze point positioning device is provided. The gaze point positioning device 1600 is applied to an electronic device, and specifically includes: a processing unit 1620 and a communication unit 1630. The processing unit 1620 is used to control and manage the actions of the electronic device. For example, the processing unit 1620 is used to support the electronic device to perform Figure 5The communication unit 1630 is used to support the communication between the electronic device and other devices. The electronic device may also include a storage unit 1610 for storing program codes and data.

[0115] Among them, the processing unit 1620 can be a processor or a controller, for example, it can be a CPU, a general processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic boxes, modules and circuits described in combination with the contents disclosed in this application. The processing unit 1620 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, etc. The communication unit 1630 can be a communication interface, a transceiver, a transceiver circuit, etc., and the storage unit 1610 can be a memory. When the processing unit 1620 is a processor, the communication unit 1630 is a communication interface, and the storage unit 1610 is a memory, the gaze point positioning device 1600 involved in the embodiment of the present application can be Fig.17 Electronic equipment shown.

[0116] In specific implementation, the processing unit 1620 is used to execute any step executed by the electronic device in the above method embodiment, and when performing data transmission such as sending, the communication unit 1630 can be selectively called to complete the corresponding operation. Detailed description is given below.

[0117] Processing unit 1620 is used to: obtain a facial image of a target user and an eyeball radius of a target eyeball of the target user; determine first coordinate information and second coordinate information based on the facial image and the eyeball radius of the target eyeball; wherein the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball in a first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball in the first coordinate system, and the first coordinate system is constructed by a two-dimensional plane where the display screen is located and a straight line perpendicular to the two-dimensional plane; determine the gaze point of the target user on the display screen based on the first coordinate information and the second coordinate information.

[0118] It can be seen that in the embodiment of the present application, first, the facial image of the target user and the eyeball radius of the target eyeball of the target user are obtained. Then, the first coordinate information and the second coordinate information are determined according to the facial image and the eyeball radius of the target eyeball. Finally, the gaze point of the target user on the display screen is determined according to the first coordinate information and the second coordinate information. Since the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball on the first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball on the first coordinate system, the gaze point of the target user on the display screen is located by the three-dimensional coordinates of the eyeball center of the target eyeball and the three-dimensional coordinates of the pupil center of the target eyeball, thereby realizing the positioning of the eyeball gaze point only through the facial image without the need for infrared equipment, which is conducive to expanding the application scope of eye tracking technology and simplifying the steps of eye tracking to improve the efficiency of gaze point positioning.

[0119] In one possible example, in terms of determining the first coordinate information and the second coordinate information based on the facial image and the eyeball radius of the target eyeball, the processing unit 1620 is specifically used to: determine the third coordinate information, the fourth coordinate information and the first size information based on the facial image; determine the first coordinate information and the second coordinate information based on the third coordinate information, the fourth coordinate information, the first size information and the eyeball radius of the target eyeball; wherein the third coordinate information is used to represent the coordinate position of the first image on the second coordinate system, the first image is used to represent the imaging of the eyeball center of the target eyeball on the facial image, the second coordinate system is constructed by a two-dimensional plane where the facial image is located, and the two-dimensional plane where the facial image is located is parallel to the two-dimensional plane where the display screen is located; the fourth coordinate information is used to represent the coordinate position of the second image on the second coordinate system, and the second image is used to represent the imaging of the pupil center of the target eyeball on the facial image; the first size information is used to represent the size of the eyeball contour of the target eyeball on the facial image in the second coordinate system.

[0120] In one possible example, in terms of determining the third coordinate information, the fourth coordinate information and the first size information based on the facial image, the processing unit 1620 is specifically used to: input the facial image into a pre-trained eye positioning model to obtain a first image, a second image and an eyeball contour; determine the coordinate position of the first image in the second coordinate system to obtain the third coordinate information; determine the coordinate position of the second image in the second coordinate system to obtain the fourth coordinate information; determine the size of the eyeball contour in the second coordinate system to obtain the first size information; wherein the eye positioning model includes a first convolutional neural network model and a second convolutional neural network model, the first convolutional neural network model is used to extract the first image, the second image and the eyeball contour from the facial image, and the second convolutional neural network model is used to determine whether the first image, the second image and the eyeball contour extracted by the first convolutional neural network model meet the preset criteria.

[0121] In a possible example, the first convolutional neural network model includes a first input layer, M first hidden layers and a first output layer, the M first hidden layers include at least one convolution layer, at least one pooling layer and at least one deconvolution layer; wherein, the convolution kernel size of the convolution layer in at least one convolution layer is k1*k1 pixels, the step size of the convolution layer in at least one convolution layer is s1, the activation function of the convolution layer in at least one convolution layer includes a rectified linear function unit ReUL function, the filter size of the pooling layer in at least one pooling layer is f1*f1, the step size of the pooling layer in at least one pooling layer is s2, the convolution kernel size of the deconvolution in at least one deconvolution layer is k1*k1 pixels, and the deconvolution kernel size in at least one deconvolution layer is k1*k1 pixels. The step size of the deconvolution layer in the convolution layer is 1 / s1, k1 is an integer of [3,6], s1 is an integer of [1,3], f1 is an integer of [2,4], s2 is 1 or 2, and M is an integer greater than 1; and the second convolutional neural network model includes a second input layer, N second hidden layers and a second output layer; wherein the convolution kernel size of each convolution layer in the N second hidden layers is k2*k2 pixels, the step size of each convolution layer in the N second hidden layers is s3, the activation function of each convolution layer in the N second hidden layers includes a leaky corrected linear function unit LeakyReUL function, k2 is 5 or 6, s3 is 1 or 2, and N is an integer greater than 1.

[0122] In one possible example, in determining the first coordinate information and the second coordinate information based on the third coordinate information, the fourth coordinate information, the first size information and the eye radius of the target eye, the processing unit 1620 is specifically used to: convert the third coordinate information from the second coordinate system to the first coordinate system to obtain the fifth coordinate information; convert the fourth coordinate information from the second coordinate system to the first coordinate system to obtain the sixth coordinate information; determine the first coordinate information and the second coordinate information based on the fifth coordinate information, the sixth coordinate information, the first size information and the eye radius of the target eye.

[0123] In one possible example, in determining the first coordinate information and the second coordinate information according to the fifth coordinate information, the sixth coordinate information, the first size information and the eyeball radius of the target eyeball, the processing unit 1620 is specifically used to: determine a first straight line according to the fifth coordinate information and a first preset mapping table; wherein the first preset mapping table is used to represent the mapping relationship between the fifth coordinate information and the first straight line, and the first straight line is formed from the coordinate origin of the first coordinate system to the eyeball center of the target eyeball; determine a first distance value according to the first size information and the second preset mapping table; wherein the second preset mapping table is used to represent the mapping relationship between the first size information and the first distance value. The first distance value is used to represent the distance from the coordinate origin of the first coordinate system to the center of the target eyeball; the first coordinate information is determined according to the first straight line and the first distance value; the first sphere equation for modeling the target eyeball is determined according to the first coordinate information and the eyeball radius of the target eyeball; the second straight line is determined according to the sixth coordinate information and the third preset mapping table; wherein the third preset mapping table is used to represent the mapping relationship between the sixth coordinate information and the second straight line, and the second straight line is formed from the coordinate origin of the first coordinate system to the pupil center of the target eyeball; the intersection of the second straight line on the first sphere equation is calculated to obtain the second coordinate information.

[0124] In one possible example, in terms of determining the position of the target user's gaze point on the display screen based on the first coordinate information and the second coordinate information, the processing unit 1620 is specifically used to: construct a straight line from the first coordinate information to the second coordinate information to obtain a third straight line; rotate the third straight line along a preset direction by a preset threshold to obtain a fourth straight line, and the fourth straight line is used to represent the visual axis of the target user's gaze on the display screen; determine the intersection of the fourth straight line on the two-dimensional plane where the display screen is located to obtain the target user's gaze point on the display screen.

[0125] The following is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. Fig.17 The electronic device 1700 includes a processor 1710 , a memory 1720 , a communication interface 1730 , and at least one communication bus for connecting the processor 1710 , the memory 1720 , and the communication interface 1730 .

[0126] The processor 1710 may be one or more central processing units (CPUs). In the case where the processor 1710 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The memory 1720 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a portable read-only memory (CD-ROM), and the memory 1720 is used for related instructions and data. The communication interface 1730 is used to receive and send data.

[0127] The processor 1710 in the electronic device 1700 is used to read one or more program codes 1721 stored in the memory 1720 for performing the following operations: obtaining a facial image of a target user and an eyeball radius of a target eyeball of the target user; determining first coordinate information and second coordinate information based on the facial image and the eyeball radius of the target eyeball; wherein the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball on the first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball on the first coordinate system, and the first coordinate system is constructed by a two-dimensional plane where the display screen is located and a straight line perpendicular to the two-dimensional plane; determining the gaze point of the target user on the display screen based on the first coordinate information and the second coordinate information, and the gaze point refers to the intersection of the visual axis of the target user gazing at the display screen and the two-dimensional plane where the display screen is located.

[0128] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program is operable to cause a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0129] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute some or all steps of any method described in the above method embodiment. The computer program product may be a software installation package.

[0130] It should be noted that, for the above-mentioned various method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present application.

[0131] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0132] In several embodiments provided in the present application, those skilled in the art should be aware that the described device can be implemented in other ways. It is understandable that the device embodiments described above are merely schematic. For example, the division of the above units is only a logical function division, and there may be other division methods in practice. In other words, multiple units or components can be combined or integrated into another software, and some features can be ignored or not executed. In addition, the coupling, direct coupling or communication connection between each other shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, or can be electrical or other forms.

[0133] The units described above as separate components may or may not be physically separated. In addition, the components shown above as units may or may not be physical units, that is, they may be located on one network unit or distributed on multiple network units. Therefore, the above embodiments may be implemented by selecting some or all of the units therein according to actual needs.

[0134] In addition, each functional unit in the above embodiments may be integrated into one processing unit, or may exist in different physical units, or two or more functional units may be integrated into one physical unit. The above units may be implemented in the form of hardware or software functional units.

[0135] If the above-mentioned units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable memory. It is understandable that the technical solution of the present application (the part of the technical solution that contributes to the prior art or all or part of the technical solution) can be embodied in the form of a computer software product. The computer software product is stored in a memory, including several instructions for a computer device (a personal computer, a server or a network device, etc.) to perform all or part of the steps of the embodiment of the present application. In addition, the above-mentioned memory includes various media that can store program codes, such as a USB flash drive, ROM, RAM, a mobile hard disk, a magnetic disk or an optical disk.

[0136] Those skilled in the art should be aware that all or part of the steps of the embodiments of the present application can be completed by a program to instruct related hardware, and the program can be stored in a memory, which can include a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0137] The embodiments of the present application are described in detail above, and the description in the embodiments of the present application is only used to help understand the method and core idea of ​​the present application. Those skilled in the art should know that the embodiments of the present application may be changed in specific implementation and application scope, and the content of this specification should not be construed as limiting the present application.

Claims

1. A method for locating a gaze point, characterized in that: Applied to an electronic device, the electronic device includes a display screen; the method includes: Acquire a face image of a target user and an eyeball radius of a target eyeball of the target user; Determine first coordinate information and second coordinate information according to the face image and the eyeball radius of the target eyeball; wherein the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball in a first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball in the first coordinate system, and the first coordinate system is constructed by a two-dimensional plane where the display screen is located and a straight line perpendicular to the two-dimensional plane; Determine a gaze point of the target user on the display screen according to the first coordinate information and the second coordinate information; Wherein, the determining of the first coordinate information and the second coordinate information according to the facial image and the eyeball radius of the target eyeball comprises: determining the third coordinate information, the fourth coordinate information and the first size information according to the facial image; determining the first coordinate information and the second coordinate information according to the third coordinate information, the fourth coordinate information, the first size information and the eyeball radius of the target eyeball; wherein the third coordinate information is used to represent the coordinate position of the first image on the second coordinate system, the first image is used to represent the imaging of the eyeball center of the target eyeball on the facial image, the second coordinate system is constructed by the two-dimensional plane where the facial image is located, and the two-dimensional plane where the facial image is located is parallel to the two-dimensional plane where the display screen is located; the fourth coordinate information is used to represent the coordinate position of the second image on the second coordinate system, and the second image is used to represent the imaging of the pupil center of the target eyeball on the facial image; the first size information is used to represent the size of the eyeball contour of the target eyeball on the facial image on the second coordinate system; Wherein, determining the first coordinate information and the second coordinate information according to the third coordinate information, the fourth coordinate information, the first size information and the eyeball radius of the target eyeball includes: converting the third coordinate information from the second coordinate system to the first coordinate system to obtain fifth coordinate information; converting the fourth coordinate information from the second coordinate system to the first coordinate system to obtain sixth coordinate information; determining the first coordinate information and the second coordinate information according to the fifth coordinate information, the sixth coordinate information, the first size information and the eyeball radius of the target eyeball; Wherein, the determining of the first coordinate information and the second coordinate information according to the fifth coordinate information, the sixth coordinate information, the first size information and the eyeball radius of the target eyeball includes: determining a first straight line according to the fifth coordinate information and a first preset mapping table; wherein the first preset mapping table is used to represent the mapping relationship between the fifth coordinate information and the first straight line, and the first straight line is formed by the coordinate origin of the first coordinate system to the eyeball center of the target eyeball; determining a first distance value according to the first size information and a second preset mapping table; wherein the second preset mapping table is used to represent the mapping relationship between the first size information and the first distance value, and the first The distance value is used to represent the distance from the coordinate origin of the first coordinate system to the center of the target eyeball; the first coordinate information is determined according to the first straight line and the first distance value; the first sphere equation for modeling the target eyeball is determined according to the first coordinate information and the eyeball radius of the target eyeball; the second straight line is determined according to the sixth coordinate information and the third preset mapping table; wherein the third preset mapping table is used to represent the mapping relationship between the sixth coordinate information and the second straight line, and the second straight line is formed from the coordinate origin of the first coordinate system to the pupil center of the target eyeball; the intersection of the second straight line on the first sphere equation is calculated to obtain the second coordinate information.

2. The method according to claim 1, characterized in that The determining the third coordinate information, the fourth coordinate information and the first size information according to the face image includes: Inputting the face image into a pre-trained eye positioning model to obtain the first image, the second image and the eyeball contour; Determining the coordinate position of the first image in the second coordinate system to obtain the third coordinate information; Determine the coordinate position of the second image in the second coordinate system to obtain the fourth coordinate information; Determine the size of the eyeball contour in the second coordinate system to obtain the first size information; wherein, The eye positioning model includes a first convolutional neural network model and a second convolutional neural network model. The first convolutional neural network model is used to extract the first image, the second image and the eyeball contour from the face image, and the second convolutional neural network model is used to determine whether the first image, the second image and the eyeball contour extracted by the first convolutional neural network model meet preset standards.

3. The method according to claim 2, characterized in that The first convolutional neural network model includes a first input layer, M first hidden layers and a first output layer, wherein the M first hidden layers include at least one convolution layer, at least one pooling layer and at least one deconvolution layer; wherein, The convolution kernel size of the convolution layer in the at least one convolution layer is k1*k1 pixels, the step size of the convolution layer in the at least one convolution layer is s1, the activation function of the convolution layer in the at least one convolution layer includes a rectified linear function unit ReLU function, the filter size of the pooling layer in the at least one pooling layer is f1*f1, the step size of the pooling layer in the at least one pooling layer is s2, the convolution kernel size of the deconvolution in the at least one deconvolution layer is k1*k1 pixels, the step size of the deconvolution layer in the at least one deconvolution layer is 1 / s1, the k1 value is an integer of [3,6], the s1 value is an integer of [1,3], the f1 value is an integer of [2,4], the s2 value is 1 or 2, and M is an integer greater than 1; and, The second convolutional neural network model includes a second input layer, N second hidden layers and a second output layer; wherein, The convolution kernel size of each convolution layer in the N-layer second hidden layer is k2*k2 pixels, the step size of each convolution layer in the N-layer second hidden layer is s3, the activation function of each convolution layer in the N-layer second hidden layer includes a leaky corrected linear function unit Leaky ReLU function, the k2 value is 5 or 6, the s3 value is 1 or 2, and N is an integer greater than 1.

4. The method according to any one of claims 1 to 3, characterized in that: The determining the gaze point position of the target user on the display screen according to the first coordinate information and the second coordinate information includes: Construct a straight line from the first coordinate information to the second coordinate information to obtain a third straight line; Rotating the third straight line along a preset direction by a preset threshold to obtain a fourth straight line, wherein the fourth straight line is used to represent the visual axis of the target user looking at the display screen; The intersection point of the fourth straight line on the two-dimensional plane where the display screen is located is determined to obtain the gaze point of the target user on the display screen.

5. A gaze point positioning device, characterized in that: Applied to an electronic device, the electronic device includes a display screen; the device includes a processing unit, and the processing unit is used to: Acquire a face image of a target user and an eyeball radius of a target eyeball of the target user; Determine first coordinate information and second coordinate information according to the face image and the eyeball radius of the target eyeball; wherein the first coordinate information is used to represent the coordinate position of the eyeball center of the target eyeball in a first coordinate system, and the second coordinate information is used to represent the coordinate position of the pupil center of the target eyeball in the first coordinate system, and the first coordinate system is constructed by a two-dimensional plane where the display screen is located and a straight line perpendicular to the two-dimensional plane; Determine a gaze point of the target user on the display screen according to the first coordinate information and the second coordinate information; Wherein, the determining of the first coordinate information and the second coordinate information according to the facial image and the eyeball radius of the target eyeball comprises: determining the third coordinate information, the fourth coordinate information and the first size information according to the facial image; determining the first coordinate information and the second coordinate information according to the third coordinate information, the fourth coordinate information, the first size information and the eyeball radius of the target eyeball; wherein the third coordinate information is used to represent the coordinate position of the first image on the second coordinate system, the first image is used to represent the imaging of the eyeball center of the target eyeball on the facial image, the second coordinate system is constructed by the two-dimensional plane where the facial image is located, and the two-dimensional plane where the facial image is located is parallel to the two-dimensional plane where the display screen is located; the fourth coordinate information is used to represent the coordinate position of the second image on the second coordinate system, and the second image is used to represent the imaging of the pupil center of the target eyeball on the facial image; the first size information is used to represent the size of the eyeball contour of the target eyeball on the facial image on the second coordinate system; Wherein, determining the first coordinate information and the second coordinate information according to the third coordinate information, the fourth coordinate information, the first size information and the eyeball radius of the target eyeball includes: converting the third coordinate information from the second coordinate system to the first coordinate system to obtain fifth coordinate information; converting the fourth coordinate information from the second coordinate system to the first coordinate system to obtain sixth coordinate information; determining the first coordinate information and the second coordinate information according to the fifth coordinate information, the sixth coordinate information, the first size information and the eyeball radius of the target eyeball; Wherein, the determining of the first coordinate information and the second coordinate information according to the fifth coordinate information, the sixth coordinate information, the first size information and the eyeball radius of the target eyeball includes: determining a first straight line according to the fifth coordinate information and a first preset mapping table; wherein the first preset mapping table is used to represent the mapping relationship between the fifth coordinate information and the first straight line, and the first straight line is formed by the coordinate origin of the first coordinate system to the eyeball center of the target eyeball; determining a first distance value according to the first size information and a second preset mapping table; wherein the second preset mapping table is used to represent the mapping relationship between the first size information and the first distance value, and the first The distance value is used to represent the distance from the coordinate origin of the first coordinate system to the center of the target eyeball; the first coordinate information is determined according to the first straight line and the first distance value; the first sphere equation for modeling the target eyeball is determined according to the first coordinate information and the eyeball radius of the target eyeball; the second straight line is determined according to the sixth coordinate information and the third preset mapping table; wherein the third preset mapping table is used to represent the mapping relationship between the sixth coordinate information and the second straight line, and the second straight line is formed from the coordinate origin of the first coordinate system to the pupil center of the target eyeball; the intersection of the second straight line on the first sphere equation is calculated to obtain the second coordinate information.

6. An electronic device, characterized in that: It comprises a processor, a memory, a communication interface and a display screen, wherein the memory stores one or more programs, and the one or more programs are executed by the processor, and the one or more programs include instructions for executing the steps in the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program is operable to cause a computer to execute the method according to any one of claims 1 to 4.

Images