Method, device, storage medium and electronic device for determining user gaze position

Abstract

The present disclosure relates to a method, device, storage medium, and electronic device for determining a user's gaze position, the method comprising: obtaining a target distance of a target user relative to the display screen; and obtaining a user image of the target user, the global image being an image of a target space in front of the display screen; determining a first space where the eyes of the target user are located from a plurality of pre-set subspaces in the target space according to the target distance and the global image; determining a gaze position of the target user on the display screen corresponding to the first space and the user image of the target user according to a correspondence between the pre-determined subspace, the user image, and the screen coordinates on the display screen. The user's gaze position can be identified according to the collected user image and user distance, and while ensuring the accuracy of the user's gaze position identification, the stability and scope of application of the user's gaze position identification process are improved, thereby improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of human-computer interaction, and in particular to a method, device, storage medium, and electronic device for determining a user's gaze position. Background Art

[0002] With the development of intelligent terminals, there are more and more ways for people to interact with terminals at a distance. Among them, interacting with terminals and controlling terminals through the gaze position of human eyes is an important development direction of long-distance interaction. The gaze position is used to characterize the position of the focus point of the human eye on the display screen when the user watches the terminal display screen. In the related technology, it is usually necessary to capture the biological characteristics of the human eye through customized hardware devices, and then identify the user's gaze position through the main device according to these biological characteristics. For example, an eye tracker with a special infrared emission device emits an infrared signal with a specific waveform, and then the main device performs data analysis through the received feedback signal to determine the user's gaze position. Alternatively, wearable devices bound to the main device, such as smart glasses, collect human eye data to assist the main device in completing the analysis of the human eye to achieve the purpose of eye tracking. The above methods all require customized hardware devices to collect user information, which has a high implementation cost, and there will be compatibility issues between the customized hardware devices and the main device. The stability of the device operation and the adaptability to different application scenarios are poor, which leads to poor user experience. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a method, an apparatus, a storage medium, and an electronic device for determining a user's gaze position.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for determining a user's gaze position is provided, which is applied to a terminal, wherein the terminal includes a display screen, and the method includes:

[0005] Obtaining a target distance of a target user relative to the display screen; and

[0006] Acquire a user image of the target user, the user image comprising: a global image, a head image, and an eye image, the global image being an image of the target space in front of the display screen;

[0007] Determining a first space where the eyes of the target user are located from a plurality of pre-set subspaces in the target space according to the target distance and the global image;

[0008] According to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen, the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user is determined.

[0009] Optionally, determining the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen includes:

[0010] Determine the screen coordinates of the display screen corresponding to the first space and the user image of the target user according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen; wherein the display screen is divided into a plurality of subareas according to the size of the display screen in advance, and the coordinates of a preset position in each subarea are used as the screen coordinates of the subarea;

[0011] The determined screen coordinates are determined as the gaze position of the target user on the display screen.

[0012] Optionally, determining the screen coordinates of the display screen corresponding to the first space and the user image of the target user according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen includes:

[0013] Determine a trained coordinate prediction model corresponding to the first space; wherein a coordinate prediction model is trained for each subspace, and the coordinate prediction model is trained based on the user image and the screen coordinates of the user's gaze position on the display screen;

[0014] The user image of the target user is input into the coordinate prediction model to obtain the screen coordinates of the display screen corresponding to the user image of the target user.

[0015] Optionally, the coordinate prediction model is trained using the following method:

[0016] For each subspace, construct an initial coordinate prediction model;

[0017] Obtain a labeled training dataset for the subspace;

[0018] The training data set includes multiple groups of training data, each group of training data includes: a user image sample of a test user and a screen coordinate label corresponding to the user image sample, and the training data is annotated with the screen coordinate label; when collecting user image samples for any subspace, the eyes of the test user are in the any subspace; the screen coordinate label represents the screen coordinate corresponding to the gaze position of the test user on the display screen when collecting user image samples, and the user image samples include: a global image sample, a head image sample and an eye image sample;

[0019] Inputting the standardized training data set into the initial coordinate prediction model;

[0020] The obtained prediction results are compared with the labeled values, and the parameters of the initial coordinate prediction model are adjusted according to the comparison results until the model converges to obtain a coordinate prediction model that has completed training.

[0021] Optionally, the multiple subspaces are set using the following method:

[0022] The target space is divided into a plurality of subspaces according to the shooting angle range and the effective shooting distance of the image acquisition device.

[0023] Optionally, the multiple subspaces are set using the following method:

[0024] Divide the horizontal shooting angle range of the image acquisition device in the horizontal direction using a first preset angle interval to obtain a plurality of first subspaces in the horizontal direction;

[0025] Using a second preset angle interval, the vertical shooting angle range of the image acquisition device is divided in the vertical direction to obtain a plurality of second subspaces in the vertical direction;

[0026] Using a preset distance interval, the effective shooting distance of the image acquisition device is divided into a third subspace in the far and near directions;

[0027] A subspace obtained by cross-intersecting the first subspace, the second subspace, and the third subspace is determined as the subspace for dividing the target space.

[0028] Optionally, determining, according to the target distance and the global image, a first space where the eyes of the target user are located from a plurality of pre-set subspaces in the target space includes:

[0029] Using a preset image recognition algorithm, identifying a target local image from the global image, wherein the target local image includes the eyes of the target user;

[0030] Determining, according to the image position information of the target partial image in the global image, the plane position information of the eyes of the target user on a first plane, wherein the first plane is parallel to a second plane where the display screen is located;

[0031] The first space where the eyes of the target user are located is determined by combining the plane position information and the target distance.

[0032] Optionally, determining, according to a correspondence between a predetermined subspace, a user image, and screen coordinates on the display screen, a gaze position of the target user on the display screen corresponding to the first space and the user image of the target user includes:

[0033] Determine a trained hierarchical coordinate prediction model corresponding to the first space; wherein a hierarchical coordinate prediction model is trained for each subspace, and the hierarchical coordinate prediction model is trained based on a user image, screen coordinates of a user gaze position on the display screen, and a plurality of pre-set hierarchical levels;

[0034] repeatedly inputting the user image of the target user into the hierarchical coordinate prediction model according to the number of hierarchical levels corresponding to the hierarchical coordinate prediction model, and obtaining the gaze position of the target user on the display screen at the corresponding hierarchical level each time;

[0035] The greater the number of levels, the more sub-areas the display screen is divided into.

[0036] According to a second aspect of an embodiment of the present disclosure, there is provided a device for determining a user's gaze position, which is applied to a terminal, wherein the terminal includes a display screen, and the device includes:

[0037] A distance acquisition module is configured to acquire a target distance of a target user relative to the display screen; and

[0038] An image acquisition module is configured to acquire a user image of the target user, wherein the user image includes: a global image, a head image, and an eye image, wherein the global image is an image of the target space in front of the display screen;

[0039] a space determination module, configured to determine a first space where the eyes of the target user are located from a plurality of pre-set subspaces in the target space according to the target distance and the global image;

[0040] The position determination module is configured to determine the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user based on the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen.

[0041] Optionally, the location determination module is configured to:

[0042] Determine the screen coordinates of the display screen corresponding to the first space and the user image of the target user according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen; wherein the display screen is divided into a plurality of subareas according to the size of the display screen in advance, and the coordinates of a preset position in each subarea are used as the screen coordinates of the subarea;

[0043] The determined screen coordinates are determined as the gaze position of the target user on the display screen.

[0044] Optionally, the location determination module is configured to:

[0045] Determine a trained coordinate prediction model corresponding to the first space; wherein a coordinate prediction model is trained for each subspace, and the coordinate prediction model is trained based on the user image and the screen coordinates of the user's gaze position on the display screen;

[0046] The user image of the target user is input into the coordinate prediction model to obtain the screen coordinates of the display screen corresponding to the user image of the target user.

[0047] Optionally, the coordinate prediction model is trained using the following method:

[0048] For each subspace, construct an initial coordinate prediction model;

[0049] Obtain a labeled training dataset for the subspace;

[0050] The training data set includes multiple groups of training data, each group of training data includes: a user image sample of a test user and a screen coordinate label corresponding to the user image sample, and the training data is annotated with the screen coordinate label; when collecting user image samples for any subspace, the eyes of the test user are in the any subspace; the screen coordinate label represents the screen coordinate corresponding to the gaze position of the test user on the display screen when collecting user image samples, and the user image samples include: a global image sample, a head image sample and an eye image sample;

[0051] Inputting the standardized training data set into the initial coordinate prediction model;

[0052] The obtained prediction results are compared with the labeled values, and the parameters of the initial coordinate prediction model are adjusted according to the comparison results until the model converges to obtain a coordinate prediction model that has completed training.

[0053] Optionally, the multiple subspaces are set using the following method:

[0054] The target space is divided into a plurality of subspaces according to the shooting angle range and the effective shooting distance of the image acquisition device.

[0055] Optionally, the multiple subspaces are set using the following method:

[0056] Divide the horizontal shooting angle range of the image acquisition device in the horizontal direction using a first preset angle interval to obtain a plurality of first subspaces in the horizontal direction;

[0057] Using a second preset angle interval, the vertical shooting angle range of the image acquisition device is divided in the vertical direction to obtain a plurality of second subspaces in the vertical direction;

[0058] Using a preset distance interval, the effective shooting distance of the image acquisition device is divided into a third subspace in the far and near directions;

[0059] A subspace obtained by cross-intersecting the first subspace, the second subspace, and the third subspace is determined as the subspace for dividing the target space.

[0060] Optionally, the space determination module is configured to:

[0061] Using a preset image recognition algorithm, identifying a target local image from the global image, wherein the target local image includes the eyes of the target user;

[0062] Determining, according to the image position information of the target partial image in the global image, the plane position information of the eyes of the target user on a first plane, wherein the first plane is parallel to a second plane where the display screen is located;

[0063] The first space where the eyes of the target user are located is determined by combining the plane position information and the target distance.

[0064] Optionally, the location determination module is configured to:

[0065] Determine a trained hierarchical coordinate prediction model corresponding to the first space; wherein a hierarchical coordinate prediction model is trained for each subspace, and the hierarchical coordinate prediction model is trained based on a user image, screen coordinates of a user gaze position on the display screen, and a plurality of pre-set hierarchical levels;

[0066] repeatedly inputting the user image of the target user into the hierarchical coordinate prediction model according to the number of hierarchical levels corresponding to the hierarchical coordinate prediction model, and obtaining the gaze position of the target user on the display screen at the corresponding hierarchical level each time;

[0067] The greater the number of levels, the more sub-areas the display screen is divided into.

[0068] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method for determining a user's gaze position provided in the first aspect of the present disclosure are implemented.

[0069] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising at least: a processor and a memory for storing executable instructions that can be run on the processor, wherein:

[0070] When the processor is used to run the executable instructions, the executable instructions execute the steps in the method for determining the user's gaze position provided in the first aspect of the present disclosure.

[0071] The technical solution provided by the embodiment of the present disclosure can obtain the target distance of the target user relative to the display screen; and obtain the user image of the target user, the user image including: a global image, a head image and an eye image, the global image being an image of the target space in front of the display screen; according to the target distance and the global image, the first space where the eyes of the target user are located is determined from a plurality of pre-set subspaces in the target space; according to the correspondence between the pre-determined subspace, the user image, and the screen coordinates on the display screen, the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user is determined. The user gaze position can be identified based on the collected user image and user distance, while ensuring the accuracy of the user gaze position identification, the equipment cost is reduced, the stability and scope of application of the user gaze position identification process are improved, and the user experience is improved.

[0072] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0074] Figure 1 is a flow chart showing a method for determining a user's gaze position according to an exemplary embodiment;

[0075] Figure 2a is a schematic diagram of the structure of a terminal according to an exemplary embodiment;

[0076] Figure 2b is a schematic structural diagram of a terminal control device according to an exemplary embodiment;

[0077] Figure 2c is based on Figure 1 A flow chart of a method for determining a gaze position is shown;

[0078] Figure 3 is a flow chart of a method for determining space according to FIG. 2 ;

[0079] Figure 4a is based on Figure 1 A flow chart of a method for determining a gaze position is shown;

[0080] Figure 4b is based on Figure 4a A schematic diagram of a display screen division process is shown;

[0081] Figure 5 is a block diagram of a device for determining a user's gaze position according to an exemplary embodiment;

[0082] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0083] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0084] Before introducing the method for determining the user's gaze position provided by the present disclosure, the application scenario involved in each embodiment of the present disclosure is first introduced. The application scenario includes a terminal, the terminal includes a display screen, and the terminal is provided with an infrared sensor and a camera device. The terminal with a display screen can also use an external infrared sensor and a camera device, then the external infrared sensor and the camera device need to transmit the acquired data to the terminal with the display screen, for example: via Bluetooth or wifi, which will not be repeated here. The infrared sensor and the camera device can be a common infrared sensor and camera on an intelligent hardware terminal. The terminal can be, for example, a mobile terminal such as a smart phone, a tablet computer, a smart watch, a smart bracelet, a PDA (English: Personal Digital Assistant, Chinese: Personal Digital Assistant) equipped with an infrared sensor and a camera device, or it can be a fixed terminal such as a desktop computer and a smart TV.

[0085] Figure 1 is a flow chart showing a method for determining a user's gaze position according to an exemplary embodiment. Figure 1 As shown, the method is applied to the terminal described in the above application scenario, and includes the following steps:

[0086] In step 101, a target distance of a target user relative to the display screen is obtained.

[0087] In step 102, a user image of the target user is obtained.

[0088] The user image includes: a global image, a head image and an eye image, and the global image is a planar image of the target space in front of the display screen.

[0089] In the embodiment of the present disclosure, the method for determining the user's gaze position is described by taking the terminal as a smart TV as an example.

[0090] For example, when the human eye control function of the smart TV is turned on, the smart TV can first obtain monitoring information in the target space in front of the smart TV through the camera device and / or infrared sensor, and select a target user from multiple users based on the monitoring information. The monitoring information can be the user's stay time, the user's distance from the TV, or the length of time the user remains still. Based on this, the target user can be the user who stays in front of the smart TV the longest, the user who is closest to the smart TV, or the user who remains still the longest. After determining the target user, the right to operate the smart TV is granted to the target user.

[0091] For example, the above statement of "granting the operation right of the smart TV to the target user" actually means starting the acquisition process of the target distance and user image corresponding to the target user. The user image includes images of three dimensions: global image, head image and eye image. Among them, the global image is a plane image of the target space in front of the smart TV. It can be understood that the plane image includes the portrait part and the background part of the target space, the eye image can be a high-precision image containing the eyes of the target user, and the head image can include the entire head of the target user. The head image is applied to the following coordinate prediction model, which can provide assistance for the analysis of the sight angle of the target user. The head image can be an image containing the head that is image recognized and then intercepted from the global image. Therefore, the head image, global image and eye image can be collected by the universal front camera in the smart TV or mobile terminal. Preferably, the pixel requirements of the head image and the global image can be set to 12 million pixels, and the aspect ratio of the head image can be 1:1, and the aspect ratio of the global image can be 16:9. On the other hand, since the eye image is the core parameter for determining the user's gaze position, the eye image can be obtained through the high-precision camera included in the camera device. The pixel requirement of the eye image is at least 48 million pixels and the aspect ratio can be 16:9.

[0092] It can be seen that the data required to determine the user's gaze position in the embodiment of the present disclosure can be collected by a common camera device on the terminal, thereby avoiding the problem of increased hardware costs caused by collecting data with customized devices such as infrared transmitting devices or wearable devices.

[0093] In step 102, a first space where the eyes of the target user are located is determined from a plurality of pre-set subspaces in the target space according to the target distance and the global image.

[0094] For example, the following method can be used to pre-set the multiple subspaces: divide the target space into multiple subspaces according to the shooting angle range and effective shooting distance of the image acquisition device. Specifically, the horizontal shooting angle range of the image acquisition device can be divided horizontally using a first preset angle interval to obtain multiple first subspaces in the horizontal direction; the vertical shooting angle range of the image acquisition device can be divided vertically using a second preset angle interval to obtain multiple second subspaces in the vertical direction; the effective shooting distance of the image acquisition device can be divided into a third subspace in the far and near direction using a preset distance interval; and the subspace obtained by the cross-interval of the first subspace, the second subspace, and the third subspace is determined as the subspace into which the target space is divided.

[0095] For example, before the smart TV is put on the market, it is necessary to pass the development stage, for example, a test machine with the same model as the terminal, or a server used for development, to divide the subspace according to the shooting angle range and effective shooting distance of the camera device. For example, for a camera with a horizontal shooting angle range of 120 degrees, a vertical shooting angle range of 60 degrees, and an effective shooting distance of 10 meters, the horizontal shooting angle range can be divided into 9 horizontal measurement spaces h_position (i.e., the above-mentioned first subspace) of 0 degrees, 15 degrees, 30 degrees, 45 degrees, 60 degrees, 75 degrees, 90 degrees, 105 degrees and 120 degrees at an interval of 15 degrees in a clockwise direction. For the vertical shooting angle range, 5 vertical measurement spaces v_position (i.e., the above-mentioned second subspace) of 60 degrees, 45 degrees, 30 degrees, 15 degrees and 0 degrees can be divided from top to bottom at an interval of 15 degrees. For the effective shooting distance, 5 distance positions (i.e., the third subspace) are divided from the effective shooting distance of 10 meters at intervals of 2 meters from far to near. A subspace can be defined by the three parameters h_position, v_position, and distance. For example, 9 h_position, 5 v_position, and 5 distance can be combined to form 225 subspaces, and the corresponding coordinate prediction models are also 225.

[0096] In step 103, the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user is determined according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen.

[0097] For example, during the development stage, a preset number of coordinate points, for example, 100, can be extracted from the countless coordinate points contained in the display screen according to the size of the display screen of the smart TV. These 100 coordinate points are the multiple screen coordinates mentioned above. The screen coordinates here can be interpreted as coordinate points, and these screen coordinates can be evenly distributed on the display screen in a grid shape; the screen coordinates can also be interpreted as a sub-area, that is, the display screen can be directly divided into a preset number of sub-areas, for example, 100 sub-areas, and the area coordinates of each sub-area are used as the screen coordinates here.

[0098] For each subspace, there is a correspondence between the subspace, the user image, and the screen coordinates on the display screen. After determining the first space where the target user's eyes are located and the user image of the target user through steps 101-103, the screen coordinates corresponding to the first space and the user image of the target user can be determined based on the correspondence, and then the gaze position of the target user on the display screen can be determined based on the screen coordinates.

[0099] In addition, after step 103, a corresponding operation can be performed according to the gaze position of the target user on the display screen, for example, a control cursor in the display screen is moved to the gaze position, or a virtual button at the gaze position is selected, and an operation instruction issued by the user through eye movements, voice or a remote control device of a smart TV is executed.

[0100] In summary, the technical solution provided by the embodiments of the present disclosure can obtain the target distance of the target user relative to the display screen; and obtain the user image of the target user, the user image including: a global image, a head image and an eye image, the global image being an image of the target space in front of the display screen; according to the target distance and the global image, the first space where the eyes of the target user are located is determined from a plurality of pre-set subspaces in the target space; according to the correspondence between the pre-determined subspace, the user image, and the screen coordinates on the display screen, the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user is determined. The user gaze position can be identified based on the collected user image and user distance, while ensuring the accuracy of the user gaze position identification, the equipment cost is reduced, the stability and scope of application of the user gaze position identification process are improved, and the user experience is thereby improved.

[0101] For example, Figure 2a is a schematic diagram of a terminal structure according to an exemplary embodiment. Figure 2a As shown, the terminal is a smart TV 200, and the smart TV 200 includes: a control device 210 and a display screen 220. The control device 210 is arranged at a middle position below the smart TV (often referred to as a chin position).

[0102] in addition, Figure 2b is a schematic diagram of a terminal control device according to an exemplary embodiment. Figure 2b As shown, the control device 210 includes: an infrared sensor 211 , a camera 212 , an indicator light 213 and a power button 214 .

[0103] For example, after the user triggers the human eye control function of the terminal 200 through voice commands and control commands of the remote control, the infrared sensor 211 and the camera device 212 can be controlled to turn on at the same time, and the target distance of the target user relative to the display screen 220 can be obtained through the infrared sensor 211, and the user image of the target user can be obtained through the camera device 212 at the same time.

[0104] Figure 2c is based on Figure 1 A flow chart of a method for determining a gaze position is shown in FIG. Figure 2c As shown, step 104 may include:

[0105] In step 1041, the screen coordinates of the display screen corresponding to the first space and the user image of the target user are determined according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen.

[0106] The display screen is divided into a plurality of sub-areas in advance according to the size of the display screen, and the coordinates of a preset position in each sub-area are used as the screen coordinates of the sub-area.

[0107] For example, step 1041 may include: determining a trained coordinate prediction model corresponding to the first space; inputting the user image of the target user into the coordinate prediction model to obtain the screen coordinates of the display screen corresponding to the user image of the target user. A coordinate prediction model is trained for each subspace, and the coordinate prediction model is trained based on the user image and the screen coordinates of the user's gaze position on the display screen.

[0108] For example, the number of coordinate prediction models is consistent with the number of spatial positions. Specifically, when the spatial position of the test user's eyes is determined, the user image sample (and its corresponding screen coordinates) of the test person at this time is obtained. The combination of multiple groups of user image samples and their corresponding screen coordinates obtained based on the same spatial position is used as a training data set. By training the preset neural network model with n different training data sets, the n above-mentioned coordinate prediction models can be obtained. Based on this training process, the smart TV purchased and used by the user already contains n pre-trained coordinate prediction models, and each coordinate prediction model corresponds to a subspace.

[0109] For example, the following steps may be used to train the coordinate prediction model:

[0110] Step a: For each subspace, construct an initial coordinate prediction model.

[0111] For example, each screen coordinate can be taken as a class, and the process of determining the coordinate label corresponding to the user image can be concretized into a multi-classification process (Multi-class classification process), wherein each screen coordinate corresponds to a coordinate label, and each coordinate label corresponds to a class in the multi-classification process. A neural network model can be used as the above-mentioned initial coordinate prediction model to implement the multi-classification process, and the neural network model includes: a convolution layer, a pooling layer, and a fully connected layer. Among them, the convolution layer is used to extract image features from the user image, and then obtain the feature vector matrix of each image. In the embodiment of the present disclosure, different convolution kernels can be used to extract local features for the three types of images, namely global images, head images, and eye images. For example, for an eye image with a 48 million pixel aspect ratio of 16:9, a larger convolution kernel, for example, a pixel matrix containing 100*100, can be used for feature extraction.

[0112] Step b: Obtain a labeled training data set for the subspace.

[0113] Step c: input the standardized training data set into the initial coordinate prediction model.

[0114] The training data set includes multiple groups of training data, each group of training data includes: a user image sample of a test user and a screen coordinate label corresponding to the user image sample, and the training data is annotated with the screen coordinate label; when collecting user image samples for any subspace, the eyes of the test user are in the any subspace; the screen coordinate label represents the screen coordinates corresponding to the gaze position of the test user on the display screen when collecting user image samples, and the user image samples include: a global image sample, a head image sample and an eye image sample.

[0115] For example, before the initial coordinate prediction model is used, a large number of user image samples need to be collected as input training data in the above training data, and the user image samples also include: global image samples, head image samples and eye image samples. In order to ensure the diversity of training data, in the embodiment of the present disclosure, in addition to the above user image samples, each training data also corresponds to a user portrait of the test user, and the user portrait may include: user identification, age, gender and race, etc. When selecting training data, it can be ensured as much as possible that the training data set corresponding to the same subspace contains user image samples corresponding to test users with different user identifications, different ages, different genders and different races. It should be noted that, without considering the user portrait, a subspace corresponds to a coordinate prediction model. After adding the user portrait, a subspace corresponds to multiple coordinate prediction models, where each coordinate prediction model corresponds to a user portrait category. In this case, the method for user gaze position provided in this embodiment also includes the following steps: obtaining a user portrait of the target user according to the user's registration information; classifying the user portrait of the target user to obtain a user portrait category. In the above step 1041, it is necessary to determine the screen coordinates of the display screen corresponding to the first space and the user image of the target user based on the correspondence between the predetermined subspace, user image and user portrait category and the screen coordinates on the display screen.

[0116] It can be seen that the addition of user portraits can increase the pertinence and accuracy of screen coordinate prediction, and further increase the accuracy of determining the user's gaze position.

[0117] For example, for the output end training data in the above training data, the display screen of the smart TV can be divided into 100 screen coordinates in advance and these screen coordinates can be marked on the display screen. Then, the test user is placed in each of the above multiple subspaces to look at these 100 screen coordinates, and the global image, head image and eye image when the test user looks at each screen coordinate are recorded to generate the user image sample, and the screen coordinates specified by the test user are recorded. It should be noted that the above statement of "making the test user in each of the above n subspaces" means that the eyes of the test user are placed in the subspace. And because the human eye is not a machine, it is not necessarily required that the focus point of the test user's gaze falls accurately on each subspace, and a certain range of deviation is allowed between the focus point of the test user's gaze and the subspace. Each screen coordinate corresponds to a unique coordinate label, and the correspondence between the user image sample and the coordinate label can be recorded in the form of (image_id, label). In addition, the screen coordinate can be used to characterize the exact position corresponding to a coordinate point, or it can be used to characterize an area within a preset range around a coordinate point.

[0118] Step d, comparing the obtained prediction results with the labeled values, and adjusting the parameters of the initial coordinate prediction model according to the comparison results until the model converges to obtain a trained coordinate prediction model.

[0119] For example, during the model training process, the number of convolution kernels in the above-mentioned convolution layer can be determined according to the actual training situation, and the convolution kernel parameters can be updated. The pooling layer is used to reduce the dimension of the feature vector matrix to avoid overfitting of the training data. In the embodiment disclosed in the present invention, the maximum value in the sampling area in each image of the above-mentioned three types of images can be used as the sampling eigenvalue, and the dimension of the feature vector matrix of the user image can be reduced by a multiple of 2 by adopting the Max_pooling (maximum pooling operation) technology and a 2*3 sampling matrix. For example, the result of this process can be to reduce the N*N feature vector matrix to (Nm)*(Nm). The fully connected layer also includes a convolution layer. After the feature vector matrix with highly concentrated features is obtained through the convolution operation of the convolution layer and the dimensionality reduction operation of the pooling layer, the highly concentrated feature vector matrix can be further convolved by the softmax function corresponding to the convolution layer contained in the fully connected layer to obtain the final classification result.

[0120] In step 1042, the determined screen coordinates are determined as the gaze position of the target user on the display screen.

[0121] For example, after the user image is used as the input of the coordinate prediction model, the output of the coordinate prediction model is actually the prediction probability corresponding to each of the above-mentioned multiple screen coordinates. The prediction probability is used to characterize the probability that the eyes of the target user in the target image are at each of the above-mentioned screen coordinates. After the user image is input into the target coordinate prediction model, the classification result output by the target coordinate prediction model is obtained, and the classification result is actually the prediction probability of the user image corresponding to each screen coordinate. For example, 100 screen coordinates correspond to 100 prediction probabilities. From these 100 screen coordinates, the screen coordinate with the highest prediction probability is determined, which is the target coordinate corresponding to the user image. The target coordinate is on the display screen of the smart TV, and can clearly indicate the position of the display screen that the target user is currently looking at. In another embodiment of the present disclosure, while determining the gaze position, the user image (or other types of user images obtained at the same time as the user image, for example, multiple consecutive eye images of the target user) can be subjected to image recognition again to determine different eye movements made by the target user. The eye movement may include: blinking, closing eyes, and staring. After the gaze position and the eye movement are determined, the operation instruction can be determined according to the binding relationship among the gaze position, the eye movement and the operation instruction, and then the operation corresponding to the operation instruction is executed through the smart TV.

[0122] For example, after step 1042, the method may further include:

[0123] In step I, the target coordinate label corresponding to the target coordinate and the user image are collected as real-time operation data corresponding to the first sub-subspace.

[0124] In step II, when a preset amount of real-time operation data is collected, the target coordinate prediction model is trained by the preset amount of real-time operation data to update the target coordinate prediction model.

[0125] For example, after the target user performs a corresponding operation at the gaze position on the display screen, the parameters involved in the identification process of the gaze position through the above steps 101 to 104, that is, the target coordinate label, the user image and the first space, can be collected. The target coordinate label and the user image can be saved as the real-time operation data corresponding to the first sub-subspace. As the target user's use of the smart TV progresses, when the real-time operation data corresponding to a subspace reaches a certain number (that is, the above-mentioned preset number), the coordinate prediction model corresponding to the trained subspace can be updated by the preset number of real-time operation data to improve the applicability of the coordinate prediction model for the target user. Alternatively, in another embodiment, a calibration game for collecting the real-time operation data can be provided in the smart TV system. The step of collecting data through the calibration game includes: determining the gaze position of the target user on the display screen through a process similar to the above steps 101 to 103; outputting the gaze position so that the target user determines whether the gaze position is accurate; when the target user determines that the gaze position is accurate, the user image and coordinate label generated in the process are recorded as the real-time operation data corresponding to the subspace where the target user is located. It should be noted that the model updating process in step 106 and the model training process in the above step 104 can be executed by the terminal, or, preferably, executed by a cloud computing platform that is connected to the above terminal and interacts with data, and the execution result (trained model) is sent to the terminal after the execution is completed.

[0126] Figure 3 FIG. 2 is a flow chart of a method for determining space, as shown in FIG. Figure 3 As shown, step 102 may include:

[0127] In step 1021, a target local image is identified from the global image using a preset image recognition algorithm, where the target local image includes the eyes of the target user.

[0128] The target partial image includes the eyes of the target user.

[0129] In step 1022, the plane position information of the eyes of the target user on the first plane is determined according to the image position information of the target partial image in the global image.

[0130] The first plane is parallel to a second plane where the display screen is located, and the distance between the first plane and the second plane is the target distance.

[0131] For example, the image recognition algorithm can be an image recognition model obtained by training a preset machine learning model (e.g., a neural network model, a support vector machine model, and a linear regression model, etc.) through global image samples and local image samples. Using the global image as the input of the image recognition model, the target local image and image position information output by the image recognition model can be obtained. The image position information is used to characterize the position of the target local image in the global image, and the target distance is used to characterize the position of the global image relative to the first plane. Therefore, based on the position of the first plane, the plane position information of the target user's eyes on the first plane can be determined according to the image position information and the target distance.

[0132] In step 1023, the first space where the eyes of the target user are located is determined by combining the plane position information and the target distance.

[0133] For example, the plane position information actually corresponds to the above-mentioned horizontal measurement space h_position (corresponding to the first subspace) and vertical measurement space v_position (corresponding to the second subspace). In step 1023, the h_position parameter and the v_position parameter can be directly represented by the horizontal coordinate value and the vertical coordinate value in the three-dimensional coordinate, or the horizontal coordinate value and the vertical coordinate value can be converted into the current horizontal shooting angle and vertical shooting angle of the camera device with the position of the camera device as the reference coordinate. By combining the horizontal shooting angle and the vertical shooting angle with the target distance, the subspace (i.e., the first space) where the eyes of the target user are located in the target space can be obtained.

[0134] Figure 4a is based on Figure 1 A flow chart of a method for determining a gaze position is shown in FIG. Figure 4a As shown, step 104 may include:

[0135] In step 1043, the trained hierarchical coordinate prediction model corresponding to the first space is determined.

[0136] A hierarchical coordinate prediction model is trained for each subspace, and the hierarchical coordinate prediction model is trained based on the user image, the screen coordinates of the user's gaze position at different levels of the display screen, and multiple pre-set levels.

[0137] In step 1044, the user image of the target user is repeatedly input into the hierarchical coordinate prediction model according to the number of levels corresponding to the hierarchical coordinate prediction model, and each input obtains the gaze position of the target user on the display screen at the corresponding level.

[0138] The greater the number of levels, the more sub-areas the display screen is divided into.

[0139] In another embodiment of the present application, the definition of a level can be added to the screen coordinates. Specifically, the display screen can be divided into multiple areas by a preset iterative division scheme. For example, the display screen is evenly divided into four areas A, B, C and D, and then each area is divided. For example, area A is evenly divided into four sub-areas A1, A2, A3 and A4, and then a smaller area is obtained for each sub-area, until each divided area coincides with the area corresponding to each screen coordinate, and the areas divided in each iteration do not overlap. Among them, each area division defines a level, for example, the four areas A, B, C and D are one level, and the four sub-areas A1, A2, A3 and A4 (also including area B) are one level. The above number of levels is equal to the number of iterations executed in the iterative division scheme.

[0140] For example, Figure 4b is based on Figure 4a A schematic diagram of a display screen division process is shown, wherein: Figure 4b The plane shown in is the plane of the display screen after gridding, and the plane can be divided by the above-mentioned iterative division scheme. Figure 4b As shown, the number of divided levels is 3, wherein the first level includes: A (upper left) area, B (lower left) area, C (upper right) area and D (lower right) area. Figure 4b The outer dotted line in the figure represents the A region. The second level in the A region includes: A1 (upper left) sub-region, A2 (lower left) sub-region, A3 (upper right) sub-region and A4 (lower right) sub-region. Figure 4b The inner dotted line in the figure represents the A1 sub-area. The sub-areas divided at the third level overlap with the screen area represented by the screen coordinates. Figure 4b The screen area is represented by a black solid rectangular area.

[0141] For example, the training process of the hierarchical coordinate prediction model may be as follows: for a certain training data, the user image of user A and the gaze position coordinates of user A at the first level of the display screen (for example, the position coordinates corresponding to area A, which may be the position coordinates of the center position of the area inside the outermost dotted line) are input into the model for the first training. When implemented, the number of categories may be reduced in dimension through the fully connected layer, so that the target dimension is the number of areas divided at the first level (in this embodiment, it may be four areas ABCD, that is, the number is 4); then the user image of user A and the gaze position coordinates of user A at the second level of the display screen (for example, the screen coordinates corresponding to the A1 sub-area, which may be the screen coordinates of the center position of the area inside the innermost dotted line) are input into the model for the second training. When implemented, the number of categories may be reduced in dimension through the fully connected layer, so that the target dimension is the number of areas divided at the second level (in this embodiment, it may be the areas divided again into four areas ABCD, that is, the number is 16); then the user image of user A and the screen coordinates corresponding to the gaze position of user A at the third level of the display screen are input into the model for the third training. The target dimension corresponding to the last level may be the number of minimum unit screen coordinates for dividing the display screen. By training the initial coordinate prediction model using multiple groups of training data similar to the above training data in the above training manner, a trained layer coordinate prediction model can be obtained.

[0142] For example, based on the hierarchical coordinate prediction model trained by the above-mentioned training method, step 1044 may include: outputting the user image of the target user to the hierarchical coordinate prediction model, obtaining the screen coordinates of the target user on the display screen at the first level (for example, the screen coordinates corresponding to area A), and then determining the first gaze position, outputting the user image of the target user to the hierarchical coordinate prediction model again, obtaining the screen coordinates of the target user on the display screen at the second level (for example, the screen coordinates corresponding to area B2), and then determining the second gaze position, and using the second gaze position as the final user gaze position. This process can be understood as, after the first input process, obtaining an approximate range of the target user's gaze position, and then narrowing the range layer by layer in subsequent input processes, and finally obtaining an accurate range. After each gaze position is obtained, the operating cursor on the display screen can be moved to the gaze position. Figure 4b For example, after obtaining the user image of the target user and inputting the user image into the hierarchical coordinate prediction model, the hierarchical coordinate prediction model will output three screen coordinates in sequence, namely, Figure 4b The screen coordinates of a preset position (such as the center position) of the screen area within the outermost dotted line, Figure 4b The screen coordinates of the preset position (such as the center position) of the screen area within the innermost dotted line and the screen coordinates of the user's actual gaze position ( Figure 4b The black rectangular area in the figure represents the screen coordinates corresponding to the screen area. The movement of the operation cursor will pass through these three screen coordinates in sequence.

[0143] It can be seen that in this way, the movement of the operation cursor will include a gradual movement trajectory, and the operation cursor will not jump directly from the initial position to the user's gaze position, thereby improving the aesthetics of the interaction and thus improving the user experience. In addition, during the training process, the prediction probability of each type of screen label is iteratively calculated layer by layer according to the hierarchical label corresponding to each output end training data, which can also improve the efficiency of the coordinate prediction model convergence and reduce the parallelism of the coordinate prediction model training process.

[0144] In summary, the technical solution provided by the embodiments of the present disclosure can obtain the target distance of the target user relative to the display screen; and obtain the user image of the target user, the user image including: a global image, a head image and an eye image, the global image being an image of the target space in front of the display screen; according to the target distance and the global image, the first space where the eyes of the target user are located is determined from a plurality of pre-set subspaces in the target space; according to the correspondence between the pre-determined subspace, the user image, and the screen coordinates on the display screen, the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user is determined. The user gaze position can be identified based on the collected user image and user distance, while ensuring the accuracy of the user gaze position identification, the equipment cost is reduced, the stability and scope of application of the user gaze position identification process are improved, and the user experience is thereby improved.

[0145] Figure 5 is a block diagram of a device for determining a user's gaze position according to an exemplary embodiment. Figure 5 As shown, the device 500 is applied to the terminal described in the above application scenario, and includes:

[0146] The distance acquisition module 510 is configured to acquire a target distance of a target user relative to the display screen; and

[0147] The image acquisition module 520 is configured to acquire a user image of the target user, the user image including: a global image, a head image and an eye image, the global image being an image of the target space in front of the display screen;

[0148] A space determination module 530 is configured to determine a first space where the eyes of the target user are located from a plurality of pre-set subspaces in the target space according to the target distance and the global image;

[0149] The position determination module 540 is configured to determine the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user based on the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen.

[0150] Optionally, the location determination module 540 is configured to:

[0151] Determine the screen coordinates of the display screen corresponding to the first space and the user image of the target user according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen; wherein the display screen is divided into a plurality of subareas according to the size of the display screen in advance, and the coordinates of a preset position in each subarea are used as the screen coordinates of the subarea;

[0152] The determined screen coordinates are determined as the gaze position of the target user on the display screen.

[0153] Optionally, the location determination module 540 is configured to:

[0154] Determine a trained coordinate prediction model corresponding to the first space; wherein a coordinate prediction model is trained for each subspace, and the coordinate prediction model is trained based on the user image and the screen coordinates of the user's gaze position on the display screen;

[0155] The user image of the target user is input into the coordinate prediction model to obtain the screen coordinates of the display screen corresponding to the user image of the target user.

[0156] Optionally, the coordinate prediction model is trained using the following method:

[0157] For each subspace, construct an initial coordinate prediction model;

[0158] Obtain a labeled training dataset for the subspace;

[0159] The training data set includes multiple groups of training data, each group of training data includes: a user image sample of a test user and a screen coordinate label corresponding to the user image sample, and the training data is annotated with the screen coordinate label; when collecting user image samples for any subspace, the eyes of the test user are in the any subspace; the screen coordinate label represents the screen coordinate corresponding to the gaze position of the test user on the display screen when collecting the user image sample, and the user image sample includes: a global image sample, a head image sample and an eye image sample;

[0160] Inputting the standardized training data set into the initial coordinate prediction model;

[0161] The obtained prediction results are compared with the labeled values, and the parameters of the initial coordinate prediction model are adjusted according to the comparison results until the model converges to obtain a coordinate prediction model that has completed training.

[0162] Optionally, the multiple subspaces are set using the following method:

[0163] The target space is divided into a plurality of subspaces according to the shooting angle range and the effective shooting distance of the image acquisition device.

[0164] Optionally, the multiple subspaces are set using the following method:

[0165] Divide the horizontal shooting angle range of the image acquisition device in the horizontal direction using a first preset angle interval to obtain a plurality of first subspaces in the horizontal direction;

[0166] Using a second preset angle interval, the vertical shooting angle range of the image acquisition device is divided in the vertical direction to obtain a plurality of second subspaces in the vertical direction;

[0167] Using a preset distance interval, the effective shooting distance of the image acquisition device is divided into a third subspace in the far and near directions;

[0168] The subspace obtained by cross-intersecting the first subspace, the second subspace, and the third subspace is determined as the subspace for dividing the target space.

[0169] Optionally, the space determination module 530 is configured to:

[0170] Using a preset image recognition algorithm, a target local image is identified from the global image, where the target local image includes the eyes of the target user;

[0171] Determining, according to the image position information of the target partial image in the global image, the plane position information of the eye of the target user on a first plane, wherein the first plane is parallel to a second plane where the display screen is located;

[0172] The first space where the eyes of the target user are located is determined by combining the plane position information and the target distance.

[0173] Optionally, the location determination module 540 is configured to:

[0174] Determine a trained hierarchical coordinate prediction model corresponding to the first space; wherein a hierarchical coordinate prediction model is trained for each subspace, and the hierarchical coordinate prediction model is trained based on a user image, screen coordinates of a user gaze position on the display screen, and a plurality of pre-set hierarchical levels;

[0175] According to the number of levels corresponding to the hierarchical coordinate prediction model, the user image of the target user is repeatedly input into the hierarchical coordinate prediction model, and each time the user image is input, the gaze position of the target user on the display screen at the corresponding level is obtained;

[0176] The greater the number of levels, the more sub-areas the display screen is divided into.

[0177] In summary, the technical solution provided by the embodiments of the present disclosure can obtain the target distance of the target user relative to the display screen; and obtain the user image of the target user, the user image including: a global image, a head image and an eye image, the global image being an image of the target space in front of the display screen; according to the target distance and the global image, the first space where the eyes of the target user are located is determined from a plurality of pre-set subspaces in the target space; according to the correspondence between the pre-determined subspace, the user image, and the screen coordinates on the display screen, the gaze position of the target user on the display screen corresponding to the first space and the user image of the target user is determined. The user gaze position can be identified based on the collected user image and user distance, while ensuring the accuracy of the user gaze position identification, the equipment cost is reduced, the stability and scope of application of the user gaze position identification process are improved, and the user experience is thereby improved.

[0178] Figure 6 6 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 600 may be, for example, a smart TV, a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0179] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 606 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0180] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method for determining the user's gaze position. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 606 and the processing component 602. The processor may perform any of the above-mentioned methods for determining the user's gaze position.

[0181] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0182] The power component 606 provides power to the various components of the electronic device 600. The power component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0183] The multimedia component 606 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 606 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0184] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0185] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0186] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0187] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0188] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned method for determining the user's gaze position.

[0189] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to complete the above method of determining the user's gaze position. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0190] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above method for determining a user's gaze position when executed by the programmable device.

[0191] In summary, the present invention can identify the user's gaze position based on the collected user image and user distance, while ensuring the accuracy of user gaze position identification, reducing equipment costs, improving the stability and applicability of the user gaze position identification process, and thus improving the user experience.

[0192] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0193] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining a user's gaze position, characterized in that: Applied to a terminal, the terminal includes a display screen, and the method includes: Obtaining a target distance of a target user relative to the display screen; and Acquire a user image of the target user, the user image comprising: a global image, a head image, and an eye image, the global image being an image of the target space in front of the display screen; Determining a first space where the eyes of the target user are located from a plurality of pre-set subspaces in the target space according to the target distance and the global image; Determine, according to a correspondence between a predetermined subspace, a user image, and screen coordinates on the display screen, a gaze position of the target user on the display screen corresponding to the first space and the user image of the target user; The determining, according to the target distance and the global image, a first space where the eyes of the target user are located from a plurality of pre-set subspaces in the target space includes: Using a preset image recognition algorithm, identifying a target local image from the global image, wherein the target local image includes the eyes of the target user; Determining, according to the image position information of the target partial image in the global image, the plane position information of the eyes of the target user on a first plane, wherein the first plane is parallel to a second plane where the display screen is located; The first space where the eyes of the target user are located is determined by combining the plane position information and the target distance.

2. The method according to claim 1, characterized in that The determining, based on the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen, a gaze position of the target user on the display screen corresponding to the first space and the user image of the target user, comprises: Determine the screen coordinates of the display screen corresponding to the first space and the user image of the target user according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen; wherein the display screen is divided into a plurality of subareas according to the size of the display screen in advance, and the coordinates of a preset position in each subarea are used as the screen coordinates of the subarea; The determined screen coordinates are determined as the gaze position of the target user on the display screen.

3. The method according to claim 2, characterized in that The determining, according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen, the screen coordinates of the display screen corresponding to the first space and the user image of the target user comprises: Determine a trained coordinate prediction model corresponding to the first space; wherein a coordinate prediction model is trained for each subspace, and the coordinate prediction model is trained based on the user image and the screen coordinates of the user's gaze position on the display screen; The user image of the target user is input into the coordinate prediction model to obtain the screen coordinates of the display screen corresponding to the user image of the target user.

4. The method according to claim 1, characterized in that: The coordinate prediction model is trained using the following method: For each subspace, construct an initial coordinate prediction model; Obtain a labeled training dataset for the subspace; The training data set includes multiple groups of training data, each group of training data includes: a user image sample of a test user and a screen coordinate label corresponding to the user image sample, and the training data is annotated with the screen coordinate label; when collecting user image samples for any subspace, the eyes of the test user are in the any subspace; the screen coordinate label represents the screen coordinate corresponding to the gaze position of the test user on the display screen when collecting user image samples, and the user image samples include: a global image sample, a head image sample and an eye image sample; Inputting the standardized training data set into the initial coordinate prediction model; The obtained prediction results are compared with the labeled values, and the parameters of the initial coordinate prediction model are adjusted according to the comparison results until the model converges to obtain a coordinate prediction model that has completed training.

5. The method according to claim 1, characterized in that The multiple subspaces are set using the following method: The target space is divided into a plurality of subspaces according to the shooting angle range and the effective shooting distance of the image acquisition device.

6. The method according to claim 5, characterized in that The multiple subspaces are set using the following method: Divide the horizontal shooting angle range of the image acquisition device in the horizontal direction using a first preset angle interval to obtain a plurality of first subspaces in the horizontal direction; Using a second preset angle interval, the vertical shooting angle range of the image acquisition device is divided in the vertical direction to obtain a plurality of second subspaces in the vertical direction; Using a preset distance interval, the effective shooting distance of the image acquisition device is divided into a third subspace in the far and near directions; A subspace obtained by cross-intersecting the first subspace, the second subspace, and the third subspace is determined as the subspace for dividing the target space.

7. The method according to claim 1, characterized in that Determining a gaze position of the target user on the display screen corresponding to the first space and the user image of the target user according to a correspondence relationship between a predetermined subspace, a user image, and screen coordinates on the display screen includes: Determine a trained hierarchical coordinate prediction model corresponding to the first space; wherein a hierarchical coordinate prediction model is trained for each subspace, and the hierarchical coordinate prediction model is trained based on a user image, screen coordinates of a user gaze position on the display screen, and a plurality of pre-set hierarchical levels; repeatedly inputting the user image of the target user into the hierarchical coordinate prediction model according to the number of hierarchical levels corresponding to the hierarchical coordinate prediction model, and obtaining the gaze position of the target user on the display screen at the corresponding hierarchical level each time; The greater the number of levels, the more sub-areas the display screen is divided into.

8. A device for determining a user's gaze position, characterized in that: Applied to a terminal, the terminal includes a display screen, and the device includes: A distance acquisition module, configured to acquire a target distance of a target user relative to the display screen; An image acquisition module is configured to acquire a user image of the target user, wherein the user image includes: a global image, a head image, and an eye image, wherein the global image is an image of the target space in front of the display screen; a space determination module, configured to determine a first space where the eyes of the target user are located from a plurality of pre-set subspaces in the target space according to the target distance and the global image; a position determination module configured to determine a gaze position of the target user on the display screen corresponding to the first space and the user image of the target user according to a correspondence between a predetermined subspace, a user image, and screen coordinates on the display screen; The space determination module is configured to: Using a preset image recognition algorithm, identifying a target local image from the global image, wherein the target local image includes the eyes of the target user; Determining, according to the image position information of the target partial image in the global image, the plane position information of the eyes of the target user on a first plane, wherein the first plane is parallel to a second plane where the display screen is located; The first space where the eyes of the target user are located is determined by combining the plane position information and the target distance.

9. The device according to claim 8, characterized in that The location determination module is configured to: Determine the screen coordinates of the display screen corresponding to the first space and the user image of the target user according to the correspondence between the predetermined subspace, the user image, and the screen coordinates on the display screen; wherein the display screen is divided into a plurality of subareas according to the size of the display screen in advance, and the coordinates of a preset position in each subarea are used as the screen coordinates of the subarea; The determined screen coordinates are determined as the gaze position of the target user on the display screen.

10. The device according to claim 9, characterized in that The location determination module is configured to: Determine a trained coordinate prediction model corresponding to the first space; wherein a coordinate prediction model is trained for each subspace, and the coordinate prediction model is trained based on the user image and the screen coordinates of the user's gaze position on the display screen; The user image of the target user is input into the coordinate prediction model to obtain the screen coordinates of the display screen corresponding to the user image of the target user.

11. The device according to claim 8, characterized in that The coordinate prediction model is trained using the following method: For each subspace, construct an initial coordinate prediction model; Obtain a labeled training dataset for the subspace; The training data set includes multiple groups of training data, each group of training data includes: a user image sample of a test user and a screen coordinate label corresponding to the user image sample, and the training data is annotated with the screen coordinate label; when collecting user image samples for any subspace, the eyes of the test user are in the any subspace; the screen coordinate label represents the screen coordinate corresponding to the gaze position of the test user on the display screen when collecting user image samples, and the user image samples include: a global image sample, a head image sample and an eye image sample; Inputting the standardized training data set into the initial coordinate prediction model; The obtained prediction results are compared with the labeled values, and the parameters of the initial coordinate prediction model are adjusted according to the comparison results until the model converges to obtain a coordinate prediction model that has completed training.

12. The device according to claim 8, characterized in that The multiple subspaces are set using the following method: The target space is divided into a plurality of subspaces according to the shooting angle range and the effective shooting distance of the image acquisition device.

13. The device according to claim 12, characterized in that The multiple subspaces are set using the following method: Divide the horizontal shooting angle range of the image acquisition device in the horizontal direction using a first preset angle interval to obtain a plurality of first spaces in the horizontal direction; Using a second preset angle interval, the vertical shooting angle range of the image acquisition device is divided in the vertical direction to obtain a plurality of second subspaces in the vertical direction; Using a preset distance interval, the effective shooting distance of the image acquisition device is divided into a third subspace in the far and near directions; The subspace obtained by cross-intersecting the first space, the second subspace, and the third subspace is determined as the subspace for dividing the target space.

14. The device according to claim 8, characterized in that The location determination module is configured to: Determine a trained hierarchical coordinate prediction model corresponding to the first space; wherein a hierarchical coordinate prediction model is trained for each subspace, and the hierarchical coordinate prediction model is trained based on a user image, screen coordinates of a user gaze position on the display screen, and a plurality of pre-set hierarchical levels; repeatedly inputting the user image of the target user into the hierarchical coordinate prediction model according to the number of hierarchical levels corresponding to the hierarchical coordinate prediction model, and obtaining the gaze position of the target user on the display screen at the corresponding hierarchical level each time; The greater the number of levels, the more sub-areas the display screen is divided into.

15. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

16. An electronic device, characterized in that: The electronic device comprises at least: a processor and a memory for storing executable instructions that can be run on the processor, wherein: When the processor is used to run the executable instructions, the executable instructions execute the steps in the method for determining the user's gaze position provided in any one of claims 1 to 7.

Images