Eyeball tracking mode determination method and device, electronic equipment and storage medium

Abstract

The invention discloses an eyeball tracking mode determination method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring precision detection information, wherein the precision detection information comprises at least one of first precision detection information, second precision detection information and third precision detection information; determining an eyeball tracking mode according to the precision detection information, wherein the eyeball tracking mode comprises a first precision mode and a second precision mode; wherein the first precision mode and the second precision mode adopt different modes to determine the fixation point of the user, the problem that data processing is complex or precision does not reach the standard due to the fact that only one mode can be used for eyeball tracking is solved, the appropriate precision mode is selected according to actual requirements, and the accuracy of eyeball tracking is improved. The high-precision requirement can be met by determining the fixation point through the first precision mode, the determined fixation point is more accurate, the common precision requirement can be met by determining the fixation point through the second precision mode, and when the fixation point is determined, the data processing complexity is low, the data size is small, and the processing speed is high.

Description

Technical Field

[0001] The present invention relates to the technical field of eye tracking, and in particular, to a method, device, electronic device and storage medium for determining an eye tracking mode. Background Art

[0002] There is a certain angle between the visual axis and the optical axis of the human eye, and the angle sizes of different people are different. In eye tracking applications, a general model can calculate the optical axis, and generally, different methods such as calibration procedures and online learning are further used to obtain the visual axis to further improve the accuracy of the gaze direction.

[0003] However, during the eye tracking process, generally only one mode can be used for eye tracking. For example, a single high-precision calibration mode is used to determine the user's fixation point or gaze direction. In some scenarios, the accuracy requirement is relatively low. If the user's fixation point or gaze direction is determined by the calibrated parameters in the same way, it will increase the amount of data processing, increase the complexity of data processing, and reduce the data processing speed. A single eye tracking mode that does not require calibration, although it omits the calibration procedure, often cannot meet the calculation accuracy of the fixation point or gaze direction in some application scenarios. Summary of the Invention

[0004] The present invention provides a method, device, electronic device and storage medium for determining an eye tracking mode to solve the problems of complex data processing or unqualified accuracy caused by only using one mode for eye tracking.

[0005] According to one aspect of the present invention, a method for determining an eye tracking mode is provided, including:

[0006] Obtain accuracy detection information, where the accuracy detection information includes at least one of first accuracy detection information, second accuracy detection information, and third accuracy detection information. Among them, the first accuracy detection information includes the user's eye features and the landmark points in the interaction interface; the second accuracy detection information includes the information of the interaction interface; the third accuracy detection information includes the information set by the user;

[0007] Determine an eye tracking mode according to the accuracy detection information, where the eye tracking mode includes a first accuracy mode and a second accuracy mode;

[0008] Among them, the first accuracy mode and the second accuracy mode determine the user's fixation point and / or gaze direction in different ways.

[0009] According to another aspect of the present invention, an apparatus for determining an eye tracking mode is provided, including:

[0010] A detection information acquisition module, configured to acquire accuracy detection information, where the accuracy detection information includes at least one of first accuracy detection information, second accuracy detection information, and third accuracy detection information. Among them, the first accuracy detection information includes user eye features and landmark points in the interaction interface; the second accuracy detection information includes information of the interaction interface; the third accuracy detection information includes information set by the user.

[0011] A mode determination module, configured to determine an eye tracking mode according to the accuracy detection information, where the eye tracking mode includes a first accuracy mode and a second accuracy mode.

[0012] Among them, the first accuracy mode and the second accuracy mode determine the user's fixation point and / or fixation direction in different ways.

[0013] According to another aspect of the present invention, there is provided an electronic device, where the electronic device includes:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; where

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the eye tracking mode determination method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, there is provided a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the eye tracking mode determination method according to any embodiment of the present invention when executed by a processor.

[0018] The technical solution of the embodiment of the present invention obtains accuracy detection information, where the accuracy detection information includes at least one of first accuracy detection information, second accuracy detection information, and third accuracy detection information. Among them, the first accuracy detection information includes user eye features and landmark points in the interaction interface; the second accuracy detection information includes information of the interaction interface; the third accuracy detection information includes information set by the user. Determine the eye tracking mode according to the accuracy detection information, where the eye tracking mode includes a first accuracy mode and a second accuracy mode. Among them, the first accuracy mode and the second accuracy mode determine the user's fixation point and / or fixation direction in different ways, solving the problem that only one mode can be used for eye tracking, resulting in complex data processing or unqualified accuracy. By analyzing the accuracy detection information to determine the eye tracking mode, the accuracy detection information includes one or more of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information. Determine whether the eye tracking mode is the first accuracy mode or the second accuracy mode through accuracy detection based on the accuracy detection information. The appropriate accuracy mode can be selected according to actual needs. The first accuracy mode and the second accuracy mode determine the user's fixation point and / or fixation direction in different ways. Determining the user's fixation point and / or fixation direction through the first accuracy mode can meet the user's high-precision requirements, and the determined fixation point is more accurate. Determining the user's fixation point and / or fixation direction through the second accuracy mode can meet the user's general accuracy requirements, with lower data processing complexity, less data volume, and faster processing speed when determining the fixation point and / or fixation direction.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 is a flowchart of a method for determining an eye tracking mode provided in Embodiment 1 of the present invention;

[0022] Figure 2 is a flowchart of a method for determining an eye tracking mode provided in Embodiment 2 of the present invention;

[0023] Figure 3 is a display example diagram of angular resolution provided in Embodiment 2 of the present invention;

[0024] Figure 4 is a schematic structural diagram of an eye tracking mode determination device provided according to Embodiment 3 of the present invention;

[0025] Figure 5 is a schematic structural diagram of an electronic device for implementing the eye tracking mode determination method of the embodiment of the present invention. Detailed implementation manners

[0026] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] Embodiment 1

[0029] Figure 1 is a flowchart of an eye tracking mode determination method provided for Embodiment 1 of the present invention. This embodiment is applicable to the situation of selecting an eye tracking mode. This method can be executed by an eye tracking mode determination device, which can be implemented in the form of hardware and / or software, and the eye tracking mode determination device can be configured in an electronic device. As Figure 1 shown, the method includes:

[0030] S101. Obtain accuracy detection information, where the accuracy detection information includes at least one of first accuracy detection information, second accuracy detection information, and third accuracy detection information. Among them, the first accuracy detection information includes user eye features and landmark points in the interaction interface; the second accuracy detection information includes information of the interaction interface; the third accuracy detection information includes information set by the user.

[0031] In this embodiment, the accuracy detection information can be specifically understood as the information used to set the eye tracking mode. The user's eye characteristics can be specifically understood as an image including the user's eye information or a signal including the user's eye information; the fiducial point in the interaction interface can be understood as a prominent point that the user will gaze at, such as a prominent point on the screen, for example, a bright star in the blue sky, a lighthouse in the sea, etc.; the information of the interaction interface can be the image information seen by the user and the object information contained in the image or the object information of the interaction interface described in words, and can also be the information of an object in the real world. The image information can be the coordinates of the objects in the image, the distance between the objects, etc., used to indicate the positional relationship between the objects in the image. The image information can also be the texture, RGB value, etc. of the image. By performing recognition processing on the image information, the coordinates of the objects in the image, the distance between the objects, etc. are obtained; the object can be an icon, text, symbol, etc. corresponding to an application program in the screen image, and can also be any object such as a person, animal, plant, cup, hat, vehicle, etc. in the image display or the real world; the information set by the user can be a mode selected by the user according to his own needs, for example, it can be the first accuracy mode or the second accuracy mode, set by the user.

[0032] The user's eye characteristics can be collected by an image acquisition device. The image acquisition device can be a camera, a video recorder, a DVS event camera, an infrared camera, an optical signal receiver in a microelectromechanical system (MEMS), etc.; the image acquisition device can be set on the eye movement device, or can be set on other terminals or other positions, communicate with the eye movement device and be used in cooperation. The interaction interface can be displayed through a display screen, or can be the real world interface viewed by the user. When the interaction interface is displayed, the information of the interaction interface is correspondingly formed. The display screen can be a screen that displays images, text, etc. for the user when the user performs interactive control through the eye movement device. The image displayed on the display screen includes one or more objects (such as interactive objects, text, people, animals, plants, items, etc.). The object can be displayed in the form of an icon. For example, office software, communication software, entertainment software, etc. Image information is generated according to the image. Among them, the content displayed by the interaction interface at different times and in different scenarios can be different. The user can adaptively set the information through manual setting, voice control, eye control, etc.

[0033] In the embodiments of the present application, a trigger condition can be set when obtaining accuracy detection information, and the accuracy detection information can be obtained when the trigger condition is met; alternatively, a period can be set to obtain the accuracy detection information periodically; or, the user can click a button or other means to trigger the acquisition of the accuracy detection information, and so on. When obtaining the accuracy detection information, three types of information can be obtained simultaneously, namely, the user's eye features and the landmark points in the interaction interface, the information of the interaction interface, and the information set by the user. It is also possible to obtain only one or two of them. For example, only obtain the user's eye features and the landmark points in the interaction interface, or only obtain the information of the interaction interface, or only obtain the information set by the user, and so on.

[0034] S102. Determine the eye tracking mode according to the accuracy detection information. The eye tracking mode includes a first accuracy mode and a second accuracy mode.

[0035] In this embodiment, in the first accuracy mode, a method with higher accuracy is used to determine the user's fixation point; in the second accuracy mode, the user has a lower requirement for accuracy but the calculation amount is greatly reduced. Therefore, a method with lower accuracy can be used to determine the user's fixation point. Among them, different methods are used to determine the user's fixation point and / or fixation direction in the first accuracy mode and the second accuracy mode. Specifically, using different methods to determine the user's fixation point and / or fixation direction can be understood as using different eye tracking algorithms for eye tracking calculation. For example, the first accuracy mode requires a calibration program while the second accuracy mode does not, resulting in different eye tracking accuracies; using different methods to determine the user's fixation point and / or fixation direction can also be understood as using the same eye tracking algorithm but using different calibration programs for calibration. For example, the first accuracy mode uses nine-point calibration while the second accuracy mode uses one-point or three-point calibration. Due to the different numbers of calibration points, the calibration coefficients are different, resulting in different eye tracking accuracies.

[0036] Analyze the accuracy detection information. Since the accuracy detection information may contain one or more types of information, when analyzing the accuracy detection information, it is possible to first analyze the types of information included in the obtained accuracy detection information. If there is only one type of information, accuracy detection can be performed on this information, and the eye tracking mode can be determined according to the detection result or directly according to the accuracy detection information. If there are multiple types of information, the priority of the information can be determined first, and the information with the highest priority can be selected to determine the eye tracking mode. Alternatively, accuracy detection can be performed on all types of information, and the detection results of each type of information can be comprehensively analyzed, and the eye tracking mode can be determined according to the detection results obtained from the comprehensive analysis. When performing accuracy detection based on the user's eye characteristics and the landmark points in the interaction interface, the user's eye fixation point and / or fixation direction can be determined by analyzing the user's eye characteristics, and then the eye tracking mode can be determined according to whether the calculated fixation point and / or fixation direction meet the requirements. When performing accuracy detection based on the information in the interaction interface, the position of the object can be determined according to the object information in the interaction interface, and then the position of the object can be analyzed, and the eye tracking mode can be determined according to the analysis result. When determining the eye tracking mode based on the information set by the user, the information set by the user can be directly used as the eye tracking mode or associated with the eye tracking mode, and the eye tracking mode can be determined according to the association relationship between the mode information and the eye tracking mode after the mode information is determined.

[0037] When the user uses eye tracking to determine the fixation point in different scenarios, different scenarios have different requirements for the eye tracking accuracy mode. Some scenarios require higher accuracy, while some scenarios require lower accuracy but faster processing speed. If the accuracy mode is set to be fixed and unchanged, it will affect the user experience. In the embodiments of the present application, the eye tracking mode can be set to the first accuracy mode or the second accuracy mode according to the accuracy detection information. When the user has a high-precision requirement, the eye tracking mode is set to the first accuracy mode, providing the user with a precise way to determine the fixation point and improving the fixation point accuracy. When the user does not need high precision, the eye tracking mode is set to the second accuracy mode. In this way, when determining the fixation point, the amount of data processed is less, the processing speed is faster, effectively reducing the complexity of data processing, saving the memory occupancy rate and CPU occupancy rate.

[0038] An embodiment of the present invention provides an eye tracking mode determination method, which solves the problems of complex data processing or unqualified accuracy caused by only using one mode for eye tracking. The eye tracking mode is determined by analyzing the accuracy detection information, and the accuracy detection information includes one or more of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information. By analyzing and detecting the accuracy detection information, it is determined whether the eye tracking mode is the first accuracy mode or the second accuracy mode. The appropriate accuracy mode can be selected according to actual needs. The first accuracy mode and the second accuracy mode determine the user's fixation point and / or fixation direction in different ways. Determining the user's fixation point and / or fixation direction through the first accuracy mode can meet the user's high-precision requirements, and the determined fixation point is more accurate. Determining the user's fixation point and / or fixation direction through the second accuracy mode can meet the user's general accuracy requirements, and the data processing complexity is lower, the data volume is smaller, and the processing speed is faster when determining the fixation point and / or fixation direction.

[0039] Embodiment 2

[0040] Figure 2 The flowchart of an eye tracking mode determination method provided by Embodiment 2 of the present invention is shown. This embodiment is refined on the basis of the above embodiment. As Figure 2 shown, the method includes:

[0041] S201. Obtain accuracy detection information, where the accuracy detection information includes at least one of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information. Among them, the first accuracy detection information includes the user's eye features and the landmark points in the interaction interface; the second accuracy detection information includes the information of the interaction interface; the third accuracy detection information includes the information set by the user.

[0042] S202. Determine the eye tracking mode according to the accuracy detection information, where the eye tracking mode includes the first accuracy mode and the second accuracy mode.

[0043] Optionally, when the accuracy detection information is one of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information, this optional embodiment further optimizes the determination of the eye tracking mode according to the accuracy detection information into steps A1-A4:

[0044] A1. When the accuracy detection information is the first accuracy detection information, perform accuracy detection according to the user's eye features and the landmark points in the interaction interface to determine the eye tracking mode; or,

[0045] A2. When the accuracy detection information is the second accuracy detection information, perform accuracy detection according to the information of the interaction interface to determine the eye tracking mode; or,

[0046] A3. When the accuracy detection information is the third accuracy detection information, determine the eye tracking mode according to the information set by the user.

[0047] The accuracy detection information can be one of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information; when the accuracy detection information is the first accuracy detection information, perform accuracy detection based on the user's eye characteristics and the fiducial points in the interaction interface, analyze the eye information of the user in the user's eye characteristics, determine the user's fixation point and / or fixation direction, and determine the eye tracking mode; when the accuracy detection information is the second accuracy detection information, perform accuracy detection based on the information of the interaction interface, analyze the information, analyze the positions, positional relationships, etc. of the objects in the interaction interface, and determine the eye tracking mode; when the accuracy detection information is the third accuracy detection information, directly determine the eye tracking mode according to the information set by the user. Since there is only one eye tracking mode, therefore, when it is determined that the eye tracking mode is the first accuracy mode, at this time, it is necessary to start the first accuracy mode for eye tracking calculation; when it is determined that the eye tracking mode is the second accuracy mode, at this time, the second accuracy mode should be used for eye tracking calculation. Since there is also only one type of information set by the user, therefore, when the information set by the user is the first accuracy mode, the eye tracking mode is the first accuracy mode; when the information set by the user is the second accuracy mode, the eye tracking mode is the second accuracy mode.

[0048] Optionally, as an optional embodiment of this embodiment, this optional embodiment further optimizes performing accuracy detection based on the user's eye characteristics and the fiducial points in the interaction interface and determining the eye tracking mode according to the accuracy detection result to:

[0049] A11. Process the user's eye characteristics according to a pre-determined general model to determine the user's first fixation direction; wherein, the user's eye characteristics are collected when the user is looking at the fiducial points in the interaction interface.

[0050] In this embodiment, the general model can be understood as a deep learning network model for determining the direction of the human eye optical axis. The first fixation direction can be specifically understood as the fixation direction predicted by the model.

[0051] Pre-acquire training samples and perform model training to obtain the general model, input the user's eye characteristics into the general model for processing, and the model processes the user's eye characteristics according to the knowledge learned during the training process and outputs the first fixation direction predicted by the model. Among them, when processing the user's eye characteristics through the model, the user's eye characteristics are collected when the user is looking at the fiducial points in the interaction interface. For example, a prominent fiducial point is displayed in the interaction interface, such as a bright star in the blue sky, etc., and it is assumed that the user has viewed this fiducial point, and then the user's eyes are collected to obtain the user's eye characteristics.

[0052] A12. Determine the user's second gaze direction based on the user's eye characteristics and the position of the landmark point.

[0053] In this embodiment, the second gaze direction is the user's true gaze direction. The position of the landmark point can be determined correspondingly when the landmark point is displayed in the interaction interface. For example, the position of the landmark point is preset, or a random coordinate is generated as the position of the landmark point, and so on.

[0054] Analyze the user's eye characteristics to determine eye information, such as pupil shape (including the size of the long axis of the pupil, the size of the short axis of the pupil, etc.), pupil position (including the center of the pupil, the coordinates of the pupil contour, etc.), the angle of the long axis of the pupil, the position information of the light spot, the position of the iris, the shape of the iris, the position of the eyelid, the position of the eye corner, etc. Further, based on the eye information and the position of the landmark point, determine the user's true gaze direction, denoted as the second gaze direction. The line connecting the center point of the user's eye pupil and the landmark point is the second gaze direction.

[0055] A13. Calculate the first included angle between the first gaze direction and the second gaze direction.

[0056] In this embodiment, the first included angle is the included angle between the first gaze direction and the second gaze direction. The first gaze direction and the second gaze direction can be represented by expressions. For example, the two gaze directions are represented by different expressions in the same coordinate system, and the included angle between the two gaze directions is calculated based on the expressions to obtain the first included angle between the first gaze direction and the second gaze direction.

[0057] A14. When the first included angle is greater than the set angle threshold, determine that the eye tracking mode is the first precision mode;

[0058] A15. When the first included angle is not greater than the set angle threshold, determine that the eye tracking mode is the second precision mode.

[0059] In this embodiment, the angle threshold is preset according to the precision requirement, such as 5°, 10°, etc. Compare the size relationship between the first included angle and the set angle threshold. If the first included angle is greater than the set angle threshold, it means that the user's eye parameter index is not applicable to the general model, and determine that the eye tracking mode corresponding to the precision detection result is the first precision mode, so as to switch to the high-precision mode to perform calibration operations for the user and obtain a high-precision eye tracking effect. If the first included angle is not greater than the set angle threshold, it means that the user's eye parameter index is more in line with the standard general model, and determine that the eye tracking mode corresponding to the precision detection result is the second precision mode. At this time, eye tracking can be performed through the normal precision mode.

[0060] As an optional embodiment of this embodiment, this optional embodiment further optimizes the accuracy detection based on the information of the interaction interface and determines the eye tracking mode as follows:

[0061] A21. Determine the first distance between each object in the image according to the information in the interaction interface.

[0062] In this embodiment, the first distance is the distance between different objects in the interaction interface.

[0063] Analyze the information in the interaction interface. If the information of the interaction interface viewed by the user includes the coordinates of several objects, the first distance between each object can be calculated according to the coordinates; if the information of the interaction interface directly includes the first distance of the object, the first distance between each object can be directly obtained. This step can determine the first distance between any two objects. It can be understood that the objects mentioned in this embodiment can be any objects in the image, such as people, animals, plants, cups, hats, vehicles, etc., and can also be graphic identifiers, such as interaction icons, graphics, numbers, etc.

[0064] A22. Compare each first distance with the set distance threshold to determine the eye tracking mode.

[0065] Among them, the distance threshold can be directly set according to the needs of the actual application scenario, or can be set according to the angular resolution. Comparing each first distance with the distance threshold can be: comparing the largest first distance with the distance threshold, comparing the smallest first distance with the distance threshold, comparing the average value of the first distances with the distance threshold, comparing each first distance with the distance threshold, etc. Correspondingly, determining the eye tracking mode can be: when the largest first distance is greater than the distance threshold, determining the eye tracking mode as the second precision mode, otherwise, determining the eye tracking mode as the first precision mode; when the smallest first distance is greater than the distance threshold, determining the eye tracking mode as the second precision mode, otherwise, determining the eye tracking mode as the first precision mode; when the average value of the first distances is greater than the distance threshold, determining the eye tracking mode as the second precision mode, otherwise, determining the eye tracking mode as the first precision mode; or, when the number of first distances greater than the distance threshold exceeds half or exceeds the set number, determining the eye tracking mode as the second precision mode, otherwise, determining the eye tracking mode as the first precision mode, etc.

[0066] Optionally, a distance threshold is set in advance according to the angular resolution. Different angular resolutions correspond to different distances. Exemplarily, the angular resolution corresponding to the distance threshold is β, and the value of β can be set according to requirements. For example, β = 5° is set. The position of the human eye is set as O. Since the vertical distance from the human eye to the screen is known, the distance from the human eye to point A (coordinate value is known) on the screen can also be calculated. Then, through the angular resolution β, the coordinates of another point B on the screen can be calculated, and the distance between two points (A and B) on the screen can be calculated. This distance is the distance threshold. The included angle between the two straight lines OA and OB is 5°.

[0067] Exemplarily, Figure 3 An exemplary diagram showing the angular resolution is provided. As shown in the figure, the angular resolution between the object 31, the human eye 32, and the object 33 is β. The distance threshold is the distance between the object 31 and the object 33.

[0068] Optionally, when the accuracy detection information includes at least two of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information, this optional embodiment further optimizes the determination of the eye tracking mode according to the accuracy detection information into steps B1 - B3:

[0069] B1. Determine the priority according to the accuracy detection information.

[0070] In this embodiment, the priority can be high priority, medium priority, low priority, etc.; it can also be first level, second level, third level, etc. In this way, it is necessary to pre - define whether the first level of priority is the highest priority or the lowest priority; the embodiments of this application do not limit the division and naming of the priority, as long as the high and low of different priorities can be distinguished. The priorities of different information are set in advance, and the priorities of different information can be the same or different. After obtaining the accuracy detection information, determine all the information included in the accuracy detection information, and determine the priority according to all the information included in the accuracy detection information and its association with the priority.

[0071] Exemplarily, the priority of the first accuracy detection information is medium, the priority of the second accuracy detection information is low, and the priority of the third accuracy detection information is high; or, the priority of the first accuracy detection information is high, the priority of the second accuracy detection information is medium, and the priority of the third accuracy detection information is high.

[0072] B2. Compare each priority, and determine the accuracy detection information with the highest priority as the target information.

[0073] In this embodiment, the target information can be specifically understood as information for determining the eye tracking mode, which is one type of information in the accuracy detection information. By comparing the priorities of each level, the accuracy detection information with the highest priority is selected and used as the target information. Here, the number or type of the target information can be one or more. That is, when there is only one piece of accuracy detection information with the highest priority, the number of target information is one; when there are two or three pieces of accuracy detection information with the highest priority, correspondingly, the number of target information is two or three.

[0074] Exemplarily, taking the case where the accuracy detection information includes the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information at the same time; if the priority of the first accuracy detection information is medium, the priority of the second accuracy detection information is low, and the priority of the third accuracy detection information is high, then the target information is the third accuracy detection information at this time; if the priority of the first accuracy detection information is high, the priority of the second accuracy detection information is medium, and the priority of the third accuracy detection information is high, then the target information is the first accuracy detection information and the second accuracy detection information at this time.

[0075] B3. Determine the eye tracking mode based on the target information.

[0076] The accuracy detection can be performed according to the target information to determine the eye tracking mode, or the eye tracking mode can be directly determined according to the target information. When there is one type of target information, the eye tracking mode is directly determined according to the accuracy detection result or directly according to the target information; when there are multiple types of target information, the eye tracking mode can be determined by comprehensively analyzing the accuracy detection result or the target information.

[0077] As an optional embodiment of this embodiment, this optional embodiment further optimizes determining the eye tracking mode based on the target information to: when the initial eye tracking mode determined based on the target information includes at least one first accuracy mode, determine the eye tracking mode as the first accuracy mode; otherwise, determine the eye tracking mode as the second accuracy mode.

[0078] In this embodiment, the initial eye tracking mode can be specifically understood as the eye tracking mode determined according to one type of target information. The initial eye tracking mode can be the first accuracy mode or the second accuracy mode. In the embodiment of the present application, when determining the final eye tracking mode, first analyze the target information to determine one or more initial eye tracking modes, and then determine the final eye tracking mode based on all the obtained initial eye tracking modes.

[0079] Based on the target information, perform accuracy detection to determine the initial eye tracking mode. If the target information is of one type and its corresponding initial eye tracking mode is the first accuracy mode, determine the eye tracking mode as the first accuracy mode; otherwise, determine the eye tracking mode as the second accuracy mode. If the target information is of multiple types, each target information can be used to determine its corresponding initial eye tracking mode through accuracy detection. If there is at least one first accuracy mode among all the initial eye tracking modes, determine the eye tracking mode as the first accuracy mode. If the initial eye tracking modes corresponding to multiple target information are all the second accuracy mode, then determine the eye tracking mode as the second accuracy mode.

[0080] As an optional embodiment of this embodiment, this optional embodiment further optimizes determining the eye tracking mode based on the target information to:

[0081] C1. When the target information includes the first accuracy detection information, perform accuracy detection based on the user's eye characteristics and the fiducial points in the interaction interface to determine the initial eye tracking mode.

[0082] C2. When the target information includes the second accuracy detection information, perform accuracy detection based on the information in the interaction interface to determine the initial eye tracking mode.

[0083] C3. When the target information includes the third accuracy detection information, determine the initial eye tracking mode according to the information set by the user.

[0084] When the target information includes the first accuracy detection information, perform accuracy detection based on the user's eye characteristics and the fiducial points in the interaction interface, analyze the eye information in the user's eye characteristics, determine the user's fixation point and / or fixation direction, and then determine the initial eye tracking mode according to the accuracy of the determined fixation point and / or fixation direction. When the target information includes the second accuracy detection information, analyze the information in the interaction interface, analyze the positions and positional relationships of various objects in the interaction interface, etc., to determine the initial eye tracking mode. When the target information includes the third accuracy detection information, directly determine the initial eye tracking mode according to the information set by the user.

[0085] In this embodiment, the target information can be one or more of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information. When the target information is multiple, determine the initial eye tracking mode according to each target information in turn.

[0086] As an optional embodiment of this embodiment, this optional embodiment further optimizes performing accuracy detection based on the user's eye characteristics and the fiducial points in the interaction interface and determining the initial eye tracking mode according to the accuracy detection result to:

[0087] D1. Process the user's eye features according to a pre-determined general model to determine the user's third gaze direction; wherein, the user's eye features are collected when the user gazes at the landmark points in the interaction interface.

[0088] In this embodiment, the third gaze direction is the gaze direction predicted by the model.

[0089] Input the user's eye features into a pre-trained general model for processing. The model processes the user's eye features according to the knowledge learned during the training process and outputs the third gaze direction predicted by the model. Among them, when processing the user's eye features through the model, the user's eye features are collected when the user gazes at the landmark points in the interaction interface. The specific collection method can refer to the user image collection process in step A11, and its implementation principle is the same. This application embodiment will not elaborate here.

[0090] D2. Determine the user's fourth gaze direction according to the position of the user's eye features and the landmark points.

[0091] In this embodiment, the fourth gaze direction is the user's true gaze direction. The position of the landmark points can be determined correspondingly when the landmark points are displayed in the interaction interface. For example, the position of the landmark points is pre-set, or a random coordinate is generated as the position of the landmark points, and so on.

[0092] Analyze the user's eye features to determine eye information, such as pupil shape (including the size of the long axis of the pupil, the size of the short axis of the pupil, etc.), pupil position (including the center of the pupil, the coordinates of the pupil contour, etc.), the angle of the long axis of the pupil, spot position information, iris position, iris shape, eyelid position, canthus position, etc. Further, determine the user's true gaze direction according to the eye information and the position of the landmark points, which is recorded as the fourth gaze direction. The line connecting the center point of the user's eye pupil and the landmark point is the fourth gaze direction.

[0093] D3. Calculate the second included angle between the third gaze direction and the fourth gaze direction.

[0094] In this embodiment, the second included angle is the included angle between the third gaze direction and the fourth gaze direction. The third gaze direction and the fourth gaze direction can be represented by expressions. For example, the two gaze directions are represented by different expressions in the same coordinate system, and the included angle between the two gaze directions is calculated based on the expressions to obtain the second included angle between the third gaze direction and the fourth gaze direction.

[0095] D4. When the second included angle is greater than the set angle threshold, determine that the initial eye tracking mode is the first precision mode;

[0096] D5. When the second included angle is not greater than the set angle threshold, determine that the initial eye tracking mode is the second precision mode.

[0097] In this embodiment, the angle threshold is set in advance according to the accuracy requirement. For example, 5°, 10°, etc. Compare the magnitude relationship between the second included angle and the set angle threshold. If the second included angle is greater than the set angle threshold, it indicates that the user's eye parameter index is not applicable to the general model. Determine that the initial eye tracking mode corresponding to the accuracy detection result is the first accuracy mode, so as to switch to the high-precision mode to perform a calibration operation for the user and obtain a high-precision eye tracking effect. If the second included angle is not greater than the set angle threshold, it indicates that the user's eye parameter index conforms to the standard general model. Determine that the eye tracking mode corresponding to the accuracy detection result is the second accuracy mode. At this time, eye tracking can be performed through the normal accuracy mode.

[0098] As an optional embodiment of this embodiment, this optional embodiment further optimizes the accuracy detection according to the information of the interaction interface and determining the initial eye tracking mode as:

[0099] E1. Determine the second distance between each object in the image according to the information in the interaction interface.

[0100] In this embodiment, the second distance is the distance between different objects in the interaction interface.

[0101] Analyze the information in the interaction interface. If the information viewed by the user includes the coordinates of several objects, the distance between each object can be calculated according to the coordinates; if the information directly includes the second distance of the objects, the second distance between each object can be directly obtained. This step can determine the second distance between any two objects. It can be understood that the objects mentioned in this embodiment can be any objects in the image, such as people, animals, plants, cups, hats, vehicles, etc., and can also be graphic identifiers, such as interaction icons, graphics, numbers, etc.

[0102] E2. Compare each second distance with the set distance threshold to determine the initial eye tracking mode.

[0103] Among them, the distance threshold can be directly set according to the needs of the actual application scenario, or can be set according to the angular resolution. Comparing each second distance with the distance threshold can be: comparing the largest second distance with the distance threshold, comparing the smallest second distance with the distance threshold, comparing the average value of the second distances with the distance threshold, comparing each second distance with the distance threshold, and so on. Correspondingly, determining the initial eye tracking mode can be: when the largest second distance is greater than the distance threshold, determining the initial eye tracking mode as the second precision mode; otherwise, determining the initial eye tracking mode as the first precision mode; when the smallest second distance is greater than the distance threshold, determining the initial eye tracking mode as the second precision mode; otherwise, determining the initial eye tracking mode as the first precision mode; when the average value of the second distances is greater than the distance threshold, determining the initial eye tracking mode as the second precision mode; otherwise, determining the initial eye tracking mode as the first precision mode; or, when the number of second distances greater than the distance threshold exceeds half or a set number, determining the initial eye tracking mode as the second precision mode; otherwise, determining the initial eye tracking mode as the first precision mode, and so on.

[0104] Optionally, the distance threshold is set in advance according to the angular resolution, and different angular resolutions correspond to different distances. The specific setting method can refer to the above implementation method of setting the distance threshold according to the angular resolution.

[0105] It should be noted that during the determination of the eye tracking mode, the accuracy detection information can include only one type of information or multiple types of information. The implementation principle of determining the eye tracking mode or the initial eye tracking mode based on the user's eye characteristics and the fiducial points in the interaction interface is the same; similarly, the implementation principle of determining the eye tracking mode or the initial eye tracking mode based on the information of the interaction interface is also the same.

[0106] In the embodiment of the present application, when determining the eye tracking mode, since the accuracy detection information includes one or more of the following: the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information; therefore, when determining the eye tracking mode, it can be determined by one or more of the above information. Exemplarily, the embodiment of the present application includes the following situations when determining the eye tracking mode:

[0107] 1. The accuracy detection information is the first accuracy detection information, and the accuracy detection is performed according to the user's eye characteristics and the fiducial points in the interaction interface to determine the eye tracking mode.

[0108] 2. The accuracy detection information is the second accuracy detection information, and the accuracy detection is performed according to the information of the interaction interface to determine the eye tracking mode.

[0109] 3. The accuracy detection information is the third accuracy detection information, and the eye tracking mode is determined according to the information set by the user.

[0110] 4. The target information is the first accuracy detection information. Accuracy detection is performed based on the user's eye characteristics and the landmark points of the interaction interface to determine the initial eye tracking mode. When the initial eye tracking mode is the first accuracy mode, the eye tracking mode is determined to be the first accuracy mode; otherwise, the eye tracking mode is determined to be the second accuracy mode.

[0111] 5. The target information is the second accuracy detection information. Accuracy detection is performed based on the information of the interaction interface to determine the initial eye tracking mode. When the initial eye tracking mode is the first accuracy mode, the eye tracking mode is determined to be the first accuracy mode; otherwise, the eye tracking mode is determined to be the second accuracy mode.

[0112] 6. The target information is the third accuracy detection information, and the eye tracking mode is determined according to the information set by the user. When the information set by the user is the first accuracy mode, the eye tracking mode is determined to be the first accuracy mode; otherwise, the eye tracking mode is determined to be the second accuracy mode.

[0113] 7. The target information is the first accuracy detection information and the second accuracy detection information. Accuracy detection is performed based on the user's eye characteristics, the landmark points in the interaction interface, and the information of the interaction interface to determine the initial eye tracking mode. When at least one of the initial eye tracking modes includes the first accuracy mode, the eye tracking mode is determined to be the first accuracy mode; otherwise, the eye tracking mode is determined to be the second accuracy mode.

[0114] 8. The target information is the second accuracy detection information and the third accuracy detection information. Accuracy detection is performed based on the information of the interaction interface to determine the initial eye tracking mode. When at least one of the initial eye tracking modes corresponding to the accuracy detection and the initial eye tracking mode corresponding to the information set by the user includes the first accuracy mode, the eye tracking mode is determined to be the first accuracy mode; otherwise, the eye tracking mode is determined to be the second accuracy mode.

[0115] 9. The target information is the first accuracy detection information and the third accuracy detection information. Accuracy detection is performed based on the user's eye characteristics and the landmark points in the interaction interface to determine the initial eye tracking mode. When at least one of the initial eye tracking modes corresponding to the accuracy detection and the initial eye tracking mode corresponding to the information set by the user includes the first accuracy mode, the eye tracking mode is determined to be the first accuracy mode; otherwise, the eye tracking mode is determined to be the second accuracy mode.

[0116] 10. The target information is the first-precision detection information, the second-precision detection information, and the third-precision detection information. Precision detection is performed based on the user's eye features, the landmark points in the interaction interface, and the information of the interaction interface to determine the initial eye-tracking mode. When at least one first-precision mode is included in the initial eye-tracking mode corresponding to the precision detection and the initial eye-tracking mode corresponding to the user-set information, the eye-tracking mode is determined as the first-precision mode; otherwise, the eye-tracking mode is determined as the second-precision mode.

[0117] S203. When the eye-tracking mode is the first-precision mode, determine the visual axis direction based on the pre-determined calibration information and the optical axis direction, and determine the user's fixation point and / or fixation direction based on the visual axis direction.

[0118] When the eye-tracking mode is the first-precision mode, calibration information is required to determine the fixation point at this time. The calibration information is determined when calibrating according to the user's eye information. When tracking the user's line of sight, if the eye-tracking mode is the high-precision mode, determine the optical axis and visual axis conversion parameters based on the pre-determined calibration information, compensate the optical axis information based on the optical axis and visual axis conversion parameters, determine the accurate visual axis direction based on the optical axis direction, and determine the user's fixation point and / or fixation direction based on this visual axis direction during the user tracking process.

[0119] The first-precision mode requires a calibration step. Taking the optical axis direction as the input and the visual axis direction as the output, it can be simply fitted with a simple polynomial model or the parameters can be fitted through a small neural network to obtain the accurate optical axis and visual axis conversion parameters for compensating the optical axis information and obtaining the accurate visual axis direction.

[0120] S204. When the eye-tracking mode is the second-precision mode, determine the optical axis direction as the visual axis direction, and determine the user's fixation point and / or fixation direction based on the visual axis direction.

[0121] The second-precision mode does not require calibration, and the user directly uses the eye-tracking function. When tracking the user's line of sight, if the eye-tracking mode is the second-precision mode, calculate the optical axis direction using a general model, determine the optical axis direction as the visual axis direction, and determine the user's fixation point and / or fixation direction based on this visual axis direction during the user line-of-sight tracking process.

[0122] In this step, when calculating the optical axis direction using the general model, it can be the same general model as that used to determine the first fixation direction.

[0123] Exemplarily, the general model can be trained in the following way:

[0124] 1. The camera internal parameters (internal parameter matrix) can perform image distortion correction so that the image differences between different datasets are only scaling.

[0125] 2. Convert the coordinates of the gaze direction label through the extrinsic parameters of the camera (rotation matrix R and translation vector t) (world coordinate system to camera coordinate system), so that the reference coordinate system of the gaze direction label obtained by the network during training is the camera coordinate system (a three-dimensional world coordinate system (X, Y, Z) presented with the optical center of the camera as the origin, and the origin of the coordinate system is the optical center of the equivalent lens of the camera). This can make the images collected by different camera modules only differ in scale, and eliminate other data interference as much as possible.

[0126] 3. Put the preprocessed images into the deep learning network for training to obtain a trained learning network model.

[0127] Through the previous preprocessing, the images taken by different camera modules can be jointly trained, increasing the training data set and obtaining a general gaze direction model.

[0128] The image of the user's eyes taken is input into the trained learning network model after the conversions in steps 1 and 2 above. Then, based on the extrinsic parameters of the camera, the gaze direction is converted into the world coordinate system to obtain the optical axis direction of the human eye.

[0129] The embodiment of the present invention provides a method for determining an eye tracking mode, which solves the problems of complex data processing, resource waste or unqualified accuracy caused by using only one mode for eye tracking, refines how to determine the eye movement tracking mode according to accuracy detection information, analyzes the user's eye features and landmark points in the interaction interface to determine whether the user is suitable for the general model, and can achieve personalized detection of different users; determines the eye movement tracking mode by comparing the distance between the object in the image and the distance threshold, and can set the eye movement tracking mode according to the actual interaction situation, select the accuracy considering the actual usage situation, and is more flexible and variable; the user can customize the accuracy detection result through the mode information, better meet the user's needs and improve the user experience; the embodiment of the present application provides multiple ways to implement accuracy detection, improving the accuracy of accuracy detection; at the same time, it is set how to determine the eye movement tracking mode when there are multiple accuracy detection information, and the appropriate accuracy mode can be selected according to the actual needs. The first accuracy mode and the second accuracy mode determine the user's fixation point in different ways. Determining the user's fixation point or gaze direction through the first accuracy mode can meet the user's high-precision requirements, and the determined fixation point and / or gaze direction are more accurate. Determining the user's fixation point and / or gaze direction through the second accuracy mode can meet the user's general accuracy requirements, and the data processing complexity is lower, the data volume is smaller, and the processing speed is faster when determining the fixation point and / or gaze direction.

[0130] Embodiment III

[0131] Figure 4The figure is a schematic structural diagram of an eyeball tracking mode determination device provided in Embodiment 3 of the present invention. As Figure 4 shown, the device includes: a detection information acquisition module 41 and a mode determination module 42.

[0132] Among them, the detection information acquisition module 41 is used to acquire accuracy detection information, and the accuracy detection information includes at least one of first accuracy detection information, second accuracy detection information, and third accuracy detection information. Among them, the first accuracy detection information includes user eye characteristics and landmark points in the interaction interface; the second accuracy detection information includes information of the interaction interface; the third accuracy detection information includes information set by the user;

[0133] The mode determination module 42 is used to determine the eyeball tracking mode according to the accuracy detection information, and the eyeball tracking mode includes a first accuracy mode and a second accuracy mode;

[0134] Among them, the first accuracy mode and the second accuracy mode determine the user's fixation point and / or fixation direction in different ways.

[0135] Embodiment of the present invention provides an eyeball tracking mode determination device, which solves the problems of complex data processing or unqualified accuracy caused by only using one mode for eyeball tracking. By analyzing the accuracy detection information, the eyeball tracking mode is determined. The accuracy detection information includes one or more of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information. By performing accuracy detection through the accuracy detection information, it is determined whether the eyeball tracking mode is the first accuracy mode or the second accuracy mode. The appropriate accuracy mode can be selected according to actual needs. The first accuracy mode and the second accuracy mode determine the user's fixation point in different ways. Determining the user's fixation point through the first accuracy mode can meet the user's high-precision requirements, and the determined fixation point and / or fixation direction are more accurate. Determining the user's fixation point and / or fixation direction through the second accuracy mode can meet the user's general accuracy requirements, and the data processing complexity is lower, the data volume is smaller, and the processing speed is faster when determining the fixation point.

[0136] Optionally, the mode determination module 42 includes:

[0137] The first mode determination unit is used to perform accuracy detection according to the user eye characteristics and landmark points in the interaction interface when the accuracy detection information is the first accuracy detection information, and determine the eyeball tracking mode; or,

[0138] The second mode determination unit is used to perform accuracy detection according to the information of the interaction interface when the accuracy detection information is the second accuracy detection information, and determine the eyeball tracking mode; or,

[0139] A third mode determination unit, configured to determine an eye tracking mode according to the information set by the user when the accuracy detection information is third accuracy detection information.

[0140] Optionally, the first mode determination unit is specifically configured to: process the user's eye features according to a predetermined general model to determine the user's first gaze direction; wherein, the user's eye features are collected when the user gazes at a landmark in the interaction interface; determine the user's second gaze direction according to the user's eye features and the position of the landmark; calculate a first included angle between the first gaze direction and the second gaze direction; when the first included angle is greater than a set angle threshold, determine that the eye tracking mode is a first accuracy mode; when the first included angle is not greater than the set angle threshold, determine that the eye tracking mode is a second accuracy mode.

[0141] Optionally, the second mode determination unit is specifically configured to: determine a first distance between each object in the interaction interface according to the information; compare each of the first distances with a set distance threshold to determine the eye tracking mode.

[0142] Optionally, when the accuracy detection information includes at least two of the first accuracy detection information, the second accuracy detection information, and the third accuracy detection information, the mode determination module 42 includes:

[0143] A priority determination unit, configured to determine a priority according to the accuracy detection information;

[0144] A target information determination unit, configured to compare each of the priorities and determine the accuracy detection information with the highest priority as the target information;

[0145] A fourth mode determination unit, configured to determine an eye tracking mode based on the target information.

[0146] Optionally, the second mode determination unit is specifically configured to: when the initial eye tracking mode determined based on the target information includes at least one first accuracy mode, determine that the eye tracking mode is a first accuracy mode; otherwise, determine that the eye tracking mode is a second accuracy mode.

[0147] Optionally, the fourth mode determination unit includes:

[0148] A first mode determination subunit, configured to perform accuracy detection according to the user's eye features and a landmark in the interaction interface to determine an initial eye tracking mode when the target information includes the first accuracy detection information;

[0149] A second mode determination subunit, configured to perform accuracy detection according to the information of the interaction interface to determine an initial eye tracking mode when the target information includes the second accuracy detection information;

[0150] A third mode determination subunit, configured to determine an eye tracking mode according to the information set by the user when the target information includes third-precision detection information.

[0151] Optionally, the precision detection according to the user's eye features and the landmark points in the interaction interface to determine the initial eye tracking mode includes: processing the user's eye features according to a pre-determined general model to determine the user's third gaze direction; wherein, the user's eye features are collected when the user gazes at the landmark points in the interaction interface; determining the user's fourth gaze direction according to the positions of the user's eye features and the landmark points; calculating a second included angle between the third gaze direction and the fourth gaze direction; when the second included angle is greater than a set angle threshold, determining the initial eye tracking mode as the first-precision mode; when the second included angle is not greater than the set angle threshold, determining the initial eye tracking mode as the second-precision mode.

[0152] Optionally, the precision detection according to the information of the interaction interface to determine the initial eye tracking mode includes: determining a second distance between each object in the interaction interface according to the information; comparing each of the second distances with a set distance threshold to determine the initial eye tracking mode.

[0153] Optionally, the apparatus further includes:

[0154] A first fixation point determination module, configured to determine an optical axis direction based on pre-determined calibration information and an optical axis direction when the eye tracking mode is the first-precision mode, and determine the user's fixation point and / or gaze direction based on the optical axis direction;

[0155] A second fixation point determination module, configured to determine the optical axis direction as the optical axis direction when the eye tracking mode is the second-precision mode, and determine the user's fixation point and / or gaze direction based on the optical axis direction.

[0156] The eye tracking mode determination apparatus provided by an embodiment of the present invention can execute the eye tracking mode determination method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0157] Embodiment 4

[0158] Figure 5FIG. 0 shows a schematic structural diagram of an electronic device 50 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0159] As Figure 5 shown, the electronic device 50 includes at least one processor 51 and a memory communicatively connected to the at least one processor 51, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc. The memory stores a computer program executable by the at least one processor. The processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 into the random access memory (RAM) 53. Various programs and data required for the operation of the electronic device 50 can also be stored in the RAM 53. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0160] Multiple components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a magnetic disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0161] The processor 51 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as the eye tracking mode determination method.

[0162] In some embodiments, the method for determining an eye tracking mode may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the method for determining an eye tracking mode described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to execute the method for determining an eye tracking mode by any other suitable means (e.g., by means of firmware).

[0163] The various embodiments of the systems and techniques described above in this document may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] The computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer programs are executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer programs may be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein can be implemented in a computing system including a backend component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a frontend component (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0168] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of high management difficulty and weak business scalability existing in traditional physical hosts and VPS services.

[0169] It should be understood that various forms of processes shown above can be used, with steps reordered, added or deleted. For example, the steps described in the present invention can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0170] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for determining an eye tracking pattern, characterized in that: include: Acquiring accuracy detection information, the accuracy detection information including at least one of first accuracy detection information, second accuracy detection information, and third accuracy detection information, wherein the first accuracy detection information includes user eye features and landmarks in an interactive interface; the second accuracy detection information includes information about the interactive interface; and the third accuracy detection information includes information set by the user; determining an eye tracking mode according to the accuracy detection information, the eye tracking mode including a first accuracy mode and a second accuracy mode; The first precision mode and the second precision mode use different methods to determine the user's gaze point and / or gaze direction.

2. The method according to claim 1, characterized in that The determining of the eye tracking mode according to the accuracy detection information includes: When the accuracy detection information is the first accuracy detection information, performing accuracy detection based on the user's eye features and the landmarks in the interactive interface to determine the eye tracking mode; or When the accuracy detection information is the second accuracy detection information, performing accuracy detection according to the information of the interactive interface to determine the eye tracking mode; or When the accuracy detection information is the third accuracy detection information, the eye tracking mode is determined according to the information set by the user.

3. The method according to claim 2, characterized in that The performing accuracy detection based on the user's eye features and the landmarks in the interactive interface to determine the eye tracking mode includes: Processing the user's eye features according to a predetermined general model to determine a first gaze direction of the user; wherein the user's eye features are collected when the user gazes at a landmark point in the interactive interface; determining a second gaze direction of the user based on the user's eye features and the position of the marker; Calculating a first angle between the first gaze direction and the second gaze direction; When the first angle is greater than a set angle threshold, determining that the eye tracking mode is a first precision mode; When the first angle is not greater than a set angle threshold, the eye tracking mode is determined to be the second precision mode.

4. The method according to claim 2, characterized in that The performing accuracy detection based on the information of the interactive interface and determining the eye tracking mode includes: determining a first distance between objects in the interactive interface according to the information; Each of the first distances is compared with a set distance threshold to determine an eye tracking mode.

5. The method according to claim 1, characterized in that When the accuracy detection information includes at least two of first accuracy detection information, second accuracy detection information, and third accuracy detection information, determining the eye tracking mode according to the accuracy detection information includes: Determining a priority based on the accuracy detection information; Comparing the priorities, and determining the precision detection information with the highest priority as the target information; An eye tracking mode is determined based on the target information.

6. The method according to claim 5, characterized in that The determining of the eye tracking mode based on the target information includes: When the initial eye tracking mode determined based on the target information includes at least one first precision mode, the eye tracking mode is determined to be the first precision mode; otherwise, the eye tracking mode is determined to be the second precision mode.

7. The method according to claim 6, characterized in that Determining an initial eye tracking mode based on the target information includes: When the target information includes the first accuracy detection information, performing accuracy detection based on the user's eye features and the landmarks in the interactive interface to determine an initial eye tracking mode; When the target information includes the second precision detection information, performing precision detection according to the information of the interactive interface to determine an initial eye tracking mode; When the target information includes the third precision detection information, an initial eye tracking mode is determined according to the information set by the user.

8. The method according to claim 7, characterized in that The performing accuracy detection based on the user's eye features and the landmarks in the interactive interface to determine the initial eye tracking mode includes: Processing the user's eye features according to a predetermined general model to determine a third gaze direction of the user; wherein the user's eye features are collected when the user gazes at a landmark point in the interactive interface; determining a fourth gaze direction of the user based on the user's eye features and the position of the marker point; Calculating a second angle between the third gaze direction and the fourth gaze direction; When the second angle is greater than a set angle threshold, determining that the initial eye tracking mode is the first precision mode; When the second angle is not greater than a set angle threshold, the initial eye tracking mode is determined to be the second precision mode.

9. The method according to claim 7, characterized in that The performing accuracy detection based on the information of the interactive interface to determine the initial eye tracking mode includes: determining a second distance between objects in the interactive interface according to the information; Each of the second distances is compared with a set distance threshold to determine an initial eye tracking mode.

10. The method according to claim 1, characterized in that Also includes: When the eye tracking mode is the first precision mode, determining a visual axis direction based on predetermined calibration information and an optical axis direction, and determining a gaze point and / or gaze direction of the user based on the visual axis direction; When the eye tracking mode is the second precision mode, the optical axis direction is determined as the visual axis direction, and the user's gaze point and / or gaze direction is determined based on the visual axis direction.

11. An eye tracking mode determination device, characterized in that: include: a detection information acquisition module, configured to acquire accuracy detection information, the accuracy detection information including at least one of first accuracy detection information, second accuracy detection information, and third accuracy detection information, wherein the first accuracy detection information includes user eye features and landmarks in the interactive interface; the second accuracy detection information includes information about the interactive interface; and the third accuracy detection information includes information set by the user; a mode determination module, configured to determine an eye tracking mode according to the accuracy detection information, wherein the eye tracking mode includes a first accuracy mode and a second accuracy mode; The first precision mode and the second precision mode use different methods to determine the user's gaze point and / or gaze direction.

12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the eye tracking pattern determination method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the eye tracking mode determination method according to any one of claims 1 to 10 when executed.

Images