An eye movement interaction method, device and electronic equipment

Abstract

The application provides an eye movement interaction method, device and electronic equipment. When it is determined according to eye lid data that the user's eyes are in an abnormal open-eye state, the visual depth of the user's eyes is adjusted, and a plurality of objects to be selected under the adjusted visual depth are obtained. When it is determined according to eye lid data that the user's eyes are in a normal open-eye angle and the gaze direction has not changed within a preset time period, the gaze point of the user's eyes in the gaze direction within the preset time period before the current time is obtained. According to the gaze point of the user's eyes in the gaze direction, the target object gazed at by the user's eyes is determined, the visual depth is adjusted by using the eye lid data, and in the case that the plurality of objects to be selected under the adjusted visual depth are obtained, the selection of the target object gazed at by the user is further performed. The triggering condition of the program for adjusting the visual depth is determined, and the program for adjusting the visual depth is prevented from being mistakenly triggered in the process in which the user normally watches images by using a wearable device such as a VR device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of eye movement interaction, in particular to an eye movement interaction method, device and electronic equipment. BACKGROUND

[0002] Currently, when a user uses a consumer electronic level virtual reality (VR) / augmented reality (AR) device, and wearable devices such as smart glasses, it is easy to trigger a program for adjusting visual depth by mistake, thereby reducing the user experience of using the wearable device. SUMMARY

[0003] To solve the above problems, the purpose of the embodiments of the present application is to provide an eye movement interaction method, device and electronic equipment.

[0004] In a first aspect, the embodiments of the present application provide an eye movement interaction method, comprising:

[0005] obtaining eyelid data and a gaze direction of a current user's eye;

[0006] when it is determined according to the eyelid data that the user's eye is in an abnormal open-eye state, adjusting the visual depth of the user's eye, and obtaining a plurality of objects to be selected under the adjusted visual depth;

[0007] when it is determined according to the eyelid data that the user's eye is in a normal open-eye angle and the gaze direction has not changed within a preset time period before the current time, determining the gaze direction as the fixation direction of the eye;

[0008] obtaining a fixation point of the user's eye in the fixation direction within a preset time period before the current time;

[0009] determining a target object of the user's eye fixation from the plurality of objects to be selected according to the fixation point of the user's eye in the fixation direction.

[0010] In a second aspect, the embodiments of the present application further provide an eye movement interaction device, comprising:

[0011] a first obtaining module configured to obtain eyelid data and a gaze direction of a current user's eye;

[0012] an adjusting module configured to, when it is determined according to the eyelid data that the user's eye is in an abnormal open-eye state, adjust the visual depth of the user's eye, and obtain a plurality of objects to be selected under the adjusted visual depth;

[0013] determine the gaze direction as the gaze direction of the eye when it is determined according to the eyelid data that the user's eye is in a normal open eye angle and the gaze direction has not changed in a preset time period before the current time;

[0014] the second obtaining module is configured to obtain a gaze point of the user's eye in the gaze direction in a preset time period before the current time;

[0015] the processing module is configured to determine, according to the gaze point of the user's eye in the gaze direction, a target object gazed at by the user's eye from the plurality of objects to be selected.

[0016] In a third aspect, an embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is run by a processor, steps of the method in the first aspect are executed.

[0017] In a fourth aspect, an embodiment of the present application further provides an electronic device, and the electronic device includes a memory, a processor and one or more programs. The one or more programs are stored in the memory and configured to be executed by the processor to execute steps of the method in the first aspect.

[0018] The above-mentioned first aspect to the fourth aspect of the embodiments of the present application provide the following solutions. When it is determined according to the eyelid data that the user's eye is in an abnormal open eye state, the visual depth of the user's eye is adjusted, and a plurality of objects to be selected under the adjusted visual depth are obtained. When it is determined according to the eyelid data that the user's eye is in a normal open eye angle and the gaze direction has not changed in a preset time period, the gaze direction is determined as the gaze direction of the eye. A gaze point of the user's eye in the gaze direction in a preset time period before the current time is obtained. According to the gaze point of the user's eye in the gaze direction, a target object gazed at by the user's eye is determined from the plurality of objects to be selected. Compared with the manner in which the target object gazed at by the user's eye cannot be accurately determined in the related art, the visual depth is adjusted by using the eyelid data, and the selection of the target object gazed at by the user is performed in the case where the plurality of objects to be selected under the adjusted visual depth are obtained. The triggering condition of the program of adjusting the visual depth is determined, the program of adjusting the visual depth is prevented from being triggered by mistake in the process in which the user normally watches an image by using a VR device or the like wearable device, and the use experience of the user in using the wearable device is improved.

[0019] In order to make the above objectives, characteristics and advantages of the present application more apparent, clear and easy to understand, the following preferred embodiments are specifically described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

[0021] Figure 1 A flow chart of an eye movement interaction method provided by Embodiment 1 of the present application is shown;

[0022] Figure 2 A structural schematic diagram of an eye movement interaction device provided by Embodiment 2 of the present application is shown;

[0023] Figure 3 A structural schematic diagram of an electronic device provided by Embodiment 3 of the present application is shown. DETAILED DESCRIPTION

[0024] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0025] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0026] In the present application, unless otherwise specifically defined and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0027] Currently, when a user uses a wearable device such as a consumer electronics level virtual reality (VR) / augmented reality (AR) device or smart glasses, a program for adjusting visual depth is easily triggered by mistake, which reduces the user experience of using the wearable device.

[0028] With the rapid development of VR / AR, people have higher and higher requirements for quickly and accurately selecting objects in a scene. Among many technical solutions, eye selection has attracted attention from the academic and industrial communities. However, current eye selection technology faces three major challenges: 1) false touch caused by the eye tracking device being always turned on; 2) low accuracy caused by natural eye jitter; and 3) occluded objects cannot be selected because of a three-dimensional scene.

[0029] The eye interaction method, device and electronic equipment provided in the present application draw inspiration from the behavior of people squinting their eyes to see objects far away, and use the control ability of people on the angle of opening and closing of the eyelids to add a new interaction dimension. For the above three problems of eye selection, the solutions provided in the present application are as follows: 1) with the help of the concept of visual depth, the nearest depth is set to a state in which all objects cannot be selected, so that the user can normally look at the scene without false touch. When the user needs to select, the user only needs to slightly open the eyes to move the visual depth backward, so that the objects in the scene are in a state that can be selected; 2) by designing an algorithm, the target is enlarged, and a probability method is used to determine whether the user's line of sight wants to select the object, so as to reduce the influence of natural eye jitter on the accuracy of selection; and 3) by adjusting the visual depth, the occluded objects are displayed, and the probability of the objects being selected by the user is calculated, so as to solve the problem of selecting occluded objects in a three-dimensional scene.

[0030] Therefore, in order to implement the eye interaction method, device and electronic equipment provided in the present application, the following two parts of operation content are described in the following embodiments: visual depth adjustment and gaze-based target selection. First, the visual depth of the current scene is adjusted by opening and squinting the eyes, so that the objects with a position depth less than the current visual depth are turned into semi-transparent and unselectable, and the selection target is ensured not to be occluded. Then, whether the user is in a gaze state is determined by using the line of sight landing point data in a time sliding window, and the selection probability of each selectable object based on the visual depth and the line of sight two-dimensional plane landing point is calculated respectively, and then the probabilities are weighted and summed to obtain the final selection probability of each object, and finally the object with the highest selection probability is selected as the selection target.

[0031] Based on this, the following embodiments of this application propose an eye-tracking interaction method, device, and electronic device. When it is determined from eyelid data that the user's eyes are in an abnormal open state, the visual depth of the user's eyes is adjusted, and multiple selectable objects are obtained under the adjusted visual depth. When it is determined from eyelid data that the user's eyes are at a normal open angle and the gaze direction has not changed within a preset time period, the gaze direction is determined as the gaze direction of the eyes. The gaze point of the user's eyes in the gaze direction within the preset time period before the current moment is obtained. Based on the gaze point of the user's eyes in the gaze direction, the target object gazed at by the user's eyes is determined from multiple selectable objects. Visual depth adjustment is performed using eyelid data, and after obtaining multiple selectable objects under the adjusted visual depth, the target object gazed at by the user is selected. The triggering conditions of the visual depth adjustment procedure are clarified, avoiding accidental triggering of the visual depth adjustment procedure during normal image viewing using VR and other wearable devices, thereby improving the user experience of wearing devices.

[0032] In the following embodiments, the user wears a wearable device that displays to the user a two-dimensional image or a projected image (also a two-dimensional image) obtained by projecting a three-dimensional image onto a plane.

[0033] In the above two-dimensional image, there is an object that can be gazed at by the user. This object that can be gazed at by the user is an object with a certain shape (such as a circular object or a rectangular object).

[0034] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Example 1

[0036] The eye-tracking interaction method proposed in this embodiment is implemented by an eye-tracking device in a wearable device.

[0037] See Figure 1 The flowchart shown illustrates an eye-tracking interaction method. This embodiment proposes an eye-tracking interaction method, including the following specific steps:

[0038] Step 100: Obtain the eyelid data and gaze direction of the current user's eyes.

[0039] In step 100 above, the eyelid data is data measured by an eye-tracking device.

[0040] The eyelid data also includes: the eye opening angle and the number of blinks.

[0041] The gaze direction refers to the direction of the user's gaze after the eye-tracking device determines the user's current visual axis. It can be obtained by using any existing algorithm that determines the gaze direction based on the visual axis. The specific process is existing technology and will not be described in detail here.

[0042] Step 102: When it is determined from the eyelid data that the user's eyes are in an abnormal open state, adjust the visual depth of the user's eyes and obtain multiple objects to be selected under the adjusted visual depth.

[0043] In step 102 above, the abnormal eye-opening state includes: squinting state and staring state.

[0044] Before adjusting the visual depth of the user's eyes, the following steps must be taken:

[0045] The eye-tracking device first obtains the user's normal eye-opening angle through calibration. Then, it needs to calibrate the user's staring and squinting degrees. This requires collecting data from the user, including the maximum staring angle and the minimum squinting angle while still being able to clearly see surrounding objects. Calibration is performed using these three data points: the normal eye-opening angle, the maximum staring angle, and the minimum squinting angle while still being able to clearly see surrounding objects. Afterward, the staring coefficient and squinting coefficient are standardized based on these three data points. The staring coefficient ranges from 0 to 1; 0 represents normal eye-opening, and 1 represents the maximum staring angle. The squinting coefficient ranges from -1 to 0; -1 represents the minimum squinting angle while still being able to clearly see surrounding objects.

[0046] The eye-tracking device obtains and stores the correspondence between the eye-opening coefficient and the eye-staring angle, as well as the correspondence between the eye-squinting coefficient and the eye-squinting angle.

[0047] Specifically, in order to adjust the visual depth of the user's eyes, step 102 above specifically performs the following steps (1) to (6):

[0048] (1) Obtain the normal opening angle of the user's eyes. When the opening angle of the user's eyes is greater than the normal opening angle, determine that the user's eyes are in a staring state and obtain the staring coefficient corresponding to the opening angle of the eyes.

[0049] (2) When the staring coefficient is greater than the first coefficient threshold and the duration of the user's eyes being in a staring state is greater than the preset duration, the visual depth of the user's eyes is increased.

[0050] (3) When the staring coefficient corresponding to the eye opening angle is less than the first coefficient threshold, the adjusted visual depth of the user's eyes is obtained;

[0051] (4) When the user's eye opening angle is less than the normal opening angle, it is determined that the user's eye is in a squinting state, and the squinting coefficient corresponding to the eye opening angle is obtained;

[0052] (5) When the squinting coefficient is less than the second coefficient threshold and the duration of the user's eyes being in a squinting state is greater than the preset duration, the visual depth of the user's eyes is reduced.

[0053] (6) When the squinting coefficient corresponding to the eye opening angle is greater than the second coefficient threshold, the adjusted visual depth of the user's eye is obtained.

[0054] In step (1) above, the normal eye opening angle of the user is pre-stored in the eye-tracking device.

[0055] The eye-opening coefficient corresponding to the eye-staring angle is obtained from the correspondence between the eye-opening coefficient and the eye-staring angle.

[0056] In step (2) above, in one embodiment, the first coefficient threshold can be set to 0.6.

[0057] The preset duration can be 0.8 seconds.

[0058] Visual depth is added to the two-dimensional image displayed to the user, and objects within the visual depth in the two-dimensional image displayed to the user by the wearable device are labeled as unselectable objects.

[0059] In step (3) above, when the staring coefficient corresponding to the eye opening angle is less than the first coefficient threshold, the adjusted visual depth of the user's eye is obtained, and multiple objects outside the adjusted visual depth in the two-dimensional image are identified as multiple selectable objects under the adjusted visual depth.

[0060] In step (4) above, the squinting coefficient corresponding to the eye opening angle is obtained from the correspondence between the squinting coefficient and the squinting angle.

[0061] In step (5) above, in one embodiment, the second coefficient threshold can be set to -0.5.

[0062] Reduce the visual depth in the 2D image presented to the user, and in the process of reducing the visual depth, mark objects in the 2D image that have been restored from inside the visual depth to outside the visual depth as selectable objects.

[0063] In step (6) above, when the squinting coefficient corresponding to the eye opening angle is greater than the second coefficient threshold, the adjusted visual depth of the user's eye is obtained, and multiple objects outside the adjusted visual depth in the two-dimensional image are identified as multiple selectable objects under the adjusted visual depth.

[0064] Step 104: When it is determined from the eyelid data that the user's eyes are at a normal eye-opening angle and the gaze direction has not changed within a preset time period before the current moment, the gaze direction is determined as the gaze direction of the eyes.

[0065] In step 104 above, in one embodiment, the preset time period can be set to 0.8 seconds.

[0066] Step 106: Obtain the gaze point of the user's eyes in the gaze direction during the preset time period before the current moment.

[0067] The gaze point is the intersection of the visual axis of the user's eye, as determined by the eye-tracking device, and the two-dimensional image displayed to the user by the wearable device.

[0068] Step 108: Based on the gaze point of the user's eyes in the gaze direction, determine the target object that the user's eyes are gazing at from a plurality of objects to be selected.

[0069] In step 108 above, in order to determine the target object that the user's eyes are gazing at, the following steps (1) to (4) can be performed:

[0070] (1) Obtain the total number of gaze points of the user's eyes in the gaze direction within a preset time period before the current time;

[0071] (2) Count the number of gaze points falling into each of the multiple objects to be selected;

[0072] (3) Among the multiple objects to be selected, the objects whose ratio of the number of gaze points falling to the total number of gaze points is greater than a preset ratio threshold are identified as candidate objects for the user's eye to gaze at.

[0073] (4) When the number of candidate objects is a preset number, the candidate objects are determined as the target objects that the user's eyes are staring at.

[0074] In step (1) above, the number of intersections between the visual axis of the user's eye and the two-dimensional image within a preset time period before the current moment is counted, and the counted number of intersections is determined as the total number of gaze points of the user's eye in the gaze direction within the preset time period before the current moment.

[0075] In step (2) above, the number of gaze points falling into the area contained by the circular or rectangular object that is to be selected is counted, thereby obtaining the number of gaze points falling into each of the multiple objects to be selected, and obtaining the planar area covered by the gaze points.

[0076] The center of the planar area covered by the gaze point is determined as the center of the user's line of sight, and the position information of the center of the user's line of sight in the two-dimensional image is obtained.

[0077] In step (3) above, the preset ratio threshold can be any value from 0.85 to 0.95, which will not be elaborated here.

[0078] In step (4) above, the preset quantity is 1.

[0079] Furthermore, if there are multiple candidate objects, it indicates that the target object being gazed at by the user is obstructed. In order to determine the target object being gazed at by the user from among the multiple candidate objects, the following steps (5) to (9) can be performed:

[0080] (5) When the number of candidate objects is greater than the preset number, obtain the depth value of each candidate object and determine the maximum value among the depth values ​​of each candidate object.

[0081] (6) Calculate the first selection probability of each candidate object using the following formula 1:

[0082]

[0083] Where, p depth (k) represents the first selection probability of the k-th candidate object; d max This represents the maximum depth value of each candidate object; d k The depth value of the k-th candidate object is represented by d; n represents the number of candidate objects; d i This represents the depth value of the i-th candidate object out of n candidate objects;

[0084] (7) Obtain the center and line-of-sight center of each candidate object, and calculate the distance between the center of each candidate object and the line-of-sight center respectively;

[0085] (8) Perform Gaussian distribution processing on the distances between the center of each candidate object and the center of the line of sight to obtain the second selection probability of each candidate object;

[0086] (9) Based on the first and second selection probabilities of each candidate object, calculate the gaze intensity of each candidate object, and determine the candidate object with the highest gaze intensity among all candidate objects as the target object that the user's eyes are gazing at.

[0087] In step (5) above, if the number of candidate objects is greater than the preset number, it means that the number of candidate objects is greater than 1, that is, there are multiple candidate objects at present.

[0088] The depth value of each candidate object is the distance between that candidate object and the user's eyes. After the wearable device generates a two-dimensional image, the distance between each object that the user can gaze at in the two-dimensional image and the user's eyes can be determined, and the distance between each object and the user's eyes is stored in the eye-tracking device as the depth value of the object in the two-dimensional image.

[0089] Furthermore, after generating a two-dimensional image, the wearable device can also obtain the position information of the center of each object that the user can gaze at, and the obtained position information of the center of each object is stored in the eye-tracking system by the wearable device.

[0090] After identifying the candidate objects, the eye-tracking device can obtain the depth values ​​of each candidate object from the depth values ​​of objects in the pre-stored two-dimensional image.

[0091] In step (6) above, when the candidate object with the smaller depth value has a higher probability of being gazed at by the user, the first selection probability of each candidate object for the user's visual depth is calculated using the above formula 1.

[0092] In step (7) above, when calculating the distance between the center of each candidate object and the center of the line of sight, the position information of the center of each candidate object is first obtained from the position information of the center of each object stored in the image. Then, the distance between the center of each candidate object and the center of the line of sight of the user's eye is calculated using the position information of the center of each candidate object and the position information of the center of the line of sight of the user's eye in the two-dimensional image.

[0093] In step (8) above, the distance between the center of each candidate object and the center of the line of sight is processed by Gaussian distribution to obtain the probability that the center of the line of sight is within each candidate object, which is used as the second selection probability.

[0094] The specific process of applying a Gaussian distribution to parameter values ​​such as distance is existing technology and will not be elaborated here.

[0095] In step (9) above, the gaze intensity of each candidate object is determined using the following formula 2:

[0096] p intended (k)=w(n)×P depth (k)+(1-w(n))×p location (k) (2)

[0097] Where w(n) represents the weighting coefficient, which is 0.3 when n = 2 and 0.9 when n ≥ 3; p intended (k) represents the gaze intensity of the k-th candidate object; p depth (k) represents the first selection probability of the k-th candidate object; p location (k) represents the second selection probability of the k-th candidate object.

[0098] The gaze intensity of each candidate object is used to represent the probability that each candidate object is being gazed at by the user's eyes at the current moment.

[0099] As can be seen from the process described in steps (5) to (9) above, when there are multiple candidate objects, it indicates that the target object may be occluded. Therefore, the first selection probability of each candidate object related to the visual depth of the user's eyes is calculated first, and the probability that the center of vision is within each candidate object is calculated as the second selection probability based on the distance between the center of each candidate object and the center of vision. Then, using the first and second selection probabilities of each candidate object, the gaze intensity of each candidate object is calculated, and the candidate object with the highest gaze intensity is determined as the target object that the user's eyes are gazing at. Even if the target object that the user's eyes are gazing at is occluded by other objects, the target object that the user's eyes are currently gazing at can be accurately determined. Moreover, the process of determining the target object that the user's eyes are currently gazing at is simple, which greatly improves the response speed of the wearable device and further improves the user's experience of using the wearable device.

[0100] The eye-tracking interaction method proposed in this embodiment has the following advantages:

[0101] 1. This eye-tracking interaction method is the first technology in the field of virtual reality to use continuous eyelid data for scene depth adjustment, and it can dynamically combine different probability distributions to predict the gaze direction and target selection in VR scenes.

[0102] 2. This eye-tracking interaction method provides three different weight selection methods to suit different task objectives, allowing users to choose different techniques in specific situations (such as when faster speed or higher accuracy is required).

[0103] 3. This eye-tracking interaction method still maintains high selection accuracy and fast speed even in severely occluded scenarios.

[0104] In summary, this embodiment proposes an eye-tracking interaction method. When eyelid data indicates that the user's eyes are not in an abnormal open state, the visual depth of the user's eyes is adjusted, and multiple selectable objects are obtained at the adjusted visual depth. When eyelid data indicates that the user's eyes are at a normal open angle and the gaze direction has not changed within a preset time period, the gaze direction is determined as the eye's gaze direction. The gaze point of the user's eyes in the gaze direction within the preset time period before the current moment is obtained. Based on the gaze point of the user's eyes in the gaze direction, the target object gazed upon by the user's eyes is determined from the multiple selectable objects. Compared with related technologies that cannot accurately determine the target object gazed upon by the user's eyes, this method uses eyelid data for visual depth adjustment and selects the target object gazed upon after obtaining multiple selectable objects at the adjusted visual depth. This clarifies the triggering conditions of the visual depth adjustment procedure, avoids accidental triggering of the visual depth adjustment procedure during normal image viewing using VR devices or other wearable devices, and improves the user experience of wearing devices.

[0105] Example 2

[0106] This embodiment proposes an eye-tracking interaction device for executing the eye-tracking interaction method proposed in Embodiment 1 above.

[0107] See Figure 2 The diagram shows the structure of an eye-tracking interaction device. This embodiment proposes an eye-tracking interaction device, comprising:

[0108] The first acquisition module 200 is used to acquire the eyelid data and gaze direction of the current user's eyes;

[0109] The adjustment module 202 is used to adjust the visual depth of the user's eyes when it is determined from the eyelid data that the user's eyes are in an abnormal open state, and to obtain multiple objects to be selected under the adjusted visual depth.

[0110] The determination module 204 is used to determine the gaze direction as the gaze direction of the eyes when it is determined from the eyelid data that the user's eyes are at a normal eye-opening angle and the gaze direction has not changed within a preset period of time before the current moment.

[0111] The second acquisition module 206 is used to acquire the gaze point of the user's eyes in the gaze direction during a preset time period before the current moment;

[0112] The processing module 208 is used to determine the target object that the user's eye is gazing at from a plurality of objects to be selected, based on the gazing point of the user's eye in the gazing direction.

[0113] Specifically, the eyelid data includes: eye opening angle; the abnormal eye-opening states include: squinting state and staring state.

[0114] The adjustment module 202 is used to adjust the visual depth of the user's eyes when it is determined from the eyelid data that the user's eyes are in an abnormally open state, including:

[0115] The normal eye opening angle of the user is obtained. When the eye opening angle of the user is greater than the normal opening angle, it is determined that the user's eyes are in a staring state, and the staring coefficient corresponding to the eye opening angle is obtained.

[0116] When the staring coefficient is greater than the first coefficient threshold and the duration of the user's eyes being in a staring state is greater than the preset duration, the visual depth of the user's eyes is increased.

[0117] When the staring coefficient corresponding to the eye opening angle is less than the first coefficient threshold, the adjusted visual depth of the user's eyes is obtained;

[0118] When the user's eyes are open at an angle less than the normal opening angle, it is determined that the user's eyes are squinting, and the squinting coefficient corresponding to the eye opening angle is obtained;

[0119] When the squinting coefficient is less than the second coefficient threshold and the duration of the user's squinting state is greater than the preset duration, the visual depth of the user's eyes is reduced.

[0120] When the squinting coefficient corresponding to the eye opening angle is greater than the second coefficient threshold, the adjusted visual depth of the user's eyes is obtained.

[0121] Specifically, the processing module 208 is used for:

[0122] Obtain the total number of gaze points of the user's eyes in the gaze direction within a preset time period prior to the current moment;

[0123] The number of gaze points falling into each of the multiple objects to be selected is counted separately.

[0124] Among the multiple objects to be selected, the objects whose ratio of the number of gaze points falling on them to the total number of gaze points is greater than a preset ratio threshold are identified as candidate objects for the user's eye to gaze at.

[0125] When the number of candidate objects is a preset number, the candidate objects are determined as the target objects that the user's eyes are staring at.

[0126] In summary, this embodiment proposes an eye-tracking interaction device. When eyelid data determines that the user's eyes are not in an abnormal open state, the device adjusts the visual depth of the user's eyes and obtains multiple selectable objects at the adjusted visual depth. When eyelid data determines that the user's eyes are at a normal open angle and the gaze direction has not changed within a preset time period, the gaze direction is determined as the eye's gaze direction. The device obtains the gaze point of the user's eyes in the gaze direction within the preset time period before the current moment. Based on the gaze point of the user's eyes in the gaze direction, the device determines the target object that the user's eyes are gazing at from the multiple selectable objects. Compared with related technologies that cannot accurately determine the target object that the user's eyes are gazing at, this device uses eyelid data for visual depth adjustment and then selects the target object after obtaining multiple selectable objects at the adjusted visual depth. This clarifies the triggering conditions of the visual depth adjustment procedure, avoids accidental triggering of the visual depth adjustment procedure while the user is normally viewing images using VR devices or other wearable devices, and improves the user experience of using wearable devices.

[0127] Example 3

[0128] This embodiment proposes a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes the steps of the eye-tracking interaction method described in Embodiment 1 above. For a detailed implementation, please refer to Method Embodiment 1, which will not be repeated here.

[0129] In addition, see Figure 3 The diagram shows the structure of an electronic device. This embodiment also proposes an electronic device, which includes a bus 51, a processor 52, a transceiver 53, a bus interface 54, a memory 55, and a user interface 56. The electronic device includes a memory 55.

[0130] In this embodiment, the electronic device further includes: one or more programs stored in memory 55 and executable on processor 52, configured to be executed by the processor to perform the one or more programs for steps (1) to (5):

[0131] (1) Obtain the eyelid data and gaze direction of the current user's eyes;

[0132] (2) When it is determined from the eyelid data that the user's eyes are in an abnormal open state, the visual depth of the user's eyes is adjusted, and multiple objects to be selected are obtained under the adjusted visual depth.

[0133] (3) When it is determined from the eyelid data that the user's eyes are at a normal eye-opening angle and the gaze direction has not changed within a preset period of time before the current moment, the gaze direction is determined as the gaze direction of the eyes;

[0134] (4) Obtain the gaze point of the user's eyes in the gaze direction during a preset time period prior to the current moment;

[0135] (5) Based on the gaze point of the user's eye in the gaze direction, determine the target object that the user's eye is gazing at from a plurality of objects to be selected.

[0136] Transceiver 53 is used to receive and send data under the control of processor 52.

[0137] The bus architecture (represented by bus 51) can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 52 and memory represented by memory 55. Bus 51 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be further described in this embodiment. Bus interface 54 provides an interface between bus 51 and transceiver 53. Transceiver 53 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. For example, transceiver 53 receives external data from other devices. Transceiver 53 is used to transmit data processed by processor 52 to other devices. Depending on the nature of the computing system, a user interface 56 may also be provided, such as a keypad, display, speaker, microphone, or joystick.

[0138] Processor 52 is responsible for managing bus 51 and general processing, such as running a general-purpose operating system as described above. Memory 55 can be used to store data used by processor 52 during operation.

[0139] Optionally, the processor 52 may be, but is not limited to, a central processing unit, a microcontroller, a microprocessor, or a programmable logic device.

[0140] It is understood that the memory 55 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 55 of the systems and methods described in this embodiment is intended to include, but is not limited to, these and any other suitable types of memory.

[0141] In some implementations, memory 55 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 551 and application programs 552.

[0142] The operating system 551 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 552 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of the embodiments of this application can be included in the application program 552.

[0143] In summary, this embodiment proposes a computer-readable storage medium and an electronic device. When eyelid data determines that the user's eyes are in an abnormally open state, the visual depth of the user's eyes is adjusted, and multiple selectable objects are obtained at the adjusted visual depth. When eyelid data determines that the user's eyes are at a normal open angle and the gaze direction has not changed within a preset time period, the gaze direction is determined as the eye's gaze direction. The gaze point of the user's eyes in the gaze direction within the preset time period before the current moment is obtained. Based on the gaze point of the user's eyes in the gaze direction, the target object gazed upon by the user's eyes is determined from multiple selectable objects. Compared with related technologies that cannot accurately determine the target object gazed upon by the user's eyes, this embodiment uses eyelid data for visual depth adjustment, and then selects the target object gazed upon after obtaining multiple selectable objects at the adjusted visual depth. This clarifies the triggering conditions of the visual depth adjustment program, avoids accidental triggering of the visual depth adjustment program during normal image viewing using VR devices or other wearable devices, and improves the user experience of wearing devices.

[0144] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An eye movement interaction method, characterized by, The method comprises the following steps: obtaining eyelid data and gaze direction of the current user's eyes; when it is determined according to the eyelid data that the user's eyes are in an abnormal open-eye state, adjusting the visual depth of the user's eyes, and obtaining a plurality of objects to be selected under the adjusted visual depth; when it is determined according to the eyelid data that the user's eyes are in a normal open-eye angle and the gaze direction has not changed within a preset time period before the current time, determining the gaze direction as the fixation direction of the eyes; obtaining the fixation point of the user's eyes in the fixation direction within a preset time period before the current time; determining the target object gazed at by the user's eyes from the plurality of objects to be selected according to the fixation point of the user's eyes in the fixation direction; the eyelid data comprises an eye opening angle; the abnormal open-eye state comprises a squinting state and a staring state; when it is determined according to the eyelid data that the user's eyes are in an abnormal open-eye state, adjusting the visual depth of the user's eyes comprises: obtaining a normal open-eye angle of the user's eyes, determining that the user's eyes are in a staring state when the eye opening angle of the user's eyes is greater than the normal open-eye angle, and obtaining a staring coefficient corresponding to the eye opening angle; when the staring coefficient is greater than a first coefficient threshold and the duration of the user's eyes in the staring state is greater than a preset duration, increasing the visual depth of the user's eyes; when the staring coefficient corresponding to the eye opening angle is less than the first coefficient threshold, obtaining the adjusted visual depth of the user's eyes; when the eye opening angle of the user's eyes is less than the normal open-eye angle, determining that the user's eyes are in a squinting state, and obtaining a squinting coefficient corresponding to the eye opening angle; when the squinting coefficient is less than a second coefficient threshold and the duration of the user's eyes in the squinting state is greater than a preset duration, decreasing the visual depth of the user's eyes; when the squinting coefficient corresponding to the eye opening angle is greater than the second coefficient threshold, obtaining the adjusted visual depth of the user's eyes.

2. The method of claim 1, wherein, determining the target object gazed at by the user's eyes from the plurality of objects to be selected according to the fixation point of the user's eyes in the fixation direction comprises: obtaining the total amount of fixation points of the fixation point of the user's eyes in the fixation direction within a preset time period before the current time; respectively counting the number of fixation points falling into each of the plurality of objects to be selected; determining the objects to be selected falling into the number of fixation points and the total amount of fixation points as candidate objects gazed at by the user's eyes when the ratio is greater than a preset proportion threshold; when the number of candidate objects is a preset number, determining the candidate objects as the target objects gazed at by the user's eyes.

3. The method of claim 2, wherein, determining the target object gazed at by the user's eyes from the plurality of objects to be selected according to the fixation point of the user's eyes in the fixation direction comprises: when the number of candidate objects is greater than a preset number, obtaining the depth values of each candidate object and determining the maximum value in the depth values of each candidate object; calculating the first selection probability of each candidate object by the following formula: wherein, denotes a first selection probability of the k candidate object; denotes a maximum value of the depth values of the candidate objects; denotes a depth value of the k candidate object; n denotes a number of candidate objects; denotes n a depth value of the i candidate object of the candidate objects; obtain the center of each candidate object and the center of the line of sight, and calculate the distance between the center of each candidate object and the center of the line of sight, respectively, wherein the center of the line of sight is the center of a planar region covered by the gaze point; perform Gaussian distribution processing on the calculated distance between the center of each candidate object and the center of the line of sight to obtain a second selection probability of each candidate object; calculate the gaze degree of each candidate object according to the first selection probability and the second selection probability of each candidate object, and determine a candidate object with the largest gaze degree as the target object gazed at by the user's eye.

4. The method of claim 3, wherein, The calculation of the gaze degree of each candidate object according to the first selection probability and the second selection probability of each candidate object comprises: The gaze degree of each candidate object is calculated by the following formula: wherein, represents a weight coefficient, when n = 2, is 0.3, when n ≥ 3, is 0.9; represents a gaze degree of the k th candidate object; represents a first selection probability of the k th candidate object; represents a second selection probability of the k th candidate object.

5. An eye movement interaction device, characterized by It comprises: The first acquisition module is configured to acquire eyelid data and a gaze direction of the current user's eye. The adjustment module is configured to adjust the visual depth of the user's eye when it is determined according to the eyelid data that the user's eye is in an abnormal open-eye state, and obtain a plurality of to-be-selected objects under the adjusted visual depth. The determination module is configured to determine the gaze direction of the eye as the gaze direction of the eye when it is determined according to the eyelid data that the user's eye is in a normal open-eye angle and the gaze direction has not changed within a preset time period before the current time. The second acquisition module is configured to acquire a gaze point of the user's eye on the gaze direction within a preset time period before the current time. The processing module is configured to determine, from the plurality of to-be-selected objects, a target object gazed at by the user's eye according to the gaze point of the user's eye on the gaze direction. The eyelid data comprises an eye opening angle, and the abnormal open-eye state comprises a squinting state and a staring state. The adjustment module is configured to adjust the visual depth of the user's eye when it is determined according to the eyelid data that the user's eye is in an abnormal open-eye state, and obtain a plurality of to-be-selected objects under the adjusted visual depth. Obtain the normal open-eye angle of the user's eye, and determine that the user's eye is in a staring state when the eye opening angle of the user's eye is greater than the normal open-eye angle, and obtain a staring coefficient corresponding to the eye opening angle. Increase the visual depth of the user's eye when the staring coefficient is greater than a first coefficient threshold and the duration of the user's eye in the staring state is greater than a preset duration. Obtain the adjusted visual depth of the user's eye when the staring coefficient corresponding to the eye opening angle is less than the first coefficient threshold. Determine that the user's eye is in a squinting state when the eye opening angle of the user's eye is less than the normal open-eye angle, and obtain a squinting coefficient corresponding to the eye opening angle. Reduce the visual depth of the user's eye when the squinting coefficient is less than a second coefficient threshold and the duration of the user's eye in the squinting state is greater than a preset duration. Obtain the adjusted visual depth of the user's eye when the squinting coefficient corresponding to the eye opening angle is greater than the second coefficient threshold.

6. The apparatus of claim 5, wherein, The processing module is specifically configured to: Obtain the total amount of gaze points of the gaze point of the user's eye on the gaze direction within a preset time period before the current time. respectively counting the number of gaze points falling in each of the plurality of objects to be selected; determining, from the plurality of objects to be selected, an object to be selected whose ratio of the number of gaze points falling in the object to the total number of gaze points is greater than a preset proportion threshold value as a candidate object of the user's eye gaze; when the number of candidate objects is a preset number, determining the candidate objects as target objects of the user's eye gaze.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program, when run by a processor, performs the steps of the method of any one of claims 1-4.

8. An electronic device, comprising: The electronic device includes a memory, a processor, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor to perform the steps of the method of any one of claims 1-4.

Images