Eye opening degree is determined using an eye-tracking device.

Abstract

A method for determining eye openness using an eye-tracking device is disclosed. The method may include determining a first sum of pixel intensities for pixels of an image sensor of the eye-tracking device during a first time period when the user's eyes are open. The method may further include determining a second sum of pixel intensities during a second time period when the user's eyes are closed. The method may further include determining a third sum of pixel intensities during a third time period. The method may additionally include determining that the user's eyes are closed after the third sum exceeds a fourth sum obtained by adding the first sum to the threshold amount, which is equal to a threshold fraction multiplied by the difference between the first and second sums.

Description

[0001] This application is a divisional application of the invention patent application filed on February 9, 2018, with international application number PCT / US2018 / 017612, national application number 201880013360.1, and entitled "Determining Eye Opening Degree Using an Eye Tracking Device".

[0002] Cross-references to related applications

[0003] This application claims priority and benefit from U.S. Provisional Patent Application No. 62 / 464,235, filed February 27, 2017, the entire contents of which are incorporated herein by reference for all purposes as if fully set forth herein. Summary of the Invention

[0004] In one embodiment, a system for adjusting the position of a lens in a wearable device is provided. The system may include the wearable device and one or more processors. The wearable device may include: a display; at least one lens disposed in front of the display; and an eye-tracking device including at least one emitter and at least one image sensor. One or more processors may be configured to receive data from the eye-tracking device and, based at least on the data, determine a first position of at least one eye of a user wearing the wearable device, said position relative to a second position, which is fixed relative to the at least one lens. One or more processors may also be configured to determine a distance between the first position and the second position, and if said distance is greater than a first value, then information is presented to the user indicating that one or more lenses should be moved to a preferred position.

[0005] In another embodiment, a method for adjusting the position of lenses in a wearable device is provided. The method may include receiving data from an eye-tracking device in the wearable device, wherein the wearable device includes a display and at least one lens movably disposed in front of the display. The method may also include receiving data from the eye-tracking device. The method may further include determining, based at least on the data, a first position of at least one eye of a user wearing the wearable device, wherein the position is relative to a second position, the second position being fixed relative to at least one lens. The method may further include determining a distance between the first position and the second position. The method may further include causing information to be presented to the user if the distance is greater than a first value or less than a second value, the information indicating that one or more lenses should be moved to a preferred position.

[0006] In another embodiment, a non-transitory machine-readable medium is provided. The non-transitory machine-readable medium may store instructions for adjusting the position of a lens in a wearable device. The instructions may be executed by one or more processors to perform a method. The method may include receiving data from an eye-tracking device in the wearable device, wherein the wearable device includes a display and at least one lens movably disposed in front of the display. The method may also include receiving data from the eye-tracking device. The method may further include determining, based at least on the data, a first position of at least one eye of a user wearing the wearable device, wherein the position is relative to a second position, the second position being fixed relative to the at least one lens. The method may further include determining a distance between the first position and the second position. The method may further include causing at least one lens to move until the distance is less than the first value and greater than the second value if the distance is greater than a first value or less than a second value.

[0007] In other embodiments, systems and methods are provided for determining eye openness using an eye-tracking device. These embodiments may include determining a first sum of pixel intensities for at least a portion of pixels of an image sensor of the eye-tracking device during a first time period when the user's eyes are open. The embodiments may further include determining a second sum of pixel intensities for at least a portion of pixels of the image sensor of the eye-tracking device during a second time period when the user's eyes are closed. The embodiments may further include determining a third sum of pixel intensities for at least a portion of pixels of the image sensor of the eye-tracking device during a third time period. The embodiments may additionally include a processor determining that the user's eyes are closed after the third sum exceeds a fourth sum obtained by adding a threshold amount, wherein the threshold amount is equal to a threshold fraction multiplied by the difference between the first and second sums.

[0008] In other embodiments, systems and methods are provided for determining eye opening using an eye-tracking device. These embodiments may include activating multiple emitters of the eye-tracking device, wherein the multiple emitters are directed toward the user's eyes. The embodiments may further include using an image sensor of the eye-tracking device to determine how much reflection from the activated emitters appears on the user's eyes. The embodiments may further include using at least one processor to determine that the user's eyes are closed based on the presence of less than a first predefined number of reflections.

[0009] In other embodiments, systems and methods are provided for determining eye opening using an eye-tracking device. These embodiments may include receiving images of a user's eyes from an image sensor of the eye-tracking device at one or more processors. The embodiments may further include determining the radius of the pupil based on the eye images using one or more processors. The embodiments may further include determining the total area of ​​the pupil based on the pupil radius using one or more processors. The embodiments may additionally include determining an amount of the total area of ​​the pupil not obscured by either eyelid using one or more processors based on the eye images and the pupil radius. The embodiments may additionally include determining whether the user's eyes are closed using one or more processors based on the amount of the total area of ​​the pupil not obscured by eyelids and the total pupil area. Attached Figure Description

[0010] Embodiments of the present invention will be described in conjunction with the accompanying drawings:

[0011] Figure 1 It is a graph showing the data used in determining the degree of eye opening according to one embodiment;

[0012] Figure 1A An image of a user's eye showing flashing dots indicating reflections from a working light source, wherein the flashing dots are spaced substantially regularly and circularly around the user's pupil;

[0013] Figure 2 It is a graph showing the data used in determining the degree of eye opening according to one embodiment;

[0014] Figures 3 to 7 It relates to an eye image and various measurement points on the eye image according to one embodiment, which are associated with a method for determining eye opening.

[0015] Figure 8 Two visual guides are shown, which indicate the position of the image sensor and the position of the user's eyes relative to the image sensor, and the two visual guides may be presented in some embodiments to assist the user in locating the eye-tracking device or a wearable device containing the eye-tracking device;

[0016] Figure 9 A wearable device illustrating an embodiment of the present invention is provided, wherein the wearable device is capable of detecting the position of a lens therein and at least assisting a user in adjusting the position of the lens;

[0017] Figure 10 A block diagram illustrating a method embodiment of the present invention for detecting and adjusting the position of a lens in a wearable device; and

[0018] Figure 11It is a block diagram of a dedicated computer system that can be used in at least a portion of the device or system of the present invention or in at least a portion of the method of the present invention.

[0019] In the accompanying drawings, similar parts and / or features may have the same reference numerals. Furthermore, various parts of the same type may be distinguished by adding a letter after the reference numeral to differentiate between similar parts and / or features. If only the first numerical reference numerals are used in this specification, then this description applies to any of the similar parts and / or features having the same first numerical reference numerals, regardless of the letter suffix. Detailed Implementation

[0020] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. In fact, the following description of exemplary embodiments will provide those skilled in the art with a feasible description for implementing one or more exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the various elements without departing from the spirit and scope of the invention as set forth in the appended claims.

[0021] For example, any details discussed with respect to one embodiment may or may not be present in all anticipated versions of that embodiment. Similarly, any details discussed with respect to one embodiment may or may not be present in all anticipated versions of other embodiments discussed herein. Ultimately, the omission of any detail discussed with respect to any embodiment herein should be construed as implying that such detail may or may not be present in any version of any embodiment discussed herein.

[0022] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other elements of the present invention may be illustrated as components in the form of block diagrams to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0023] Additionally, it should be noted that each embodiment can be described as a process, depicted as a flowchart, diagram, data flow diagram, structural diagram, or block diagram. While a flowchart may describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process may terminate upon completion of its operations, but may have additional steps not discussed or included in the figures. Moreover, not all operations in any specifically described process may appear in all embodiments. A process may correspond to a method, function, process, subroutine, subroutine, etc. When a process corresponds to a function, its termination corresponds to the function returning to the calling function or the main function.

[0024] The term "machine-readable medium" includes (but is not limited to) transient and non-transient, portable or fixed storage devices, optical storage devices, wireless channels, and various other media capable of storing, containing, or carrying instructions and / or data. A code segment or machine-executable instruction may represent a process, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted via any appropriate means, including memory sharing, message passing, token passing, network transmission, etc.

[0025] Furthermore, embodiments of the invention can be implemented, at least partially manually or automatically. Manual or automatic implementations can be performed, or at least assisted thereto, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented as software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks can be stored in a machine-readable medium. The processor can perform the necessary tasks.

[0026] This invention generally relates to systems and methods for eye tracking and their uses, and more specifically, to systems and methods for eye tracking in wearable devices. Wearable devices have received considerable attention in recent years; these devices typically contain at least one display and are commonly used in virtual reality (VR) or augmented reality (AR) applications.

[0027] Eye tracking is the process of determining a person's gaze direction. This is typically done using an image sensor-based eye tracking device, where an emitter projects infrared light onto the user's eyes, and the image sensor captures an image containing the infrared light reflected from the eyes. The gaze direction can be determined from the position of the reflection on the eyes. As will be readily understood by those skilled in the art, eye tracking devices have various configurations of image sensors, emitters, and processing structures.

[0028] Using eye-tracking technology in wearable devices can have certain advantages. For example, it can enable more natural interactions in virtual or augmented reality environments.

[0029] In one embodiment, a wearable device is provided for wearing on a user's head. The wearable device may include at least one display and an eye-tracking device. The eye-tracking device may, for example, include at least one image sensor and at least one infrared emitter, as previously discussed. As will be readily understood by those skilled in the art, the image sensor may be any conventional complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor containing multiple pixels.

[0030] An eye-tracking device can be placed within a wearable device so that it is close to the user's eyes when the wearable device is worn by the user. A processing unit can be operatively coupled to an image sensor and a light emitter to process the image captured by the image sensor to determine the direction of gaze.

[0031] As discussed, in some embodiments, the infrared emitter projects infrared light toward the user's eyes, and the image sensor captures an image of the area of ​​the user's eyes. The sensor and / or processor can then sum the received values ​​or intensities (where higher values / intensities are characteristics of a closed eye) from all pixels of the captured image to obtain a sum X. This process is repeated for each image captured by the image sensor. An example of this repeated monitoring is shown in... Figure 1 In the curve graph, where Figure 1 The relationship between the total intensity (Y-axis) and time (X-axis) is shown. This achieves a binary determination of the eye opening degree at the highest and lowest Y-axis values ​​(i.e., closed (0) or open (1)), but also enables a more simulated determination of various opening degree states between closed and open, i.e., one-quarter open, half open, three-quarter open, and all variations in between.

[0032] exist Figure 1 In this example, the value of X changes over time, resulting in three distinct peaks. These peaks represent the sum of the intensities of the pixels in the image sensor as it changes from a previous time ( Figure 1 The peaks (valleys) change and increase, which occurs when the user closes their eyelids (e.g., blinks). These peaks can be determined by the processing unit, and thus provide another application with an indication that the user is blinking.

[0033] Figure 1 The Y data in the figure is a baseline, which can be adjusted according to various embodiments and can be different for each user. One method of adjusting the baseline Y is to determine a value of a type of low-pass filter (e.g., a moving average filter value) for an image of the user's eyes when the user's eyes are open.

[0034] Figure 1 The Z data therein is the threshold when the eyes are considered to be approximately closed, and the threshold is compared with the value of X to determine whether the user is blinking. If X exceeds the threshold Z, it can be determined that the user's eyes are performing a closing movement until the value of X returns below the threshold Z. In some embodiments, the threshold Z can also be adjusted based on the data collected during the operation of the methods disclosed herein.

[0035] Next, in one embodiment, the following formula can be used to analyze the sum of pixels X to determine whether the user is blinking:

[0036] X = the current sum of pixels;

[0037] Y = the baseline;

[0038] A = the blink amplitude;

[0039] f = the blink threshold score (in some embodiments, between about 1 / 8 and about 1 / 2);

[0040] If X >= Y + f*A, then C = 0;

[0041] If X < Y + f*A, then C = 1; and

[0042] where C = 0 is the closed eyes and C = 1 is the open eyes.

[0043] Simply put:

[0044] Z = Y + f*A;

[0045] If X >= Z, then the eyes are closed; and

[0046] If X < Z, then the eyes are open.

[0047] A is updated only during the time when C = 0 (eyes closed) with the moving average filtered value of the value of X.

[0048] Y is updated only during the period when C=1 (eyes open) using a moving average filtered value of X. More specifically, the value of Y is updated based on images of at least one of the captured user's eyes (indicating the eye is open). The value of Y is set to a predetermined default value at the start of system initialization or at another time. Furthermore, the value of Y is set to be updated based on each image frame of the captured user's eye, or is set to be updated according to a predetermined time series and based on images of the user's eyes captured at specific time points. Preferably, the value of Y is set to a specific value when a predetermined flashing point pattern is detected, which is adapted to the number of emitters that may cause flashing points. The predetermined flashing point pattern indicates that the number of flashing points meets a predetermined threshold and that the shape of the flashing points is substantially circular, and that the location of the flashing points is within a predefined area (e.g., near the pupil). The distance between adjacent flashing points can also be substantially the same. And the flashing points can be aligned on both sides of the pupil in a similar circular pattern. See also Figure 1A Examples to understand the above characteristics.

[0049] Figure 2 Showing about Figure 1 The results of these calculations on instance data, where the Y-axis represents eye opening (1 = open, 0 = closed) and the X-axis represents time. By normalizing the baseline Y based on the value of X during the eye-open state (C = 1), the embodiments described herein can eliminate interference during monitoring, such as head movement, where the sum of pixels X varies due to external factors such as ambient light, movement of the wearable device, etc.

[0050] In another embodiment, instead of using the sum of pixels X to determine whether the eye is open or closed, it is better to analyze more specific characteristics of the reflection of infrared emission. For example, in a system with multiple infrared emitters pointing towards the user's eye (e.g., an eye-tracking system), the captured image of the eye can be analyzed to determine whether there is any reflection on the eye from the infrared emitters. If so, the amount of reflection, the location of the reflection, etc., can be used to infer whether the eye is open or closed.

[0051] For example, consider a scenario where eight emitters arranged in a circle face the user's eyes. If the image of the eye shows flickering dots caused by all eight current emitters, then it can be determined that the user's eyes are open. If the image does not show any flickering dots, then it can be determined that the user's eyes are closed. If the image shows 1 to 7 flickering dots caused by the emitters, then it can be determined that the user's eyes are in the process of opening or closing. Depending on the embodiment, these intermediate states occurring during opening or closing can be classified as open or closed states to make the data binary for analytical purposes.

[0052] In some embodiments, methods for determining eye opening are also provided, wherein the methods are adaptable to eyes of different shapes and sizes, and to individuals whose eyes may not open as wide as those of other users. These methods may be based on the fact that during blinking, regardless of a person's normal eye opening characteristics, a large portion of the pupil is obscured by the eyelids. In contrast, when a user is born with squinting or normal squinting, a large portion of their pupil will not be obscured, regardless of whether the overall eyelid position appears to be a blinking motion.

[0053] Regarding these embodiments, please refer to the following text. Figure 3 The various pieces of information can be related:

[0054] p top = The point where the upper eyelid opens to its maximum extent;

[0055] p botttom =The point where the lower eyelid opens to its maximum extent;

[0056] c pupil = Pupil center; and

[0057] r pupil = Pupil radius.

[0058] This information can be collected using the image sensor of an eye-tracking device and can be used to estimate the distance between a user's upper and lower eyelids. Conversely, to accommodate naturally squinty eyes, the maximum eye opening can be tracked over time, and the methods described herein can be used relative to this maximum eye opening to determine whether the eyes are open or closed.

[0059] The above information can also be used to determine an estimate of the pupillary visibility score. (See reference...) Figure 4 One method is to estimate how much of the pupil is visible by approximating the eyelids with horizontal lines:

[0060]

[0061] and

[0062] area botttom It is done in the same way from p botttom Calculated.

[0063] Reference Figures 5 to 8 By tracking the maximum fraction of the user's visible pupil area (f visible,max The degree of eye opening can be calculated using the following formula:

[0064] Opening degree = f visible / f visible,max ;and

[0065] fvisible =area visible / area pupil .

[0066] Alternatively, in the binary determination of eye opening, it is further advantageous to determine whether the pupil center is located at d. top With d bottom between.

[0067] Once the degree of eye opening of a user is determined, this information can be used by applications that can be displayed on the wearable device or other applications. For example, in an application that displays a virtual avatar (a computer representation of a person), the single or double eye on the avatar can be updated to reflect the real-life condition of the user's single or double eye. The application can run on a local wearable device containing eye-tracking devices or on a remote wearable device that communicates with the user's wearable device.

[0068] In other embodiments, a specific action may be initiated by a processor or other device when it is determined that the user's eyes are open, closed, or in a state between open and closed. For example, if the user's eyes are detected as open, the eye-tracking device may be switched on to a higher power and / or higher activity / ready state. If the user's eyes are detected as closed, the eye-tracking device may be switched off or switched to a lower power and / or lower activity / ready state. Any other resources, input devices, output devices, or other devices coupled to the processor managing the eye-tracking device (or within the tracking device) may also operate in a similar manner based on the determination of eye openness using any of the methods disclosed herein.

[0069] Therefore, in some embodiments, systems and methods are provided for determining eye openness using an eye-tracking device. These embodiments may include determining a first sum of pixel intensities for at least a portion of pixels of an image sensor of the eye-tracking device during a first time period when the user's eyes are open. In some embodiments, the first sum may be a moving average of the pixel intensities of the image sensor. The embodiments may also include determining a second sum of pixel intensities for at least a portion of pixels of the image sensor of the eye-tracking device during a second time period when the user's eyes are closed. The embodiments may also include determining a third sum of pixel intensities for at least a portion of pixels of the image sensor of the eye-tracking device during a third time period. The embodiments may further include determining, by a processor, that the user's eyes are closed after the third sum exceeds a fourth sum obtained by adding a threshold amount, where the threshold amount is equal to a threshold fraction multiplied by the difference between the first and second sums. The threshold fraction may be between approximately 1 / 8 and approximately 1 / 2.

[0070] In some embodiments, the method and system may further include activating the emitter of the eye-tracking device during each of the first, second, and third time periods. Furthermore, in various embodiments, the first time period may be dynamic and precede the third time period.

[0071] In other embodiments, systems and methods are provided for determining eye openness using an eye-tracking device. These embodiments may include activating a plurality of emitters of the eye-tracking device, wherein the plurality of emitters are directed toward the user's eyes. The embodiments may further include using an image sensor of the eye-tracking device to determine how many reflections from the activated emitters appear on the user's eyes. The embodiments may further include using at least one processor to determine that the user's eyes are closed based on the presence of fewer than a first predefined number of reflections. In some embodiments, the first predefined number may be one. In various embodiments, the first predefined number may be freely selected from the group of all integers between zero and the total number of emitters. In various embodiments, the methods and systems may further include using at least one processor to determine whether the user's eyes are open or closed based on the number of reflections being greater than zero but less than the total number of emitters.

[0072] In other embodiments, systems and methods are provided for determining eye opening using an eye-tracking device. These embodiments may include receiving images of a user's eyes from an image sensor of the eye-tracking device at one or more processors. The embodiments may further include determining the radius of the pupil based on the eye images using one or more processors. The embodiments may further include determining the total area of ​​the pupil based on the pupil radius using one or more processors. The embodiments may additionally include determining an amount of the total area of ​​the pupil not obscured by either eyelid using one or more processors based on the eye images and the pupil radius. The embodiments may additionally include determining whether the user's eyes are closed using one or more processors based on the amount of the total area of ​​the pupil not obscured by eyelids and the total pupil area.

[0073] Therefore, in some embodiments, the user's eyes are open when the total area of ​​the unobstructed pupils is greater than the remaining portion of the pupil. In these or other embodiments, the user's eyes are closed when the total area of ​​the unobstructed pupils is less than the remaining portion of the pupil.

[0074] In some embodiments, the system and method may further include using one or more processors to determine, based on an image of the eye, that the center of the pupil is obscured by either eyelid, and using one or more processors to determine, based on the obscuration of the pupil center by either eyelid, that the user's eye is closed. In these or other embodiments, the system and method may further include using one or more processors to determine, based on an image of the eye, the maximum opening degree between either upper or lower eyelid over time, and using one or more processors to determine, based on the maximum opening degree over time, whether the user's eye is closed. Determining whether the user's eye is closed based on the maximum opening degree over time may include determining, based on an image of the eye, that the eye opening degree is less than a predefined proportion of the maximum opening degree over time.

[0075] In the above or other embodiments, systems and methods for instructing a user to properly position a wearable device on their head may also be provided, wherein the wearable device includes an eye-tracking device. The wearable device may be in the form of a VR or AR headset having at least one display. The wearable device may include an eye-tracking device having at least one image sensor and at least one infrared emitter, wherein the image sensor may be arranged to capture an image of at least a portion of the wearer's eyes, and the infrared emitter may be arranged to project infrared light onto the wearer's eyes.

[0076] According to these embodiments, the application is executed by a processor, resulting in items being displayed on the wearable device's display. These items can be graphical guides for the user wearing the wearable device, assisting the user in positioning the wearable device on the user's head in a manner that enables functional eye tracking via an eye-tracking device.

[0077] like Figure 8 As shown, these graphic guides can take the form of a first graphic guide 810 and a second graphic guide 820. The first graphic guide 810 may represent an approximate form of a lens incorporated into an image sensor in an eye-tracking device, and the second graphic guide 820 may represent an approximate form of a user's eye as detected by the eye-tracking device.

[0078] In practice, when the second graphic guide 820 overlays the first graphic guide 810, this provides the user with a visual representation of the position of their monocular or binocular eyes relative to the lens of the image sensor that images their eyes.

[0079] For eye tracking to function correctly, it is advantageous for the user's monocular or binocular position to be as close as possible to the center of the lens. As the user repositions or moves the wearable device, the position of their monocular or binocular position relative to the lens will change, and therefore, the position of the second graphic guide 820 relative to the first graphic guide 810 will also change.

[0080] One or both of the graphic guides 810 and 820 can change color to indicate the suitability of the user's current position for eye tracking (either monocular or binocular). For example, green can be used to indicate a highly relevant position, while red can indicate a less relevant position. Other colors (e.g., orange) can be used to indicate a sufficiently relevant position (i.e., not perfectly relevant, but sufficient for useful eye tracking).

[0081] In some embodiments, text may also be displayed on the display to instruct the user on how to reposition the wearable device. Alternatively, the text may adjust in response to the currently determined position of the user's single or binocular eyes.

[0082] The position of the wearable device can be adjusted by moving the entire unit, or the two lenses can be adjusted relative to each other. This can be done by a mechanical device (e.g., rotating a knob connected to gears to make the two lenses slide closer to or further apart from each other) or by an electronic device that controls a motor, etc.

[0083] In some embodiments, the position of the lens of the wearable device relative to the user's eyes may be determined by an automated process described below and subsequently optimized by the user with assistance / instructions from the automated process. In some of these embodiments, the automated process may also automatically optimize the position even after the position of the lens relative to the user's eyes has been determined.

[0084] Such a system may include a wearable device and one or more processors. The system and / or a user may perform various methods, and / or perform various methods stored on a machine-readable medium. In some embodiments, one, more, or all of the processors will be located within the wearable device. In these embodiments, one, more, or all of the processors will be located within the eye-tracking device of the wearable device. In some embodiments, one, more, or all of the processors will be located within a separate but communicatively coupled computing device (e.g., a mobile device, tablet, laptop, computer, desktop computer, or remote / cloud computing device).

[0085] The wearable device may include: a display; at least one lens movably disposed in front of the display; and an eye-tracking device including at least one emitter and at least one image sensor. The display may be a display module such as an LCD and / or LED display characteristic of VR headsets, such as a retinal scan display and / or a retinal projector, and the lens may be a Fresnel lens and / or other lenses designed to assist the wearable device in providing a three-dimensional immersive effect to the user viewing the display. In some embodiments, multiple Fresnel lenses or other lenses in series may be combined into a lens stack and used instead of a single lens. While the distance between the display and the lens may be fixed in many embodiments, the distance between the lens and the user's eyes is adjustable in the wearable device. Optimizing this distance improves the display's presentation to the user, thereby providing an enhanced three-dimensional immersive effect.

[0086] An eye-tracking device can be any eye-tracking device known in the art capable of determining various data related to a user's eyes, including data indicating the direction of gaze and data indicating the position of the eye or a portion thereof relative to the image sensor of the eye-tracking device (or another component whose position is fixed relative to the image sensor).

[0087] One or more processors may be configured to receive data from an eye-tracking device and, based at least on the data, determine a first position of at least one eye of a user wearing the wearable device, wherein the position is relative to a second position that is fixed relative to at least one lens. As described above, data from the eye-tracking device may be evaluated by the processor to determine the first position of the user's eye. However, in other embodiments, the eye-tracking device may determine the first position of the user's eye and report it directly to the processor.

[0088] In some embodiments, data from the eye-tracking device may be representative of data from a single point in time during which the image sensor of the eye-tracking device captures a single image frame. In other embodiments, data from the eye-tracking device may be representative of data from multiple consecutive image frames, while in still other embodiments, the multiple image frames are captured at different time intervals or at the same time interval. In any of the above cases, the data can be used to determine the position of the user's eyes. Embodiments using multiple images to derive average / mean data may be advantageous to ensure the elimination of minor fluctuations caused by variations / errors in the measurements from the eye-tracking device, minor movements of the wearable device relative to the user's eyes during normal use, etc.

[0089] The first location can be any consistently positioned location on the user's eye. For example, the first location could represent the position of the pupil of one of the user's eyes. In another instance, the first location could represent the position of the spherical region of the cornea of ​​one of the user's eyes. Any other consistently positioned feature on the user's eye can also be used as the first location.

[0090] The first and second positions may have any one or more of the three-axis components. For example only, any position may have an X-axis position, a Y-axis position, and a Z-axis position. In some embodiments, the X and Y axes may be coplanar with the display (and the lens positioned in front of the display), while the Z axis may be perpendicular to the X and Y axes (e.g., towards and away from the user when viewing the display; also referred to as the lens undulation direction).

[0091] In either case, the first position of the user's eye can be determined relative to a fixed second position. The fixed position can be located at any coherent point in space and can be, for example, the position of any component discussed herein or any position fixed relative to those components. In some embodiments, the second position can be fixed at a location between two lenses of the wearable device. In some augmented reality or other types of displayless head-mounted devices, light guides or waveguides can be used to transmit video display to the user. In these embodiments, the second position can be a point on a light guide or waveguide configured to transmit video projection to one or both eyes of the user, or a fixed position relative to a point on a plurality of light guides or waveguides (possibly between them).

[0092] One or more processors may also be configured to determine the distance between the first position and the second position. Since the first position is determined relative to the second position, the distance can be determined by the processors and / or the eye-tracking device. As described above, both the first and second positions may have X-axis, Y-axis, and Z-axis, and therefore the distance between the two positions can be determined as a vector in each of these directions.

[0093] Once the distance is determined, it is compared with one or more thresholds. A threshold may represent a minimum distance for the preferred positioning of the lens relative to the user's eye, a maximum distance for the preferred positioning of the lens relative to the user's eye, or both. In any given embodiment, all or some of the aforementioned direction vectors of the distance (i.e., the x-axis, y-axis, and / or z-axis) may be compared with all or some of the direction vectors of the thresholds (i.e., the x-axis, y-axis, and / or z-axis). Thus, by way of example only, the z-axis vector of the distance may be compared with the z-axis vectors of one or more thresholds to determine whether the lens is too far from or too close to the user's eye. In other embodiments, the x-axis and / or y-axis are also taken into account.

[0094] Furthermore, in some embodiments, multiple thresholds may be used. For example, an upper threshold may represent a maximum distance for which the method herein defines a preferred lens position relative to the user's eye, while a lower threshold may represent a minimum distance for which the method herein defines a preferred lens position relative to the user's eye.

[0095] In some embodiments, the threshold can be dynamic and change over time based on other variables. For example only, the processor may be notified or able to analyze the content being presented to the user on the display and determine that one or more different thresholds are preferred during this period of time (e.g., different thresholds may represent different convexity distances required for certain content), visual characteristics (e.g., brightness, contrast, percentage change in display pixels, etc.), the presence of eye protection (i.e., glasses), and / or other factors may cause changes in the threshold.

[0096] For example, if the distance is greater than a first value, the methods of various embodiments may result in information indicating that one or more lenses should be moved to a preferred position being presented to the user. This information may be presented audibly, visually, or tactilely. In various instances: the method may result in content being presented on a display, wherein the content informs the user that a lens is too close to or too far from the user's eye. The content also informs the user how to manually reposition the lens accordingly. For example, the user's wearable device may have controls to move the lens mechanically or electromechanically as described herein. The displayed content may indicate the user the location and details of this control. In some embodiments, auditory information may be presented to the user via a speaker, etc. In some embodiments, haptic feedback is presented to the user, which may be done through intermittent / pulsed operation of an automatonomechanical system that can reposition the lens without manual user input.

[0097] Therefore, in some embodiments, an electromechanical movement system may be provided to automatically reposition the lens relative to the user's eye based on an earlier distance determination (and subsequently compared with a threshold). This automatic repositioning may occur after the comparison of the determined distance with the threshold has ended. This process of determining the distance and comparing it with the threshold may be automatically repeated over time and / or repeated under the guidance of the user (via instructions from the user to the processor, etc.). In some embodiments, an automechanical movement system may also be present to potentially move each of the two lenses in the wearable device closer to or further away from each other in the x-axis direction (i.e., in the left-right direction relative to the user). These automatic movement systems can be used to implement other embodiments of the invention (e.g., those discussed above) to account for changes in inter-eye or inter-pupil distances.

[0098] Figure 9A wearable device 900 having the components described above is shown. The wearable device 900 includes a display 905 (some embodiments may have a separate display for each eye), two lenses 910, and an eye-tracking device 915. Each part of the eye-tracking device 915 may have at least one emitter 920 and at least one image sensor 925. A processor 930 may receive data from the components of the device 900 as described above and issue instructions to the components of the device 900.

[0099] The first electromechanical movement system 935 provides means for moving the lens 910 relative to each other, while the second electromechanical movement system 940 provides means for moving the lens 910 relative to the user's eye (not shown, but may be present on the left side of the device 900 during use). In some embodiments, the two movement systems 935, 940 may be manually operated by the user (i.e., via a knob, etc.). Orientation axes 945 in the X, Y, and Z directions as discussed above are also shown.

[0100] Figure 10 This is a block diagram of a method 1000 for adjusting the position of a lens in a wearable device, as described. In block 1010, a reference position (which may be predefined) is determined or otherwise identified. In block 1020, data is received from an eye-tracking device.

[0101] In box 1030, the eye position is determined from the data. In box 1040, the distance from the reference position to the eye position is determined. In box 1050, the distance is compared with a threshold.

[0102] In box 1060, if the distance is within the threshold, then method 1000 terminates at box 1080. If the distance is not within the threshold in box 1060, then in box 1070, motion guidance is provided to the user, or in box 1075, the motion system is automatically actuated to bring the distance within the threshold. Method 1000 may be repeated whenever the system resets, between application execution times on the wearable device, or at regular or irregular intervals, to ensure a preferred lens distance.

[0103] Figure 11This is a block diagram illustrating a dedicated computer system 1100 that can implement embodiments of the present invention. This example illustrates the dedicated computer system 1100, which may be used, for example, wholly, partially, or with various modifications, to provide the functionality of, for example, the display 905, eye-tracking device 915, processor 930, mobile systems 935, 940, and / or other components of the present invention discussed above. For example, various functions of the processor 930 may be controlled by the dedicated computer system 1100, including (by example only) determining the distance from eye features to a reference point, comparing the distance to a predefined threshold, etc.

[0104] A dedicated computer system 1100 is shown to include hardware elements electrically coupled via a bus 1190. The hardware elements may include one or more central processing units 1110, one or more input devices 1120 (e.g., mouse, keyboard, touchpad, eye-tracking device, etc.), and one or more output devices 1130 (e.g., display device, printer, etc.). The dedicated computer system 1100 may also include one or more storage devices 1140. For example, the storage devices 1140 may be disk drives, optical storage devices, solid-state storage devices, such as random access memory (RAM) and / or read-only memory (ROM), which may be programmable, flash-updatable, etc.

[0105] The dedicated computer system 1100 may additionally include a computer-readable storage medium reader 1150, a communication system 1160 (e.g., a modem, a network card (wireless or wired), an infrared communication device, Bluetooth), etc. TM The dedicated computer system 1100 may include a dedicated computer system 1180 (such as a cellular communication device, etc.) and a working memory 1180, wherein the working memory 1180 may include RAM and ROM devices as described above. In some embodiments, the dedicated computer system 1100 may further include a processing acceleration unit 1170, wherein the processing acceleration unit 1170 may include a digital signal processor, a dedicated processor, etc.

[0106] The computer-readable storage medium reader 1150 may be further connected to a computer-readable storage medium, which, together with (and, where appropriate, in conjunction with storage device 1140) broadly refers to a remote, local, fixed, and / or removable storage device plus storage medium for temporarily and / or more permanently storing computer-readable information. The communication system 1160 may allow data exchange with networks, systems, computers, and / or other components described above.

[0107] The dedicated computer system 1100 may also include software elements shown as currently residing in working memory 1180, including an operating system 1184 and / or other code 1188. It should be understood that alternative embodiments of the dedicated computer system 1100 may have various variations relative to what has been described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software, such as applets, etc.), or both. Furthermore, connections to other computing devices, such as network input / output and data acquisition devices, may also occur.

[0108] The software of the dedicated computer system 1100 may contain code 1188 for implementing any or all of the functions of various elements of the architecture described herein. For example, software stored on and / or executed by a dedicated computer system such as dedicated computer system 1100 may provide the functions of, for example, the display 905, eye-tracking device 915, mobile systems 935, 940, processor 930, and / or other components of the invention discussed above. Methods that can be implemented by software on some of these components have been discussed in more detail above.

[0109] The invention has been described in detail for clarity and ease of understanding. However, it should be understood that certain changes and modifications may be practiced within the scope of the appended claims.

Claims

1. A method for determining eye opening using an eye-tracking device, wherein the method comprises: Images of the user's eyes are received from the image sensor of the eye-tracking device at one or more processors; The radius of the pupil is determined by one or more processors based on the image of the eye; The total area of ​​the pupil is determined by the one or more processors based on the radius of the pupil; The one or more processors determine the amount of the total area of ​​the pupil that is not obscured by either eyelid, based on the image of the eye and the radius of the pupil; as well as The one or more processors determine whether the user's eyes are closed based on the amount of the total area of ​​the pupil not obscured by the eyelids and the total area of ​​the pupil.

2. The method for determining eye opening degree using an eye-tracking device according to claim 1, wherein: The user's eyes are open when the total unobstructed area of ​​the pupil is greater than the remaining portion of the pupil.

3. The method for determining eye opening degree using an eye-tracking device according to claim 1, wherein: The user's eyes are closed when the total unobstructed area of ​​the pupil is smaller than the remaining portion of the pupil.

4. The method for determining eye opening degree using an eye-tracking device according to claim 1, wherein the method further comprises: The one or more processors determine, based on the image of the eye, that the center of the pupil is obscured by either eyelid; as well as The one or more processors determine that the user's eyes are closed based on the fact that the center of the pupil is obscured by either eyelid.

5. The method for determining eye opening degree using an eye-tracking device according to claim 1, wherein the method further comprises: The one or more processors determine the maximum opening degree between any upper and lower eyelids over time based on the image of the eye; as well as The one or more processors determine whether the user's eyes are closed based on the maximum degree of eye opening over time.

6. The method for determining eye opening using an eye-tracking device according to claim 5, wherein determining whether the user's eyes are closed based on the maximum eye opening over time comprises: Based on the image of the eye, it is determined that the eye's opening degree is less than a predefined proportion of the maximum opening degree over time.

7. A computer-readable medium storing instructions that, when executed on a computer, perform the method according to claim 1.

8. A system for determining eye opening using an eye-tracking device, the system comprising a processor configured to: Receives images of the user's eyes from the image sensor of the eye-tracking device; The radius of the pupil is determined based on the image of the eye; The total area of ​​the pupil is determined based on the radius of the pupil; The amount of the total area of ​​the pupil that is not obscured by either eyelid is determined based on the image of the eye and the radius of the pupil; as well as Whether the user's eyes are closed is determined based on the amount of the total area of ​​the pupil not obscured by the eyelids and the total area of ​​the pupil.

9. The system of claim 8, wherein the processor is further configured to determine that the user's eyes are open when the total unobstructed area of ​​the pupil is greater than the remaining portion of the pupil.

10. The system of claim 8, wherein the processor is further configured to determine that the user's eyes are closed when the total unobstructed area of ​​the pupil is less than the remaining portion of the pupil.

11. The system of claim 8, wherein the processor is further configured to: Based on the image of the eye, it is determined that the center of the pupil is obscured by either eyelid; and The user's eyes are determined to be closed based on the fact that the center of the pupil is obscured by either eyelid.

12. The system of claim 8, wherein the processor is further configured to: The processor determines, based on the image of the eye, the maximum opening degree between any upper and lower eyelids over time; and Whether the user's eyes are closed is determined based on the maximum degree of eye opening over time.

13. The system of claim 12, wherein the processor is further configured to: By determining, based on the image of the eyes, that the degree of eye opening is less than a predefined proportion of the maximum degree of eye opening over time, the user's eyes are determined to be closed based on the maximum degree of eye opening over time.

Images