Eye tracking enabled wearable device

Abstract

A device (1300) adapted to be worn by a user is disclosed, the device comprising an optical element, a light source, and a sensor. The optical element is adapted to be arranged in front of an eye (1312) of the user and formed of a light transmissive material that allows the user to see the optical element, where the light source is arranged on the optical element and adapted to illuminate at least a portion of the eye of the user. Further, the sensor is adapted to capture light that has been emitted from the light source and reflected on the eye. Also disclosed is a system comprising such a device and a processor adapted to determine a gaze direction of a user based on light captured by the sensor.

Description

[0001] This application is a divisional application of Chinese invention patent application filed on December 30, 2016, with international application number PCT / EP2016 / 082946, national application number 201680087040.1, and entitled "Wearable device with eye tracking enabled".

[0002] Cross-reference to related applications This application claims priority to U.S. Provisional Patent Application No. 62 / 329,719, filed April 29, 2016, entitled “WEARABLE DEVICE COMPRISING ADISPLAY AND IMAGE SENSOR”, and International Application No. PCT / US16 / 53783, filed September 26, 2016, entitled “EYE-TRACKING ENABLED WEARAL BEDEVICES”, the entire disclosure of which is hereby incorporated herein by reference for all purposes, as if fully set forth herein. Summary of the Invention

[0003] In one embodiment, a method for panning content on a display of a wearable device is provided. The method may include: determining a user's gaze direction using an eye-tracking device. The method may further include: determining the user's head direction using a motion detection system. The method may further include: causing panning of the content displayed on the display of the wearable device in a specific direction, based at least in part on the fact that both the gaze direction and the head direction are aligned with that specific direction. The method may additionally include: determining, during panning of the content, that the user's gaze direction has returned to a neutral position using the eye-tracking device. The method may further include: causing cessation of panning of the content displayed on the display, based at least in part on the fact that the user's gaze direction has returned to the neutral position.

[0004] In another embodiment, a non-transitory machine-readable medium is provided, on which instructions are stored for panning content on a display of a wearable device. These instructions can be executed to perform a method. The method may include: determining the gaze direction of a user of the wearable device using an eye-tracking device. The method may further include: determining the head orientation of the user of the wearable device using a motion detection system. The method may further include: causing panning of content displayed on the display of the wearable device in a specific direction, at least in part based on the fact that both the gaze direction and the head orientation are aligned with that specific direction. The method may additionally include: determining, during panning of the content, that the user's gaze direction has returned to a neutral position using the eye-tracking device. The method may further include: causing cessation of panning of the content displayed on the display of the wearable device, at least in part based on the user's gaze direction returning to the neutral position.

[0005] In another embodiment, a system for panning content on a display of a wearable device is provided. The system may include an eye-tracking device, a motion detection system, and one or more processors. The eye-tracking device may be used to determine the gaze direction of a user of the wearable device. The motion detection system may be used to determine the head orientation of the user of the wearable device. The one or more processors may be configured to: at least in part based on the gaze direction and the head orientation both being aligned with a specific direction, cause panning of content displayed on the display of the wearable device in that specific direction. The processors may also be configured to: determine, during panning of the content, that the user's gaze direction has returned to a neutral position via the eye-tracking device. The processors may be further configured to: at least in part based on the user's gaze direction returning to the neutral position, cause cessation of panning of content displayed on the display of the wearable device. Attached Figure Description

[0006] The present invention has been described in conjunction with the accompanying drawings: Figure 1 An exemplary wearable device of the present invention, having a display device and an eye-tracking device, is shown; Figure 2 Exemplary contour sensor outputs of various embodiments of the present invention are shown; Figure 3A The horizontal and vertical pixel bins used in various embodiments of the invention are shown; Figure 3B This invention illustrates a possible embodiment of a method for determining gaze direction; Figure 4 An embodiment of the application of the cold light mirror of the present invention to a lens of a head-mounted device is shown; Figure 5This invention illustrates a possible method embodiment for panning a virtual reality display; Figure 6 This invention illustrates a possible method embodiment for mapping an input device to a virtual object; Figure 7A A system embodiment of the present invention, comprising a sensor / illuminator array disposed on and / or in a lens of a wearable device, is shown; Figure 7B A close-up view of an embodiment of the present invention, in which the optical fiber is disposed within a groove on the main viewing lens, is shown. Figure 7C It shows Figure 7B A cross-sectional view of the endpoint of the optical fiber; Figure 8 It is a block diagram of an exemplary computer system that can be used in at least a portion of the apparatus or system of the present invention or that can implement at least a portion of the method of the present invention; Figure 9 This is a view of a display device that modifies the image in response to the user's gaze point according to the present invention; Figure 10A It is a simplified diagram showing how image quality can continuously change within a modified image region; Figure 10B It is a simplified diagram showing how image quality can gradually change within a modified image area; Figure 11 This is a view of a display device that modifies an image in response to a detected change in the user's gaze point, according to the present invention. Figure 12 This is a flowchart of a possible method for modifying an image based on the user's gaze point, according to the present invention.

[0007] Figure 13 This is a schematic cross-section of a device according to some embodiments; Figure 14 The cross-section of the optical element is shown; Figure 15 It is a cross-section of a part of an optical element; Figure 16 This is a front view of the optical element; and Figures 17a to 17c show different examples of wearable devices that include optical elements.

[0008] In the accompanying drawings, similar parts and / or features may have the same numerical reference numerals. Furthermore, various parts of the same type may be distinguished by appending a letter after the reference numeral, which distinguishes similar parts and / or features. If only the first numerical reference numeral is used in the specification, the description applies to any of the similar parts and / or features having the same first numerical reference numeral, regardless of the letter suffix. Detailed Implementation

[0009] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing one or more exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the invention as set forth in the appended claims.

[0010] For example, any details discussed with respect to one embodiment may or may not be present in all contemplative versions of that embodiment. Similarly, any details discussed with respect to one embodiment may or may not be present in all contemplative versions of other embodiments discussed herein. Finally, the absence of any discussion of details regarding embodiments herein should be understood implicitly to be that such details may or may not be present in any version of any embodiment discussed herein.

[0011] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other elements in this invention may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0012] Furthermore, it should be noted that individual embodiments may be described as processes depicted as flowcharts, data flow diagrams, structural diagrams, or block diagrams. Although flowcharts may describe operations as sequential processes, many of these operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process may terminate upon completion of its operations, but may have additional steps not discussed or included in the diagrams. Furthermore, not all operations in any specifically described process may occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination corresponds to the function returning to the calling function or the main function.

[0013] The term "machine-readable medium" includes, but is not limited to, transient or non-transient, portable or fixed storage devices, optical storage devices, wireless channels, or any other medium capable of storing, containing, or carrying (multiple) instructions and / or data. A code segment or machine-executable instruction can represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or software circuit by transmitting and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be transmitted, forwarded, or transported by any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0014] Furthermore, embodiments of the invention can be implemented, at least partially, manually or automatically. Manual or automatic implementations can be performed or at least assisted by using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented as software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks can be stored in a machine-readable medium. Multiple processors can perform the necessary tasks.

[0015] This invention generally relates to wearable devices comprising a display and an image sensor, which use information obtained through the image sensor to modify information on the display. Specifically, this invention relates to systems and methods for modifying information on a display using information about the eyes.

[0016] Wearable devices containing displays are well-known and are commonly used in virtual reality (VR) and augmented reality (AR) systems. In these systems, the display is used to provide users with experiences that simulate different realities (in the case of VR) or augmented reality (in the case of AR).

[0017] In some cases, wearable devices do not need to include any kind of display. For example, the system described in U.S. Patent No. 9,041,787 does not require a display. The entire contents of the aforementioned patent are hereby incorporated herein by reference for all purposes, as if fully set forth herein.

[0018] In some embodiments, the use of eye-tracking devices, etc., can be combined with these wearable devices to improve their performance. For example, U.S. Patent Application Publication No. 2015 / 0062322 describes a wearable eye-tracking device. The entire contents of the aforementioned patent disclosure are hereby incorporated herein by reference for all purposes, as if fully set forth herein.

[0019] The paper "Using an Eye-Tracking System to Improve Camera Motions and Depth-of-Field Blur Effects in Virtual Environments" (Hillaire et al., Virtual Reality Conference, 2008) describes an implementation of eye tracking in virtual reality devices, whereby eye tracking is used to determine the user's focal point in the virtual environment. This focal point is then used when rendering the virtual environment to improve the user's perception of movement within it. The entirety of the above disclosure is hereby incorporated herein by reference for all purposes, as if fully described herein.

[0020] Embodiments of the present invention seek to provide improved solutions for eye tracking in wearable devices and improved use of eye tracking information in VR, AR, or other environments. These improvements may involve hardware solutions used in wearable devices and software solutions used with the same or similar devices.

[0021] Therefore, the object of at least some embodiments of the present invention is to provide an improved wearable eye-tracking system. This or other objects of the embodiments of the present invention will become apparent from the description and claims together with the accompanying drawings.

[0022] Various embodiments and aspects of the invention will be arranged using headings herein to facilitate a more readily understood understanding of the invention.

[0023] According to a first aspect of the invention, a wearable device is provided, which includes at least a display and an eye-tracking device. Information from the eye-tracking device can be used by an associated computing device (e.g., a computer, tablet computer, mobile device, or other processor-enabled device) to influence or change items displayed on the display.

[0024] In some embodiments, the eye-tracking device includes at least one image sensor and at least one illumination source. In some embodiments, the eye-tracking device may include two image sensors and two illumination sources. According to embodiments, any number of image sensors and illumination sources is possible, and a considerable number of image sensors and illumination sources may or may not exist. The illumination source projects illumination onto the eyes (of the wearer) of the wearable device, and the image sensors capture one or more images of the wearer's (of the wearer's) eyes (of the wearer). Based on the position of the illumination reflected on the eyes, the user's gaze direction can be determined. For example, a suitable system for determining a user's gaze direction using a wearable eye-tracking device is described in, for example, U.S. Patent Application Publication No. 2015 / 0061996. The entire contents of the foregoing disclosure are hereby incorporated herein by reference for all purposes, as if fully set forth herein.

[0025] The computing unit or other associated processing device can be directly or wirelessly connected to the eye-tracking device and the display. The computing unit or other processing device performs calculations based on information from the eye-tracking device and, possibly after considering these calculations, controls or modifies the information or other content displayed on the display.

[0026] Figure 1 A block diagram of a typical VR headset 100 with gaze tracking functionality, according to various embodiments of the present invention, is shown. Different embodiments of the invention may have a smaller or larger number of components and may be variably positioned / configured as discussed herein. The headset 100 may include a display device 110 and an eye-tracking device 120, which includes multiple illuminators 123 and multiple image / light sensors 126. The illuminators 123 and image / light sensors 126 may include dedicated lenses. A processor 130 may provide computational / operational control for these components. In some embodiments, the headset 100 may also include a motion / movement detection system 140 that detects movement of the headset 100. A communication bus, not shown, may provide wired and / or wireless communication with an associated computing device.

[0027] Contour sensor In some embodiments, a contour sensor can be used to determine the user's gaze direction. In these embodiments, a wearable device may be provided that includes at least one contour sensor pointing towards at least one of the user's eyes. An example of a suitable contour sensor is the S9132 contour sensor manufactured by Hamamatsu. Contour sensors operate by aggregating the values ​​of all pixels in a row and / or column into a single value, as will be understood by those skilled in the art.

[0028] Furthermore, at least one infrared light emitter can be provided and directed towards at least one of the user's eyes. In this way, a contour sensor can be used to determine the location where infrared light is reflected from the user's cornea—otherwise referred to as a "glint." Therefore, basic gaze tracking can be performed by analyzing the location of the glint on the user's cornea, as will be readily understood by those skilled in the art. In a further improvement, two or more contour sensors can be used. This offers several advantages.

[0029] First, if more than one two-dimensional contour sensor is used, the corneal radius and flicker location can be determined in three dimensions, and then the corneal center of the user's eye can be determined in three dimensions, which is quite different from two dimensions.

[0030] Secondly, arranging at least two contour sensors such that the resulting flicker does not overlap allows for more accurate flicker detection. For example, consider the case where two flickers are caused by illuminators A and B. If these flickers are imaged by a one-dimensional contour sensor aligned in the same direction as the flicker, the sensor will only register a single response caused by the two flickers, making it difficult or impossible to determine whether the flicker is caused by illuminator A, illuminator B, or both. Therefore, aligning the illuminators and contour sensors in a manner that prevents the flicker from overlapping in any readout of any contour sensor is advantageous.

[0031] Alternatively, the illuminators can be adjusted to ensure that only one illuminator is lit at any given time. Contour sensors using this arrangement can be designed to operate at a very fast sampling rate, enabling multiple samples and capturing flicker from only one illuminator at each sample within a short time frame to ensure minimal eye movement between samples.

[0032] Furthermore, multiple one-dimensional profile sensors can be used. For this system to function accurately, each sensor must be positioned and rotated relative to each other. In this way, the individual dimensions of each sensor can alternate between horizontal and vertical configurations, but the relative orientation difference does not need to be limited to 90 degrees. Additionally, in these arrangements, it is desirable to add cylindrical lenses to each one-dimensional profile sensor.

[0033] In some embodiments, the contour sensor may output to a processing device the sum of all rows and / or the sum of all columns of the sensor's pixel matrix. The reflection of infrared light from an infrared illuminator onto the user's cornea—known as flicker—can be detected using a technique called peak detection. Peak detection can be performed on both the row sums and the column sums. Figure 2 This technique demonstrates the output of the sum of rows or columns, where the x-axis represents the pixels of the contour sensor and the y-axis represents the magnitude, with examples highlighting the peak values.

[0034] In an optional improvement, to facilitate high-speed eye tracking where the previously calculated flicker location is known, only a subset of pixels must be analyzed to find the peak. For example, only 10 to 20 pixels close to the known previous flicker location can be analyzed.

[0035] Once the flicker location is known, the gaze direction can be determined using, for example, a polynomial flicker-to-gaze model. gaze X =C 1X +C 2y +C 3xy +C 4X 2 +C 5y 2 +C6 gaze y =C 7x +C 8y +C 9xy +C 10x 2 +C 11y 2 +C 12 , Wherein, gaze x and gaze y are the x and y positions of the gaze point, x and y are the x and y positions of the blink, and c1…c12 are calibration model parameters.

[0036] In some embodiments, contour sensors can be combined to provide more than one illuminator. These illuminators can be selectively enabled or adjusted, and the processing device can determine which illuminator to enable based on metrics derived from the captured image data. Alternatively, the illuminators can be illuminated in a predefined order, thereby ensuring that only one illuminator is illuminated at any given time.

[0037] In another embodiment, the wearable device may further include at least one image sensor of a conventional area sensor type. This conventional sensor may also be pointed at at least one of the user's eyes. The conventional sensor can capture images of the user's eyes, and the system can perform conventional pupil-center corneal reflection (PCCR) eye tracking. PCCR is a well-known and easily understood method for determining a user's gaze. Further information on this method can be found in many places, including "General theory of remote gaze estimation using the pupil center and corneal reflections" by Guestrin, ED, and Eizenman, E. (Biomedical Engineering, IEEE Transactions, Vol. 53, No. 6, pp. 1124, 1133, June 2006).

[0038] By combining a contour sensor, which will be enabled to output the sum of all columns and / or the sum of all rows, with a conventional image sensor, the system can perform flicker tracking using the contour sensor and PCCR tracking using the conventional sensor. Due to the additional information provided by the contour sensor, the conventional sensor only needs to operate at 0.5Hz to 10.0Hz. Therefore, the system can achieve low power consumption, low latency, and high frame (or sampling) rates.

[0039] As long as the sensor remains fixed relative to the user's face, a contour sensor tracking the flicker will provide sufficient gaze data, while images from a conventional image sensor allow for slip compensation whenever the sensor moves relative to the face. However, eye movement typically occurs much faster than any potential slippage of the wearable device on the user's head. Therefore, it is of interest to find a method that tracks the flicker position with only low power and low latency. This could allow for foveated rendering in VR headsets, where a relatively low-power eye-tracking solution could allow for significant savings in the overall power consumption of the VR system, as this would substantially reduce the power requirements for graphics rendering.

[0040] For example, the sensor can be configured to operate in a mode where it cycles through two or more illuminators, illuminating only one illuminator per sensor exposure. For instance, the sensor can be configured to operate in a loop where, during a first sensor exposure, a first illuminator is illuminated, and then the sum of at least 10% of the pixel elements in at least 10% of the rows and at least 10% of the pixel elements in at least 10% of the columns of the sensitive region is calculated (and a flicker location is detected). Subsequently, a second illuminator is illuminated during a second sensor exposure, and the sum of at least 10% of the pixel elements in at least 10% of the rows and at least 10% of the pixel elements in at least 10% of the columns of the sensitive region is calculated. Afterward, the sensor captures a regular image from at least one sub-region of the sensor's sensitive region while at least one of the illuminators is illuminated.

[0041] In an alternative implementation, the sensor can be configured to operate in a loop such that, during a first sensor exposure, a first illuminator is illuminated, and then the sum of at least 10% of the pixel elements in at least 10% of the rows of the sensitive region is calculated. Next, the first illuminator is illuminated during a second sensor exposure, and the sum of at least 10% of the pixel elements in at least 10% of the columns of the sensitive region is calculated. Thereafter, the sensor captures a regular image from at least one sub-region of the sensor's sensitive region while at least one of the illuminators is illuminated.

[0042] An image sensor that is enabled to operate as a conventional image sensor but is also enabled to output the outline of the sum of pixel rows and / or the sum of pixel columns of a sensitive area may include output pins for synchronizing the exposure of one or more illuminators with the sensor exposure.

[0043] Image sensors that are enabled to operate as conventional image sensors but are also enabled to output the outline of the sum of pixel rows and / or pixel columns of sensitive areas can also support daisy-chaining, allowing two or more sensors to be connected to the processing unit via the same data bus, such as the MIPI CSI-2 interface.

[0044] An image sensor that is enabled to operate as a conventional image sensor but is also enabled to output the contour of the sum of pixel rows and / or pixel columns of a sensitive region may further include means for detecting the distance to objects in its field of view. This can be accomplished through time-of-flight analysis.

[0045] To compensate for ambient light profile data and / or regular image data, the image sensor can be sampled periodically without active illumination, i.e. without turning on any illuminators in the illuminator.

[0046] Image sensors can also be used to identify users via iris recognition for login, security, and / or other reasons.

[0047] Image sensors that are enabled to operate as regular image sensors but are also enabled to output the contours of pixel rows and / or pixel columns of sensitive regions can be designed such that each pixel can only be included in the column sum or row sum when the sensor is operating in contour mode. For example, pixel elements can be laid out in a checkerboard pattern, where only every other pixel can be totaled in the row contour, and other pixels can be totaled in the column contour.

[0048] An alternative implementation of the image sensor is to divide the sensitive region into a checkerboard pattern, where every other pixel is read out in the rows and every other pixel is read out in the columns, essentially having an A / D converter next to each row and every other column. This means that the typical image from this sensor would be two images with half the resolution of the sensor, one read out vertically and one read out horizontally. This increases the computational complexity of conventional PCCR eye tracking. Figure 3 illustrates an example and such a configuration in other examples discussed above, where a region image sensor, a 2D contour sensor, and / or two orthogonally oriented 1D contour sensors, simulating contour sensors through their individual pixels, can be employed as follows: every other pixel in every row (as marked with 'X') is used for horizontal blinking / image data acquisition (horizontal bins 1 to 5), and every other pixel in every column (as also marked with 'X') is used for vertical blinking / image data acquisition (vertical bins 1 to 5).

[0049] The benefit will be that the sensor is designed in such a way that it supports horizontal pixel binning of the image read from each line and vertical pixel binning of the image read from each column, thereby facilitating low-power flicker detection. For example, the sensor can be designed to sum values ​​from 8 to 16 pixel elements or even more into a single value, meaning it can operate as a contour sensor that supports sub-windowing, which can help suppress irrelevant signals and reduce noise.

[0050] Sensors enabled to operate as contour sensors with or without sub-windowing support may include power logic for detecting the center of flicker, thereby further reducing power consumption and the amount of data that needs to be sent to an external processing unit. When the sensor supports sub-windowing, the sensor can change the position of the sub-window after flicker center detection to ensure that subsequent contour images include the flicker.

[0051] An alternative implementation of an eye tracker that supports both traditional PCCR eye tracking and flicker tracking to allow low-power eye tracking at latency and data rates supporting foveated rendering would have a conventional sensor for imaging the eye, but would include HW logic for flicker center detection when the sensor operates in a predefined sub-window mode. This is possible, for example, only with sub-windows of 24×24 pixels, 32×32 pixels, 48×24 pixels, or some other suitable resolution.

[0052] In some implementations, an organic light-emitting diode (OLED) display can be used. OLED displays are typically transparent, with a reflector placed behind them to ensure all light is emitted forward toward the viewer. For eye-tracking purposes, a cold-light mirror can be positioned behind the OLED display to reflect virtually all visible light from the display toward the user's eyes while allowing NIR light to pass through. In this way, an eye-tracking sensor that detects NIR light can be placed behind the display, looking toward the viewer through the display, thus achieving a favorable viewing angle toward the user's eyes.

[0053] In VR headsets, it's common to use Fresnel lenses to position the display at a further distance from the user compared to its actual distance. A drawback of such lenses is that they distort the image from the display, and similarly, they distort the image of the eye as seen by an eye-tracking sensor through the lens. Therefore, it's likely preferable to compensate for this distortion within the eye-tracking algorithm.

[0054] An additional effect of Fresnel lenses is that they can cause circular defects in images of the eye, such as those seen from an eye-tracking sensor. When you throw in a small stone and try to look beneath the surface, the pattern resembles the distorted effect of ripples on water. Therefore, it may be preferable to calibrate the eye-tracking sensor viewing the eye through a Fresnel lens before the machine vision algorithm attempts to detect different eye features or flickering, ensuring that the image from the sensor is adjusted to compensate for the limitations of the Fresnel lens.

[0055] Image sensors that are enabled to operate as regular image sensors but are also enabled to output the contours of the sum of pixel rows and / or the sum of pixel columns in sensitive areas can be designed to support sub-windowing. This is common in regular image sensors, but by allowing sub-windowing when operating in contour mode, many potentially destructive reflections or light sources from the sensor's field of view can be removed before pixel elements are summed into row and / or column contours, thus ensuring higher accuracy in flicker location determination and allowing sub-windows to be recentered for subsequent sampling.

[0056] Figure 3B A block diagram of a possible method 300 for determining a user's gaze direction, as described above, is shown. At block 310, one or more eyes are illuminated using one or more available illuminators. At block 320, one or more eyes are imaged using one or more available sensors (contour sensors and / or region image sensors). Blocks 310 and 320 may be repeated as needed. At block 330, as described above, the number of sensors and / or which portion thereof may be limited to focus on sub-windows of possible pixel / resolution, where imaging may be necessary for determining the gaze position. Blocks 310, 320, and 330 may be repeated as needed. At block 340, the gaze direction can be determined, and blocks 310, 320, 330, and 340 may be repeated as needed to continuously redetermine changing gaze directions and / or sub-windowing as required.

[0057] heat mirror In one embodiment, such as Figure 4 As shown, a lens 400 is provided in a wearable device such as a virtual reality headset. The lens 400 is positioned so that the wearer of the device can view the display 110 through the lens 400. A heat mirror coating or film 410 is fixed to the top of the lens 400, and a plurality of illuminators 420 are fixed to the top of the heat mirror film 410. The film 410 has a cutout portion 430 toward the center of the lens 400 to allow an eye-tracking camera to view through the lens 400.

[0058] The heat mirror coating or film 410 can be applied directly to the lens 400, or for ease of manufacturing, the heat mirror coating or film 410 can also be in the form of a separate part that does not function as a lens. Furthermore, and also for ease of manufacturing, a separate glass and plastic part can be placed on top of the heat mirror, and the illuminator 420 can be attached to this separate part.

[0059] Illuminator 420 emits infrared light toward the eyes of wearer 440, which is then reflected back from the wearer's eyes toward lens 400. Film 410 has the properties of a thermal mirror; in other words, film 410 allows light in the visible spectrum to pass through while preventing light in the infrared spectrum from passing through. In this way, visible light emitted by display 110 behind lens 400 can reach the wearer's eyes while preventing most infrared light from passing through.

[0060] An image sensor positioned behind the lens and viewed through the cutout portion 430 captures an image of the wearer's eye containing the reflection of infrared light emitted by the illuminator 420. The image sensor is connected to a processing device that then analyzes those images to determine the user's gaze direction based on the reflection of infrared light.

[0061] Although this embodiment has been described with reference to several infrared illuminators 420, the present invention can be fully utilized with only one illuminator.

[0062] The illuminator 420 can be applied to the membrane 410 in several ways, first simply by using adhesive, or alternatively, by printing electronic equipment directly onto the membrane 410. The illuminator 420 will further need to have communication lines applied to the membrane 410, allowing the illuminator to receive power for being switched on or off.

[0063] In one embodiment, the wearable device includes two lenses 400 having an associated film 410 and an illuminator 420. The film 410 can be applied to the lens using adhesive or some other semi-permanent material. In a system with two lenses, light can be reflected from the user's skin into the lenses, whereby the lenses have a waveguide effect and direct the light away from the wearable device. The presence of the heat mirror 410 serves to reduce the occurrence of this light.

[0064] In an alternative embodiment, the illuminator 420 may be placed on the side of the lens 400 (or an additional part as previously described). The illumination from the illuminator 420 can then travel through the lens 400 to be emitted in front of the user's eyes. This light guidance occurs in the lens area where the heat mirror 410 has been applied to prevent the illumination from being emitted toward the display 110.

[0065] In a further aspect of the invention, an angled thermal mirror 410 may be added in front of the display 110, and an image sensor may be arranged to view the thermal mirror. The lens 400 is then placed in front of the user's eye, with an illuminator 420 positioned near the lens. The illuminator 420 illuminates the user's eye, and the diffuse illumination from the illuminator 420 is reduced due to the thermal mirror film or coating, as previously described, on the lens 400. Cutouts in the thermal mirror film or coating allow the image sensor to capture an image of the user's eye through the angled thermal mirror.

[0066] algorithm In various embodiments, an algorithm is used to determine the gaze direction based on an image containing the eye and the reflection of infrared light from the eye. The processing unit performs calculations based on the algorithm and the captured image to determine the gaze direction.

[0067] The algorithms used in the wearable devices described in this paper are substantially similar to those used in existing remote eye-tracking units. Therefore, the basic methods for determining gaze direction should be better understood by those skilled in the art.

[0068] However, several improvements are discussed below.

[0069] Pupil position One step in determining gaze direction in an eye-tracking device is estimating the position and size of the pupil. In some embodiments, methods can be used to estimate pupil size and / or position.

[0070] In the captured image, the reflected position of the infrared light emitted by the illuminator is analyzed to determine its position relative to previously captured images. The displacement of the reflected light, combined with the pupil position from the previously captured images, is then used to determine the pupil's position in the current image.

[0071] Fresnel lens When a Fresnel lens is present in a wearable device, the image captured by the eye-tracking image sensor through the Fresnel lens typically contains concentric circles, as is present in the Fresnel lens. When attempting to determine gaze detection, the processing unit may mistakenly identify these concentric circles as the pupil edge, thus necessitating the rapid and accurate removal of these concentric circles.

[0072] In some embodiments, erosion is used to remove these concentric circles from the captured image. Erosion is an image processing concept that those skilled in the art will well understand. Using erosion, a small kernel, such as 3×3 or 5×5 pixels, is passed through the captured image pixel by pixel. For each pixel, the intensity value is replaced with the darkest pixel among the pixel's neighbors, where the neighbor size is defined by the kernel. Since the pupil in the captured image is dark, the bright concentric circles in the captured image are thus replaced with the dark pupil.

[0073] Blocking light Another embodiment allows the use of hardware or software to perform mathematical operations along a line across a 2D image sensor to provide an output similar to that of a contour sensor. The line can typically be rows and columns of the sensor, but is not necessarily limited to these orientations. This allows for other operations beyond simply calculating the average and / or sum of all pixel values ​​along the line, and makes it possible to mask light contributions from image portions known not to contain any flicker. Masking light contributions from image portions not containing any flicker increases the signal-to-noise ratio and thus helps detect flicker by allowing examination of the intensity contour of the contour response. In an exemplary embodiment, the area for masking can be anything outside the cornea, and the latest output from the eye-tracking algorithm can be used to give an approximate area for masking.

[0074] Using a traditional image sensor to simulate a 2D contour sensor reduces the computational load required for eye tracking and thus lowers power consumption. However, the frame rate is limited by the frame rate of the 2D imaging sensor.

[0075] When using a real contour sensor, it is possible to block the contribution of light from parts of the image that do not contain any flicker.

[0076] One way to block light from parts of an image that are known not to contain any flicker is to use one or more illuminators whose light can be spatially controlled (e.g., an infrared OLED array behind a lens, any device with a DLP or LCOS projector, or a variety of other solutions that are readily understood by those skilled in the art).

[0077] Another way to block light from image portions known not to contain any flicker is by using light-blocking elements before they enter the contour sensor. These blocking elements can be LCD-based, mechanical-based, or based on a variety of other solutions that are easily understood by those skilled in the art.

[0078] Using a traditional image sensor to simulate a contour sensor A contour sensor, as discussed earlier, can be simulated using a conventional image sensor that includes a pixel matrix. To achieve this, hardware or software can perform mathematical operations (such as calculating the average intensity level along a line on the sensor or the sum of all intensity levels along a line on the sensor) to provide an output similar to that of a contour sensor. Typically, this would be equivalent to outputting rows or columns from a conventional sensor. However, any pixel configuration, such as a diagonal, can be output. By using such a simulation system, more operations can be performed than simply the conventional average and sum of all pixel values ​​along a line, such as the masking described earlier. Additionally, it will be possible to mask detected light from areas of the captured image that are known not to contain any flicker (pixels in the image sensor). By performing this masking function, the signal-to-noise ratio will increase. An example of an area to be masked is the region outside the user's cornea, as this region does not contribute to flicker.

[0079] Blocking light from non-contributing flickering image areas can be performed using conventional contour sensors. Further options for blocking light include utilizing an illuminator whose light can be spatially controlled, such as an infrared OLED array behind a lens, any device with a DLP or LCOS projector, or any other solution readily understood by those skilled in the art. Another option is to block light from non-contributing areas from reaching the sensor, which can be achieved through mechanical solutions, LCD solutions, or any other solution understood by those skilled in the art. Mechanical LCD solutions may include placing a transparent LCD in front of the contour sensor.

[0080] Eye tracker synchronized with display For some eye-tracking applications, it is valuable to synchronize the eye-tracking device with a display, particularly a display in a wearable device. According to this aspect of an embodiment of the invention, the wearable device is provided with a display, at least one camera, and at least one illuminator. The cameras and / or illuminators form an eye-tracking device. The cameras and / or illuminators can be synchronized with the display. Synchronization can be characterized as synchronizing the camera flicker rate with the vertical sync (v-sync) of the display.

[0081] Further exciting is the ability to synchronize the eye-tracking device with one or more positioning devices. For example, the eye-tracking device could be synchronized with an inertial measurement unit or a room positioning device using infrared light or other non-visible light. Valve® has presented such a system under the name "Lighthouse." Those skilled in the art will readily understand how this synchronization can operate.

[0082] Removable eye tracker According to another aspect of an embodiment of the present invention, a removable eye tracker is provided, which can be inserted into a wearable device. The eye tracker can then be integrated with another device, such as a telephone, tablet computer, watch, or display.

[0083] An eye tracker may include at least one camera and at least one illuminator, and its primary function may be to track a user's gaze relative to a device into which the eye tracker is integrated, such as a phone, tablet, or watch. As an auxiliary function, the eye tracker is sometimes integrated into a device that can be plugged into a wearable device. This device can then provide functions such as a display for the wearable device, and the eye tracker can be used to determine the direction of the wearer's gaze. The operation of the eye tracker can be any conventional method or any method described herein.

[0084] Smile identification According to one aspect of the invention, an image sensor for eye tracking in a wearable device can also be used to capture images of the area around a user's eyes. For example, these images can be analyzed to determine whether the user is smiling and whether the smile is genuine or fake. Characteristics of the area around the user's eyes can be used to determine whether the smile is genuine or fake. See, for example, "Areyour eyes smiling? Detecting Genuine Smiles with Support Vector Machines and Gabor Wavelets" by Littlewort-Ford, Gwen, Marian Stewart Bartlett, and Javier R. Movilland (Proceedings of the 8th Joint Symposium on Neural Computation, 2001). The full disclosure of the aforementioned publications is hereby incorporated herein by reference for all purposes, as if fully set forth herein.

[0085] According to an embodiment of the invention, an image sensor for eye tracking captures at least a portion of the area around the eyes when capturing eye images, and these images can then be analyzed using known smile or other detection algorithms to determine whether the user's smile or other facial features are fake or real.

[0086] Iris recognition-based calibration According to an embodiment of the present invention, an image sensor for eye tracking captures information related to a user's iris. This iris information can be used to identify the user for input into a system connected to the eye tracker.

[0087] For example, according to some embodiments of the invention, a wearable device is provided, in which at least one image sensor and at least one infrared illuminator are provided. The image sensor and illuminator are oriented toward the eyes (one or more) of the user of the wearable device. Optionally, the device further includes a display, such as that in a virtual reality display in a wearable head-mounted device.

[0088] An image sensor captures an image of the user's iris and transmits the image to a processing device, which may be located on a wearable device or located remotely from the processing device. In this case, communication may be achieved via wired or wireless means, as those skilled in the art will understand.

[0089] Iris recognition is a known technology that uses mathematical pattern recognition techniques to uniquely identify a user’s pattern on one or both irises in order to identify or authenticate the user. The most basic form of iris recognition includes the following steps: (1) localization – calculating the inner and outer boundaries of the iris; (2) normalization – normalizing the captured data to achieve consistency; (3) feature extraction – forming feature vectors of features extracted from the captured image; and (4) matching – classifying the feature vectors using thresholding techniques.

[0090] Many algorithms have been proposed that allow for iris recognition; see, for example, Daugman JG, High Confidence Visual Recognition of Persons by a Test of Statistical Independence (IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15, No. 1I, 1993, pp. 1148–1161). The full contents of the aforementioned publications are hereby incorporated herein by reference for all purposes, as if fully expounded herein.

[0091] Based on an image of the user's iris, a processing unit connected to a wearable device (wired or wireless) can use user identification to influence its functionality. For example, when using an eye tracker, the processing unit can load a calibration curve that provides user-specific information about the deviation between the user's calculated gaze position and their actual gaze position. As another example, this identification can be used to authenticate the user as authorized to operate the wearable device or a processing unit connected to it.

[0092] Eye rotation compensation According to another aspect of an embodiment of the invention, eye rotation can be tracked. By way of illustration, the human eye is attached to muscles in such a way that, in addition to left / right and up / down movement, the eye can rotate such that the top of the iris can rotate closer to the nose, while the bottom of the iris rotates further away from the nose. The opposite direction is also possible. This type of rotation is commonly referred to as eye rotation.

[0093] When a person turns their head slightly to the side, most people automatically turn their eyes slightly in the opposite direction, bringing their eyes closer to the horizon. This effect is only noticeable during minute turns, as it is impossible to rotate the eyes by a large number of degrees in this manner.

[0094] For all individuals whose fovea is not perfectly aligned with the optical axis of the eye, this phenomenon introduces an additional source of error in eye-tracking systems.

[0095] This invention can track eye rotation by observing other features on the iris and / or the eyeball and / or using directional information from an eye tracker. This can provide the eye tracker with a better estimate of the fovea position and therefore a better estimate of the subject's gaze when the subject's head is tilted during eye tracking.

[0096] corneal curvature According to one embodiment of the present invention, the at least one image sensor captures an image of the user's eye. A computing unit uses this information to determine the shape of the user's cornea. The curvature of the cornea can be measured based on the position and orientation of the reflection from an infrared light source. In some individuals, the cornea has an abnormal curvature. This can be referred to as astigmatism.

[0097] By using information obtained from image sensors and infrared light sources, the curvature of the cornea can be modeled, and users with corneal shape abnormalities can be identified. By determining the shape of the cornea, corrective measures can be implemented, such as prescribing appropriate lenses.

[0098] Reduced power consumption and processing power In some embodiments, to further reduce the power consumption of wearable devices that may include virtual reality displays or other display systems, the eye-tracking device may directly track only the gaze position of the user's first eye. The gaze position of the user's second eye can then be determined by prediction based at least in part on the directly determined gaze position of the first eye.

[0099] This can be achieved based on the assumption that the second eye is roughly at the same level as the first eye, which is biologically typical in humans. In this way, power that would otherwise be used by additional eye-tracking devices (including illuminators, image sensors, and / or processing power) can be saved. In some embodiments, a combination of the determined gaze positions and the predicted gaze positions of both eyes can then be used to determine the area on the wearable device's display for gaze rendering, thereby providing further power savings due to improved video rendering efficiency.

[0100] In some embodiments, specific software, such as gaming applications and / or other high-end graphics applications, may request gaze data from an eye-tracking device to inform how the application might interact with the user in the future, how much graphics information should be provided to the graphics processing unit, and / or to achieve other purposes. While in other embodiments all gaze direction information may be transmitted to such an application, allowing individual applications to request this information and providing it only upon request can result in power and processing savings.

[0101] Virtual reality panning and centering In some embodiments, the ability is provided to facilitate a user panning the virtual reality display or other display system in the user's wearable device and subsequently centering it is provided. Typically, because virtual reality displays are infinitely large in any direction (i.e., a user looking to the right in the virtual display will eventually rotate 360 ​​degrees to view the original scene), panning can become awkward for the user, as they may only turn their head to a degree that makes them uncomfortable before continuing to turn their head.

[0102] To mitigate this problem, the system described in this paper allows users to pan the view on a virtual reality display by rotating their head and moving their gaze in a given direction in which panning will occur, but allows the displayed scene to be recentered near a point when the user's head has been rotated while the user's gaze has been refocused on the scene (i.e., the user's gaze is no longer panning).

[0103] In some embodiments, recentering of the gaze direction must occur and last for at least a predefined period of time to ensure that the user is indeed panning their view. For example, the user's gaze must return to a relatively central area of ​​the current view. In some embodiments, the size, shape, and position of this area can be set by the user. In some embodiments, this area can be visible in different ways, possibly as set by the user (i.e., a light-colored indicator of the area can be overlaid on other content on the display). In some embodiments, the user can be provided with an indication (i.e., audible or visual cues) to confirm that the virtual display view has been recentered, and the user can then move their head position back to a directly forward, center-neutral position. This recentering of the user's head can be achieved by the system without causing the display to be re-panned in the opposite direction to the original panning.

[0104] Similarly, if the user's gaze direction is the same as or within a certain / predefined angular direction of their head movement, the panning speed may begin and / or increase while the user's head remains straight forward (i.e., the panning speed will stop and / or decrease), but their gaze direction indicates that panning is desired (i.e., by moving the gaze direction to the edge of the peripheral view). Additionally, the panning speed may depend on and / or be proportional to the magnitude of the user's head movement. That means a large or sudden movement of the user's head may rapidly increase the panning speed, while a small or slow movement of the user's head may slowly increase the panning speed.

[0105] Therefore, in one embodiment, a system and method may be provided for panning (and recentering) content on the display of a wearable device. For example... Figure 5 As shown, the system described herein can perform method 500, which includes: at box 510, determining the gaze direction of a user of a wearable device using an eye-tracking device. The method may further include: at box 520, determining the head orientation of a user of the wearable device using a motion detection system. The motion detection system may include any internal / integrated or external / remote device for determining the user's head position. The motion detection system can determine the head position in any manner now known or hereafter known in the art.

[0106] The method may further include, at frame 530, at least in part based on both the gaze direction and head direction being aligned with a specific direction, causing a panning motion on the display of the wearable device in that specific direction. The method may further include, at frame 540, determining, via an eye-tracking device, that the user's gaze direction has returned to a neutral position during the panning motion. The method may further include, at least in part based on the user's gaze direction returning to a neutral position, causing a cessation of the panning motion on the display of the wearable device.

[0107] Additionally, the user's head orientation beyond a certain angle deviating from the neutral / forward direction can also be the panning direction / velocity. Generally, the panning speed can be based on both the gaze direction and the head orientation. The user can provide additional input to command the system to stop panning. Furthermore, in some embodiments, the system can provide the user with visual and / or audible indications that panning has stopped, and the user can move their head back to the forward / neutral orientation without considering that panning in the opposite direction would occur (as shown in box 550).

[0108] Virtual reality mapping of input devices In some embodiments, gaze direction can allow input device actions to be mapped onto the surface of a virtual object in a virtual reality display. For example, if a virtual sphere is positioned in virtual space, an eye-tracking device can determine that the user's gaze is directed toward the sphere at a given moment. Upon receiving a second input or when a predefined time period expires, the associated input device can map its control onto the virtual object.

[0109] The second input can include tactile input from traditional input devices such as a keyboard, touchpad (or other touch-sensitive device), mouse, or trackball; or non-tactile input such as voice commands or gaze input. More specifically, tactile input can have directional input on the touch-sensitive surface that controls the mapping on the virtual sphere to a user-specified location; where the directional input refers to a touch gesture; and tactile input can also be a pressure signal received from a pressure-sensitive device associated with the touch-sensitive surface; last but not least, tactile input can be combined with pressure touch and touch gestures. It may also be proximity input, where proximity touch input is received from a touch-sensitive device associated with a proximity sensor that does not physically contact the touch-sensitive surface.

[0110] The mapping control of input devices to objects can take many forms. For example, if the mouse is the mapped input device, input from the mouse can determine the rotation of the object in virtual space, or can move virtual indicators on the object, such as pointers, around the virtual object. This allows users to perform complex actions using virtual objects.

[0111] In another example, a virtual control panel with various virtual inputs, such as switches and knobs, can exist in a virtual reality display. Once it is determined that the user's gaze has been directed to the virtual control panel and a predetermined time period has elapsed, or a second input has been received, keyboard or mouse controls can be mapped to the control panel. This allows keyboard keys to be mapped to the control panel accordingly, or allows mouse movement to move visual indicators or pointers (i.e., a virtual representation of the user's hand) across the control panel to select or activate various virtual inputs thereon.

[0112] In some embodiments, a predefined portion of the virtual control panel (or other virtual object) may be an area where the user's gaze must be directed to activate the control mapping (rather than the user's gaze being positioned anywhere on the control panel). In some embodiments, this predefined portion may be visually indicated and / or labeled in the virtual display.

[0113] Therefore, in some embodiments, a system for performing the methods discussed herein can be provided. Figure 6In one example shown, a method 600 for mapping an input device to a virtual object displayed on a display device in a virtual space may include, at box 610, determining the user's gaze direction using an eye-tracking device.

[0114] The method may further include, at box 620, modifying at least part of the action that one or more processors would take in response to receiving a first input from an input device, based at least in part on gaze direction guidance to a virtual object displayed on a display device in a virtual space. This may also occur in response to gaze direction guidance to the virtual object for at least a predefined time period or upon receiving a second input. The user may also be notified of the possible modification of the action via visual / auditory cues.

[0115] Modifying the action may include changing the action from a normal action to be taken in response to receiving a first input to a specific action in a virtual space associated with the virtual object. The specific action in the virtual space associated with the virtual object may be causing the first input to rotate the virtual object and / or move the virtual object. In these or other embodiments, the specific action in the virtual space associated with the virtual object may be causing the first input to move a second object about the surface of the virtual object. The second object may move in a direction corresponding to the direction associated with the first input and may be a pointer or other virtual position marker.

[0116] The method may further include: thereafter, in response to receiving input from the input device at box 630, causing an action to occur at box 640, wherein the action associates the first input with the interaction with the virtual object.

[0117] In response to a user input, which could be a third input or other input, the mapping of the input to the virtual object can be terminated after a certain amount of time or because the user's gaze has moved away from the virtual object in question, as shown in box 650.

[0118] For example only, input devices that can be mapped include: touch-sensitive devices, touchscreens, pressure-sensitive devices, mice, trackballs, joysticks, handheld controllers, styluses, keyboards, voice input devices, and / or any other input devices discussed herein or known in the art.

[0119] In some embodiments, the apparent effects that an input device mapped to a virtual object as described herein will be magnified or reduced, such as those perceived by a user during operation. For example, in an embodiment where the user is rotating a virtual representation of a planet, a very small motion input of the mouse mapped to that virtual object may cause a large degree of rotation of the virtual planet. Conversely, in the case of a virtual representation where the user is rotating a molecule, a very large input of the mouse mapped to that virtual object may cause only a small degree of rotation of the virtual molecule. The amount of magnification or reduction may depend on another input, such as one set by the user, or may be pre-programmed into a particular application. The magnification or reduction may also vary according to various user and program setting variables.

[0120] Interest saccade prediction area in the display In some embodiments, saccades performed by the user's (multiple) eyes can be detected and these saccades can be used to estimate or determine where the user's gaze direction will fall upon completion of the saccade. Additionally, saccades performed by different individual users can be analyzed and information associated with these saccades can be stored to predict future changes in the gaze direction of a particular user when a saccade is performed by that user (i.e., saccades at different speeds result in different angular changes in the gaze direction for different users).

[0121] Additionally, known salient regions or objects of interest can inform any prediction algorithm of the likelihood that the user's gaze will ultimately fall upon them when completing a scan in that direction. User gaze patterns, including the length of time the user's gaze falls on a given region or object, can also be analyzed to determine whether such a region or object will be particularly salient when displayed in the future.

[0122] Therefore, the information above regarding changes in user saccades and gaze direction, whether in the direction of a known or unknown salient region or object, can be used to implement foveated rendering. For example, if a known salient region or object is to be displayed on a virtual display, and the user's (multiple) eyes are saccaded toward that region or object while it is being displayed, then the region or object can be rendered with higher quality before the user's gaze direction reaches that region or object.

[0123] Even when the salience of the displayed area or object is unknown, or when no particularly salient area or object is displayed, the gaze direction and the corresponding angular change in the viewing area can be determined or estimated by detecting the saccades that are occurring. The amount of angular change in the gaze direction determined or estimated can be based on the user's previously recorded saccade data, or it may be user-specific, or it may be default data applicable to all users before data for a specific user becomes available.

[0124] In some embodiments, if the saccade information does not allow for precise determination of the user's likely gaze direction, multiple known salient regions or objects located at the most probable gaze direction can be rendered with higher quality. In some embodiments, the improvement in rendering quality for each region or object can be proportional to the probability that the user's gaze direction may ultimately fall on that region or object. For example, if two regions have an equal probability of becoming the synthetic gaze region of the user's saccade, the rendering quality of both regions can be improved in a similar manner. However, if a region is the more probable synthetic gaze region, its rendering quality is improved to a greater extent compared to the less probable synthetic gaze region.

[0125] The manner in which the image displayed on the display device 110 can be modified by the graphics processing device may vary depending on the embodiment. However, in any case, the manner in which the image is displayed may be designed to improve the image quality of the image portions to which the user's gaze or focus is directed, relative to those portions of the image not directed to the user's gaze or focus. In this way, the use of the available resources of the graphics processing device and / or other system resources is maximized to provide the image quality that is most important for the display device 110. For demonstration purposes, Figure 9 A display device 110 is shown, which illustrates a user's gaze point 910 and a region 920 surrounding the user's gaze point 910. Embodiments of the present invention can improve image quality in this region relative to the remaining region 930 of the display device 110. Therefore, in various embodiments of the present invention, the quality of the image generated across the display device 110 can be improved in region 920 relative to the remaining region 930.

[0126] When “modification” of an image present on display device 110 is discussed herein, it should be understood that the intent is that a subsequent image displayed on display device 110 differs from a previous image displayed on display device 110. Therefore, the graphics processing device and display device 110, or the other devices discussed herein, “modify” an image by displaying a first image and then displaying a second image different from the first image. Any other changes to an image discussed herein, such as improving or reducing image quality, should also be understood to mean that a subsequent image differs from a previous image. Note that an image change or modification may include changing or modifying only a portion of the image. Thus, some portions of a previous image may be identical to a subsequent image, while others may differ. In some cases, the entire previous image may differ from the subsequent image. It should be understood that modifying a region or the entire image does not necessarily mean that every finite portion (e.g., each pixel) of that region or the entire image is changed, but rather that the region or the entire image can be changed in some possibly consistent, predefined, or ordered manner (e.g., changing the quality of the image).

[0127] Improving image quality can include improving the quality of any one or more of the following non-exclusive list of graphics properties, in addition to those known in the art: - Resolution: The number of different pixels that can be displayed in one or more dimensions. For example, "1024×768" means 1024 pixels displayed in height and 768 pixels displayed in width.

[0128] - Shading: The color and brightness of the graphic objects vary according to artificial lighting projected by a light source simulated by the graphics processing device 130.

[0129] - Texture mapping: Mapping a graphic image, or "texture," onto a graphic object to give the object a specific appearance. The resolution of the texture affects the quality of the graphic object on which the texture is applied.

[0130] - Bump mapping: Simulates small-scale bumps and roughness gradients on the surface of a graphic object.

[0131] -Atomizing / Participating medium: Light dims when it passes through an impure atmosphere or air.

[0132] -Shadow: Simulates the blocking of light.

[0133] - Soft shadows: Shadows and variations in darkness caused by partially obscured light sources.

[0134] - Reflection: A representation of a mirror-like or highly reflective surface.

[0135] - Transparency / Opacity (Optical / Graphic): Light is rapidly transmitted through a solid object.

[0136] - Semi-transparency: Light is transmitted through a solid object in a highly dispersed manner.

[0137] - Refraction: The bending of light, which is associated with transparency.

[0138] - Diffraction: The bending, propagation, and interference of light passing through an object or aperture that disrupts its propagation.

[0139] - Indirect lighting: The surface is illuminated by light reflected from other surfaces rather than directly from the light source (also known as global lighting).

[0140] - Caustic (a form of indirect lighting): Light reflected from a glossy object or light focused through a transparent object to create a bright spot on another object.

[0141] - Anti-aliasing: The process of bending the edges of a displayed object to reduce the appearance of sharpening or jagged lines. Typically, it uses an algorithm that samples the color around the edges of the displayed object to blend the edges into its surrounding environment.

[0142] - Frame rate: For animated images, the number of individual frames rendered during a certain period of time to depict movement within the image.

[0143] -3D: The visual and temporal characteristics of an image that make it appear to the viewer as three-dimensional.

[0144] The size and shape of the image region that can be modified to appear with higher quality can vary depending on the embodiment. By way of example only, the shape of the region can be circular, elliptical, square, rectangular, or polygonal. In some embodiments, the image quality within the region can be improved non-uniformly. In other embodiments, the improvement in image quality may be greatest at the center of the region (i.e., near the gaze point) and decrease towards the edges of the region (i.e., away from the gaze point), possibly to match the image quality around the region. For illustrative purposes, Figure 10A This illustrates how image quality can degrade linearly or non-linearly in a continuous manner from the center of the gaze region outwards. Figure 10B It shows how image quality can gradually decrease from the center of the gaze area outwards.

[0145] In some embodiments, modification of the image displayed on the display device 110 may occur in response to a detected change in the gaze point. This can occur in a variety of ways, some of which are described below.

[0146] In some embodiments, the entire image can be modified during a change in the user's gaze, and once the change in the user's gaze ceases, the area around the user's final gaze or the remainder of the image (image portions not around the final gaze) can be modified. For example, in one embodiment, the quality of the entire image can be improved during a movement of the user's gaze (sometimes referred to as a saccade), but once the saccade is complete, the quality improvement only persists in the area around the user's final gaze (i.e., the quality of the remainder of the image can be reduced upon completion of the saccade). In different embodiments, the quality of the entire image can be reduced during a saccade, but once the saccade is complete, the quality reduction only persists in areas outside the user's final gaze (i.e., the quality of the image region around the user's final gaze can be improved upon completion of the saccade).

[0147] Additionally, the use of other system resources, including, for example, processors / computers and related resources, can be modified during a user scan. For instance, non-graphics operations can be supplemented with processor / computer and graphics processing device resources during a scan. More specifically, non-graphics computations necessary for other system operations can be performed at a higher speed or efficiency during a scan because additional resources associated with the processor / computer and graphics processing device are available for such operations.

[0148] In some embodiments, modifying an image displayed on display device 110 may include modifying a portion of the image in the area surrounding the user's expected gaze point, possibly by improving its quality. The expected gaze point may be determined based on changes in the user's gaze point. To determine the user's expected gaze point, an eye-tracking device and / or another processor (i.e., the processor of a computer or game console) may determine the rate of change of the user's gaze point on the display device and determine the expected gaze point at least in part based on this rate of change.

[0149] The rate of change of a user's fixation point—also known as the user's saccade rate or speed—depends directly on the total change of the user's fixation point (often referred to as the amplitude of the saccade). Therefore, as the expected amplitude of a user's saccade increases, the saccade speed also increases. While in humans, the saccade speed of a user can be as fast as 900° / second, for saccades less than or approximately 60°, the saccade rate is generally linearly and directly related to the saccade amplitude. For example, an amplitude of 10° is associated with a rate of 300° / second, and an amplitude of 30° is associated with a rate of 500° / second. For saccades greater than 60°, the peak rate begins to stabilize towards the maximum rate achievable by the eye (900° / second). In response to an unexpected stimulus, it typically takes about 200 milliseconds (ms) to initiate a saccade, and then the saccade lasts for about 20 ms to about 200 ms. Based on these relationships between saccade speed and amplitude, embodiments of the present invention can determine the expected fixation point based on the saccade rate. Various embodiments of the present invention can also employ other predetermined models of the mathematical relationship between saccade speed and amplitude to determine the expected fixation point.

[0150] In some embodiments, the modified image portion around the intended gaze point may further include the image portion around the original gaze point (i.e., the gaze point from which the user's saccade begins). While the shape of the modified image portion can be any of the shapes described above, in some embodiments, the shape may be a triangular or trapezoidal shape with a gradually increasing width perpendicular to the saccade direction, such as... Figure 11 As shown.

[0151] exist Figure 11 The diagram illustrates a display device 110, on which an initial user gaze point 1110 is shown. Embodiments of the invention can provide improved graphic quality in region 1120 prior to any changes to the initial gaze point 1110. The size and shape of region 1120 can be altered to accommodate both the initial gaze point 1110 and the expected gaze point 1140 when an eye-tracking device detects a user saccade, indicated by arrow 1130. While this embodiment is triangular and / or trapezoidal, in other embodiments, the altered region 1150 can take on different shapes and sizes. For example, the entire edge of the display device 110 in the saccade direction from the initial gaze point to the edge of the display can be included in the altered region 1150 to account for more possibilities of where the user's gaze might end. In other embodiments, the altered region 1150 can be circular, elliptical, or square. In yet another embodiment, the altered region 1150 can include separate and distinct areas surrounding the initial gaze point 1110 and the expected gaze point 1140.

[0152] In some embodiments, the size or shape of the area around the gaze point that modifies the image (or, in various embodiments, remains unmodified from the improved quality) is dynamic. This can occur at least in part based on any number of factors, including the gaze point's current position relative to the image or display device. For example, if a user moves their gaze point to a portion of the screen, a predefined portion of the screen (e.g., a corner portion of a display mapped to a virtual area in a video game) can be modified by increasing the quality therein. In some embodiments, if sufficient user gazing with one or more predefined characteristics is detected within a predefined amount of time, the entire display can be modified to render at a higher quality.

[0153] In another embodiment of the invention, a non-transitory computer-readable medium is provided having instructions thereon for rendering graphics on a display device 110. These instructions are executable by one or more processors to display an image at least on the display device 110. These instructions are also executable to receive information from an eye-tracking device indicating at least one of a user's gaze point on the display device 110 or a change in the user's gaze point on the display device 110. These instructions can be further executed to cause a graphics processing device to modify an image displayed on the display device 110 at least in part based on the user's gaze point on the display device 110 or a change in the user's gaze point on the display device 110. Therefore, a non-transitory computer-readable medium is also provided that is capable of implementing any of the features described herein with respect to other embodiments.

[0154] In another embodiment of the present invention, a method 1200 for presenting graphics on a display device 110 is provided, such as... Figure 12 As shown. At step 1210, method 1200 may include: displaying an image on display device 110. At step 1220, method 1200 may further include: receiving information from an eye-tracking device indicating at least one of a user's gaze point on display device 110 or a change in the user's gaze point on display device 110. At step 1230, method 1200 may further include: causing a graphics processing device to modify the image displayed on display device 110 at least in part based on the user's gaze point on display device 110 or a change in the user's gaze point on display device 110. Step 1230 may include: at step 1233, improving the image quality in a region surrounding the user's gaze point relative to the outside of that region. Step 1230 may further include: at step 1236, improving the image quality outside the region surrounding the user's gaze point relative to the inside of that region. Therefore, methods for implementing any of the features described herein with respect to other embodiments are also provided.

[0155] In some embodiments, a user can toggle the systems and methods described herein on and off to accommodate situations where multiple additional viewers are present on display device 110. In other embodiments, the systems and methods described herein can automatically toggle on when only one user is viewing display device 110 (as detected by an eye-tracking device) and toggle off when more than one user is viewing display device 110 (as detected by an eye-tracking device). Additionally, in some embodiments, the systems and methods described herein can allow for a reduction in the overall rendering quality of display device 110 when no viewer is detected, thereby saving system resources and power consumption when display device 110 is not the primary focus for any viewer.

[0156] In other embodiments, the systems and methods described herein can allow modification of multiple portions of an image on display device 110 to account for multiple viewers, such as those detected by an eye-tracking device. For example, if two distinct users are focused on different portions of display device 110, the two distinct image regions that are focused on can be rendered at a higher quality to provide an enhanced viewing experience for each viewer.

[0157] In other embodiments, image-associated data can inform the systems and methods described herein to allow prediction of which areas of the image a user is likely to focus on next. This data can supplement data provided by the eye-tracking device to allow for faster and smoother adjustments to image quality in areas where the user is likely to focus. For example, during a sporting event, a picture-in-picture interview with a coach or athlete can be displayed in the corner of the image. Metadata associated with the image feed can inform the systems and methods described herein of the potential importance of sub-sections of the image and, consequently, the viewer's interest and likely focus.

[0158] Trajectory Adjustment of Projectiles in Virtual Reality In virtual reality implementations, when a virtual object is thrown or otherwise projected by a user's action, there is a problem that the user cannot easily estimate the object's weight and therefore how far the object—in this case, a projectile—will travel upon being launched. Given that accurate distance may be necessary for making virtual interaction useful or successful, gaze detection can help improve the accurate placement of the projectile at the desired target location.

[0159] In some embodiments, the eye-tracking device can be used to determine the gaze direction of a user projecting a virtual object, and allows the position of the gaze direction on the virtual display to be used to determine or assist in determining the intended destination of the projectile. In some embodiments, the precise gaze position on the virtual display can be used as the user's intended target.

[0160] In other embodiments, the gaze position on the virtual display may only inform the software process of the intended target, and this may affect the calculated target of the projectile to varying degrees depending on the algorithm. Therefore, while an initial target can be calculated algorithmically, and the initial target may depend on factors such as the launch velocity or other motions initiating the projectile's launch and the virtual weight allocation of the object, the calculated initial target can be modified using the gaze direction to a degree of variability, such as the algorithmic allocation.

[0161] Distributed multi-camera array on / in VR lens In attached virtual reality or other wearable devices, positioning the image sensor and camera associated with the eye-tracking device in a way that allows for good imaging without interfering with the viewing lens of the display system can be challenging. To address this issue, various embodiments of the present invention provide a solution by placing one or more arrays of miniature cameras / image / light sensors (and illuminators as discussed herein) on or in a lens directly in front of the user's eyes. Because the camera is very small (e.g., 1...),... mm3 And it's very close to the user's eye, so the camera will essentially appear invisible to the user because it's not within the user's effective focal length. Figure 7A As shown, a viewing / main lens 710 may be present in front of the display device 110 of a given system, and an array of cameras / illuminators 720 may also be present on or in the lens 710. For illustrative purposes in this figure, the cameras / illuminators 720 are shown as larger than their actual scale. This will be discussed in further detail below. Figure 7A .

[0162] Although each camera in the array can only have a limited resolution, a standard image can be obtained when the associated processing equipment combines and reconstructs the images from all the cameras. Furthermore, because an array is used, additional depth information of the image can be obtained using a single camera or image sensor that observes (multiple) eyes.

[0163] User and obstruction presence verification is performed for safe operation of the light source. Consumer devices must meet various safety standards regarding light sources and laser sources near the user's eyes. The closer the user is to the light / laser source of the consumer device, the lower the permissible illumination values ​​for meeting various standards.

[0164] In some embodiments, the device described herein may include a security mechanism arranged to protect the user and verify that authentication criteria are met. In one embodiment, the eye-tracking device described herein may use an image sensor to determine whether a user is in front of the device and thus react accordingly.

[0165] If no user is present, the illumination of the display and / or eye-tracking illuminators is reduced to ensure safety at any distance. Because no user is present, the display and / or illuminators will illuminate at a safe level for any situation that arises. However, if a user is determined to be present, the distance to any display and / or eye-tracking subsystem such as illuminators can be calculated, and the maximum permissible illumination allowed by the system can be restored. In some embodiments, if the eye-tracking device determines that no user is present, the display device can be disconnected to save power.

[0166] In some embodiments, the system described herein can also determine whether any illuminator is blocked by: directly detecting the blockage via an image sensor; or by the image sensor not detecting light from a particular illuminator. If it is determined that an illuminator or other light-emitting device, such as a display, is blocked, the light-generating device can be dimmed or completely disabled to save power.

[0167] Reduced visibility of lighting devices and image sensors in VR devices In some embodiments, to position the lighting device and image sensor of the present invention in the most advantageous location, the flexible printed circuit (FPC) supported illuminator and image sensor can be recessed into the lens of a virtual reality or other display device, possibly in the following manner: Figure 7A In the position shown, LEDs or other illuminators, along with an image sensor, can be positioned on a very thin FPC and then oriented to minimize its visible outline.

[0168] For example, an illuminator and image sensor can be mounted on the FPC such that the illuminator or image sensor is mounted facing the user, but the outline of the FPC (i.e., the thickness of the FPC) is only directly visible to the user of the device. In some embodiments, the FPC can be attached to the illuminator or image sensor such that it is coupled to more than one side of the illuminator / sensor. In this way, the capture of light going to or guided from the illuminator / sensor can be improved.

[0169] In these and other embodiments, another method can be used to minimize the presence of the illuminator in virtual reality or other display devices. In these embodiments, the optical fiber can be embedded in a groove on the front (user) side of the device's viewing lens. Figure 7B This groove 730 and the optical fiber 740 deposited on the surface of the lens 710 are shown. The optical fiber can be distributed extensively across the front side of the lens to generate the illumination necessary for eye tracking. LEDs or other illumination sources, such as light-emitting diodes, can be coupled to the ends of the optical fiber at the edges of the lens or away from the lens.

[0170] Any gaps in the groove where the optical fiber is placed can be filled using substances such as glue or other adhesives to maintain the fiber's position and minimize refraction. This substance can have the same or similar refractive index as the lens itself to minimize visible distortion of the lens for the user.

[0171] like Figure 7C As shown, the angled and / or reflective surface 760 within the lens 710, located at the end point 745 of the optical fiber 740, can guide light from the end of the optical fiber toward the eye. To further minimize potential stray light from the optical fiber within the lens, a light-absorbing material 770 can also be disposed near the end point of the optical fiber, perhaps behind the angled surface. A beam-shaping material 780, such as epoxy resin, silicone resin, or similar substances, can also be applied to the lens near the end point to assist in guiding light toward the appropriate location on the user's eye.

[0172] According to aspects of the present invention, a device suitable for wear by a user is provided, the device comprising an optical element, a light source, and a sensor. The optical element is adapted to be positioned in front of the user's eyes when the device is worn, and may be formed of a light-transmitting material that allows the user to see through the optical element. The light source may be positioned on the optical element and adapted to illuminate at least a portion of the user's eyes, while the sensor is adapted to capture light emitted from the light source and reflected off the eyes.

[0173] The device can be, for example, an eye-tracking device used to determine the user's gaze direction and can be similarly configured as described above. Figure 1 and Figure 4 The described eye-tracking device can also be understood as a virtual reality or augmented reality head-mounted device similar to the embodiments discussed earlier, as well as other wearable devices such as helmets and glasses.

[0174] Optionally, the device may be combined with (or include) a display, which may be arranged such that a user can view the display through optical elements. The display may be similarly configured to combine with, for example... Figure 4 and Figure 11 The monitor in question.

[0175] Optical elements are objects or devices selected or primarily used for concentrating or otherwise modifying the direction of motion of light, such as visible light, near-infrared (NIR), or infrared (IR). The above description, for example, uses a Fresnel lens. Figure 4 and Figures 7A to 7CExamples of such elements have been discussed. Thus, an optical element can, for example, be used to make the display of a VR headset appear farther away than its actual location. Furthermore, an optical element can also refer primarily to an object or device that enables its support, such as a light source (or other structure), and simultaneously allows the user to view, for example, a display or other surrounding objects through the optical element. Therefore, an optical element can refer to, for example, a lens in a pair of glasses and may or may not be corrective. Alternatively or additionally, an optical element can refer to the visor of a helmet.

[0176] Examples of light sources can include those mentioned above, such as the references. Figure 1 , Figure 4 and Figures 7A to 7C The discussion focuses on lighting sources or illuminators. In the context of this application, the terms "light source" or "illuminator" are used to define virtually any device or element capable of emitting radiation in any region of the electromagnetic spectrum, such as the visible, infrared, and / or ultraviolet regions or combinations thereof, when activated, for example, by applying a potential difference across the device or element or by allowing current to flow through the device or element. Examples of light sources include lasers such as vertical cavity surface-emitting lasers (VCSELs) and semiconductor, organic, or polymer / polymer light-emitting diodes (LEDs), blue LEDs, light-pumped phosphor-coated LEDs, light-pumped nanocrystal LEDs, or any other similar devices as known to those skilled in the art. The light source may be disposed on a support such as a substrate, such that an electrical connection is provided between the substrate and the light source, or may be disposed directly on (or therein) the surface of an optical element, such that an electrical connection is provided. The light source may be arranged on or coupled to a substrate or optical element to be generally directed toward the eyes of a user wearing the device.

[0177] The light source can be formed from a bare die, i.e., a light-emitting die without any conventional packaging or housing, in order to reduce the size of the light source and thus its visibility. Advantageously, the die size is 300 x 300 micrometers or smaller, such as 200 x 200 micrometers or smaller.

[0178] A "sensor" is a device capable of measuring, recording, indicating, or otherwise responding to electromagnetic radiation such as NIR or IR. Examples of sensors include, in particular, those mentioned above. Figure 2 and Figure 3A and Figure 3B The discussion focuses on image sensors. A sensor can further refer to a camera or image capturing device used to determine the user's gaze direction. A sensor can include, for example, a photodiode or phototransistor. A sensor can utilize, for example, a one-dimensional or two-dimensional pixel array. A sensor can be positioned on the surface of an optical element, within an optical element, at a location between the optical element and the user's eye, and / or at a location behind the optical element (as seen from the user's eye).

[0179] In attached virtual reality headsets or other wearable devices, positioning a light source so that it illuminates the eyes for eye tracking without blurring or interfering with the user's field of vision can be challenging. Positioning the light source outside the user's field of vision can make eye tracking difficult because positioning the light source outside the user's cornea is risky. Therefore, positioning the light source on the optics allows for more accurate gaze tracking compared to implementations where the light source is positioned outside the optics. Furthermore, integrating the light source directly on the optics allows for a more compact design because the light source can be positioned to illuminate the eyes without requiring or occupying additional space, such as in front of, behind, or to the sides of the optics. Therefore, the inventive concept allows for easier integration of eye-tracking technology into wearable devices such as VR headsets or glasses.

[0180] The light source can be mounted on a substrate, which can be disposed on the surface of the optical element. The substrate can be disposed, for example, to cover a secondary portion, a major portion, or the entire surface of the optical element. The substrate and the light source can be assembled in a process separate from the manufacture of the optical element and can be used on the optical element as pre-assembled units. Alternatively, the light source can be mounted on the substrate after the substrate has been attached to the optical element.

[0181] The substrate can be, for example, a flexible printed circuit (FPC), which may include conductive structures such as conductive rails for supplying power to a light source. The conductive rails can be disposed on or defined within a flexible film or carrier that can be attached to the surface of an optical element. The conductive rails can be formed or defined, for example, by screen printing, ion implantation, photolithography, or jet printing. In some examples, the conductive rails can be formed from a conductive polymer or a metal such as copper or silver. The conductive material can be, for example, forged (rolled) or electrodeposited onto the carrier. Further, the conductive rails can be formed from a light-transmitting material such as a conductive polymer that is at least partially transparent or translucent, and / or can be so thin that these conductive rails are difficult to detect with the naked eye. The conductive rails can be disposed on one side, both sides, or laminated between two films or carriers supporting the film or carrier.

[0182] The substrate may include a support film or carrier formed of, for example, a glass material or a polymeric material, such as polyester (PET), polyimide (PI), polyethylene ester (PEN), polyetherimide (PEI), fluoropolymer (FEP), polyaryletherketone (PEAK), nylon, polyamide (PA), or polyetheretherketone (PEEK).

[0183] In one example, the light source can be mounted to the substrate, for example, by means of a pick-and-place machine surface, and can be fixed in place by adhesive. The light source can also be mounted to one or more contact pads arranged on the substrate. Depending on the number and arrangement of the electrical terminals of the light source, the light source can be contacted from below by at least two different contact pads, from above by two bonding wires, or by a combination of contacting the first terminal of the light source with a pad and contacting the second terminal with a bonding wire.

[0184] The light source and substrate can be provided as prefabricated separate units. However, the light source can also be directly attached to the surface of the optical element without using any intermediate substrate. In that case, the optical element itself can be provided with conductive rails similar to those described above. These conductive rails can be printed or formed directly on the surface of the substrate to provide power to the light source. The light source can then be connected to the conductive rails by means of, for example, conductive adhesives and / or bonding wires as described above.

[0185] A protective layer can be provided to at least partially surround or enclose the light source or to provide a flat surface for the device. Therefore, the protective layer can be disposed on a substrate (or, if no substrate is used, directly on the surface of the optical element). The protective layer can cover at least a portion of the substrate and, in some examples, can be arranged as a foil or film extending over a major portion or the entire surface of the optical element. By providing the protective layer, the device can have a substantially flat or uniform surface and can protect the light source and / or conductive rails from, for example, mechanically induced damage. Thus, the protective layer can act as a filler to flatten the surface of the optical element after the light source has been installed.

[0186] The protective layer can be formed, for example, from a material that is at least partially transparent, such as transparent or translucent glass or a polymer material. The protective layer can be provided as a flexible film or foil, which can be attached to or soldered to a substrate, for example, by an adhesive. Alternative methods for applying the protective layer can include, for example, spin coating, lamination, and screen printing.

[0187] Cutouts can be provided for the light source, and these cutouts can therefore be at least partially surrounded or not completely covered by a protective material. The protective layer can be attached to the substrate, for example, before the substrate is mounted on the surface of the optical element or in an additional step performed after the substrate has been mounted.

[0188] The light source may be provided with a (preferably, light-transmitting) material for dissipating the heat generated by the light source. This thermally conductive material may be arranged to be in thermal contact with the light source and may, for example, be arranged to at least partially embed, surround, or cover the light source. If the device includes a protective layer as described above, the thermally conductive material may be configured to fill the gap or space between the light source and the protective layer.

[0189] According to some examples, the device may include a cover layer that provides a hard, abrasion-resistant surface to protect the underlying light source (and possibly, the substrate / protective layer). The cover layer may be formed, for example, from a glass layer and may preferably be flexible to facilitate handling during manufacturing and reduce the risk of breakage or shattering. The cover layer may be attached to the protective layer, which can therefore be sandwiched between the substrate and the cover layer in a separate step, either before the substrate is mounted to the optical element or after the substrate has been mounted. Cover layers are of particular interest in applications such as consumer wearables and commercial VR headsets, which may experience high levels of wear and frequent cleaning. Furthermore, the cover may be removable to allow for replacement in case of damage or wear.

[0190] The device may further include a light guiding device, such as a reflective material, arranged to guide at least a portion of the light emitted by the light source. The light guiding device may be arranged, for example, to reduce the amount of stray light entering the optical element and, conversely, to guide light toward the user's eye. The reflective material may be arranged, for example, below the light source or substrate, i.e., between the light source and the optical element, and / or on the side of the light source. In some examples, the reflective material may be provided in the form of a layer or coating applied to, for example, the surface of the substrate, the light source, and / or the optical element. The light guiding device may be adapted to be more reflective of IR or NIR light than visible light, which is of particular interest for applications where the light source is an IR or NIR source. Thus, the light guiding device can act as a reflector guiding emitted IR or NIR light toward, for example, the eye, however, it is less visible to the user. The light guiding device can therefore be similarly configured as described above. Figure 4 The discussion is in the heat of the moment.

[0191] The device may include one or more of the light sources described above. The light sources may be arranged around the periphery of the optical elements, preferably as different or separate elements, between which the user can still see through the optical elements. The device may include, for example, two or more light sources, such as six, eight, ten, or sixteen, at least some of which are arranged around or near the periphery of the optical device. As already mentioned, the positioning of the light sources can be determined by the quality of eye tracking (which can be assumed to improve closer to the center of the field of view) and the visibility of the light sources (which can be assumed to decrease with increasing distance from the center of the field of view).

[0192] Figure 13A schematic cross-section of a device 1300 according to some embodiments is shown, wherein the device includes an optical element 1320 arranged in front of a user's eye 1312, allowing the user to view a display 1390 through the optical element 1320. Further, sensors such as photodiodes or camera units 1340 are arranged between the display 1390 and the optical element 1320 to track the eye 1312 through the optical element 1320. Figure 13 The disclosed components can form part of the VR headgear 1300 worn by the user.

[0193] As shown in this figure, an optical element 1320, such as a Fresnel lens 1320, used to make the display 1390 appear farther away from the user, may include a light source or illuminator 1330, such as an LED 1330, arranged at the periphery 1322 of the optical element 1320. The light source 1330 may be attached to the optical element 1320 by means of a substrate 1350, which may, for example, be formed with a flexible printed circuit (FPC) having conductive rails for powering the light source 1330. For visible light, the FPC may be at least partially transparent or translucent to reduce its visibility to the user. The light transmission characteristics may also be applied to the conductive rails. However, alternatively or additionally, the conductive rails may be thin or narrow enough to be difficult to detect with the naked eye. Furthermore, a reflective material 1380 may be provided to reflect at least some of the light emitted by the light source 1330. The reflective material may be provided, for example, as a coating or film on the light source 1330, the substrate 1350, and / or the optical element 1320. As shown in this figure, reflective material 1380 can be disposed in layer 1380 between the substrate 1350 and the surface of optical element 1320. Therefore, reflective material 1380 can act as a reflector that guides the emitted light away from optical element 1320 and reduces stray light entering optical element 1320, preferably a thermal mirror 1380.

[0194] Figure 14 A cross-section of optical element 1320 is shown, which can be similarly configured to combine... Figure 13The optical element under discussion. For example, an optical element 1320, which may be a lens formed of transparent glass or polymeric material, may include a stack of layers surrounding or sandwiching one or more light sources 1330. First, a substrate 1350, such as a PET foil, may be attached to the eye-facing surface of the optical element 1320. However, it is also conceivable to arrange the substrate 1350 on the eye-repellent side of the optical element 1320. In one example, the substrate 1350 may be 50 to 75 micrometers thick and may be, for example, glued to the optical element 1320. Second, a light source, such as an LED 1330, may be attached to the substrate 1320 and electrically connected to it via wire bonding 1354. The wire bonding may, for example, include metal wires, such as copper or silver wires. The LED 1330 may be formed from a bare die—i.e., without any packaging—to reduce the size of the light source and thus reduce its visibility.

[0195] To protect the light source 1330, a cover layer 1370, such as a glass sheet (e.g., Corning® Gorilla® glass or Corning® Willow® glass), can be disposed to cover the light source 1330. According to a non-limiting example, the thickness of this cover layer can be 250 micrometers. The cover layer 1370 can be attached to a protective layer 1360 formed of, for example, a light-transmitting polymer disposed on a substrate 1320. The protective layer 1360 can be formed, for example, of a PA film with a thickness of 200 to 250 micrometers. As shown in this figure, the protective layer 1360 may include a cutout portion to accommodate the light source 1330, which can be installed before or after the protective layer 1360 has been attached. Furthermore, a heat-transferring material 1362 can be disposed at the light source 1330 to at least partially embed or encapsulate the light source 1330.

[0196] If a cover layer is not used, it should be recognized that the light source 1330 can be equipped with a light-transmitting capsule or sphere to protect the die from mechanical and environmental damage.

[0197] Furthermore, it should be understood that the light source 1330 can be mounted on the substrate 1350 before the substrate 1350 is attached to the optical element 1320. Therefore, the substrate 1350 and the light source 1330 (and possibly the protective layer 1360 and / or the cover layer 1370) can be assembled into a separate unit that can be disposed on the optical element 1320.

[0198] Figure 15 It shows the relationship with Figure 13 and Figure 14The optical element 1320 is similar to the optical element disclosed herein. However, in this example, the light source 1330 and the conductive rail 1352 can be disposed directly on the surface of the optical element 1320 without any intermediate substrate. The conductive rail 1352 can be formed, for example, from a conductive (and preferably, light-transmitting) layer patterned to form the circuit structure required to supply power to the light source 1330. A first terminal of the light source 1330 can be connected from below to a pad 1356 disposed on the substrate, while a second terminal of the light source 1330 can be connected to the conductive rail 1352 by means of a bonding wire 1354. Furthermore, a protective layer 1360, for example made of polymer foil, can be disposed to protect and seal the light source 1330 and the conductive structures 1352, 1354, 1356 from the influence of the surrounding environment.

[0199] Figure 15 Further illustrated is a reflective coating 1380 disposed on the lateral sidewall 1332 of the light source 1330. This reflective coating is used to guide the emitted light away from the surface of the substrate 1320 and reduce the amount of stray light transmitted into the optical element 1320 and / or the protective layer 1360. In some examples, the reflective coating can be disposed, for example, on... Figure 14 The reflective coating is arranged on the sidewall of the cut portion of the protective layer 1360 to extend toward or all the way to the top surface of the optical element 1320. The reflective coating 1380 may be formed, for example, of a metal such as aluminum or silver, or a dielectric such as a metal oxide or silicon dioxide.

[0200] Figure 16 A front view of an optical element 1320 in the shape of a spectacle lens 1320 is shown. The lens 1320 can be similarly configured as the optical element discussed with reference to the previous figures and can include a plurality of light sources 1330 arranged at the periphery of the lens 1320. The light sources 1320 can be electrically connected to conductive rails 1352 and can be spaced apart such that a user is allowed to see through portions 1324 of the lens 1320 arranged between adjacent light sources 1330.

[0201] Figures 17a to 17c illustrate different examples of wearable devices according to embodiments of the concepts of the present invention. Each of these devices includes, for example, a combination of... Figures 13 to 16 The optical components, light sources, and sensors described.

[0202] Figure 17a illustrates a pair of glasses 1300 including two optical elements 1320 or lenses, through which a user can view their surroundings when wearing the glasses. Lens 1320 may include multiple light sources (not shown) for illuminating the eyes and sensors (such as a camera, not shown) for tracking the user's gaze direction. The sensors may be arranged on (or in) the frame 1301 of the glasses.

[0203] Figure 17b illustrates a VR headset 1300 according to an embodiment, which includes optical elements and sensors (not shown) that capture images of the illuminated eyes to track gaze on a display 1390.

[0204] Figure 17c shows a helmet 1300, in which optical elements 1320 can form a visor. Therefore, the helmet 1300 can be used to track the gaze direction when a user is looking at the surrounding environment or the display (not shown) of the helmet 1300.

[0205] Exemplary computer system The above demonstrates how Figure 8 The block diagram of the exemplary computer system 800 shown herein illustrates any embodiment of the present invention that may be implemented within this computer system. This example demonstrates a computer system 800 that can be used, for example, to provide the functionality of the systems and methods described above, either wholly, partially, or with various modifications.

[0206] Computer system 800 is shown as including hardware elements electrically coupled via bus 890. The hardware elements may include one or more central processing units 810, one or more input devices 820 (e.g., eye-tracking devices, whether or not integrated with another device; mouse; keyboard; touchpad; microphone; handheld controller, etc.), and one or more output devices 830 (e.g., display devices, wearable devices with displays, printers, etc.). Computing system 800 may also include one or more storage devices 840. For example, the storage devices 840 may be transient and / or non-transitory disk drives, optical storage devices, solid-state storage devices such as random access memory (“RAM”) and / or read-only memory (“ROM”), which may be programmable, flash-updatable, etc.

[0207] The computer system 800 may further include a computer-readable storage medium 850, a communication system 860 (e.g., a modem, a network interface card (wireless or wired), an infrared communication device, a Bluetooth™ device, a cellular communication device, etc.), and a working memory 880, which may include RAM devices and ROM devices as described above. In some embodiments, the computer system 800 may also include a processing acceleration unit 870, which may include a digital signal processor, a dedicated processor, etc.

[0208] The computer-readable storage medium reader 850 can be further connected to computer-readable storage media, which together (and optionally, in combination of storage devices 840) comprehensively represent remote, local, fixed, and / or removable storage devices plus storage media for temporarily and / or more permanently containing computer-readable information. The communication system 860 can allow the exchange of data with networks, systems, computers, and / or other components described above.

[0209] Computer system 800 may also include software elements, including operating system 884 and / or other code 878, shown as currently located within working memory 880. It should be understood that alternative embodiments of computer system 800 may have various variations derived from the computer system described above. For example, custom hardware may be used and / or specific elements may be implemented in hardware / software (including portable software, such as applets) or both. Additionally, connections to other computing devices, such as network input / output and data acquisition devices, may also occur.

[0210] The software of computer system 800 may include code 878 for implementing any or all of the functionality of the various elements of the architecture described herein. For example, software stored on and / or executed by a computer system such as system 800 may provide the functionality of the methods and systems discussed above. The methods that can be implemented by software on some of these components have been discussed in more detail above.

[0211] The invention has been described in detail for clarity and understanding. However, it should be understood that certain changes and modifications may be practiced within the scope of the appended claims.

[0212] List of Implementation Examples: 1. A method for panning and tilting content on a display of a wearable device, wherein the method includes: Determine the user's gaze direction using eye-tracking devices; The user's head orientation for the wearable device is determined using a motion detection system. At least in part, based on the fact that both the gaze direction and the head direction are aligned with a specific direction, the content displayed on the wearable device's screen is panned in that specific direction. During the panning of this content, the eye-tracking device determines that the user's gaze has returned to a neutral position; and The camera returns to the neutral position, at least in part, based on the user's gaze direction, causing the panning of the content displayed on the wearable device's screen to stop.

[0213] 2. The method for panning the content on a display of a wearable device as described in Example 1, wherein the motion detection system is selected from the group consisting of: The motion detection system is located away from the wearable device; and The motion detection system integrated with this wearable device.

[0214] 3. The method for panning content on a display of a wearable device as described in Example 1, wherein the panning of the content displayed on the display of the wearable device is further based at least in part on: The user's head direction exceeds a certain angular distance from the forward direction.

[0215] 4. The method for panning content on a display of a wearable device as described in Example 3, wherein the angular distance is at least partially based on: The angular direction of the head.

[0216] 5. The method for panning the content on the display of a wearable device as described in Example 1, wherein: The panning speed is at least partially based on the gaze direction.

[0217] 6. The method for panning the content on a display of a wearable device as described in Example 1, wherein: The panning speed is at least partially based on the head orientation.

[0218] 7. The method for panning content on a display of a wearable device as described in Example 1, wherein causing a halt to panning content displayed on the display of the wearable device is further based at least in part on: The user's gaze direction returns to that neutral position for at least a predefined period of time.

[0219] 8. The method for panning content on a display of a wearable device as described in Example 1, wherein causing a halt to panning content displayed on the display of the wearable device is further based at least in part on: Additional input was received from the user.

[0220] 9. The method for panning the content on a display of a wearable device as described in Example 1, wherein the method further comprises: This prompts the user to receive an instruction indicating that the panning has stopped.

[0221] 10. The method for panning content on a display of a wearable device as described in Example 1, wherein the method further comprises: After the panning of the content has stopped, determine that the user's head is returning to the forward direction; and The panning of the content displayed on the monitor is stopped during the return from the head direction to the forward direction.

[0222] 11. A non-transitory machine-readable medium storing instructions for panning content on a display of a wearable device, wherein the instructions are executable by one or more processors to at least: Determine the user's gaze direction using eye-tracking devices; The user's head orientation for the wearable device is determined using a motion detection system. At least in part, based on the fact that both the gaze direction and the head direction are aligned with a specific direction, the content displayed on the wearable device's screen is panned in that specific direction; During the panning of this content, the eye-tracking device determines that the user's gaze has returned to a neutral position; and The camera returns to the neutral position, at least in part, based on the user's gaze direction, causing the panning of the content displayed on the wearable device's screen to stop.

[0223] 12. The non-transitory machine-readable medium as described in Example 11, wherein the content causing the panning display on the display of the wearable device is further based at least in part on: The user's head direction exceeds a certain angular distance from the forward direction.

[0224] 13. The non-transitory machine-readable medium as described in Example 11, wherein: The panning speed is at least partially based on the gaze direction.

[0225] 14. The non-transient machine-readable medium as described in Example 11, wherein the content displayed on the display of the wearable device that causes the panning to stop is further based at least in part on: The user's gaze direction returns to that neutral position for at least a predefined period of time.

[0226] 15. The non-transitory machine-readable medium as described in Example 11, wherein these instructions can be further executed by one or more processors to at least: After the panning of the content has stopped, determine that the user's head is returning to the forward direction; and The panning of the content displayed on the monitor is stopped during the return from the head direction to the forward direction.

[0227] 16. A system for panning and tilting content on a display of a wearable device, wherein the system comprises: An eye-tracking device used to determine the gaze direction of a user of a wearable device; A motion detection system is used to determine the head orientation of the user of the wearable device; and One or more processors, wherein the one or more processors are configured to at least: At least in part, based on the fact that both the gaze direction and the head direction are aligned with a specific direction, the content displayed on the wearable device's screen is panned in that specific direction; During the panning of this content, the eye-tracking device determines that the user's gaze direction has returned to a neutral position; and The camera returns to the neutral position, at least in part, based on the user's gaze direction, causing the panning of the content displayed on the wearable device's screen to stop.

[0228] 17. The system for panning content on a display of a wearable device as described in Example 16, wherein the panning of content displayed on the display of the wearable device is further based at least in part on: The user's head direction exceeds a certain angular distance from the forward direction.

[0229] 18. The system for panning content on a display of a wearable device as described in Example 16, wherein: The panning speed is at least partially based on the gaze direction.

[0230] 19. The system for panning content on a display of a wearable device as described in Example 16, wherein causing a halt to panning content displayed on the display of the wearable device is further based at least in part on: The user's gaze direction returns to that neutral position for at least a predefined period of time.

[0231] 20. The system for panning content on a display of a wearable device as described in Example 16, wherein the one or more processors are further configured to at least: After the panning of the content has stopped, it is determined that the user's head is returning to the forward direction; and The panning of the content displayed on the monitor is stopped during the return from the head direction to the forward direction.

[0232] 21. The system for panning the content on a display of a wearable device as described in Example 16, wherein the motion detection system is selected from the group consisting of: The motion detection system is located away from the wearable device; and The motion detection system integrated with this wearable device.

[0233] 22. The system for panning content on a display of a wearable device as described in Example 17, wherein the angular distance is based at least in part on: The angular direction of the head.

[0234] 23. The system for panning content on a display of a wearable device as described in Example 16, wherein: The panning speed is at least partially based on the head orientation.

[0235] 24. The system for panning content on a display of a wearable device as described in Example 16, wherein causing a halt to panning content displayed on the display of the wearable device is further based at least in part on: Additional input was received from the user.

[0236] 25. The system for panning content on a display of a wearable device as described in Example 16, wherein the one or more processors are further configured to include at least: This prompts the user to receive an instruction indicating that the panning has stopped.

[0237] 26. The system for panning the content on a display of a wearable device as described in Example 16, wherein the eye-tracking device includes a contour sensor.

Claims

1. A device (1300) suitable for user wear, said device comprising: An optical element (1320) adapted to be positioned in front of the user's eyes (1312) when the device is worn, the optical element being formed of a light-transmitting material that allows the user to see through it, wherein the optical element is a lens; and A light source (1330) is arranged on the optical element and positioned to face the user's eyes when the device is worn, wherein the light source is adapted to illuminate at least a portion of the user's eyes; as well as Sensor (1340), the sensor being adapted to capture light that has been emitted from the light source and reflected on the eye; as well as Display (1390), which can be viewed by the user through the optical element; as well as The sensor is disposed between the display and the optical element, and the sensor is arranged to view the eye through the optical element; The device is characterized in that it further comprises: A reflective material (1380) is arranged to guide at least a portion of the light emitted by the light source, wherein the reflective material is arranged in a layer between the light source and the optical element.

2. The device according to claim 1, wherein, The light source is mounted on a substrate (1350), which is disposed on the surface of the optical element.

3. The device according to claim 2, wherein, The substrate is transparent.

4. The device according to claim 2 or 3, wherein, The substrate is formed of a flexible film.

5. The device according to claims 2 to 4, wherein, The substrate is arranged to cover the main portion of the surface of the optical element.

6. The device according to claims 2 to 5, wherein, The substrate includes conductive rails (1352) for supplying power to the light source.

7. The device according to claim 6, wherein, The conductive rail is transparent.

8. The device according to claims 2 to 7, further comprising a protective layer (1360) disposed on the substrate and at least partially surrounding the light source.

9. The device according to claim 8, wherein, The protective layer is formed of a flexible membrane.

10. The device according to any one of the preceding claims, further comprising a cover layer (1370) arranged to at least partially cover the optical element.

11. The device according to any one of the preceding claims, wherein, The optical element is a Fresnel lens.

12. The device according to claims 1 to 11, wherein, Compared to visible light, the reflective material is more reflective of infrared or near-infrared light.

13. The device according to claim 12, wherein, The reflective material is suitable for transmitting visible light.

14. The device according to any one of the preceding claims, wherein, The light source includes light-emitting diodes (LEDs).

15. The device according to any one of the preceding claims, wherein, The light source is suitable for emitting infrared or near-infrared light.

16. The device according to any one of the preceding claims, wherein, The light source is arranged at the periphery (1322) of the optical element.

17. The device according to any one of the preceding claims, comprising a plurality of spaced-apart light sources, the light sources allowing the user to see through portions (1324) of the optical element, the portions being arranged between adjacent light sources.

18. The device according to any one of the preceding claims, wherein the device is a pair of glasses or a helmet.

19. A system for determining a user's gaze direction, the system comprising: An optical element adapted to be positioned in front of the user's eyes, the optical element being formed of a light-transmitting material that allows the user to see through the optical element, wherein the optical element is a lens; as well as A light source, the light source being arranged on the optical element and positioned facing the user's eyes, wherein the light source is adapted to illuminate the user's eyes; A sensor adapted to capture light that has been emitted from the light source and reflected off the eye; as well as Display (1390), which can be viewed by the user through the optical element; as well as The sensor is disposed between the display and the optical element, and the sensor is arranged to view the eye through the optical element; and A reflective material (1380) is arranged to guide at least a portion of the light emitted by the light source, wherein the reflective material is arranged in a layer between the light source and the optical element; A processor adapted to determine the user's gaze direction based on the light captured by the sensor.

Images