Wearable devices capable of eye tracking
By combining eye tracking devices and motion detection systems in wearable devices, dynamically adjusting the translation direction and speed of the display content, the problems of gaze direction detection accuracy and power consumption in virtual reality devices are solved, and efficient and low-power virtualization is achieved Realistic experience.
Patent Information
- Application Number
- CN202111314880.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2016-04-29
- Filing Date
- 2016-09-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2036-09-26
AI Technical Summary
In the virtual reality and augmented reality environment, eye tracking technology is difficult to achieve high-precision and low-power gaze direction detection, resulting in high graphics rendering power and increased overall power consumption of virtual reality devices.
Using a combination scheme including an eye tracking device, a motion detection system and a processor, the user's gaze direction is determined through the eye tracking device, the motion detection system determines the user's head direction, and dynamically adjusts the content on the display based on the collaborative information of the two. Translate direction and speed to ensure accurate tracking of gaze direction and efficient rendering of content.
It realizes high-precision gaze direction detection and content translation in virtual reality and augmented reality environments, reducing the overall power consumption of virtual reality devices and improving graphics rendering efficiency and user experience.
Smart Images

Figure CN114035685B_ABST
Abstract
Description
[0001] This application is a divisional application of application number 201680068859.3, application date September 26, 2016, and titled “Wearable device capable of eye tracking”.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 329,719, filed on April 29, 2016, entitled “WEARABLE DEVICE COMPRISING A DISPLAY AND IMAGE SENSOR,” and U.S. Provisional Patent Application No. 62 / 232,268, filed on September 24, 2015, entitled “WEARABLE DEVICE COMPRISING A DISPLAY AND IMAGE SENSOR,” the entire disclosures of which are incorporated by reference and used for all purposes as if fully set forth herein. Summary of the invention
[0004] In one embodiment, a method for translating content on a display of a wearable device is provided. The method may include determining a gaze direction of a user via an eye tracking device. The method may also include determining a head direction of the user via a movement detection system. The method may also include, based at least in part on a gaze direction and a head direction consistent with a particular direction, causing content displayed on the display of the wearable device to be translated in a particular direction. The method may additionally include, during the translation of the content, determining via the eye tracking device that the gaze direction of the user has returned to a neutral position. The method may further include, based at least in part on the gaze direction of the user returning to a neutral position, causing the content displayed on the display to stop translating.
[0005] In another embodiment, a non-transitory machine-readable medium is provided on which instructions for translating content on a display of a wearable device are stored. The instructions are executable to perform a method. The method may include determining a gaze direction of a user of the wearable device via an eye tracking device. The method may also include determining a head direction of a user of the wearable device via a movement detection system. The method may also include, at least in part based on a gaze direction and a head direction consistent with a particular direction, causing content displayed on the display of the wearable device to be translated in a particular direction. The method may also additionally include, during the translation of the content, determining via the eye tracking device that the user's gaze direction has returned to a neutral position. The method may further include, at least in part based on the user's gaze direction returning to a neutral position, causing the content displayed on the display of the wearable device to stop translating.
[0006] In another embodiment, a system for translating content on a display of a wearable device is provided. The system may include an eye tracking device, a movement detection system, and one or more processors. The eye tracking device may be used to determine a gaze direction of a user of the wearable device. The movement detection system may be used to determine a head direction of a user of the wearable device. One or more processors may be configured to cause content displayed on the display of the wearable device to be translated in a particular direction based at least in part on a gaze direction and a head direction consistent with a particular direction. The processor (or processors) may also be configured to determine, during the translation of the content, via the eye tracking device, that the user's gaze direction has returned to a neutral position. The processor (or processors) may be further configured to stop translating the content displayed on the display of the wearable device based at least in part on the user's gaze direction returning to a neutral position. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The present invention is described in conjunction with the accompanying drawings:
[0008] Figure 1 An exemplary wearable device of the present invention is shown, which has a display device and an eye tracking device;
[0009] Figure 2 Exemplary profile sensor outputs of various embodiments of the present invention are shown;
[0010] Figure 3A Demonstrates horizontal and vertical pixel binning used in various embodiments of the present invention;
[0011] Figure 3B An embodiment of a possible method of the invention for determining gaze direction is presented;
[0012] Figure 4 An embodiment of the present invention is shown in which a cold mirror is applied to a headphone lens;
[0013] Figure 5 An embodiment of one possible method of the present invention for panning a virtual reality display is presented;
[0014] Figure 6 An embodiment of one possible method of the present invention for mapping an input device to a virtual object is presented;
[0015] Fig. 7A A system embodiment of the present invention is presented wherein a sensor / illuminator is disposed on and / or in a lens of a wearable device;
[0016] Figure 7B shows a detailed view of an embodiment of the present invention, wherein the optical fiber is arranged in a groove on the primary viewing lens;
[0017] Figure 7C Shown Figure 7B A profile view of the end point of the optical fiber;
[0018] Figure 8 is a block diagram of an exemplary computer system that can be used to implement at least some portions of the apparatus or system of the present invention, or to implement at least some portions of the method of the present invention;
[0019] Fig. 9 is a view of a display device of the present invention wherein image modification occurs in response to a user's gaze point;
[0020] Fig. 10A is a schematic diagram of how the image quality varies continuously within the modified image area;
[0021] Fig. 10B is a schematic diagram of how the image quality changes gradually within the modified image area;
[0022] Fig.11 is a view of a display device of the present invention wherein modification of the image occurs in response to a detected change in the user's gaze point; and
[0023] Fig.12 is a flow chart of one possible method of the present invention for modifying an image based on a user's gaze point.
[0024] In the accompanying drawings, similar components and / or features may have the same reference numeral. In addition, various components of the same type may be distinguished by a letter following the reference numeral that distinguishes the similar components and / or features. If only the first reference numeral is used in the specification, the description applies to any one of the similar components and / or features having the same first reference numeral, regardless of the letter suffix. DETAILED DESCRIPTION
[0025] Only exemplary embodiments are provided in the subsequent description, and are not intended to limit the scope, applicability, and configuration of the present disclosure. On the contrary, the exemplary embodiments described subsequently will provide an enabling description for implementing one or more exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present invention as set forth in the appended claims.
[0026] For example, any detail discussed with respect to one embodiment may or may not be present in all contemplated versions of that embodiment. Similarly, any detail discussed with respect to one embodiment may or may not be present in all contemplated versions of another embodiment described herein. Finally, the lack of discussion of any detail of the embodiments herein should be an implicit recognition that such detail may or may not be present in any version of any embodiment discussed herein.
[0027] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those skilled in the art that the embodiments can be implemented without these specific details. For example, the circuits, systems, networks, processors and other elements in the present invention can be shown as components in the form of block diagrams so as not to obscure the embodiments in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details to avoid obscuring the embodiments.
[0028] In addition, it should be noted that a separate embodiment may be described as a process depicted as a flow chart, a job diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. When the operation is completed, the process may be terminated, but there may be other steps not discussed or included in the figure. In addition, not all operations in any specific description process may occur in all embodiments. A process may correspond to a method, a function, a step, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to the return of the function to the calling function or the main function.
[0029] The term "computer-readable medium" includes, but is not limited to, temporary and non-temporary portable or fixed storage devices, optical storage devices, wireless channels, and various other media capable of storing, containing or carrying instructions and / or data. Code segments or machine executable instructions can represent any combination of programs, functions, subroutines, steps, routines, subroutines, modules, software packages, classes or instructions, data structures or program statements. Code segments can be connected to another code segment or hardware circuit by transmitting and / or receiving information, data, variables, parameters or memory contents. Information, variables, parameters, data, etc. can be transmitted, forwarded or transmitted by any suitable means including memory sharing, message passing, token passing, network transmission, etc.
[0030] In addition, embodiments of the present invention may be implemented at least in part manually or automatically. Manual or automatic implementation may be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored in a machine-readable medium. The processor will perform the necessary tasks.
[0031] The present invention generally relates to a wearable device including a display and an image sensor, the wearable device using information obtained by the image sensor to change information on the display. In particular, the present invention relates to a system and method for using information about the eyes to change information on the display.
[0032] Wearable devices that include displays are well known, and typically they are used in virtual reality (VR) and augmented reality (AR) systems. In these systems, the display is used to provide the user with an experience that simulates a different reality in the case of VR, or an experience of augmented reality in the case of AR.
[0033] In some cases, the wearable device need not include a display of any kind. For example, the system described by U.S. Patent No. 9,041,787 does not require a display. The entire contents of the above patent are incorporated herein by reference for all purposes, as if fully set forth herein.
[0034] In some embodiments, the use of eye tracking devices and the like can be incorporated into these wearable devices to improve their performance. For example, U.S. Patent Application Publication No. 2015 / 0062322 describes a wearable eye tracking device. The entire contents of the above patent publication are incorporated herein by reference for all purposes, as if fully set forth herein.
[0035] In Using an Eye-Tracking System to Improve Camera Motions and Depth-of-Field Blur Effects in Virtual Environments (Hillaire et al, 2008, Virtual Reality Conference), an implementation of eye tracking in a virtual reality device is described, whereby eye tracking is used to determine the focus of a user in a virtual embodiment. The focus is then used when rendering the virtual environment in order to improve the user's perception when navigating the virtual environment. For all purposes, the entire contents of the above disclosure are incorporated herein by reference as if fully set forth herein.
[0036] Embodiments of the present invention attempt to provide improved solutions for eye tracking in wearable devices, and improved use of eye tracking information in VR, AR or other environments. These improvements may involve hardware solutions for use in wearable devices, as well as software solutions for use with the same or similar devices.
[0037] It is therefore an object of at least some embodiments of the present invention to provide an improved wearable eye tracking system. This and other objects of embodiments of the present invention will become apparent from the specification and claims as well as from the drawings.
[0038] The various embodiments and aspects of the invention will be arranged using the headings herein to facilitate easier understanding of the invention.
[0039] According to a first aspect of the present invention, there is provided a wearable device comprising at least a display and an eye tracking device. Information from the eye tracking device can be used by an associated computing device (e.g., a computer, tablet, mobile device, or other processor-enabled device) to affect or change items displayed on the display.
[0040] In some embodiments, the eye tracking device includes at least one image sensor and at least one illumination source. In some embodiments, the eye tracking device may include two image sensors and two illumination sources. Depending on the embodiment, any number of image sensors and illumination sources are possible, and the image sensors and illumination sources may be equal or may be unequal. The illumination source projects illumination onto the eyes of the wearer of the wearable device, and the image sensor captures one or more images of the wearer's eyes. Based on the reflection position of the illumination on the eye, the direction of the user's gaze can be determined. For example, a suitable system for determining the direction of a user's gaze using a wearable eye tracking device is described in U.S. Patent Application Publication No. 2015 / 0061996. For all purposes, the entire contents of the above disclosure are incorporated herein by reference as if fully set forth herein.
[0041] Directly or wirelessly connected to the eye tracking device and the display may be a computing unit or other associated processing device that performs calculations based on information from the eye tracking device and may control information or other content displayed on the display after taking into account the calculations to modify the information or other content displayed on the display.
[0042] Figure 1A block diagram of a typical VR headset 100 with gaze tracking of various embodiments of the present invention is shown. As described herein, different embodiments of the present invention have a greater or lesser number of components and may be variably positioned / configured. The headset 100 may include a display device 110 and an eye tracking device 120, wherein the eye tracking device 120 includes an illuminator 123 and an image / light sensor 126. The illuminator 123 and the image / light sensor 126 may include dedicated lenses. The processor 130 may provide computational / operational control of the components. In some embodiments, the headset 100 may also include a motion / movement detection subsystem 140 that detects movement of the headset 100. A communication bus (not shown) may provide wired and / or wireless communications with an associated computing device.
[0043] Profile Sensor
[0044] In some embodiments, a profile sensor may be used to determine the direction of a user's gaze. In these embodiments, a wearable device may be provided that includes at least one profile sensor directed toward at least one of the user's eyes. An example of a suitable profile sensor is the profile sensor model S9132 manufactured by Hammamatsu. As will be appreciated by those skilled in the art, a profile sensor operates by aggregating the values of all pixels in a row and / or column into a single value.
[0045] Additionally, at least one infrared light emitter may be provided and directed toward at least one eye of the user. In this way, the profile sensor may be used to determine the location of reflections (or "flashes") of infrared light from the user's cornea. As will be readily appreciated by those skilled in the art, basic gaze tracking may therefore be performed by analyzing the location of the flashes on the user's cornea. In a further refinement, two or more profile sensors may also be used. This provides several advantages.
[0046] First, if more than one 2D profile sensor is used, after determining the radius of the cornea and the location of the glint in three dimensions, it is possible to determine the center of the cornea of the user's eye in three dimensions rather than in two dimensions.
[0047] Second, arranging at least two profile sensors so that the flashes produced do not overlap allows for more accurate flash detection. For example, consider the case of two flashes caused by illuminators A and B. If they are imaged by a one-dimensional profile sensor aligned in the same direction as the flashes, the sensor will only record a single response caused by the two flashes, making it difficult or impossible to determine whether the flash was caused by illuminator A, illuminator B, or both. It is therefore advantageous to align the illuminators and profile sensors in such a way that the flashes do not overlap in any readout of any profile sensor.
[0048] Alternatively, the illuminators may be adjusted to ensure that only one is illuminated at any given time. A profile sensor in this arrangement may be designed to operate at a very fast sampling rate, enabling many samples, and capturing flash from only one illuminator at each sample within a short time frame to ensure only minimal eye movement between samples.
[0049] Third, multiple one-dimensional profile sensors can be used. In order for such a system to function accurately, each sensor must be placed and rotated relative to each other. In this way, the single dimension of each sensor can alternate between being in horizontal and vertical configurations, although the relative orientation difference does not have to be limited to 90 degrees. In addition, it may be desirable to add a cylindrical lens to each one-dimensional profile sensor in these configurations.
[0050] In some embodiments, the profile sensor may output the sum of all rows and / or the sum of all columns of the sensor's pixel matrix to a processing device. Reflections of infrared light from an infrared illuminator on the user's cornea (called glints) may be found using a technique known as peak detection. Peak detection may be performed on the row sums and column sums. Figure 2 The technique is illustrated on the output of a row or column sum, where the x-axis represents the pixels of the profile sensor and the y-axis represents the amplitude, with the peak highlighted as an example.
[0051] In an optional refinement, to facilitate high-speed eye tracking (where the location of a previously calculated flash is known), only a subset of pixels need to be analyzed for peaks. For example, only 10-20 pixels close to a known previous flash location may be analyzed.
[0052] Once the flash position is known, the direction of gaze can be determined using, for example, the polynomial flash of the gaze point model:
[0053] gaze x =c1X+c2y+c3xy+c4X 2 +c5y 2 +c6
[0054] gaze y =c7x+c8y+c9xy+c 10 X 2 +c 11 y 2 +C 12 ,
[0055] where fixationx and fixationy are the x and y positions of the fixation point, x and y are the x and y positions of the flash, and c1...c12 are the calibrated model parameters.
[0056] In some embodiments, more than one illuminator may be provided in conjunction with a profile sensor. These illuminators may be selectively enabled or adjusted, and the processing device may determine which illuminator to enable based on metrics derived from the captured image data. Alternatively, the illuminators may be illuminated in a predetermined order, ensuring that only one illuminator is illuminated at any given time.
[0057] In another embodiment, the wearable device may also include at least one conventional area sensor type image sensor. Such a conventional sensor may also be directed to at least one eye of the user. The conventional sensor may capture an image of the user's eye, and the system may perform conventional pupil center corneal reflection (PCCR) eye tracking. PCCR is a well-known and well-understood method for determining the gaze of a user. Further information about this method may be found in a number of places, including Guestrin, ED, Eizenman, E., "General theory of remote gaze estimation using the pupil center and corneal reflections", IEEE Transactions on Biomedical Engineering, Volume 6, 63, pp. 1124-1133, June 2006.
[0058] By combining a profile sensor that is capable of outputting the sum of all columns and / or the sum of all rows with a conventional image sensor, the system can use the profile sensor for flash tracking and the conventional sensor for PCCR tracking. Due to the additional information provided by the profile sensor, the conventional sensor then only needs to run at 0.5-10.0 Hz. Therefore, the system can achieve low power consumption, low latency, and high frame rate (or sampling rate).
[0059] A profile sensor that tracks the flash will give sufficient gaze data as long as the sensor remains fixed relative to the user's face, while the image from a traditional image sensor allows for slide compensation as the sensor moves relative to the face. However, typically, eye movement is much faster than any potential slide of the wearable device on the user's head. Therefore, it makes sense to find a way to track the position of the flash only at low power and low latency. This could allow for foveated rendering in VR headsets, where a relatively low-power eye tracking solution could allow for substantial savings in the overall power consumption of a VR system, as the graphics rendering power requirements could be significantly reduced.
[0060] For example, the sensor may be set to a mode in which the sensor cycles through two or more illuminators, with only one illuminator being illuminated for one sensor exposure. For example, the sensor may be set to operate in a cycle in which, during a first sensor exposure, a first illuminator is illuminated, and then the sum of at least 10% of the pixel elements in at least 10% of the rows of the sensitive area and the sum of at least 10% of the pixel elements in at least 10% of the columns of the sensitive area are calculated (and the location of the flash is detected). Thereafter, during a second sensor exposure, a second illuminator is illuminated, and then the sum of at least 10% of the pixel elements in at least 10% of the rows of the sensitive area and at least 10% of the pixel elements in at least 10% of the columns of the sensitive area are calculated. Thereafter, the sensor captures a conventional image from at least a sub-portion of the sensitive area of the sensor while at least one of the illuminators is illuminated.
[0061] In an alternative implementation, the sensor may be configured to operate in a cycle in which, during a first sensor exposure, the first illuminator is illuminated and then the sum of at least 10% of the pixel elements in at least 10% of the rows of the sensitive area is calculated. Next, during a second sensor exposure, the first illuminator is illuminated and the sum of at least 10% of the pixel elements in at least 10% of the columns of the sensitive area is calculated. Thereafter, the sensor captures a conventional image from at least a sub-portion of the sensitive area of the sensor while at least one of the illuminators is illuminated.
[0062] An image sensor that can operate like a conventional image sensor and also output a plot of the sum of pixel lines and / or the sum of pixel columns of a sensitive area may include an output pin for synchronizing one or more illuminators with the exposure of the sensor.
[0063] An image sensor that can operate like a conventional image sensor and also output a curve of the sum of pixel lines and / or the sum of pixel columns of a sensitive area can also support daisy chaining, allowing two or more sensors to be connected to a processing unit via the same data bus (e.g., a MIPI CSI-2 interface).
[0064] An image sensor capable of operating like a conventional image sensor and also capable of outputting a curve of the sum of pixel lines and / or the sum of pixel columns of a sensitive area may also include means for detecting the distance of an object in its field of view. This may be done by "time of flight" analysis.
[0065] To compensate for ambient light curve data and / or conventional image data, the image sensor may be sampled from time to time without active illumination, ie, without illuminating any illuminator.
[0066] The image sensor may also be used to identify a user through iris recognition for login, security, and / or other reasons.
[0067] An image sensor that can operate like a conventional image sensor and also output curves for sums of pixel lines and / or sums of pixel columns of a sensitive area can be designed so that when the sensor is operated in profile mode, each pixel can only be included in the sum of the line or the sum of the column. For example, the pixel elements can be laid out in a grid pattern, where only every other pixel can be summed into a row profile, and the other pixels can be summed into a column profile.
[0068] An alternative implementation of the image sensor is to divide the sensitive area into a grid pattern where every other pixel is read out in a row and every other pixel is read out in a column, essentially with an AD converter next to each row and an AD converter next to each column. This means that the conventional image of the sensor would be two images of half the resolution of the sensor, one read out vertically and the other read out horizontally. This increases the computational complexity of conventional PCCR eye tracking. FIG3 illustrates in one example, as well as other examples discussed above, a configuration where an area image sensor that simulates a profile sensor via its individual pixels, a 2D profile sensor and / or two vertically oriented 1D profile sensors can be employed in a manner where every other pixel in every other row (marked with an "X") is used for horizontal flash / image data acquisition (horizontal grids 1-5) and every other pixel in every other column (also marked with an "X") is used for vertical flash / image data acquisition (vertical grids 1-5).
[0069] Advantageously, the sensor can be designed in such a way that it supports horizontal pixel binning for the image read out from each row, and vertical pixel binning for the image read out from each column, thus facilitating low power flash detection. For example, the sensor can be designed to sum values from 8-16 pixel elements, or even more, into one value, which means that it can operate as a profile sensor supporting sub-window functionality, which can help suppress irrelevant signals and reduce noise.
[0070] With or without sub-window support, a sensor capable of operating as a profile sensor may include HW logic for detecting the center of a flash, further reducing power consumption and the amount of data required to be sent to an external processing unit. In the case where the sensor supports sub-windows, the sensor may change the sub-window position after the center of the flash is detected to ensure that subsequent profile images include the flash.
[0071] An alternative embodiment of an eye tracker that supports traditional PCCR eye tracking as well as flash tracking, which allows low power eye tracking at latencies and data rates that support foveated rendering, having a traditional sensor for imaging the eye, but including HW logic for flash center detection when the sensor is operated in a specific predefined sub-window mode. For example, this may only work for sub-windows of 24×24 pixels, 32×32 pixels, 48×24 pixels, or other appropriate resolutions.
[0072] In some embodiments, an organic light emitting diode (OLED) display may be employed. OLED displays are typically transparent, with a mirror placed behind them to ensure that all light is sent forward (toward the viewer). For eye tracking purposes, a cold mirror may be placed behind the OLED display, reflecting essentially all visible light from the display toward the user's eyes, but allowing NIR light to pass through. In this way, an eye tracking sensor that detects NIR light can be placed behind the display, looking through the display toward the viewer, thereby achieving a favorable viewing angle toward the user's eyes.
[0073] It is also common in VR headsets to use a Fresnel lens to make the display appear to be at a greater distance from the user than it actually is. The disadvantage of having a lens like this is that it distorts the image from the display, and likewise distorts the image of the eye seen through the lens from the eye tracking sensor. Therefore, it may be preferable to compensate for this distortion in the eye tracking algorithm.
[0074] An additional effect of the Fresnel lens is that it can cause circular imperfections in the image of the eye as seen from the eye tracking sensor. The pattern is similar to the distorting effect of waves on water when you throw in a small stone and try to look at something below the surface. Therefore, it is advantageous to calibrate the eye tracking sensor that views the eye through a Fresnel lens to ensure that the image from the sensor is adjusted to compensate for the imperfections of the Fresnel lens before the machine vision algorithm attempts to detect different eye features or glints.
[0075] An image sensor that can operate like a conventional image sensor and also output a curve of the sum of pixel lines and / or the sum of pixel columns of a sensitive area can also be designed to support sub-windowing. This is common in conventional image sensors, but by allowing sub-windowing when operating in profile mode, many potentially disruptive reflections or light sources from the sensor's field of view can be removed before the pixel elements are summed to the row profile / column profile, thus ensuring greater accuracy in the determination of the flash position and allowing the sub-window to be re-centered for subsequent sampling.
[0076] Figure 3BA block diagram of a possible method 300 for determining the gaze direction of a user described above is shown. At block 310, one or more eyes are illuminated by one or more available illuminators. At block 320, one or more eyes are imaged via one or more available sensors (profile sensors and / or area image sensors). Blocks 310 and 320 may be repeated as needed. At block 330, as described above, the number of sensors and / or where they are located may be limited to focus on a sub-window of possible pixels / resolutions, where imaging may be necessary to determine the location of gaze. Blocks 310, 320, and 330 may be repeated as needed. At block 340, the direction of gaze may be determined, and blocks 310, 320, 330, and 340 may be repeated as needed to continuously re-determine the changed gaze direction and / or sub-window as needed.
[0077] Thermal Mirror
[0078] In one embodiment, Figure 4 As shown, lens 400 is disposed in a wearable device (e.g., a virtual reality headset). Lens 400 is positioned so that a wearer of the device can view display 110 through lens 400. Affixed to the top of lens 400 is a heat mirror coating or film 410, and affixed to the top of heat mirror film 410 are a plurality of illuminators 420. Film 410 has a cutout portion 430 toward the center of lens 400 to allow an eye tracking camera to view through lens 400.
[0079] The heat mirror coating or film 410 can be applied directly to the lens 400, or for ease of manufacturing, the heat mirror coating or film 410 can also be in the form of a separate piece that does not function as a lens. In addition, also for ease of manufacturing, a separate piece of glass or plastic can be placed on top of the heat mirror, and the illuminator 420 can be fixed to the separate piece.
[0080] Illuminator 420 may emit infrared light outwardly toward the eyes of wearer 440, which is then reflected by the wearer's eyes toward lens 400. Film 410 has the properties of a hot mirror, in other words, film 410 allows light in the visible spectrum to pass through while preventing light in the infrared spectrum from passing through. In this way, visible light emitted by display 110 behind lens 400 may pass through to the wearer's eyes, while most infrared light is blocked from passing through.
[0081] An image sensor located behind the lens and viewing through the cutout portion 430 captures images of the wearer's eyes that include reflections of infrared light emitted by the illuminator 420, and a processing device connected to the image sensor then analyzes those images to determine the direction of the user's gaze based on the reflections of the infrared light.
[0082] Although this embodiment has been described with reference to a plurality of infrared illuminators 420, the present invention may function adequately with only one illuminator.
[0083] The illuminator 420 may be applied to the film 410 in a variety of ways, first simply by glue or alternatively possible application by printing the electronics directly onto the film 410. The illuminator 420 will further have to have a communication line applied to the film 410 so that the illuminator can receive power to be switched on and off.
[0084] In one embodiment, the wearable device includes two lenses 400 with associated films 410 and illuminators 420. The films 410 may be applied to the lenses by glue or some other semi-permanent substance. In a system with two lenses, light may be reflected from the user's skin into the lenses, whereby the lenses have a waveguide effect and direct the light away from the wearable device. The presence of the thermal mirror 410 serves to reduce the appearance of such light.
[0085] In an alternative embodiment, the illuminator 420 can be placed on the side of the lens 400 (or an additional piece as described above). The illumination from the illuminator 420 can then pass through the lens 400 to be emitted in front of the user's eyes. This directing of light occurs in the area of the lens where the hot mirror 410 has been applied to prevent the illumination from being emitted toward the display 110.
[0086] In a further addition to the present invention, an angled hot mirror 410 may be added in front of the display 110, and the image sensor is arranged to view the hot mirror. The lens 400 is then placed in front of the user's eye, with the illuminator 420 located next to it. The illuminator 420 illuminates the user's eye, and because the lens 400 has a hot mirror film or coating, as previously described, stray illumination from the illuminator 420 is reduced. The cutouts in the hot mirror film or coating allow the image sensor to capture an image of the user's eye through the angled hot mirror.
[0087] algorithm
[0088] In various embodiments, an algorithm is used to determine the gaze direction of the eye from an image containing the eye and reflections of infrared light from the eye.The processing unit performs calculations based on the algorithm using the captured image to determine the gaze direction.
[0089] The algorithms used in the wearable device described here are substantially similar to those used in remote, existing eye tracking units. Therefore, the basic method of determining gaze direction should be well understood by those skilled in the art.
[0090] However, several improvements are discussed below.
[0091] Pupil position
[0092] One step in determining gaze direction in an eye tracking device is to estimate the position and size of the pupil. In some embodiments, the following method can be used to estimate pupil size and / or position.
[0093] In the captured image, the positions of reflections of the infrared light emitted by the illuminator are analyzed to determine their positions relative to the previously captured image. The displacement of the reflections is combined with the position of the pupil from the previously captured image and then used to determine the position of the pupil in the current image.
[0094] Fresnel lens
[0095] When a Fresnel lens is present in a wearable device, the image captured by the eye tracking image sensor through the Fresnel lens typically contains concentric circles as present in the Fresnel lens. When attempting to determine gaze detection, these concentric circles may be incorrectly determined by the processing unit as pupil edges, so it is necessary to quickly and accurately remove these concentric circles.
[0096] In some embodiments, these concentric circles are removed from the captured image using erosion. Erosion is an image processing concept that will be well understood by those skilled in the art. Using erosion, a small kernel (e.g., of 3×3 or 5×5 pixels) is passed element-by-element to the captured image. For each pixel, the intensity value is replaced by the darkest pixel in the pixel's neighborhood, where the size of the neighborhood is defined by the kernel. Since the pupil in the captured image is dark, the light concentric circles in the captured image are therefore replaced by the dark pupil.
[0097] Blocking light
[0098] Another embodiment allows the use of hardware or software to perform mathematical operations along a line across the 2D image sensor to provide an output similar to the output of a profile sensor. The line may typically be a row and column of the sensor, but is not necessarily limited to these orientations. This will allow not only the average and / or sum of all pixel values on the line to be calculated, but other operations may also be calculated, and light from portions of the image known not to contain any glitter may be masked. Masking light from portions of the image that do not contain any glitter increases the signal to noise ratio and therefore aids in detecting glitter by allowing the intensity distribution of the profile response to be examined. In an exemplary embodiment, the area to be masked may be all of the area outside the cornea, and the most recent output from the eye tracking algorithm may be used to give an approximation of the area that is masked.
[0099] Using a conventional image sensor to emulate a 2D profile sensor reduces the computational load required for eye tracking and therefore reduces power consumption. However, the frame rate is limited to that of the 2D image sensor.
[0100] Even when a true profile sensor is used, light from parts of the image that do not contain any flash may be masked.
[0101] One way to mask light from portions of an image known not to contain any flash is by using one or more illuminators whose light can be spatially controlled (e.g., an infrared OLED array behind a lens, any arrangement with a DLP or LCOS projector, or numerous other schemes that will be readily apparent to those skilled in the art).
[0102] Another way to mask light from portions of the image that are known not to contain any flash is by using elements that block portions of the light before entering the profile sensor. Those blocking elements can be LCD-based, mechanical-based, or based on numerous other schemes that are readily apparent to those skilled in the art.
[0103] Emulating a profile sensor using a conventional image sensor
[0104] As previously described, a profile sensor can be simulated using a conventional image sensor that includes a matrix of pixels. To achieve this, hardware or software can perform mathematical operations (e.g., calculating the average intensity level along a line on the sensor, or calculating the sum of all intensity levels along a line on the sensor) to provide an output similar to a profile sensor. Typically, this will be equivalent to an output row or column from a conventional sensor. However, any pixel configuration, such as a diagonal line, may be output. By using this simulation system, more operations can be performed than just the traditional mean and sum of all pixel values on a row, such as the previously described masking. In addition, detection light from areas of the captured image (pixels in the image sensor) that are known not to contain any flash can be masked. By performing this masking function, the signal-to-noise ratio can be increased. An example of an area to be masked is an area outside the user's cornea, as this area cannot provide flash.
[0105] The shielding of light from areas where the image of the flash cannot be provided can be performed using a conventional profile sensor. Further options for shielding the light include: using an illuminator whose light can be spatially controlled, such as an infrared OLED array behind a lens, any arrangement with a DLP or LCOS projector, or numerous other solutions readily understood by those skilled in the art. Another option is to prevent light from non-provided areas from reaching the sensor, which can be achieved by a mechanical solution, an LCD solution, or any other solution understood by those skilled in the art. The mechanical LCD solution can include placing a transparent LCD in front of the profile sensor.
[0106] Eye tracker synced with display
[0107] For certain applications of eye tracking, it is valuable to synchronize the eye tracking device with the display, particularly in a wearable device. According to this aspect of an embodiment of the invention, the wearable device is provided with a display, at least one camera and at least one illuminator. The camera and the illuminator form the eye tracking device. The camera and / or the illuminator can both be synchronized with the display. The synchronization can be characterized as synchronizing the camera strobe rate with the v-sync of the display.
[0108] It is further desirable to synchronize the eye tracking device with one or more position devices. For example, the eye tracking device may be synchronized with an inertial measurement unit or the like, or with a room position device using infrared or other invisible light. Such systems have been developed by It is proposed that the name "Lighthouse" be used. Those skilled in the art will easily understand how this synchronization works.
[0109] Removable eye tracker
[0110] According to another aspect of an embodiment of the present invention, a removable eye tracker is provided, whereby the eye tracker can be inserted into a wearable device. The eye tracker can then be integrated with another device (e.g., a phone, tablet, watch, display, etc.).
[0111] An eye tracker may include at least one camera and at least one illuminator, and its primary function may be to track the gaze of a user relative to a device (e.g., a phone, tablet, or watch) into which the eye tracker is integrated. As a secondary function, the device into which the eye tracker is sometimes integrated may be inserted into a wearable device. The device may then provide functionality such as a display to the wearable device, and the eye tracker may be used to determine the gaze direction of the wearer of the wearable device. The method of operation of the eye tracker may be any conventional method or any method described herein.
[0112] Smile certification
[0113] According to one aspect of the present invention, the image sensor used for eye tracking in the wearable device can also be used to capture images of the area around the user's eyes. For example, these images can be analyzed to determine whether the user is smiling, and whether the smile is real or fake. Features of the area around the user's eyes can be used to determine whether the smile is fake or real. For example, see "Are your eyes smiling? Detecting Genuine Smiles with Support Vector Machines and Gabor Wavelets" by Littlewort-Ford, Gwen, Marian Stewart Bartlett and Javier R. Movellan in the proceedings of the 8th Joint Symposium on Neural Computing in 2001. For all purposes, the entire disclosure of the above disclosure is incorporated herein by reference as if fully set forth herein.
[0114] According to an embodiment of the present invention, an image sensor for eye tracking captures at least a portion of the area around the eye when capturing eye images, and then known smile or other detection algorithms can be used to analyze these images to determine whether the user's smile or other facial features are fake or real.
[0115] Iris recognition based calibration
[0116] According to an embodiment of the present invention, the image sensor used for eye tracking further captures information related to the user's iris. This iris information can be used to determine the identity of the user for input to a system connected to the eye tracker.
[0117] For example, according to some embodiments of the present invention, a wearable device is provided, wherein at least one image sensor and at least one infrared illuminator are provided. The image sensor and the illuminator face one or more eyes of a user wearing the device. Optionally, the device also includes a display, such as a virtual reality display in a wearable headset.
[0118] The image sensor captures an image of the user's iris and transmits the image to a processing device, which may be located on the wearable device or may be located remotely from the processing device, in which case communication may be achieved by wired or wireless means, as will be understood by those skilled in the art.
[0119] Iris recognition is a known technology and uses mathematical pattern recognition techniques to uniquely identify patterns on one or both of a user's irises for identification or authentication of the user. In its most basic form, iris recognition involves the following steps:
[0120] (1) Localization - calculating the inner and outer boundaries of the iris; (2) Normalization - normalizing the captured data for consistency; (3) Feature Extraction - forming a feature vector of features extracted from the captured image; and (3) Matching - classifying the feature vector using thresholding techniques.
[0121] Many algorithms that allow iris recognition have been proposed, see for example Daugman JG, High Confidence Visual Recognition of Persons by a Test of Statistical Independence, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15, No. 11, pp. 1148-1161, 1993. The entire disclosure of the above publication is incorporated herein by reference for all purposes as if fully set forth herein.
[0122] Based on an image of the user's iris, a processing unit connected to the wearable device (wired or wireless) can use the user's identification to affect its functionality. For example, when using an eye tracker, the processing unit can load a calibration distribution that provides user-specific information about the offset between their calculated gaze position and the true gaze position. As another example, the identification can be used to authenticate that the user is authorized to operate the wearable device, or is authorized to operate the processing unit connected to it.
[0123] Eye rotation compensation
[0124] According to another aspect of an embodiment of the present invention, eye torsion can be tracked. As an explanation, a person's eyes are attached to muscles in such a way that in addition to moving left / right and up / down, the eyes can also rotate so that the top of the iris can be rotated closer to the nose while the bottom of the iris rotates further away from the nose. The opposite rotation is also possible. This type of rotation is often referred to as eye torsion.
[0125] When a person rotates their head slightly to their side, most people will automatically turn their eyes slightly in the opposite direction and keep their eyes nearly level with the horizon. This effect is only effective during small rotations, as it is not possible to rotate the eyes by large degrees in this manner.
[0126] This phenomenon introduces an additional source of error in the eye tracking system for all people whose retina's fovea is not perfectly centered along the eye's optical axis.
[0127] The present invention can track the torsion of the eye by observing the iris and / or other features on the eyeball, and / or using orientation information from an eye tracker. This can provide the eye tracker with a better estimate of the location of the fovea of the retina, and therefore provide a better estimate of the gaze of the eye tracked subject when the head of the eye tracked subject is tilted.
[0128] Corneal curvature
[0129] According to one of the embodiments of the present invention, at least one image sensor captures an image of the user's eye. The computing unit uses this information to determine the shape of the user's cornea. Based on the position and orientation of the reflection from the infrared light source, the curvature of the cornea can be measured. In some people, their cornea has an abnormal curvature. This can be called astigmatism.
[0130] By using the information obtained from the image sensor and the infrared light source, the curvature of the cornea can be simulated and thus users with abnormally shaped corneas can be identified. By determining the shape of the cornea, corrective measures can be taken, such as prescribing a suitable lens.
[0131] Reduced power consumption and processing power
[0132] In some embodiments, to further reduce power consumption of a wearable device that may include a virtual reality display or other display system, the eye tracking device may directly track only the gaze position of a first eye of the user. The gaze position of a second eye of the user may then be determined by a prediction based at least in part on the gaze position directly determined for the first eye.
[0133] This can be done based on the assumption that the second eye is at approximately the same level as the first eye, as is typical for human biology. In this way, power used by additional eye tracking equipment (including illuminators, image sensors, and / or processing power) can be saved. In some embodiments, a combination of both the determined and predicted gaze positions of both eyes can then be used to determine an area for gaze point rendering on the display of the wearable device, which can further save power due to improved video rendering efficiency.
[0134] In some embodiments, particular software (e.g., games and / or other high-end graphics applications) may request gaze data from an eye tracking device to inform the application how to potentially interact with the user in the future, how much graphics information to provide to a graphics processing unit, and / or other purposes. While in other embodiments, all gaze direction information may be passed to such applications, allowing individual applications to request such information, and providing such information only upon request, which may result in power and processing savings.
[0135] VR Pan and Center
[0136] In some embodiments, schemes are provided for mitigating the user's ability to pan and then center a virtual reality display or other display system in a wearable device of the user. Typically, because a virtual reality display is infinite in any direction (i.e., a user looking to the right in a virtual display will ultimately turn 360 degrees to view the original scene), panning can become awkward for the user because the user can only turn their head so much before turning further becomes uncomfortable.
[0137] To mitigate this problem, the system described herein may allow a user to pan their field of view on a virtual reality display by turning their head and moving their gaze position in a given direction in which the translation will occur, but while the user's head is still turning, allow the displayed scene to be re-centered about a point when it is determined that the user's gaze direction has been re-centered in the scene (i.e., their gaze position is no longer translating).
[0138] In some embodiments, the re-centering of the gaze direction must occur for at least a predetermined period of time to ensure that the user has actually completed their field of view translation. For example, the user's gaze must return to a relatively central area of the current view. In some embodiments, the size, shape, and position of the area can be set by the user. In some embodiments, the area can be visible in different ways, possibly as set by the user (i.e., the area indicated by the light color can be covered on other content on the display). In some embodiments, an indication (i.e., an audible or visual prompt) can be provided to the user to confirm that the virtual display view has been re-centered and that they can move their head position to a straight-ahead, central, neutral position again. The system can allow the user's head to be re-centered without causing the display to be re-translated in the opposite direction of the original translation.
[0139] Similarly, if the user's gaze direction is also the same as the direction their head is moving, or within a certain / predefined angular direction of each other, the panning speed may start and / or increase (i.e., where the panning would stop and / or decrease in speed) with the user's head remaining straight ahead, but their gaze direction indicates that panning is desired (i.e., by moving the gaze direction to the edge of the surrounding view). Additionally, the panning speed may depend on, and / or be proportional to, the magnitude of the user's head movement. This means that a large or sudden movement of the user's head may increase the panning speed quickly, while a small or slow movement of the user's head may increase the panning speed slowly.
[0140] Thus, in one embodiment, systems and methods for translating (and re-centering) content on a display of a wearable device may be provided. Figure 5 As shown, the system described herein can perform method 500, which includes: at box 510, determining the gaze direction of the user of the wearable device via an eye tracking device. The method can also include: at box 520, determining the head direction of the user of the wearable device via a motion detection system. The motion detection system can include any internal / integrated or external / remote device to determine the head position of the user. The motion detection system can determine the position of the head by any method known in the art or later known.
[0141] The method may also include, at block 530, causing content displayed on the display of the wearable device to be translated in a particular direction based at least in part on the gaze direction and the head direction consistent with the particular direction. The method may also include, at block 540, determining, via an eye tracking device, that the user's gaze direction has returned to a neutral position during the translation of the content. The method may also include, at least in part on the user's gaze direction returning to a neutral position, causing the content displayed on the display of the wearable device to stop translating.
[0142] In addition, the direction of the user's head that exceeds a certain angle direction from the neutral / forward direction can also be the direction / speed of translation. Generally, the speed of translation can be based on the gaze direction and the head direction. Additional input can be provided by the user to instruct the system to stop translation. In addition, in some embodiments, the system can provide the user with a visual and / or audible indication that the translation has stopped, and the user can move their head back to the forward / neutral direction without worrying about translation in the opposite direction (as shown in box 550).
[0143] VR Mapping of Input Devices
[0144] In some embodiments, gaze detection may allow the mapping of input device actions to the surface of a virtual object in a virtual reality display. For example, if a virtual sphere is located in virtual space, the eye tracking device may determine that the user's gaze direction is toward the sphere at a certain moment. Once a secondary input is received or a predetermined time period expires, the associated input device may map its controls to the virtual object.
[0145] Secondary inputs may include contact inputs from conventional input devices such as a keyboard, a touchpad (or other touch-sensitive device), a mouse, or a trackball; or may also include non-contact inputs such as voice commands or gaze inputs. More specifically, contact inputs may be directional inputs on a touch-sensitive surface that controls the mapping to a user-specified position on a virtual sphere; wherein the directional inputs are touch gestures; and contact inputs may also be pressure signals received from a pressure-sensitive device associated with the touch-sensitive surface; and last but not least, contact inputs may be combined with pressure inputs and touch gestures. Proximity inputs are also possible where proximity touch inputs are received from a touch-sensitive device associated with a proximity sensor without physical contact with the touch-sensitive surface.
[0146] Mapping the controls of an input device to an object can take a variety of forms. For example, if a mouse is the input device being mapped, then input from the mouse can determine the rotation of an object in virtual space, or can cause a visual indicator (e.g., a pointer) on an object to move with respect to a virtual object. This can allow complex actions to be performed by a user using a virtual object.
[0147] In another example, there may be a virtual control panel with many virtual inputs (e.g., switches and knobs) in the virtual reality display. When it is determined that the user's gaze direction is directed to the virtual control panel, and a predetermined time period has passed or a secondary input has been received, keyboard or mouse controls may be mapped to the control panel. This may allow keyboard keys to be mapped accordingly to the control panel, or mouse movement to cause a visible indicator or pointer (i.e., a visual representation of the user's hand) to move on the control panel to select or activate various virtual inputs thereon.
[0148] In some embodiments, the predefined portion of the virtual control panel (or other virtual object) may be the area where the user's gaze direction must be located to activate the control mapping (rather than the user's gaze direction being located anywhere on the control panel). In some embodiments, the predetermined portion may be visually indicated and / or labeled in the virtual display.
[0149] Thus, in some embodiments, a system may be provided that performs the methods discussed herein. Figure 6 In one example shown in , a method 600 for mapping an input device to a virtual object in a virtual space displayed on a display device may include: at block 610, determining a gaze direction of a user via an eye tracking device.
[0150] The method may also include, at block 620, modifying an action to be taken by the one or more processors in response to receiving the first input from the input device based at least in part on the gaze direction directed to the virtual object in the virtual space displayed on the display device. This may also occur in response to the gaze direction directed to the virtual object for at least a predetermined period of time or when the secondary input is received. The user may also be notified by a visible / audible indication that the possible action has been modified.
[0151] Modification of the action may include modifying the action from a normal action taken in response to receiving the first input to a specialized action in the virtual space associated with the virtual object. The specialized action in the virtual space associated with the virtual object may be causing the first input to rotate the virtual object and / or causing the first input to move the virtual object. In these or other embodiments, the specialized action in the virtual space associated with the virtual object may be causing the first input to move a secondary object around the surface of the virtual object. The secondary object may move in a direction corresponding to the direction associated with the first input, and the secondary object may be a pointer or other virtual position marker.
[0152] Thereafter, the method may further include, in response to receiving input from the input device, at block 630 , causing an action to occur at block 640 , wherein the action associates the first input with an interaction with the virtual object.
[0153] In response to some input from the user (perhaps tertiary or other input), after a period of time has passed, or because their gaze has moved away from the virtual object in question, the mapping of input to virtual objects can cease, as shown in box 650.
[0154] By way of example only, input devices that may be mapped include: touch-sensitive devices, touch-sensitive screens, pressure-sensitive devices, mice, trackballs, joysticks, handheld controllers, styluses, keyboards, voice input devices, and / or other input devices discussed herein or known in the art.
[0155] In some embodiments, the apparent effect of an input device mapped to a virtual object as described herein can be magnified or reduced as the user would perceive when operating. By way of example only, in one embodiment, where the user is rotating a virtual representation of a planet, a very small movement input of a mouse mapped to the virtual object can result in a large degree of rotation of the virtual planet. Conversely, when the user is rotating a virtual representation of a tiny particle, a very large input of a mouse mapped to the virtual object can result in a small degree of rotation of the virtual tiny particle. The amount of magnification or reduction can depend on another input set by the user, or can be pre-programmed into a particular application. The magnification or reduction can also vary according to different users and program settings.
[0156] Scan to predicted areas of interest in the display
[0157] In some embodiments, the pan of the user's eyes can be detected and used to estimate or determine where the user's line of sight will be when the pan is completed. In addition, the pans of different individual users can be analyzed, and information related thereto can be stored to predict future changes in the gaze direction of a particular user when the pan occurs (i.e., pans of different speeds for different users cause different angles of gaze direction to change).
[0158] Additionally, known salient regions or objects of interest can also inform any prediction algorithm of the likelihood that the user's gaze direction will ultimately be located upon that region or object when a glance is completed in that direction. The user's gaze direction pattern, including the length of time the user's gaze direction is located upon a given region or object, can also be analyzed to determine whether those regions or objects are particularly salient when displayed in the future.
[0159] Therefore, the above information about the user's panning and gaze direction changes, whether in the direction of a known or unknown salient region or object, can be used to implement foveated rendering. For example, if a known salient region or object is to be displayed in a virtual display, and the user's eyes pan toward the region or object while the region or object is displayed, the region or object can be rendered at a higher quality before the user's gaze direction reaches it.
[0160] Even when the prominence of a displayed area or object is unknown when displayed, or when no particularly salient area or object is displayed, an angular change in gaze direction, and a corresponding angular change in the viewing area that may result, may be determined or estimated by detecting that a glance is occurring. The amount of angular change in gaze direction that is determined or estimated may be specific to the user, based on glance data previously recorded therefor, or may be default data applicable to all users until recorded data for a particular user is available.
[0161] In certain embodiments, if the pan information does not allow to accurately determine the possible result gaze direction of the user, a plurality of known prominent regions or objects that can be positioned at the most likely gaze direction can be rendered with higher quality so.In certain embodiments, the increase of the rendering quality for each region or object can be proportional to the possibility that the gaze direction of the user is finally positioned thereon.For example, if two regions are likely to be the result gaze region of the user's pan, the rendering quality in two regions can increase similarly so.But, if a region is more likely to be the result gaze region, its rendering quality can improve to a greater extent than unlikely result gaze region so.
[0162] The manner in which an image displayed on display device 110 may be modified by a graphics processing device may vary depending on the embodiment, but in any event, the image may be displayed in a manner intended to increase image quality in portions of the image to which a user's gaze or focused gaze is directed, as compared to portions of the image to which the user's gaze or focused gaze is not directed. In this manner, the available resources of the graphics processing device and / or the use of other system resources are maximized to provide image quality in the most important locations of display device 110. For demonstration purposes, Fig. 9 The display device 110 is illustrated showing a user's gaze point 910 and an area 920 surrounding the user's gaze point 910, wherein embodiments of the present invention may increase the quality of an image relative to a remaining area 930 of the display device 110. Thus, in various embodiments of the present invention, the quality of an image produced across the display device 110 may be increased in the area 920 relative to the remaining area 930.
[0163] When discussing "modification" of an image presented on the display device 110 herein, it should be understood that the intention is that a subsequent image displayed on the display device 110 is different from a previous image displayed on the display device 110. Thus, the graphics processing device and display device 110 or other device discussed herein "modifies" an image by causing a first image to be displayed and then displaying a second image that is different from the first image. Any other changes to the image discussed herein (e.g., an increase or decrease in image quality) should also be understood to indicate that the subsequent image is different from the previous image. Note that the change or modification of an image may include changing or modifying only a portion of the image. Thus, some portions of a previous image may be the same as a subsequent image, while other portions may be different. In other cases, the entirety of a previous image may be different from a subsequent image. It should be understood that modification of an area or the entirety of an image does not necessarily mean that every finite portion of the area or the entirety is changed (e.g., every pixel), but rather that the area or the entirety may be changed in some potentially consistent, predetermined, or ordered manner (e.g., changing the quality of the image).
[0164] Increasing the quality of an image may include increasing the quality of any one or more of the following non-exclusive graphical features, in addition to other possible features known in the art:
[0165] Resolution: The number of distinct pixels that can be displayed in one or more dimensions. For example, "1024×768" means 1024 pixels are displayed in height and 768 pixels are displayed in width.
[0166] Shadows: The color and brightness of graphical objects vary depending on the artificial lighting cast by the light sources simulated by the graphics processing device 130 .
[0167] Texture Mapping: Mapping a graphic image or "texture" onto a graphic object to give the object a specific appearance. The resolution of textures affects the quality of the graphic object to which they are applied.
[0168] Bump mapping: simulation of small-scale bumps and rough gradients on the surface of a graphic object.
[0169] Atomization / Participating Media: Modulation of light as it passes through a non-clear atmosphere or air.
[0170] Shadow: Simulation of obstruction of light.
[0171] Soft Shadows: Variations in shadow and darkness caused by partially blurred light sources.
[0172] Reflection: A representation of a mirrored or high-gloss reflective surface.
[0173] Transparency / Opacity (optical or graphic): The rapid transmission of light through a solid object.
[0174] Translucency: Highly diffuse transmission of light through a solid object.
[0175] Refraction: The bending of light associated with transparency.
[0176] Diffraction: The bending, propagation, and interference of light rays by an object or void that destroys them.
[0177] Indirect Illumination: Surfaces illuminated by light reflected by other surfaces, rather than directly from a light source (also called global illumination).
[0178] Caustics (a form of indirect lighting): Reflection of light from a shiny object, or the focusing of light through a transparent object, to produce a bright spot on another object.
[0179] Anti-aliasing: The process of blending the edges of display objects to reduce the appearance of sharp or jagged lines. An algorithm is usually used to sample the colors around the edge of a display object to blend the edge with its surroundings.
[0180] Frame rate: For animated images, the number of individual frames rendered during a specific period of time to render motion within the image.
[0181] 3D: The visual and temporal properties of an image make it appear three-dimensional to the viewer.
[0182] The size and shape of the image region that may be modified to appear at a higher quality may vary depending on the embodiment. By way of example only, the shape of the region may be circular, elliptical, square, rectangular, or polygonal. In some embodiments, the quality of the image within the region may be increased uniformly. In other embodiments, the increase in image quality may be greatest at the center of the region (i.e., near the point of fixation) and decrease toward the edges of the region (i.e., far from the point of fixation), perhaps to match the quality of the image surrounding the region. For demonstration purposes, Fig. 10A shows how image quality can decrease in a linear or nonlinear continuous manner from the center of the fixation area outward, while Fig. 10B It shows how the image quality decreases in a step-wise manner from the center of the fixation area outwards.
[0183] In some embodiments, in response to detection of a change in gaze point, modifying the image displayed on display device 110 may occur. This may occur in a variety of ways, some of which are described below.
[0184] In some embodiments, the entirety of the image may be modified during the period of change of the user's gaze point, and once the change of the user's gaze point stops, the area around the user's terminal gaze point or the remainder of the image (the portion of the image that is not around the terminal gaze point) may be modified. By way of example only, in one embodiment, the quality of the entire image may increase during the movement of the user's gaze (sometimes referred to as a glance), but once the glance is completed, the increase in quality may only last in the area around the user's final gaze point (i.e., the quality of the rest of the image may decrease when the glance is completed). In different embodiments, the quality of the entire image may be reduced during the glance, but once the glance is completed, the reduction in quality may only last in areas other than around the user's final gaze point (i.e., the quality of the image area around the user's final gaze point when the glance is completed).
[0185] In addition, the use of other system resources (comprising, for example, processor / computer and related resources) also can be modified during the user's pan.For example, during the pan, the operation of non-graphics can be supplemented by the resource of processor / computer and graphics processing device.More specifically, during the pan, the non-graphics calculation required for other system operations can be carried out with higher speed or efficiency, because the additional resources associated with processor / computer and graphics processing device can be used for such operation.
[0186] In some embodiments, modifying an image displayed on display device 110 may include modifying a portion of the image in an area around an expected gaze point of the user, possibly by increasing its quality. The expected gaze point may be determined based on a change in the user's gaze point. To determine the user's expected gaze point, the eye tracking device and / or another processor (i.e., a processor of a computer or game controller) may determine a rate of change of the user's gaze point on the display device and determine the expected gaze point based at least in part on the rate of change.
[0187] The rate of change of user's fixation point, also referred to as the speed or rate of user's pan, directly depends on the total change (commonly referred to as the amplitude of pan) of user's fixation point.Therefore, along with the increase of the amplitude of the intention of user's pan, the speed of pan also increases.Although the pan of human user can reach 900 ° / second in human, for the pan that is less than or about 60 °, the speed of pan usually depends on the amplitude of pan linearly and directly.For example, the amplitude of 10 ° is associated with the speed of 300 ° / second, and the amplitude of 30 ° is associated with the speed of 500 ° / second.For the pan that is greater than 60 °, the peak speed begins to reach steady state towards the maximum speed (900 ° / second) that eyes can reach.In response to unexpected stimulation, pan usually needs about 200 milliseconds (ms) to start, then continues about 20ms to about 200ms.Based on these relations between pan speed and amplitude, embodiments of the present invention can determine the fixation point of expectation based on pan speed.Other predetermined models of the mathematical relationship between pan speed and amplitude can also be adopted by various embodiments of the present invention, to determine the fixation point of expectation.
[0188] In some embodiments, the portion of the image modified around the expected gaze point may also include a portion of the image around the original gaze point (i.e., the gaze point where the user's saccade begins). Although the shape of the portion of the image modified can be any of those shapes described above, in some embodiments, it can be as follows: Fig.11 The triangles or trapezoids shown have gradually increasing widths perpendicular to the scanning direction.
[0189] exist Fig.1110 is shown in FIG. 1 , and an initial user gaze point 1110 is shown thereon. Prior to any change in the initial gaze point 1110, embodiments of the present invention may provide increased graphic quality in region 1120. When a user pan is detected by an eye tracking device, represented by arrow 1130, the size and shape of region 1120 may change to accommodate the initial gaze point 1110 and the expected gaze point 1140. In this embodiment, the changed region 1150, while triangular and / or trapezoidal, may be a different shape and size in other embodiments. By way of example only, the entire side of the display device 110 from the initial gaze point to the edge of the display in the direction of the pan may also be included in the changed region 1150 to account for more possibilities of where the user's gaze point may end up. In other embodiments, a circular, elliptical, or square changed region 1150 may be provided. In other embodiments, the changed region 1150 may include separate and distinct regions around the initial gaze point 1110 and the expected gaze point 1140.
[0190] In some embodiments, the size or shape of the area around the gaze point where the image is modified (or the size or shape of the area around the gaze point that remains unchanged from the increased quality in various embodiments) is dynamic. This may occur at least in part based on any number of factors, including the current position of the gaze point relative to the image or display device. By way of example only, if the user moves their gaze point to a certain portion of the screen, a predetermined portion of the screen may be modified by increasing the quality therein (e.g., a corner portion of a display having a mapping of a virtual area in a video game). In some embodiments, if enough user glances having one or more predetermined characteristics are detected within a predetermined amount of time, the entire display may be modified to be rendered at a higher quality.
[0191] In another embodiment of the present invention, a non-transitory computer-readable medium having instructions thereon for presenting graphics on a display device 110 is provided. The instructions may be executed by one or more processors to at least display an image on the display device 110. The instructions may also be executable to receive information indicating at least one of a gaze point of a user on the display device 110 or a change in the gaze point of a user on the display device 110 from an eye tracking device. The instructions may also be executable to cause a graphics processing device to modify an image displayed on the display device 110 based at least in part on the gaze point of a user on the display 110 or a change in the gaze point of a user on the display device. Therefore, a non-transitory computer-readable medium capable of implementing any of the features described herein with respect to other embodiments is also provided.
[0192] In another embodiment of the present invention, Fig.12As shown, a method 1200 for presenting graphics on a display device 110 is provided. At step 1210, the method 1200 may include displaying an image on the display device 110. At step 1120, the method 1200 includes: receiving information indicating at least one of a gaze point of a user on the display device 110 or a change in the gaze point of the user on the display device 110 from an eye tracking device. At step 1230, the method 1200 also includes: causing a graphics processing device to modify the image 110 displayed on the display device 110 based at least in part on the gaze point of the user on the display device 110 or the change in the gaze point of the user on the display device 110. Step 1230 may include: at step 1233, increasing the quality of the image in the area around the user's gaze point relative to outside the area. Step 1230 may also include: at step 1236, reducing the quality of the image outside the area around the user's gaze point relative to within the area. Therefore, a method capable of implementing any of the features described herein with respect to other embodiments is also provided.
[0193] In some embodiments, the systems and methods described herein may be turned on and off by a user, perhaps to account for additional viewers present at multiple display devices 110. In other embodiments, the systems and methods described herein may be automatically turned on when only one user is viewing display device 110 (as detected by an eye tracking device), and turned off when more than one user is viewing display device 110 (as detected by an eye tracking device). Additionally, in some embodiments, the systems and methods described herein may allow for the rendering quality of the entire display device 110 to be reduced when no viewers are detected, thereby saving system resources and power consumption when display device 110 is not the primary focus of any viewer.
[0194] In other embodiments, the systems and methods described herein may allow for modification of multiple portions of an image on display device 110 to account for multiple viewers detected by an eye tracking device. For example, if two different users focus on different portions of display device 110, then two different areas of the focused pattern may be rendered at a higher quality to provide enhanced viewing for each viewer.
[0195] In other embodiments, data associated with the image can inform the systems and methods described herein to allow prediction of which areas of the image are likely to be focused on next by the user. This data can supplement the data provided by the eye tracking device to allow for faster and smoother adjustment of the quality of the image in areas where the user is likely to focus. For example, during viewing of a sporting event, a picture-in-picture of an interview with a coach or player may be presented in a corner of the image. Metadata associated with the image feed can inform the systems and methods described herein of the likely importance in a sub-portion of the image, and therefore inform the viewer of interest and likely focus.
[0196] Adjusting the trajectory of projectiles in VR
[0197] In virtual reality embodiments, when a virtual object is thrown or otherwise launched by a user action, there is a problem whereby the user cannot easily estimate the weight of the object, and therefore how far the object (now a projectile) will fly when launched. Given that accurate distance is necessary for virtual interaction to be usable or successful, gaze detection can be used for the purpose of helping to increase the accuracy of placing a projectile at a desired target location.
[0198] In some embodiments, an eye tracking device can be used to determine the gaze direction of the user launching the virtual object, and allow the location of the gaze direction on the virtual display to be used to determine or assist in determining the intended destination of the projectile. In some embodiments, the exact gaze location on the virtual display can be used as the user's intended target.
[0199] In other embodiments, the gaze position on the virtual display may simply inform the software process of the intended target, and this may affect (depending on the degree of variability of the algorithm) the calculated target for the projectile. Thus, while the initial target may be calculated by the algorithm, and depending on factors such as the speed of the throw or other motion that initiates the projectile and the virtual assigned weight of the object, the gaze direction may be used to modify the calculated initial target to a variable degree (as specified by the algorithm).
[0200] Distributed multi-camera array turns VR lenses on / off
[0201] In traditional VR or other wearable devices, this can be challenging: positioning the image sensor and camera associated with the eye tracking device so that they can get good imaging but do not interfere with the viewing lens of the display system therein. To address this problem, one solution provided by various embodiments of the present invention is to place one or more small camera / image / light sensor arrays (and the illuminators discussed herein) on or in the lens directly in front of the user's eye. Because the camera is very small (e.g. 1mm 3), and are very close to the user's eyes, so they essentially appear invisible to the user because they are not within the user's effective focal distance. Fig. 7A As shown, in front of the display device 110 of a given system, there may be a viewing lens / main lens 710, and an array of cameras / illuminators 720 may also be present on or in the lens 710. In this figure, the cameras / illuminators 720 are shown larger than to scale for demonstration purposes. Fig. 7A This will be discussed in further detail below.
[0202] Although each camera of the array may have only limited resolution, when all the images envisioned are combined and reconstructed by the associated processing equipment, a standard image can be obtained. In addition, because an array is used, additional depth information of the image can be obtained by using a single camera or image sensor for observing the eye.
[0203] User and fault presence verification for safe light source operation
[0204] The customer's equipment must comply with various safety standards for light sources and laser sources near the user's eyes. The closer the customer's equipment light source / laser source is to the user, the lower the amplitude of illumination allowed to meet the various standards.
[0205] In some embodiments, the device herein may include a safety mechanism arranged to protect the user and verify that the criteria are met. In one embodiment, the eye tracking device herein may determine via an image sensor whether a user is present in front of the device and react accordingly.
[0206] If a user is not present, the lighting levels of the displays and / or eye tracking illuminators are reduced to be safe at any distance. Since there is no user present, the displays and / or illuminators will be illuminated at safe levels for any conditions that arise. However, if a user is determined to be present, the distance to any display and / or eye tracking subsystem (e.g., illuminator) can be calculated and the maximum allowable lighting allowed by the system can be restored. In some embodiments, if the eye tracking device determines that no user is present, the display device can be turned off to save power.
[0207] In some embodiments, the system herein may also be able to determine whether any illuminator is blocked by directly detecting an obstacle by the image sensor, or by the image sensor detecting no light from a particular illuminator. If a illuminator or other light emitting device (e.g., a display) is determined to be blocked, such light emitting device may be completely dimmed or disabled to save power.
[0208] Reduced visibility of lighting equipment and image sensors in VP devices
[0209] In some embodiments, in order to place the lighting device and image sensor of the present invention in the most advantageous position, the lighting device and image sensor supported by the flexible printed circuit (FPC) may be embedded in the lens of the virtual reality or other display device, such as Fig. 7A The LEDs or other illuminators and image sensors can be located on a very thin FPC and then oriented to minimize their visible profile.
[0210] By way of example only, the illuminator and image sensor may be mounted on an FPC such that the illuminator or image sensor is mounted facing the user, but only the outline of the FPC (i.e., the thickness of the FPC) is directly viewed by the user of the device. In some embodiments, the FPC may be attached to the illuminator or image sensor such that it is connected to more than one side of the illuminator / sensor. In this way, the light captured from the illuminator / sensor or the light directed from the illuminator / sensor may be improved.
[0211] In these and other embodiments, another approach can also be used to minimize the appearance of the illuminator in a virtual reality or other display device. In these embodiments, the optical fiber can be embedded in a groove on the front side (user side) of the viewing lens of the device. Figure 7B Such grooves 730 and optical fibers 740 are shown deposited in the surface of lens 710. The optical fibers may be distributed across the front side of the lens in a plurality to generate the illumination required for eye tracking. LEDs or other illumination sources (e.g., light emitting diodes) may be connected to the ends of the optical fibers at the edge or away from the lens.
[0212] Any voids in the grooves where the optical fibers are placed can be filled with a substance 750 such as glue or other adhesive to hold the optical fibers in place and minimize refraction therein. The substance can have the same or similar refractive index as the lens itself to minimize visible distortion of the lens to the user.
[0213] like Figure 7C As shown, an angled and / or reflective surface 760 within the lens 710 at the termination point 745 of the optical fiber 740 can direct light from the end of the optical fiber toward the eye. To further minimize potential stray light from the optical fiber in the lens, a light absorbing material 770 can also be disposed near the termination point of the optical fiber, possibly behind the angled surface. A beam shaping material 780 (e.g., epoxy, silicone, or the like) can also be applied to the lens near the termination point to assist in directing light to the appropriate location on the user's eye.
[0214] Exemplary Computer System
[0215] The above is as Figure 8The block diagram shown illustrates an exemplary computer system 800 in which any embodiment of the present invention may be implemented. The example illustrates a computer system 800 that may be used, for example, in whole, in part, or in various modifications to provide the functionality of the above-described systems and methods.
[0216] The computer system 800 is shown as including hardware elements that can be electrically connected via a bus 890. The hardware elements can include one or more central processor units 810, one or more input devices 820 (e.g., an eye tracking device (whether or not integrated with other devices), a mouse, a keyboard, a touchpad, a microphone, a handheld controller, etc.), and one or more output devices 830 (e.g., a display device, a wearable device with a display, a printer, etc.). The computer system 800 can also include one or more storage devices 840. By way of example, the storage device 840 can be a temporary and / or non-temporary disk drive, an optical storage device, a solid-state storage device (e.g., a random access memory ("RAM") and / or a read-only memory ("ROM")), which can be programmable, flash-updatable, and / or the like.
[0217] The computer system 800 may additionally include a computer-readable storage medium reader 850, a communication system 860 (e.g., a modem, a network card (wireless or wired), an infrared communication device, a Bluetooth device, a cellular communication device, etc.), and a working memory 880, which may include RAM and ROM devices as described above. In some embodiments, the computer system 800 may also include a processing acceleration unit 870, which may include a digital signal processor, a special-purpose processor, and / or other similar processors.
[0218] Computer-readable storage media reader 850 may further be connected to computer-readable storage media, collectively (and optionally in combination with storage device 840) representing comprehensively remote, local, fixed and / or removable storage devices and storage media for temporarily and / or permanently containing computer-readable information. Communication system 860 may allow data to be exchanged with the networks, systems, computers and / or other components described above.
[0219] The computer system 800 may also include software elements, which are shown as currently located in the working memory 880, including an operating system 884 and other code 878. It should be understood that alternative embodiments of the computer system 800 may vary from those described above. For example, customized hardware may also be used and / or specific elements may be implemented in hardware, software (including portable software, such as applets), or both. In addition, connections to other computing devices (such as network input / output and data acquisition devices) may also occur.
[0220] The software of the computer system 800 may include code 878 for implementing any or all of the functionality of the various elements of the architecture described herein. For example, software stored on and / or executed by a computer system (e.g., system 800) may provide the functionality of the methods and systems discussed above. Methods implemented by software on some of these components have been discussed in more detail above.
[0221] The present invention has now been described in some detail for purposes of clarity and understanding. It will be understood, however, that certain changes and modifications may be practiced within the scope of the appended claims.
Claims
1. A method for translating content on a virtual reality display, the method comprising: Using eye tracking devices to determine the user's gaze direction; Determining the user's head direction using a head detection system; as well as Panning content on a virtual reality display in a specific direction and at a specific speed, wherein: the specific direction is based on the gaze direction and the head direction; and the specific speed is based on the gaze direction and the head direction, wherein the specific speed increases when the gaze direction corresponds to the head direction; determining, via the eye tracking device, that the gaze direction of the user has returned to a neutral position during the translation of the content; and Based at least in part on the gaze direction of the user returning to the neutral position, the content is stopped from translating and the content on the virtual reality display is automatically re-centered.
2. The method according to claim 1, characterized in that The specific speed is based on the gaze direction and the head direction and includes: The specific speed is reduced when the gaze direction differs from the head direction by at least a predetermined angle.
3. The method according to claim 1, characterized in that The gaze direction corresponds to the head direction including: The gaze direction is within a predetermined angular direction of the head direction.
4. The method according to claim 1, characterized in that: The specific speed is based on the gaze direction and the head direction and includes: The specific speed is zero when the gaze direction is determined to remain in a neutral direction for a predetermined period of time.
5. The method according to claim 1, characterized in that The specific speed is also based on a signal from a user input device.
6. The method according to claim 1, characterized in that The specific speed is based on the gaze direction and the head direction and includes: The specific speed is adjusted in proportion to the change in the head direction.
7. A system for translating digitized content on a virtual reality display, the system comprising: Virtual reality displays; Eye tracking devices; Head detection system; and One or more processors configured to: Determining the user's gaze direction using the eye tracking device; Determining the user's head direction using the head detection system; Panning the content on the virtual reality display in a specific direction and at a specific speed, wherein: the specific direction is based on the gaze direction and the head direction; and the specific speed is based on the gaze direction and the head direction, wherein the specific speed increases when the gaze direction corresponds to the head direction; determining, via the eye tracking device, that the gaze direction of the user has returned to a neutral position during the translation of the content; and Based at least in part on the gaze direction of the user returning to the neutral position, the content is stopped from translating and the content on the virtual reality display is automatically re-centered.
8. The system according to claim 7, characterized in that The gaze direction corresponds to the head direction including: The gaze direction is within a predetermined angular direction of the head direction.
9. The system according to claim 7, characterized in that The specific speed is based on the gaze direction and the head direction and includes: The specific speed is reduced when the gaze direction differs from the head direction by at least a predetermined angle.
10. The system according to claim 7, characterized in that The specific speed is based on the gaze direction and the head direction and includes: The specific speed is zero when the gaze direction is determined to remain in a neutral direction for a predetermined period of time.
11. The system according to claim 7, characterized in that The specific speed is also based on an input signal from a physical input device.
12. The system according to claim 7, characterized in that The specific speed is based on the gaze direction and the head direction and includes: The specific speed is adjusted in proportion to the change in the head direction.
13. A non-transitory computer-readable medium having stored thereon instructions for translating content on a virtual reality display, the instructions when executed causing a computing device to perform operations comprising: determining a user's gaze direction based on data from an eye tracking device; determining a head orientation of the user based on data from a motion detection system; as well as Panning content on a virtual reality display in a specific direction and at a specific speed, wherein: the specific direction is based on the gaze direction and the head direction; and the specific speed is based on the gaze direction and the head direction, wherein the specific speed increases when the gaze direction corresponds to the head direction; determining, via the eye tracking device, that the gaze direction of the user has returned to a neutral position during the translation of the content; and Based at least in part on the gaze direction of the user returning to the neutral position, the content is stopped from translating and the content on the virtual reality display is automatically re-centered.
14. The non-transitory computer readable medium of claim 13, wherein: The gaze direction corresponds to the head direction including: The gaze direction is within a predetermined angular direction of the head direction.
15. The non-transitory computer readable medium of claim 13, wherein: The specific speed is based on the gaze direction and the head direction and includes: The specific speed is reduced when the gaze direction differs from the head direction by at least a predetermined angle.
16. The non-transitory computer readable medium of claim 13, wherein: The specific speed is based on the gaze direction and the head direction and includes: The specific speed is zero when the gaze direction is determined to remain in a neutral direction for a predetermined period of time.
17. The non-transitory computer readable medium of claim 13, wherein: The specific speed is also based on an input signal from a physical user input device.
18. The non-transitory computer readable medium of claim 13, wherein: The specific speed is based on the gaze direction and the head direction and includes: The specific speed is changed in proportion to a change in the gaze direction or the head direction.
