Eye accommodation detection based on retinal imaging

Abstract

Some implementations disclosed herein provide systems, methods (200) and devices (10) that use a retinal imaging technique to assess a user's eye (45) accommodation (i.e., focusing) during use of an electronic device (10), e.g., in real-time. One or more light sources (322a-d, 1022a-d) produce one or more illuminated spots (320, 420, 520, 620) on the retina that are detectable via a sensor (324, 1024). The size and shape of the spots (320, 420, 520, 620) depend on the eye accommodation / focusing, and thus are used to identify accommodation / focusing changes or measure the eye's accommodation / focusing. Eye accommodation can be determined in real-time while a user is viewing, interacting with, or otherwise experiencing electronic content via the electronic device.

Description

Technical Field

[0001] This disclosure relates in general to providing an improved user experience on electronic devices, and more specifically to systems, methods, and apparatus for detecting eye accommodation (i.e., focusing) during the use of electronic devices. Background Technology

[0002] Various eye-tracking techniques exist. For example, some gaze-tracking techniques are based on detecting reflections on the outer parts of the eye. Light is directed to the eye to induce detectable reflections on the pupil and cornea, and these reflections are tracked by a camera to determine the gaze direction, for example, by determining a vector between the cornea and pupil corresponding to the gaze direction. Existing eye-tracking techniques may not provide sufficient tracking of eye accommodation (i.e., focusing). Summary of the Invention

[0003] Some specific embodiments disclosed herein provide systems, methods, and apparatuses for evaluating a user's eye accommodation (i.e., focus) during, for example, in real time, using retinal imaging techniques. One or more light sources direct light onto one or more spots on the retina that can be detected by sensors. The size and shape of the spots depend on eye accommodation / focus and are therefore used to identify changes in accommodation / focus or to measure eye accommodation / focus. Eye accommodation can be determined in real time while the user is viewing, interacting with, or otherwise experiencing electronic content via the electronic device.

[0004] Some specific implementations involve methods executed via a processor that executes instructions stored in the non-transitory memory of an electronic device. This method may involve using the light source at an electronic device having a processor and a light source to generate light (e.g., infrared or visible light) to direct the light to a spot on the retina of the eye. For example, one or more collimating illuminators may be used to direct light to one or more spots on the retina. The light source may be oriented off-axis relative to the axis of the eye. The method receives sensor data (e.g., images) at a sensor (e.g., a camera). The sensor data corresponds to the illuminated spot on the retina. For example, the sensor data may include one or more camera images having image portions, for example, pixels corresponding to the retinal appearance including light reflected at the illuminated spot.

[0005] This method determines eye accommodation characteristics based on sensor data. In some implementations, eye accommodation characteristics are determined based on the size or location of illuminated spots on the retina. Determining eye accommodation characteristics may involve detecting changes in eye accommodation or finding an estimate of eye accommodation (e.g., a numerical value representing eye accommodation). In some implementations, determining eye accommodation characteristics involves comparing an image of the retina with a previous image of the retina to identify changes in the size and / or location of the illuminated spots. In some implementations, the spots are identified (e.g., their size, shape, and / or location) and / or distinguished from other, less illuminated portions of the retina. Spots can be identified via algorithms (e.g., threshold-based) and / or using machine learning (ML) models. ML models can be used to evaluate / compare the size and / or location of spots and / or relationships between multiple spots.

[0006] In some embodiments, a non-transitory computer-readable storage medium stores instructions that are computer-executable to perform or cause to perform any of the methods described herein. In some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. Attached Figure Description

[0007] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to some exemplary embodiments, some of which are shown in the accompanying drawings.

[0008] Figure 1 This illustrates a device that provides a user interface and obtains data corresponding to the user when the user is using the electronic device.

[0009] Figure 2 This is a flowchart illustrating an exemplary method for assessing a user’s eye accommodation (i.e., focus) using retinal imaging techniques according to some specific implementations.

[0010] Figure 3 One or more illuminators are shown, according to some specific implementations, that direct light to a spot on the retina of the eye in a first adjustment / focusing state.

[0011] Figure 4 It shows including Figure 3 An image of the retina illuminated by a spot.

[0012] Figure 5 This illustrates, according to some specific embodiments, the guidance of light to a spot on the retina of the eye in a second accommodation / focusing state. Figure 3One or more illuminators.

[0013] Figure 6 It shows including Figure 5 An image of the retina illuminated by a spot.

[0014] Figure 7 This is a block diagram illustrating device components according to some specific implementations of an exemplary device.

[0015] Figure 8 It is a block diagram of an exemplary head-mounted device (HMD) based on some specific implementations.

[0016] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0017] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will recognize that other effective aspects or variations do not include all the specific details set forth herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0018] Figure 1 An example of the use of electronic device 10 in physical environment 5 is shown. Physical environment refers to the physical world that people can interact with and / or sense without the assistance of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with physical environments through senses such as sight, touch, hearing, taste, and smell.

[0019] exist Figure 1In the examples, device 10 is shown as a single device. Some embodiments of device 10 are handheld. For example, device 10 may be a mobile phone, tablet, laptop computer, etc. In some embodiments, device 10 is worn by a user. For example, device 10 may be a watch, head-mounted display (HMD), etc. In some embodiments, the functionality of device 10 is implemented via two or more devices (e.g., additionally including optional base stations). Other examples include laptop computers, desktop computers, servers, or other such devices that include additional capabilities in terms of power, CPU capacity, GPU capacity, storage capacity, memory capacity, etc. Multiple devices that can be used to implement the functionality of device 10 may communicate with each other via wired or wireless communication.

[0020] In some embodiments, device 10 includes an eye-tracking system for detecting eye position and eye movement. The eye-tracking system may include one or more illuminators 30 (e.g., one or more infrared (IR) light-emitting diodes (LEDs)) that emit light 40 reflected from the eye 45 and captured by a sensor 35 (e.g., a near-IP (NIR) camera). In one example, one or more illuminators 30 of device 10 may emit NIR light directed to one or more spots on the retina of the user 25's eye 45, and the sensor 35 captures an image of the user 25's eye 45 including the illuminated spots. In some embodiments, the images captured by the eye-tracking system may be analyzed to detect the position and movement of the user 25's eyes, detect other information about the eyes such as gaze direction, and / or detect eye accommodation / focus. Furthermore, the gaze point estimated from the eye-tracking images may enable gaze-based interaction with content.

[0021] In some embodiments, device 10 has a user interface (e.g., a graphical user interface (GUI)), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing multiple functions. In some embodiments, user 25 interacts with the user interface through finger contact and gestures on a touch-sensitive surface. In some embodiments, these functions include image editing, drawing, rendering, word processing, web page creation, disk editing, spreadsheet creation, playing games, making and receiving phone calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, and / or digital video playback. Executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0022] According to some specific implementations Figure 2This is a flowchart representation of method 200 for assessing a user's eye accommodation (i.e., focusing) using retinal imaging technology during use of an electronic device. In some embodiments, method 200 is performed by one or more devices (e.g., device 10). Method 200 may be performed on a mobile device, HMD, desktop computer, laptop computer, or server device. In some embodiments, method 200 is performed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 200 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory).

[0023] At box 210, method 200 uses a light source to generate light (e.g., infrared or visible light) directed toward a spot on the retina of the eye. For example, one or more collimating illuminators may be used to direct light toward one or more spots on the retina. In some embodiments, multiple light sources may simultaneously direct light toward multiple separate spots on the retina. In some embodiments, the light source has a collimating illuminator that simultaneously directs light toward multiple spots on the retina. In some embodiments, the light source provides light of various wavelengths and / or polarization states.

[0024] The light source can be oriented off-axis relative to the eye's axis. Using an off-axis light source increases the effect of accommodative differences on one or more illuminated spots on the retina; that is, the more off-axis, the greater the change in the size and / or position of the spots as the eye's accommodation changes. In some implementations, off-axis illumination can facilitate the use of a dedicated camera for accommodation detection without interfering with retinal eye tracking.

[0025] At block 220, method 200 receives sensor data (e.g., an image) at a sensor (e.g., a camera). The sensor data corresponds to illuminated spots on the retina. For example, the sensor data may include one or more camera images having image portions, such as pixels corresponding to the retinal appearance including light reflected at the illuminated spots. In some specific implementations, the sensor data corresponds to multiple spots.

[0026] At box 230, method 200 determines eye accommodation characteristics based on sensor data. In some implementations, eye accommodation characteristics are determined based on the size or position of an illuminated spot on the retina. Eye accommodation can be determined in real time while content is being presented via an electronic device (e.g., while a user is viewing, interacting with, or otherwise experiencing electronic content via the electronic device). In implementations that illuminate multiple spots, one or more of those spots can be used to determine eye accommodation characteristics. In one example, one of the multiple spots is selected to determine eye accommodation characteristics based on the attributes of that spot (e.g., size, shape, position, etc.) relative to the attributes of the other spots.

[0027] Determining eye accommodation characteristics may involve detecting changes in eye accommodation or finding an estimate of eye accommodation (e.g., a numerical value representing eye accommodation). In some embodiments, determining eye accommodation characteristics involves comparing an image of the retina with a previous image of the retina to identify changes in the size and / or position of one or more illuminated spots. In some embodiments, eye accommodation characteristics are determined based on the distance between two or more simultaneously illuminated spots on the retina or the size of the pattern of illuminated spots on the retina.

[0028] In some implementations, blobs are identified (e.g., their size, shape, and / or location) and / or distinguished from other, less illuminated parts of the retina. Blobs can be identified via algorithms (e.g., threshold-based) and / or using machine learning (ML) models. ML models can be used to evaluate / compare the size and / or location of blobs and / or the relationships between multiple blobs. In one example, a threshold (e.g., a threshold for pixel brightness, relationship to nearby pixels, etc.) is used to identify blobs and / or their size, location, and / or other detectable properties. An example of this is varying the divergence of one or more point illumination sources until the smallest possible blob appears in the imaging camera. When this occurs and / or is detected, the eye's accommodation precisely offsets the calibrated divergence of the sources. Examples of machine learning models include, but are not limited to, neural networks (e.g., artificial neural networks), decision trees, support vector machines, or Bayesian networks.

[0029] In some implementations, the light source provides light of various wavelengths and / or polarization states, and these differences are used to help interpret images obtained via sensors, for example, by facilitating the identification of spots (e.g., their size, shape, and / or location) and / or distinguishing spots from other parts of the retina.

[0030] Eye accommodation determination can be used for various purposes. In some implementations, for example, method 200 also involves determining the object the eye is focusing on based on eye accommodation characteristics. In some implementations, method 200 also involves improving user comfort during the user experience, for example, by positioning virtual objects at a depth in the graphical environment determined based on eye accommodation characteristics. For example, the virtual object may be placed at approximately the depth / distance from the user's focus.

[0031] Eye accommodation can be used to provide information, messages, feedback, or otherwise enhance content offered to a user. For example, content can be highlighted or differentiated based on what the user is focusing on. In some implementations, method 300 also involves tracking the gaze direction based on eye accommodation characteristics. The gaze direction can be used to identify whether the user is focusing on a real or virtual object.

[0032] In some implementations, method 300 also involves altering the optical focus of an outward-facing camera based on determined eye accommodation characteristics. Information obtained from such a camera includes image data that can be presented to a user, modified and presented by the user, or otherwise used by method 300 to provide content to the user.

[0033] In some implementations, eye-adjustment is used to enhance the rendering of apertures on a display. For example, eye-adjustment can be used to determine what the user is focusing on or where the user is focusing relative to the display. This information can be used, for example, to configure the display for optimal or efficient display by adjusting the display settings of different parts of the display accordingly.

[0034] In some implementations, eye accommodation and / or the identification of what the user is focusing on are used to improve communication, efficiency, or other aspects of the system providing method 300. For example, content associated with an object (e.g., enhancements, supplementary information, related objects, etc.) can be extracted or processed based on determining that the user is focusing on the object.

[0035] Figure 3 One or more illuminators 322a-d are shown that direct light to a spot 320 on the retina of the eye 45 in a first adjustment / focusing state. In this example, one or more illuminators 322a-d direct light 325 to the retinal portion of the eye 45. The light 325 travels through the lens 310 to create the spot 320 on the retina of the eye 45. A sensor 324 captures an image of the retina of the eye 45 including the spot 320.

[0036] Figure 4 The retina is shown to include Figure 3 An image 400 depicting a spot 320 illuminated in the image 420. Eye accommodation characteristics are determined based on the size or location of the spot 320 depicted in the image 400 (e.g., relative to the vessels and other aspects of the retina depicted in the image 400). Determining eye accommodation characteristics may involve, for example, determining an estimate (e.g., a value such as 50 diopters) of the accommodation of the eye 45 based on the relative size and / or location of the spot 320 depicted in the image 400 relative to the vessels and other aspects of the retina depicted in the image 400.

[0037] Figure 5 This illustrates how light is directed onto different spots 520 on the retina of the eye 45 in the second accommodation / focusing state. Figure 3 One or more illuminators 322a-d. In this example, one or more illuminators 322a-d direct light 525 to the retinal portion of the eye 45. The light 525 travels through the lens 310 to create a spot 520 on the retina of the eye 45. Because the lens 310 has a... Figure 3 Due to the different shapes of the lens in the eye, spot 520 varies in size and location relative to the surrounding environment. Figure 3 The spots in the image are different. Sensor 324 captures an image of the retina of eye 45, including the spots 520.

[0038] Figure 6 The retina is shown to include Figure 5 Image 600 shows a depiction 620 of a spot 520 illuminated in the image 600. In some embodiments, eye accommodation characteristics are determined based on the size or position of the depiction 620 of spot 520 (e.g., relative to the vessels and other aspects of the retina depicted in image 600). Determining eye accommodation characteristics may involve, for example, determining an estimate (e.g., a numerical value representing the accommodation) of the eye 45 based on the relative size and / or position of the depiction 620 of spot 520 relative to the vessels and other aspects of the retina depicted in image 400. In some embodiments, eye accommodation characteristics are determined based on the relative size or position of the depiction 620 of spot 520 compared to the depiction 420 of spot 320 (e.g., by comparing aspects of image 400 and image 600 to compare data from two different time points and accommodation states).

[0039] Figure 7 This is a block diagram of an example of a device 10 according to some specific implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure further relevant aspects of the specific implementations disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 10 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, SPI, I2C and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more displays 712, one or more internal and / or external image sensor systems 714, memory 720, and one or more communication buses 704 for interconnecting these components and various other components.

[0040] In some embodiments, the one or more communication buses 704 include circuitry for communication between interconnecting system components and control system components. In some embodiments, one or more I / O devices and sensors 706 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an electromyogram (EMG) sensor, a functional near-infrared spectroscopy (fNIRS) sensor, a skin conductance sensor, or an image sensor, such as for pupil response, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), etc.

[0041] In some embodiments, one or more displays 712 are configured to present a user experience to user 25. In some embodiments, one or more displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), retinal projection systems, and / or similar display types. In some embodiments, one or more displays 712 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, device 10 includes a single display. As another example, device 10 includes displays for each eye of user 25, such as an HMD. In some embodiments, one or more displays 712 are capable of presenting extended reality (XR) content, such as augmented reality content, virtual reality content, etc.

[0042] In some embodiments, one or more image sensor systems 714 are configured to acquire image data corresponding to at least a portion of the face of user 25, including the eyes of user 25. For example, the one or more image sensor systems 714 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, event-based cameras, etc. In various embodiments, the one or more image sensor systems 714 may also include an illumination source, such as a flash or flash source, that emits light to the portion of the user 25's face.

[0043] Memory 720 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 720 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 720 optionally includes one or more storage devices remotely located to one or more processing units 702. Memory 720 includes a non-transitory computer-readable storage medium. In some embodiments, memory 720 or the non-transitory computer-readable storage medium of memory 720 stores programs, modules, and data structures, or subsets thereof, including optional operating system 720 and instruction set 740.

[0044] Operating system 730 includes procedures for handling various basic system services and for performing hardware-related tasks. In some implementations, instruction set 740 is configured to implement gaze tracking and / or related features. For this purpose, in various implementations, instruction set 740 includes light generator 742 and tracker 744.

[0045] In some embodiments, the light generator 742 is configured to control the generation of such a spot on the user's retina, for example, by determining when and / or how the illumination source will operate to generate the illuminated spot. For these purposes, in various embodiments, the light generator 742 includes instructions and / or logic for those instructions, as well as heuristics and metadata for the heuristics.

[0046] In some implementations, tracker 744 is configured to determine the eye's accommodation / focus based on sensor data. This may involve detecting one or more illuminated spots on the retina via sensors and using the size and / or shape of the spots to determine eye accommodation / focus characteristics, such as state, changes, etc. To this end, in various implementations, tracker 744 includes instructions and / or logic for those instructions, as well as heuristics and metadata for those heuristics.

[0047] although Figure 7 The units and modules are shown as residing on a single device (e.g., device 10), but it should be understood that in other specific implementations, any combination of these units may reside in a separate computing device.

[0048] also, Figure 7 This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 8Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0049] Figure 8 A block diagram of an exemplary head-mounted device 1000 according to some specific embodiments is shown. The head-mounted device 1000 includes a housing 1001 (or shell) that houses various components of the head-mounted device 1000. The housing 1001 includes (or is coupled to) eye pads (not shown) disposed at a proximal (user 25) end of the housing 1001. In various specific embodiments, the eye pads are plastic or rubber components that comfortably and snugly hold the head-mounted device 1000 in a proper position on the face of the user 25 (e.g., around the eyes of the user 25).

[0050] Housing 1001 houses display 1010, which displays images, emits light toward or onto the eyes of user 25. In various embodiments, display 1010 emits light through an eyepiece having one or more lenses 1005 that refract the light emitted by display 1010, causing the display to appear to user 25 at a virtual distance greater than the actual distance from the eyes to display 1010. In order for user 25 to focus on display 1010, in various embodiments, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). Furthermore, to provide a better user experience, in various embodiments, the virtual distance is greater than 1 meter.

[0051] The housing 1001 also houses a tracking system comprising one or more light sources 1022, a camera 1024, and a controller 1080. One or more light sources 1022a-d emit light toward the user's eye 45, which is reflected and detectable by the camera 1024. Based on the reflection, the controller 1080 can determine eye-tracking characteristics such as the user's eye accommodation characteristics. In some embodiments, the controller 1080 can determine the user's gaze direction and / or blinking state (open or closed eyes). For example, the controller 1080 can determine the pupil center, pupil size, or point of focus. For example, the controller 1080 can determine the user's eye accommodation characteristics. Thus, in various embodiments, light is emitted by one or more light sources 1022a-d, reflected from the user's eye 45, and detected by the camera 1024. In various embodiments, light from the user's eye 45 is reflected from a thermal mirror or passes through an eyepiece before reaching the camera 1024.

[0052] Display 1010 emits light within a first wavelength range, and one or more light sources 1022a-d emit light within a second wavelength range. Similarly, camera 1024 detects light within the second wavelength range. In various specific embodiments, the first wavelength range is the visible wavelength range (e.g., a wavelength range of approximately 400 nm to 700 nm within the visible spectrum), and the second wavelength range is the near-infrared wavelength range (e.g., a wavelength range of approximately 700 nm to 1400 nm within the near-infrared spectrum).

[0053] In various specific implementations, eye tracking (or specifically, determining the direction of gaze and / or accommodation) is used to enable user interaction (e.g., user 25 selects an option by looking at and focusing on the display 1010), to provide punched rendering (e.g., presenting a higher resolution in the area of ​​display 1010 that user 25 is viewing and a lower resolution elsewhere on display 1010), or to correct distortion (e.g., for the image to be presented on display 1010).

[0054] In various embodiments, camera 1024 is a frame / shutter-based camera that generates images of the user 25's eye at one or more time points at a frame rate. Each image includes a matrix of pixel values ​​corresponding to the pixels in the image, the pixels corresponding to the positions of the camera's light sensor matrix. In specific embodiments, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0055] It should be understood that the specific embodiments described above are cited by way of example, and this disclosure is not limited to what has been specifically shown and described above. Rather, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to those skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

[0056] As described above, one aspect of the present invention is the collection and use of physiological data to improve the user's electronic device experience. This disclosure envisions that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0057] This disclosure recognizes that the use of such personal information data in the present invention can benefit users. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control over electronic devices. Furthermore, this disclosure also anticipates other uses of personal information data that benefit users.

[0058] This disclosure also envisions that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate purposes. Furthermore, such collection should only be conducted with the user's informed consent. Additionally, such entities should take any necessary steps to safeguard and protect access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and procedures. Furthermore, such entities may be subject to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices.

[0059] Regardless of the foregoing, this disclosure also contemplates specific implementations allowing users to selectively block the use or access to personal information data. That is, this disclosure contemplates providing hardware or software components to prevent or block access to such personal information data. For example, with regard to a content delivery service tailored to a user, the technology of this invention can be configured to allow a user to choose to "join" or "opt out" of the collection of personal information data during service registration. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow the transmission of anonymous information for improving device functionality.

[0060] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not be rendered inoperable due to the absence of all or part of such personal information data. For example, preferences or settings can be inferred based on non-personal information data or a minimal amount of personal information, such as content requested by a device associated with a user, other non-personal information available to the content delivery service, or publicly available information, thereby selecting content and delivering it to the user.

[0061] In some implementations, data is stored using a public / private key system that allows only the data owner to decrypt the stored data. In other implementations, data may be stored anonymously (e.g., without requiring identification and / or personal information about the user, such as legal name, username, time, and location data). This prevents other users, hackers, or third parties from identifying the user associated with the stored data. In some implementations, users can access their stored data from a different user device than the one used to upload the stored data. In these cases, users may need to provide login credentials to access their stored data.

[0062] This document sets forth numerous specific details to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods, apparatus, or systems known to a person of ordinary skill have not been described in detail so as not to obscure the claimed subject matter.

[0063] Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as “processing,” “calculating,” “calculating,” “determining,” and “identifying” refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data representing physical electronic or magnetic quantities within the memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0064] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein can be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0065] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the above examples can be changed; for example, the boxes can be reordered, grouped, or divided into sub-boxes. Some boxes or procedures can be executed in parallel.

[0066] The use of “applies to” or “configured to” in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Similarly, the use of “based on” implies openness and inclusivity, as processes, steps, calculations, or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included in this document are for illustrative purposes only and are not intended to be restrictive.

[0067] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various objects, these objects should not be limited by these terms. These terms are merely used to distinguish one object from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0068] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and the appended claims, the singular forms “a” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “comprising” or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, objects, or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, components, or groups thereof.

[0069] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" are interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when the prerequisite is detected" or "in response to detection" that the prerequisite is true, depending on the context.

[0070] The foregoing description and summary of the present invention should be understood as illustrative and exemplary in every respect, and not restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the illustrative specific embodiments, but also by the full extent permitted by patent law. It should be understood that the specific embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method for detecting eye accommodation based on retinal imaging, the method comprising: In electronic devices that have processors and light sources: Illuminate the light source to direct the light to the retina of the eye; Sensor data is received at the sensor, the sensor data corresponding to the illuminated spot on the retina; Eye accommodation characteristics are determined based on the sensor data, the eye accommodation characteristics being determined based on the size or position of the illuminated spot on the retina, wherein determining the eye accommodation characteristics includes comparing the size or position of the illuminated spot associated with a first time point with the size or position of a corresponding illuminated spot associated with a second time point, wherein the size or position of the illuminated spot associated with the first time point and the size or position of the corresponding illuminated spot associated with the second time point are determined based on distinguishing other less illuminated portions of the retina; as well as The virtual object is positioned at a depth in the graphical environment based on the eye accommodation characteristics.

2. The method of claim 1, wherein the eye accommodation characteristics are determined during the presentation of content via the electronic device.

3. The method of claim 1, wherein the light source comprises a collimating illuminator that directs light to a plurality of spots on the retina.

4. The method of claim 1, wherein the axis of the light generated using the light source is off-axis relative to the lens of the eye.

5. The method of claim 1, wherein the sensor data includes images.

6. The method of claim 1, wherein determining the eye accommodation characteristic includes determining that eye accommodation has changed.

7. The method of claim 1, wherein determining the eye accommodation characteristics includes determining the eye accommodation state.

8. The method of claim 1, wherein determining the eye accommodation characteristics comprises comparing the size of the illuminated spot associated with the first time point with the size of the corresponding illuminated spot associated with the second time point, wherein the divergence of the illuminated spot associated with the first time point and the second time point varies over time or at its position on the eye.

9. The method of claim 1, wherein determining the eye accommodation characteristics includes comparing the position of the illuminated spot associated with the first time point with the position of the corresponding illuminated spot associated with the second time point.

10. The method of claim 1, wherein determining the eye accommodation characteristics includes comparing the relative positions of a plurality of illuminated spots associated with the first time point with the relative positions of corresponding illuminated spots associated with the second time point.

11. The method of claim 1, wherein the size or location of the illuminated spot is determined via a machine learning model.

12. The method of claim 1, further comprising determining the object focused by the eye based on the eye accommodation characteristics.

13. The method of claim 1, further comprising tracking the gaze direction of the eye based on the eye accommodation characteristics.

14. An apparatus for detecting eye accommodation based on retinal imaging, the apparatus comprising: light source; Non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations including: Illuminate the light source to direct the light to a spot on the retina of the eye; Sensor data is received at the sensor, the sensor data corresponding to the illuminated spot on the retina; Eye accommodation characteristics are determined based on the sensor data, the eye accommodation characteristics being determined based on the size or position of the illuminated spot on the retina, wherein determining the eye accommodation characteristics includes comparing the size or position of the illuminated spot associated with a first time point with the size or position of the corresponding illuminated spot associated with a second time point, wherein the size or position of the illuminated spot associated with the first time point and the size or position of the corresponding illuminated spot associated with the second time point are determined based on distinguishing other less illuminated portions of the retina; as well as The virtual object is positioned at a depth in the graphical environment based on the eye accommodation characteristics.

15. The device of claim 14, wherein the light source comprises a collimating illuminator configured to illuminate a plurality of spots on the retina.

16. The device of claim 14, wherein determining the eye accommodation characteristic includes determining that eye accommodation has changed.

17. The device of claim 14, wherein determining the eye accommodation characteristics includes determining the eye accommodation state.

18. The device of claim 14, wherein the device is a head-mounted device (HMD).

19. A non-transitory computer-readable storage medium that stores computer-executable program instructions on a computer to perform operations including: Illuminate the light source to direct the light to the spot on the retina of the eye; Sensor data is received at the sensor, the sensor data corresponding to the illuminated spot on the retina; Eye accommodation characteristics are determined based on the sensor data, the eye accommodation characteristics being determined based on the size or position of the illuminated spot on the retina, wherein determining the eye accommodation characteristics includes comparing the size or position of the illuminated spot associated with a first time point with the size or position of a corresponding illuminated spot associated with a second time point, wherein... The size or location of the illuminated spot associated with the first time point and the size or location of the corresponding illuminated spot associated with the second time point are determined based on distinguishing other less illuminated portions of the retina; as well as The virtual object is positioned at a depth in the graphical environment based on the eye accommodation characteristics.

Images