Adaptive algorithm for high efficiency eye tracking

Abstract

Systems and techniques for eye tracking are provided. For example, a computing device may obtain an image of an eye. The image of the eye includes light flicker from a light source. The computing device may determine information associated with the light flicker based on the image of the eye. The computing device may obtain one or more target flicker parameter values, and may compare information associated with the light flicker to the one or more target flicker parameter values. The computing device may adjust an amount of power for the light source based on comparing information associated with the light flicker to the one or more target flicker parameter values.

Description

Technical Field

[0001] The present disclosure generally relates to eye tracking. For example, aspects of the present disclosure relate to systems and techniques for executing adaptive algorithms for power-efficient eye tracking solutions. Background Art

[0002] Extended reality technology can be used to present virtual content to users and / or can combine real environments from the physical world with virtual environments to provide users with XR experiences. The term "XR" can include VR, AR, mixed reality, etc. Extended reality systems can allow users to experience XR environments by superimposing virtual content on images of real-world environments, and users can view the extended reality environment through XR devices (e.g., head-mounted displays, extended reality glasses, or other devices). Extended reality (XR) devices are devices that display an environment to users, such as through a head-mounted display (HMD) or other device. The environment is at least partially different from the real-world environment in which the user is located. Users can typically interactively change their view of the environment, for example by tilting or moving the HMD or other device.

[0003] In some cases, an XR system may include a "see-through" display that allows a user to see their real-world environment based on light from the real-world environment passing through the display. In some cases, an XR system may include a "transparent" display that allows a user to see their real-world environment, or a virtual environment based on their real-world environment, based on a view of the environment captured by one or more cameras and displayed on the display. A user may wear a "see-through" or "transparent" XR system while engaging in activities in their real-world environment.

[0004] In some cases, XR systems can include eye imaging (also referred to herein as gaze detection or eye tracking) systems. In some cases, eye tracking can be used to analyze eye movements to estimate gaze for certain applications, such as XR applications. Eye tracking can be used to help utilize information such as pupil location, gaze vector for each eye, and gaze point. Summary of the Invention

[0005] This document describes systems and techniques for eye tracking. The following presents a simplified summary related to one or more aspects disclosed herein. Therefore, the following summary should neither be considered an exhaustive overview related to all contemplated aspects nor be considered to identify key or critical elements related to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form prior to the detailed description presented below.

[0006] In one illustrative example, a method for eye tracking is provided. The method includes obtaining an image of an eye, the image of the eye including light glint from a light source; determining information associated with the light glint based on the image of the eye; obtaining one or more target glint parameter values; comparing the information associated with the light glint with the one or more target glint parameter values; and adjusting an amount of power to the light source based on the comparison of the information associated with the light glint with the one or more target glint parameter values.

[0007] In another example, an apparatus for eye tracking is provided. The apparatus includes: at least one memory; and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain an image of an eye, the image of the eye including light glint from a light source; determine information associated with the light glint based on the image of the eye; obtain one or more target glint parameter values; compare the information associated with the light glint with the one or more target glint parameter values; and adjust the amount of power applied to the light source based on the comparison of the information associated with the light glint with the one or more target glint parameter values.

[0008] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided. The instructions, when executed by at least one processor, cause the at least one processor to: obtain an image of an eye, the image of the eye including light glint from a light source; determine information associated with the light glint based on the image of the eye; obtain one or more target glint parameter values; compare the information associated with the light glint with the one or more target glint parameter values; and adjust an amount of power to the light source based on the comparison of the information associated with the light glint with the one or more target glint parameter values.

[0009] In another example, an apparatus for eye tracking is provided. The apparatus includes: means for obtaining an image of an eye, the image of the eye including light glint from a light source; means for determining information associated with the light glint based on the image of the eye; means for obtaining one or more target glint parameter values; means for comparing the information associated with the light glint with the one or more target glint parameter values; and means for adjusting the amount of power applied to the light source based on the comparison of the information associated with the light glint with the one or more target glint parameter values.

[0010] In some aspects, one or more of the devices described herein may include or be part of an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device (e.g., a web-connected watch or other wearable device), a personal computer, a laptop computer, a server computer, a television, a video game console, or other device. In some aspects, the device also includes at least one camera for capturing one or more images or video frames. For example, the device may include one or more cameras (e.g., an RGB camera, an IR camera) for capturing one or more images and / or one or more videos comprising video frames. In some cases, the device may also include an assembly of IR light-emitting diodes (LEDs) or lasers and / or diffractive optical elements. In some aspects, the device includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the device includes a transmitter configured to transmit data or information to at least one device via a transmission medium. In some aspects, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or other processing device or component.

[0011] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. This subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0012] The foregoing and other features and examples will become more apparent after reference to the following description, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Illustrative examples of the present application are described in detail below with reference to the following drawings:

[0014] Figure 1 is a block diagram illustrating the architecture of an image capture and processing system according to some examples;

[0015] Figure 2 is a diagram illustrating the architecture of an example extended reality (XR) system according to some examples;

[0016] Figure 3 is a diagram illustrating a simplified cross-sectional view of a lens assembly according to some examples;

[0017] Figure 4 illustrates how eye and facial shape can affect the lighting of the eyes for eye tracking according to some examples;

[0018] Figure 5 is a block diagram illustrating a system implementing an adaptive algorithm for power-efficient eye tracking according to aspects of the present disclosure;

[0019] Figure 6 is a graph illustrating brightness distribution of pixels for flickering in an image according to aspects of the present disclosure;

[0020] Figure 7A and Figure 7B Illustrating the use of multiple light sources to illuminate an eye in accordance with aspects of the present disclosure;

[0021] Figure 8 is a flowchart illustrating a process for image processing according to aspects of the present disclosure;

[0022] Figure 9A is a perspective view illustrating a head-mounted display (HMD) performing feature tracking and / or visual simultaneous localization and mapping (VSLAM) according to some examples.

[0023] Figure 9B is an example based on some examples Figure 9A A perspective view of a head-mounted display (HMD) being worn by a user.

[0024] Figure 10A is a perspective view of a front surface of a mobile device illustrating the use of one or more front-facing cameras to perform feature tracking and / or visual simultaneous localization and mapping (VSLAM) according to some examples.

[0025] Figure 10B is a perspective view illustrating a rear surface of a mobile device according to aspects of the present disclosure.

[0026] Figure 11 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. DETAILED DESCRIPTION

[0027] Provided below are certain aspects and examples of the present disclosure. As will be apparent to those skilled in the art, some of these aspects and examples can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the subject matter of the present application. However, it will be apparent that each example can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0028] The following description provides only illustrative examples and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description will provide a feasible description for realizing the illustrative examples to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the application as set forth in the appended claims.

[0029] A camera is a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). The terms "image," "image frame," and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f-stop, shutter speed, focus, and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters can be applied to the image sensor used to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color. For example, settings or parameters can be applied to the processor (e.g., an image signal processor, or ISP) used to process one or more image frames captured by the image sensor.

[0030] Various devices or systems can use eye tracking to perform one or more operations, including extended reality (XR) systems or devices, vehicles (or computing systems within vehicles), robotic systems or devices, and the like. For example, XR systems or devices can provide virtual content to users and / or can combine a real-world physical environment with a virtual environment (composed of virtual content) to provide users with an XR experience. The real-world environment can include real-world objects (also known as physical objects) such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. XR systems or devices can facilitate interaction with different types of XR environments (e.g., users can use the XR system or device to interact with the XR environment). XR systems can include virtual reality (VR) systems that facilitate interaction with VR environments, augmented reality (AR) systems that facilitate interaction with AR environments, mixed reality (MR) systems that facilitate interaction with MR environments, and / or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, and the like. In some cases, the XR system can track user parts (e.g., the user's hands and / or fingertips) to allow the user to interact with items of virtual content.

[0031] In some cases, an XR system may include an optical "see-through" display or "pass-through" display (e.g., a see-through or pass-through AR HMD or AR glasses), allowing the XR system to display XR content (e.g., AR content) directly onto a real-world view without displaying video content. For example, a user may view physical objects through a display (e.g., glasses or lenses), and the AR system may display AR content onto the display to provide the user with an augmented visual perception of one or more real-world objects. In one example, the display of an optical see-through AR system may include a lens or glasses positioned in front of each eye (or a single lens or glasses positioned above both eyes). The see-through display may allow the user to directly see real-world objects or physical objects, and may display (e.g., project or otherwise display) an augmented image of the object or additional AR content to enhance the user's visual perception of the real world.

[0032] An XR system may include one or more user-facing sensors facing the user, such as a user-facing image sensor (or camera). In some cases, the user-facing sensors may face the user's face, eyes, one or more other parts of the user's body, and / or a combination thereof. In some cases, the XR system may include an eye-tracking system for tracking one or more of the user's eyes. Eye tracking can be used by applications executing on the XR system to utilize information (such as pupil location, gaze vector for each eye, and gaze point) to, for example, control the XR system and / or applications. In some cases, the eye-tracking system may reflect and / or scatter light from a light source away from the user's eyes and detect glint of the reflected light from the light source. Eye movement can be tracked based on pupil movement relative to the glint. In some cases, the brightness (e.g., intensity) of the glint can be based on the amount of light output by the light source. This brightness can vary based on the distance of the light source from the eye. Facial and eye shapes can affect how far the light source is from the eye. Because the intensity of light (and the resulting glint) is inversely proportional to the square of the distance, relatively small changes in the distance between the eye and the light source can make a large difference in the amount of light sufficient to generate useful glint. Additionally, if the light source is too bright, in addition to wasting power, the brightness of the light at certain distances may exceed exposure guidelines. Therefore, it may be useful to adjust the brightness, for example by adjusting the power of the illumination source used for eye tracking.

[0033] This document describes systems, apparatus, electronic devices, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for adaptive algorithms for power-efficient eye tracking. For example, an eye tracker can be configured to determine certain information associated with light glint reflected from the eye, such as the brightness of the glint and the area of ​​the glint. The eye tracker can also obtain a target glint parameter value. The eye tracker can then adjust the power level of the light source to match the parameter value of the information associated with the glint with the target glint parameter value.

[0034] In some cases, adjusting the power level of the light source allows the eye tracker to dynamically adjust how far the light source is from the eye. This adjustment helps allow the brightness of the light to be adjusted for different users or even for movement of the eye tracker. This can help save power and ensure that the brightness of light, such as that reflected / scattered by the eye, does not exceed exposure guidelines.

[0035] Various aspects of the application will be described with respect to the accompanying drawings. Figure 1 is a block diagram illustrating the architecture of image capture and processing system 100. Image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., images of scene 110). Image capture and processing system 100 can capture individual images (or photographs) and / or can capture videos comprising multiple images (or video frames) in a particular sequence. In some cases, lens 115 and image sensor 130 may be associated with an optical axis. In one illustrative example, both the photosensitive area of ​​image sensor 130 (e.g., a photodiode) and lens 115 may be centered about the optical axis. Lens 115 of image capture and processing system 100 faces scene 110 and receives light from scene 110. Lens 115 bends incoming light from the scene toward image sensor 130. Light received by lens 115 passes through an aperture. In some cases, the aperture (e.g., aperture size) is controlled by one or more control mechanisms 120 and received by image sensor 130. In some cases, the aperture may have a fixed size.

[0036] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms beyond those illustrated, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0037] Focus control mechanism 125B of control mechanism 120 may obtain a focus setting. In some examples, focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, focus control mechanism 125B may adjust the position of lens 115 relative to the position of image sensor 130. For example, based on the focus setting, focus control mechanism 125B may actuate a motor or servo (or other lens mechanism) to move lens 115 closer to or further away from image sensor 130, thereby adjusting the focus. In some cases, image capture and processing system 100 may include additional lenses, such as one or more microlenses positioned above each photodiode of image sensor 130, each of which bends light received from lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using control mechanism 120, image sensor 130, and / or image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.In some cases, the lens 115 may be fixed relative to the image sensor, and the focus control mechanism 125B may be omitted without departing from the scope of the present disclosure.

[0038] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f-stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the duration that the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0039] Zoom control mechanism 125C of control mechanism 120 may obtain a zoom setting. In some examples, zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including lens 115 and one or more additional lenses. For example, zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanisms) to move one or more lenses relative to one another. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, this focusing lens may be lens 115) that first receives light from scene 110, where the light then passes through an afocal zoom system between the focusing lens (e.g., lens 105) and image sensor 130 before reaching image sensor 130. In some cases, an afocal zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference of each other), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses. In some cases, zoom control mechanism 125C may control zoom by capturing images from an image sensor (e.g., including image sensor 130) from a plurality of image sensors at a zoom corresponding to a zoom setting. For example, image capture and processing system 100 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, based on the selected zoom setting, zoom control mechanism 125C may capture images from the corresponding sensor.

[0040] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes may be covered in different color filters and may therefore measure light that matches the color of the filter covering the photodiode. Various color filter arrays may be used, including a Bayer color filter array, a four-color color filter array (also known as a four-color Bayer color filter array or QCFA), and / or any other color filter array. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of the image is generated based on red light data from at least one photodiode covered in the red filter, blue light data from at least one photodiode covered in the blue filter, and green light data from at least one photodiode covered in the green filter.

[0041] Back to Figure 1 Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") filters as an alternative to or in addition to red, blue, and / or green filters. In some cases, some photodiodes may be configured to measure infrared (IR) light. In some implementations, the photodiodes measuring IR light may not be covered by any filters, thereby allowing the IR photodiodes to measure both visible light (e.g., color) and IR light. In some examples, the IR photodiodes may be covered by IR filters, thereby allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 130) may lack filters entirely (e.g., color, IR, or any other part of the spectrum) and may instead use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors may also lack filters and, therefore, lack color depth.

[0042] In some cases, image sensor 130 may alternatively or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, opaque and / or reflective masks may be used for phase detection autofocus (PDAF). In some cases, opaque and / or reflective masks may be used to block portions of the electromagnetic spectrum from reaching the image sensor's photodiodes (e.g., IR cutoff filters, UV cutoff filters, bandpass filters, low-pass filters, high-pass filters, etc.). Image sensor 130 may also include analog gain amplifiers for amplifying analog signals output by the photodiodes and / or analog-to-digital converters (ADCs) for converting the analog signals output by the photodiodes (and / or the analog signals amplified by the analog gain amplifiers) into digital signals. In some cases, certain components or functionality discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0043] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including the ISP 154), one or more host processors (including the host processor 152), and / or related Figure 11The host processor 152 may be a digital signal processor (DSP) and / or other types of processors discussed above for the computing system 1100. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) ports 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0044] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in a random access memory (RAM) 140 / 1025, a read-only memory (ROM) 145 / 1020, a cache, a memory unit, another storage device, or some combination thereof.

[0045] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device, any other input device, or some combination thereof. In some cases, subtitles may be entered into the image processing device 105B via a physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad of the touch screen of the I / O device 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that enable wired connections between the image capture and processing system 100 and one or more peripheral devices, via which the image capture and processing system 100 can receive data from and / or send data to the one or more peripheral devices. The I / O devices 160 may also include one or more wireless transceivers that enable wireless connections between the image capture and processing system 100 and one or more peripheral devices, via which the image capture and processing system 100 can receive data from and / or send data to the one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0046] In some cases, the image capture and processing system 100 can be a single device. In some cases, the image capture and processing system 100 can be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B can be disconnected from each other.

[0047] like Figure 1 As shown, the vertical dotted line will Figure 1 1. The image capture and processing system 100 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 105, control mechanism 120, and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145, and I / O device 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0048] The image capture and processing system 100 may include an electronic device, such as a mobile or landline telephone handset (e.g., a smartphone, a cell phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.10 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0049] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include and / or be implemented using electronic circuitry or other electronic hardware that may include one or more programmable electronic circuits (e.g., a microprocessor, GPU, DSP, CPU, and / or other suitable electronic circuitry) and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0050] Figure 2 is a diagram illustrating the architecture of an example extended reality (XR) system 200 according to some aspects of the present disclosure. In some examples, Figure 2The XR system 200 may include the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof. The XR system 200 may run (or execute) XR applications and implement XR operations. In some examples, as part of the XR experience, the XR system 200 may perform tracking and positioning, mapping of an environment in the physical world (e.g., a scene), and / or positioning and rendering of virtual content on a display 209 (e.g., a screen, a visible plane / area, and / or another display). For example, the XR system 200 may generate a map (e.g., a three-dimensional (3D) map) of the environment in the physical world, track the pose (e.g., position and positioning) of the XR system 200 relative to the environment (e.g., relative to the 3D map of the environment), position and / or anchor virtual content in a specific location on the map of the environment, and render the virtual content on the display 209 so that the virtual content appears to be at a location in the environment corresponding to the specific location on the map of the scene at which the virtual content is positioned and / or anchored. The display 209 may include glass, screens, lenses, projectors, and / or other display mechanisms that allow a user to see the real-world environment and also allow XR content to be overlaid, superimposed, blended, or otherwise displayed thereon.

[0051] In this illustrative example, XR system 200 includes one or more image sensors 202, accelerometer 204, gyroscope 206, storage 207, computing component 210, XR engine 220, image processing engine 224, rendering engine 226, communication engine 228, and model generation engine 230. It should be noted that Figure 2 The components 202 to 230 shown are non-limiting examples provided for purposes of illustration and explanation, and other examples may include those related to Figure 2 For example, in some cases, the XR system 200 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 2 One or more other software and / or hardware components not shown. Although various components of XR system 200 (such as image sensor 202) may be referred to herein in the singular, it should be understood that XR system 200 may include multiple components of any component discussed herein (e.g., multiple image sensors 202).

[0052] The XR system 200 includes or communicates (wired or wirelessly) with an input device 208. The input device 208 can include any suitable input device, such as a touch screen, a pen or other pointer device, a keyboard, a mouse, buttons or keys, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, a video game controller, a steering wheel, a joystick, a set of buttons, a trackball, a remote control, any other input device 1145 discussed herein, or any combination thereof. In some cases, the image sensor 202 can capture images that can be processed for interpreting gesture commands.

[0053] The XR system 200 may also communicate with one or more other electronic devices (wired or wirelessly). For example, the communication engine 228 may be configured to manage connections and communicate with one or more electronic devices. In some cases, the communication engine 228 may correspond to Figure 11 Communication interface 1140.

[0054] In some cases, XR system 200 may use, for example, a sequence of frames of a target object to generate a 3D reconstruction of the object. For example, model generation engine 230 may be configured to obtain images and generate a 3D reconstruction of the object based on the obtained images.

[0055] In some implementations, one or more image sensors 202, accelerometer 204, gyroscope 206, storage 207, computing component 210, XR engine 220, image processing engine 224, rendering engine 226, communication engine 228, and model generation engine 230 can be part of the same computing device. For example, in some cases, one or more image sensors 202, accelerometer 204, gyroscope 206, storage 207, computing component 210, XR engine 220, image processing engine 224, and rendering engine 226 can be integrated into an HMD, extended reality glasses, a smartphone, a laptop, a tablet, a gaming system, and / or any other computing device. However, in some implementations, one or more image sensors 202, accelerometer 204, gyroscope 206, storage 207, computing component 210, XR engine 220, image processing engine 224, rendering engine 226, communication engine 228, and model generation engine 230 can be part of two or more separate computing devices. For example, in some cases, some of the components 202 - 230 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0056] Storage 207 can be any storage device for storing data. Furthermore, storage 207 can store data from any of the components of XR system 200. For example, storage 207 can store data from image sensor 202 (e.g., image or video data), data from accelerometer 204 (e.g., measurements), data from gyroscope 206 (e.g., measurements), data from compute component 210 (e.g., processing parameters, preferences, virtual content, rendered content, scene maps, tracking and positioning data, object detection data, privacy data, XR application data, facial recognition data, occlusion data, etc.), data from XR engine 220, data from image processing engine 224, data from rendering engine 226 (e.g., output frames), data from the communication engine, and / or data from model generation engine 230 (e.g., 3D reconstructions). In some examples, storage 207 can include a buffer for storing frames for processing by compute component 210.

[0057] One or more computing components 210 may include a central processing unit (CPU) 212, a graphics processing unit (GPU) 214, a digital signal processor (DSP) 216, an image signal processor (ISP) 218, and / or other processors (e.g., a neural processing unit (NPU) implementing one or more trained neural networks). Computing component 210 may perform various operations such as image enhancement, computer vision, graphics rendering, extended reality operations (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image and / or video processing, sensor processing, recognition (e.g., text recognition, facial recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), trained machine learning operations, filtering, and / or any of the various operations described herein. In some examples, computing component 210 may implement (e.g., control, operate, etc.) an XR engine 220, an image processing engine 224, a rendering engine 226, and a model generation engine 230. In other examples, computing component 210 may also implement one or more other processing engines.

[0058] Image sensor 202 may include any image and / or video sensor or capture device. In some examples, image sensor 202 may be part of a multi-camera assembly (such as a dual-camera assembly). Image sensor 202 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by compute component 210, XR engine 220, image processing engine 224, rendering engine 226, and / or model generation engine 230, as described herein. In some examples, image sensor 202 may include image capture and processing system 100, image capture device 105A, image processing device 105B, or a combination thereof.

[0059] In some examples, image sensor 202 may capture image data and may generate an image (also referred to as a frame) based on the image data and / or may provide the image data or frame to XR engine 220, image processing engine 224, and / or rendering engine 226 for processing. An image or frame may include a video frame from a video sequence or a still image. An image or frame may include an array of pixels representing a scene. For example, the image may be a red-green-blue (RGB) image with red, green, and blue color components per pixel; a luminance, redness, blueness (YCbCr) image with a luminance component and two chrominance (color) components (redness and blueness) per pixel; or any other suitable type of color or monochrome image.

[0060] In some cases, image sensor 202 (and / or other cameras of XR system 200) may also be configured to capture depth information. For example, in some implementations, image sensor 202 (and / or other cameras) may include an RGB depth (RGB-D) camera. In some cases, XR system 200 may include one or more depth sensors (not shown) that are separate from image sensor 202 (and / or other cameras) and that can capture depth information. For example, such depth sensors may obtain depth information independently of image sensor 202. In some examples, the depth sensor may be physically mounted in the same general location as image sensor 202 but may operate at a different frequency or frame rate than image sensor 202. In some examples, the depth sensor may take the form of a light source that projects a structured or textured light pattern (which may include one or more narrowband lights) onto one or more objects in a scene. Depth information can then be obtained by exploiting the geometric deformation of the projected pattern caused by the surface shape of the objects. In one example, depth information can be obtained from a stereo sensor, such as a combination of an infrared structured light projector and an infrared camera registered to a camera (e.g., an RGB camera).

[0061] XR system 200 may also include other sensors within its sensor(s). The sensor(s) may include one or more accelerometers (e.g., accelerometer 204), one or more gyroscopes (e.g., gyroscope 206), and / or other sensors. The sensor(s) may provide velocity, orientation, and / or other positioning-related information to computing component 210. For example, accelerometer 204 may detect acceleration of XR system 200 and may generate acceleration measurements based on the detected acceleration. In some cases, accelerometer 204 may provide one or more translation vectors (e.g., up / down, left / right, forward / backward) that may be used to determine the position or pose of XR system 200. Gyroscope 206 may detect and measure the orientation and angular velocity of XR system 200. For example, gyroscope 206 may be used to measure the pitch, roll, and yaw of XR system 200. In some cases, gyroscope 206 may provide one or more rotation vectors (e.g., pitch, yaw, roll). In some examples, image sensor 202 and / or XR engine 220 may use measurements obtained by accelerometer 204 (e.g., one or more translation vectors) and / or measurements obtained by gyroscope 206 (e.g., one or more rotation vectors) to calculate the pose of XR system 200. As previously described, in other examples, XR system 200 may also include other sensors, such as an inertial measurement unit (IMU), a magnetometer, gaze and / or eye tracking sensors, machine vision sensors, smart scene sensors, voice recognition sensors, shock sensors, vibration sensors, positioning sensors, tilt sensors, and the like.

[0062] As mentioned above, in some cases, the one or more sensors may include at least one IMU. An IMU is an electronic device that uses a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers to measure specific forces, angular velocities, and / or orientations of XR system 200. In some examples, the one or more sensors may output measured information associated with the capture of images captured by image sensor 202 (and / or other cameras of XR system 200) and / or depth information obtained using one or more depth sensors of XR system 200.

[0063] The XR engine 220 can use the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs, and / or other sensors) to determine the pose (also referred to as head pose) of the XR system 200 and / or the pose of the image sensor 202 (or other camera of the XR system 200). In some cases, the pose of the XR system 200 and the pose of the image sensor 202 (or other camera) can be the same. The pose of the image sensor 202 refers to the position and orientation of the image sensor 202 relative to a reference frame (e.g., with respect to the scene 110). In some implementations, the camera pose can be determined for six degrees of freedom (6DoF), which refers to three translational components (e.g., which can be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame (such as the image plane)) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame). In some implementations, the camera pose can be determined for 3 degrees of freedom (3DoF), which refers to three angular components (eg, roll, pitch, and yaw).

[0064] In some cases, a device tracker (not shown) may use measurements from one or more sensors and image data from image sensor 202 to track the pose (e.g., a 6DoF pose) of XR system 200. For example, the device tracker may fuse visual data from the image data (e.g., using a visual tracking solution) with inertial data from the measurements to determine the position and motion of XR system 200 relative to the physical world (e.g., a scene) and a map of the physical world. As described below, in some examples, while tracking the pose of XR system 200, the device tracker may generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates to the 3D map of the scene. 3D map updates may include, for example, but not limited to, new or updated features and / or feature or landmark points associated with the scene and / or the 3D map of the scene, position updates that identify or update the position of XR system 200 within the scene and the 3D map of the scene, and the like. The 3D map may provide a digital representation of the scene in the real / physical world. In some examples, the 3D map can anchor location-based objects and / or content to real-world coordinates and / or objects. The XR system 200 can use the mapped scene (e.g., a scene in the physical world represented by a 3D map and / or associated with the map) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.

[0065] In some aspects, computing component 210 may use a visual tracking solution to determine and / or track the pose of image sensor 202 and / or the XR system 200 as a whole based on images captured by image sensor 202 (and / or other cameras of XR system 200). For example, in some examples, computing component 210 may perform tracking using computer vision-based tracking, model-based tracking, and / or simultaneous localization and mapping (SLAM) techniques. For example, computing component 210 may perform SLAM or may communicate (wired or wirelessly) with a SLAM engine (not shown). SLAM refers to a class of technologies that simultaneously track the pose of a camera (e.g., image sensor 202) and / or the XR system 200 relative to a map of an environment (e.g., a map of the environment modeled by XR system 200) while creating the map. This map may be referred to as a SLAM map and may be three-dimensional (3D). SLAM techniques can be performed using color or grayscale image data captured by image sensor 202 (and / or other cameras of XR system 200) and can be used to generate estimates of 6DoF pose measurements of image sensor 202 and / or XR system 200. Such SLAM techniques configured to perform 6DoF tracking may be referred to as 6DoF SLAM. In some cases, the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs, and / or other sensors) may be used to estimate, correct, and / or otherwise adjust the estimated pose.

[0066] In some cases, 6DoF SLAM (e.g., 6DoF tracking) can associate features observed from certain input images from image sensor 202 (and / or other cameras) to a SLAM map. For example, 6DoF SLAM can use feature point associations from an input image to determine the pose (position and orientation) of image sensor 202 and / or XR system 200 for that input image. 6DoF map building can also be performed to update the SLAM map. In some cases, a SLAM map maintained using 6DoF SLAM can include 3D feature points triangulated from two or more images. For example, keyframes can be selected from an input image or video stream to represent the observed scene. For each keyframe, a corresponding 6DoF camera pose associated with the image can be determined. The pose of image sensor 202 and / or XR system 200 can be determined by projecting features from the 3D SLAM map into the image or video frame and updating the camera pose based on verified 2D-3D correspondences.

[0067] In one illustrative example, the computing component 210 may extract feature points from certain input images (e.g., each input image, a subset of the input images, etc.) or from each keyframe. As used herein, a feature point (also referred to as a registration point) is a unique or identifiable portion of an image, such as a portion of a hand, the edge of a table, and other examples. Features extracted from a captured image may represent different feature points along three-dimensional space (e.g., coordinates on the X, Y, and Z axes), and each feature point may have an associated feature location. Feature points in a keyframe may match (be identical to or correspond to) or fail to match feature points from a previously captured input image or keyframe. Feature detection may be used to detect feature points. Feature detection may include image processing operations for examining one or more pixels of an image to determine whether a feature exists at a particular pixel. Feature detection may be used to process the entire captured image or portions of an image. For each image or keyframe, once a feature has been detected, a local image patch surrounding the feature may be extracted. Features may be extracted using any suitable technique, such as Scale Invariant Feature Transform (SIFT) (which localizes features and generates their descriptions), Learned Invariant Feature Transform (LIFT), Speeded Up Robust Features (SURF), Gradient Location Orientation Histogram (GLOH), Orientation Rapid Rotation Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), KAZE, Accelerated KAZE (AKAZE), Normalized Cross Correlation (NCC), descriptor matching, another suitable technique, or a combination thereof.

[0068] As an illustrative example, the computing component 210 may extract a message corresponding to a mobile device (e.g., Figure 2 XR system, Figure 9A and Figure 9B HMD 910, Figure 11 A and Figure 11 B) and the like. In some cases, feature points corresponding to the mobile device may be tracked to determine the pose of the mobile device. As described in more detail below, the pose of the mobile device may be used to determine the location of a projection of AR media content that may augment media content displayed on a display of the mobile device.

[0069] In some cases, the XR system 200 may also track the user's hands and / or fingers to allow the user to interact with and / or control virtual content in the virtual environment. For example, the XR system 200 may track the pose and / or movement of the user's hands and / or fingertips to identify or translate user interactions with the virtual environment. User interactions may include, for example, but are not limited to, moving virtual content items, resizing virtual content items, selecting input interface elements in a virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and / or other virtual interfaces), providing input through the virtual user interface, and the like.

[0070] Figure 3 is a diagram illustrating a simplified cross-sectional view of a lens assembly 300 (eg, of an HMD). Figure 3 In the illustrated example, lens assembly 300 includes a lens system 302 , a display 304 , an illumination source 306 , a light directing assembly 308 , and image sensors 310A / 310B (collectively, image sensors 310 ).

[0071] As illustrated, light from the display 304 can pass through the light directing assembly 308 and be focused by the lens system 302 onto the user's eye 315. In some implementations, the light directing assembly 308 can be configured to allow visible light from the display 304 to pass through. Figure 3 In the illustrated example, the light directing assembly 308 is located within the cavity 320 of the lens assembly 300 between the lens system 302 and the display 304. Figure 3 As illustrated, visible light 312 can be focused at the location of a user's eye 315, as illustrated by line 314. Illumination source 306 can be an IR illumination source (e.g., an IR LED) that illuminates the user's eye 315. In some cases, multiple illumination sources 306 can be used. When the IR light reaches the user's eye, scattered and / or reflected portions of the light (such as example light ray 316A) can reach light directing assembly 308 and reflect toward image sensor 310A.

[0072] In some cases, image sensor 310B can be positioned so that image sensor 310B faces toward a user's eye 315. Illumination source 306 (or multiple illumination sources) can be positioned to direct light (such as IR light) so that the light is scattered and / or reflected (such as shown in exemplary light rays 316B) and can reach image sensor 310B. In some cases, light directing assembly 308 can be omitted when image sensor 310B is configured to face toward a user's eye 315.

[0073] Image sensor 310 may be an infrared (IR) image sensor capable of detecting scattered and / or reflected light (e.g., glint) from the eye to form one or more images. In some cases, the XR system may obtain image data from image sensor 310 and track the user's eye positioning and / or gaze direction based on the obtained data. For example, image sensor 310 may track eye movement based on how IR light glint reflected or scattered by the eye moves.

[0074] While this article describes examples of eye tracking for use in XR systems, the eye tracking systems and techniques described herein can be used to perform eye tracking with other types of devices and using other geometries. Figure 3 The illustrations in the drawings are not to scale and are provided for illustration purposes only. In addition, more or fewer components may be included without departing from the scope of the present disclosure. Figure 3 In the lens assembly 300.

[0075] In some cases, the positioning of a user's eyes relative to lens assembly 300 can vary. For example, each individual user can have different eye size, facial shape, facial symmetry, eye spacing, facial feature alignment, and / or combinations thereof. In some implementations, the eye tracking system can be configured to perform eye tracking over a specified range of eye positions and / or rotations using, for example, image data collected from image sensor 310.

[0076] In some cases, eye tracking can be performed using multiple illumination sources 306, and these illumination sources can be configured to emit a constant amount of light. Generally speaking, how light reflects off the eye can vary based on the user's eyes and face. Figure 4This example illustrates how eye and facial shape can affect eye illumination for eye tracking. For example, if a first user 402 has relatively more sunken eyes, a first distance 404 between the first user's eyes 402 and a first IR LED plane 406 (e.g., the plane where the illumination source 306 resides) can be greater than a second distance 408 between a second user's eyes 410, where the second user's eyes 410 are relatively more protruding, and a second IR LED plane 412. In some cases, if the illumination source is configured to emit a constant amount of light, a darker, smaller glint can be generated for the first user 402, compared to a brighter, larger glint for the second user's eyes 410. Furthermore, the distances between the IR LED planes 406 and 412 and the first user's eyes 402 and the second user's eyes 410 can vary for the same user based on, for example, the positioning of a portable device (such as an HMD), and this varying distance can cause the glint to vary for the same user. Because the intensity of light (and the resulting glint) is inversely proportional to the square of the distance, a relatively small change in the distance between the eyes and the light source can make a large difference in the amount of light sufficient to generate useful glint. Additionally, IR LEDs or other components of the eye tracker (such as Figure 5 The IR light sources discussed above may age over time and may become dimmer for a given amount of input power over time, thereby reducing the amount of IR light that produces flicker. In some cases, smaller and / or dimmer flickers may lead to inaccuracies for eye tracking, while overly bright flickers may be inefficient and / or potentially harmful to the eyes in the case of prolonged exposure. Therefore, it may be useful to adjust the brightness, for example, by adjusting the amount of power supplied to the illumination source used for eye tracking.

[0077] Figure 5 FIG2 is a block diagram illustrating a system 500 that implements an adaptive algorithm for power-efficient eye tracking according to aspects of the present disclosure. The system 500 includes a processor 502, which can execute an LED control engine 504. The LED control engine 504 can be coupled to an IR LED driver 506 and control the IR LED driver. For example, the LED control engine 504 can instruct the IR LED driver 506 to increase or decrease (or turn off) the brightness of one or more IR LEDs 508 coupled to the IR LED driver 506. The IR LED driver 506 can control the operation of the one or more IR LEDs 508, for example, by controlling the amount of current flowing to the one or more IR LEDs 508.

[0078] One or more IR LEDs 508 can transmit IR light 510 that can be reflected / scattered 512 by an eye 514. This reflected / scattered 512 IR light from the eye 514 can be imaged by an eye-tracking camera 516. The eye-tracking camera 516 can be coupled to a glint analysis engine 518, which can analyze the images from the eye-tracking camera 516. In some cases, the glint analysis engine 518 can analyze the glint (e.g., the IR light 512 reflected / scattered by the eye 514) to determine information associated with the glint (e.g., target glint parameters 501). In some cases, the information associated with the glint can include the brightness of the pixel that glints, the area of ​​the glint, etc.

[0079] The flicker analysis engine 518 can communicate information associated with flicker to the LED control engine 504. The LED control engine 504 can also receive target flicker parameters 501. In some cases, the target flicker parameters 501 can include target values ​​for the information associated with flicker. The flicker parameters 501 can include parameter values ​​corresponding to the values ​​in the information associated with flicker. The LED control engine 504 can adjust the power / duty cycle of the IR LEDs so that the information associated with flicker aligns with (e.g., matches) the values ​​in the target flicker parameters 501. This allows the LED control engine 504 to balance flicker magnitude / brightness and power usage while remaining within guidelines for light exposure, such as IEC 62471. In some cases, the target flicker parameters 501 can be predetermined, for example, through experimentation during research and development. In some cases, the target flicker parameter values ​​can be fixed for a group of devices.

[0080] The LED control engine 504 can then compare the information associated with the flicker to the target flicker parameters 501 and adjust the brightness of one or more IR LEDs 508 based on the comparison. For example, if the flicker parameters of the information associated with the flicker are lower than the target flicker parameters 501, the LED control engine 504 can increase the brightness of one or more IR LEDs 508. Similarly, if the flicker parameters of the information associated with the flicker are higher than the target flicker parameters 501, the LED control engine 504 can decrease the brightness of one or more IR LEDs 508. In some cases, adjusting the brightness of the IR LEDs 508 can be performed by increasing or decreasing the amount of power supplied to one or more IR LEDs 508. In some cases, as LEDs age, they may become dimmer for a certain amount of power. Adjusting the brightness (e.g., the amount of power supplied to the LEDs) based on the target flicker parameters 501 can help compensate for the dimming of the LEDs as they age.

[0081] Figure 6Graph 600 illustrates the brightness distribution of pixels for glints in an image, according to aspects of the present disclosure. In graph 600, brightness (intensity) is plotted on a vertical axis 602, while pixels of a glint image (e.g., an image of glints) captured by an eye-tracking camera are plotted on a horizontal axis 604. In some cases, the brightness of the pixels has been Gaussianized to normalize the brightness curve for clarity. In some cases, the target glint parameter value may include a set of thresholds and / or a set of target values. As an example of using thresholds, a first curve 606 may illustrate the brightness levels across a set of pixels for glints at a particular IR LED power level for a first distance between the IR LED and the eye. In some cases, if the peak brightness 608 (e.g., maximum brightness) of first curve 606 or multiple pixels of first curve 606 have a brightness less than a first threshold 610, it may be difficult to perform eye tracking using the image. In some cases, this first distance may be relatively longer than the distances associated with second curve 612 or third curve 620.

[0082] A second curve 612 can illustrate the brightness level across a set of pixels of the flicker at the same power level for a second distance between the IR LED and the eye. The second distance can be shorter than the first distance. In some cases, if the peak 614 brightness of the second curve 612 or a plurality of pixels 616 of the second curve 612 have a brightness greater than a second threshold 618, the brightness may be too high, potentially leading to excessive power usage and possibly exceeding exposure guidelines over time, or potentially saturating the camera sensor. Such high brightness values ​​can be inefficient and potentially dangerous to the eye. Therefore, the target flicker parameter value can include a set of thresholds, such as an upper threshold and a lower threshold, that can be used to adjust the brightness of the illumination source (e.g., the IR LED).

[0083] As another example, third curve 620 may represent an ideal distance scenario where peak 622 brightness and / or the number of pixels 624 having at least a certain brightness can be easily tracked and are within eye-safe levels. In some cases, peak 622 brightness and / or the number of pixels 624 having at least a certain brightness can be set as a target value for a target flicker parameter. Third curve 620 may be associated with a certain brightness (e.g., based on the peak value in vertical axis 602) and a flicker area (e.g., based on the amount by which third curve 620 exceeds first threshold 610). In some cases, an eye tracking system may be associated with certain eye tracker flicker characteristics, such as minimum and maximum brightness (e.g., corresponding to first threshold 610 and second threshold 618, respectively) and a minimum area of ​​flicker (e.g., the number of pixels on horizontal axis 604 exceeding first threshold 610). In some cases, target flicker parameter values ​​(e.g., such as those in third curve 620) may be determined based on the eye tracker flicker characteristics.

[0084] In some cases, a brightness curve may be determined based on the brightness level required for a particular eye tracking system and the output of the illumination source (e.g., for the same IR LED power level as the other curves), such as Figure 6 For example, the output of the illumination source can be measured based on the intensity of the light. In some cases, the intensity I at a point on the eye can be a function of the distance r and the LED radiation characteristic. is a function of where φ is the angle of incidence and I f is the forward current of the LED. I can be expressed as The intensity may be determined at different expected distances to the eye tracker to generate a predicted intensity and a threshold (or target value) selected based on factors such as the sensitivity of the eye tracker, the eye-safe intensity value, the length of the expected exposure, the saturation light level, etc.

[0085] In some cases, brightness adjustment can be performed on a single light source or multiple light sources. In some cases, multiple light sources can be adjusted together. For example, Figure 7A The eye 702 is illustrated as being illuminated by eight light sources 704 spaced equidistant from one another. Where multiple light sources are used, they can be configured so that each light source produces a separate flash to increase tracking accuracy. In some cases, for example, if the light sources 704 are equidistant from the eye 702, the power to all light sources 704 can be controlled together.

[0086] For example, Figure 7B An eye 752 is illustrated as being illuminated by eight light sources 754 that are spaced differently around the eye 752. In this case, assuming that the power level of each of the eight light sources 754 is individually controlled, the power levels for the eight light sources 754 can be adjusted together, but the exact amount of power provided to each of the eight light sources 754 can vary based on the distance (shown as d1, d2, d3, d4, d5, d6, d7, and d8) of the respective light source from the eye 752. In this case, each light source 754 can be adjusted independently.

[0087] Figure 8 is a flow chart illustrating a process 800 for image processing according to aspects of the present disclosure. The process 800 may be performed by a computing device (or apparatus) or a component (eg, a chipset, a codec, etc.) of a computing device (eg, Figure 1 Image processor 150, Figure 2 The computing component 210, Figure 11The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device (e.g., Figure 9A 、 Figure 9B HMD 910, Figure 10A 、 Figure 10B The operations of process 800 may be implemented as a processor (e.g., a mobile phone 1050), a vehicle or a component or system of a vehicle, or other type of computing device. Figure 1 Image processor 150, Figure 2 CPU 212, GPU 214, DSP 216 and / or ISP 218, Figure 11 In some cases, the operations of process 800 may be performed by a processor having Figure 11 The architecture 1100 is implemented in a system.

[0088] At block 802, a computing device (or a component thereof) may obtain an image of an eye including images from a light source (e.g., illumination source 306, Figure 5 IR LED 508, Figure 7A Light source 704 or Figure 7B In some cases, the light glint comprises light from the light source 754 that is scattered or reflected from the eye.

[0089] At block 804, the computing device (or a component thereof) may determine information associated with the light glint based on the image of the eye. In some cases, the information associated with the light glint includes a set of parameters including an area of ​​the light glint and a maximum brightness value of the light glint.

[0090] At block 806, the computing device (or a component thereof) may obtain one or more target flicker parameter values.In some cases, the one or more target flicker parameter values ​​are predetermined based on eye tracker flicker characteristics associated with the eye tracker.

[0091] At block 808, the computing device (or a component thereof) may compare the information associated with the light flicker to one or more target flicker parameter values. In some cases, to compare the information associated with the light flicker to the one or more target flicker parameter values, the computing device (or a component thereof) may compare parameters of the information associated with the light flicker to a set of threshold values ​​for corresponding parameter values ​​of the one or more target flicker parameter values. In some cases, to compare the information associated with the light flicker to the one or more target flicker parameter values, the computing device (or a component thereof) may compare parameters of the information associated with the light flicker to corresponding target parameter values ​​of the one or more target flicker parameter values.

[0092] At block 810, the computing device (or a component thereof) may adjust the amount of power provided to the light source based on comparing the information associated with the light flicker to one or more target flicker parameter values. In some cases, to adjust the amount of power, the computing device (or a component thereof) may increase or decrease the amount of power provided to the light source. In some cases, adjusting the amount of power provided to the light source compensates for aging of the light source.

[0093] Figure 9A 9 is a perspective view 900 illustrating a head-mounted display (HMD) 910 that performs feature tracking and / or visual simultaneous localization and mapping (VSLAM) according to some examples. The HMD 910 can be, for example, an augmented reality (AR) head-mounted device, a virtual reality (VR) head-mounted device, a mixed reality (MR) head-mounted device, an extended reality (XR) head-mounted device, or some combination thereof. The HMD 910 can be an example of an XR system 200. The HMD 910 can include Figure 3 The HMD 910 includes a first camera 930A and a second camera 930B along the front of the HMD 910. The first camera 930A and the second camera 930B can be two of the one or more cameras. In some examples, the HMD 910 may have only a single camera. In some examples, the HMD 910 may include one or more additional cameras in addition to the first camera 930A and the second camera 930B. In some examples, the HMD 910 may include one or more additional sensors in addition to the first camera 930A and the second camera 930B.

[0094] Figure 9B is an example based on some examples Figure 9AA perspective view 930 of a head-mounted display (HMD) 910 being worn by a user 920. User 920 wears HMD 910 on their head, above their eyes. HMD 910 may capture images using a first camera 930A and a second camera 930B. In some examples, HMD 910 displays one or more display images based on the images captured by first camera 930A and second camera 930B, toward the eyes of user 920. The displayed images may provide a stereoscopic view of the environment, in some cases with superimposed information and / or other modifications. For example, HMD 910 may display a first display image based on the image captured by first camera 930A to the right eye of user 920. HMD 910 may display a second display image based on the image captured by second camera 930B to the left eye of user 920. For example, HMD 910 may provide overlay information in the displayed images overlaid on the images captured by first camera 930A and second camera 930B.

[0095] HMD 910 may not include wheels, propellers, or other forms of transport of its own. Instead, HMD 910 relies on the movement of user 920 to move HMD 910 around the environment. In some cases, such as when HMD 910 is a VR headset, the environment may be fully or partially virtual. If the environment is at least partially virtual, movement through the virtual environment may also be virtual. For example, movement through the virtual environment may be controlled by input device 208. Movement actuators may include any such input device 208. Movement through the virtual environment may not require wheels, propellers, legs, or any other form of transport. Even if the environment is virtual, SLAM technology may still be valuable because the virtual environment can be mapped and / or generated by devices other than HMD 910, such as a remote server or console associated with a video game or video game platform. In some cases, feature tracking and / or SLAM can even be performed in the virtual environment by a vehicle or other device that has its own physical transport system that allows it to physically move around in the physical environment. For example, SLAM can be performed in a virtual environment to test whether the SLAM system is working properly without wasting time or energy on movement and without wearing out the physical conveyor system.

[0096] Figure 10A1000 is a perspective view of a front surface 1055 of a mobile device 1050 illustrating the use of one or more front-facing cameras 1030A-1030B to perform features described herein, including, for example, feature tracking and / or visual simultaneous localization and mapping (VSLAM), according to some examples. Mobile device 1050 can be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a fitness tracking device, a wearable device, a wireless communication device, a laptop, a mobile device, any other type of computing device or computing system 1200 discussed herein, or a combination thereof. Front surface 1055 of mobile device 1050 includes display 1045. Front surface 1055 of mobile device 1050 includes first camera 1030A and second camera 1030B. First camera 1030A and second camera 1030B are illustrated in a bezel surrounding display 1045 on front surface 1055 of mobile device 1050. In some examples, first camera 1030A and second camera 1030B can be positioned in a notch or cutout cut out of display screen 1045 on front surface 1055 of mobile device 1050. In some examples, first camera 1030A and second camera 1030B can be under-display cameras positioned between display screen 1045 and the rest of mobile device 1050, such that light passes through a portion of display screen 1045 before reaching first camera 1030A and second camera 1030B. First camera 1030A and second camera 1030B in perspective view 1000 are front-facing cameras. First camera 1030A and second camera 1030B face in a direction perpendicular to the planar surface of front surface 1055 of mobile device 1050. First camera 1030A and second camera 1030B can be two of the one or more cameras. In some examples, front surface 1055 of mobile device 1050 can have only a single camera. In some examples, the mobile device 1050 can include one or more additional cameras in addition to the first camera 1030A and the second camera 1030B. In some examples, the mobile device 1050 can include one or more additional sensors in addition to the first camera 1030A and the second camera 1030B.

[0097] Figure 10B1090 is a perspective view 1030 illustrating a rear surface 1065 of a mobile device 1050. The mobile device 1050 includes a third camera 1030C and a fourth camera 1030D on the rear surface 1065 of the mobile device 1050. The third camera 1030C and the fourth camera 1030D of the perspective view 1090 are rear-facing. The third camera 1030C and the fourth camera 1030D face a direction perpendicular to the planar surface of the rear surface 1065 of the mobile device 1050. Although the rear surface 1065 of the mobile device 1050 does not have a display screen 1045 as illustrated in the perspective view 1090, in some examples, the rear surface 1065 of the mobile device 1050 can have a second display screen. If the back surface 1065 of the mobile device 1050 has a display screen 1045, any positioning of the third camera 1030C and the fourth camera 1030D relative to the display screen 1045 can be used, as discussed with respect to the first camera 1030A and the second camera 1030B on the front surface 1055 of the mobile device 1050. The third camera 1030C and the fourth camera 1030D can be two of the one or more cameras. In some examples, the back surface 1065 of the mobile device 1050 may have only a single camera. In some examples, the mobile device 1050 may include one or more additional cameras in addition to the first camera 1030A, the second camera 1030B, the third camera 1030C, and the fourth camera 1030D. In some examples, the mobile device 1050 may include one or more additional sensors in addition to the first camera 1030A, the second camera 1030B, the third camera 1030C, and the fourth camera 1030D.

[0098] Like HMD 910, mobile device 1050 does not include wheels, propellers, or other means of transport of its own. Instead, mobile device 1050 relies on the movements of a user holding or wearing mobile device 1050 to move it around the environment. In some cases, such as when mobile device 1050 is used for AR, VR, MR, or XR, the environment can be fully or partially virtual. In some cases, mobile device 1050 can be inserted into a head-mounted device (HMD) (e.g., into a cradle of the HMD) so that mobile device 1050 serves as the display of the HMD, with display 1045 of mobile device 1050 serving as the display of the HMD. If the environment is at least partially virtual, movement through the virtual environment can also be virtual. For example, movement through the virtual environment can be controlled by one or more joysticks, buttons, video game controllers, mice, keyboards, trackpads, and / or other input devices coupled to mobile device 1050, either wired or wirelessly.

[0099] Figure 111105 is a diagram illustrating an example of a computing system 1100 for implementing certain aspects of the present technology. Computing system 1100 can be or can be part of any computing device, such as, for example, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device (e.g., a network-connected watch or other wearable device), a personal computer, a laptop computer, a server computer, a television, a video game console, or other device, or any component thereof, where the components of the system communicate with each other using connection 1105. Connection 1105 can be a physical connection using a bus, or a direct connection to processor 1110, such as in a chipset architecture. Connection 1105 can also be a virtual connection, a networked connection, or a logical connection.

[0100] In some examples, computing system 1100 can be a distributed system, in which the functionality described in this disclosure can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a plurality of such components, each of which performs some or all of the functionality for which the component is described. In some cases, these components can be physical or virtual devices.

[0101] The example computing system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 that couple various system components including system memory 1115 such as read-only memory (ROM) 1120 and random access memory (RAM) 1125 to the processor 1110. The computing system 1100 may include a cache 1112 of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1110.

[0102] Processor 1110 may include any general-purpose processor and hardware or software services, such as services 1132, 1134, and 1136 stored in storage device 1130, configured to control processor 1110 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1110 may be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0103] To enable user interaction, the computing system 1100 includes an input device 1145 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, a camera, an accelerometer, a gyroscope, and the like. The computing system 1100 may also include an output device 1135 that can be one or more of a plurality of output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140 that generally governs and manages user input and system output. The communication interface may perform or facilitate the use of wired and / or wireless transceivers to receive and / or send wired or wireless communications, including utilizing an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple ® Lightning ® Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ® Low energy (BLE) wireless signal transmission, IBEACON ® The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1100 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and thus the base features herein may be readily substituted for improved hardware or firmware arrangements as they are developed.

[0104] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick ® card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0105] Storage devices 1130 may include software services, servers, services, etc. that, when code defining such software is executed by processor 1110, cause the system to perform a function. In some examples, hardware services that perform a particular function may include software components stored in a computer-readable medium connected to the necessary hardware components (such as processor 1110, connections 1105, output devices 1135, etc.) to perform the function.

[0106] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media that can store data and does not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or storage devices. A computer-readable medium may have stored thereon code and / or machine-executable instructions, which may represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means, including memory sharing, message passing, token passing, network transmission, and the like.

[0107] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0108] Specific details are provided in the above description to provide a thorough understanding of the examples provided herein. However, it will be understood by those skilled in the art that examples can be implemented without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks that include devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form so as not to obscure the examples in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details in order to avoid obscuring the examples.

[0109] The various examples above may be described as processes or methods, which may be depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. Furthermore, the order of the operations may be rearranged. A process is terminated when its operations are completed, but a process may have additional steps not included in the accompanying figures. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0110] The processes and methods according to the examples described above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0111] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards in different chips or different processes executed on a single device.

[0112] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0113] In the foregoing description, various aspects of the present application have been described with reference to the specific examples of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or in combination. Further, without departing from the broader essence and scope of this specification, each example can be utilized in any number of environments and applications other than the environment and application described herein. Therefore, the description and the accompanying drawings should be considered to be illustrative and not restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order.

[0114] It should be understood by those skilled in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to (" ") and greater than or equal to (" ) symbol instead.

[0115] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0116] The phrase “coupled to” refers to any component being directly or indirectly physically connected to another component, and / or any component being in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0117] Claim language or other language that recites "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that recites "at least one of A and B" means A, B, or A and B. In another example, claim language that recites "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language that recites "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0118] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Technicians may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of this application.

[0119] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium containing program code, including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0120] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0121] Illustrative aspects of the present disclosure include:

[0122] Aspect 1. A method for eye tracking, the method comprising: obtaining an image of an eye, the image of the eye including light flickers from a light source; determining information associated with the light flickers based on the image of the eye; obtaining one or more target flicker parameter values; comparing the information associated with the light flickers with the one or more target flicker parameter values; and adjusting an amount of power for the light source based on comparing the information associated with the light flickers with the one or more target flicker parameter values.

[0123] Aspect 2. The method according to aspect 1, wherein the information associated with the light flash comprises a set of parameters, the set of parameters comprising an area of ​​the light flash and a maximum brightness value of the light flash.

[0124] Aspect 3. A method according to any one of Aspects 1 to 2, wherein comparing the information associated with the light flicker with the one or more target flicker parameter values ​​includes comparing parameters of the information associated with the light flicker with a set of threshold values ​​of corresponding parameter values ​​of the one or more target flicker parameter values.

[0125] Aspect 4. A method according to any one of Aspects 1 to 3, wherein comparing the information associated with the light flash with the one or more target flash parameter values ​​includes comparing parameters of the information associated with the light flash with corresponding target parameter values ​​of the one or more target flash parameter values.

[0126] Aspect 5. The method according to any one of aspects 1 to 4, wherein adjusting the amount of power comprises increasing or decreasing the amount of power provided to the light source.

[0127] Aspect 6. The method according to any one of aspects 1 to 5, wherein the light source comprises one or more infrared light sources.

[0128] Aspect 7. A method according to any one of aspects 1 to 6, wherein the light glint comprises light from the light source that is scattered or reflected from the eye.

[0129] Aspect 8. The method according to any one of aspects 1 to 7, wherein the one or more target flicker parameter values ​​are predetermined based on eye tracker flicker characteristics associated with the eye tracker.

[0130] Aspect 9. The method of any one of aspects 1 to 8, wherein adjusting the amount of power directed to the light source compensates for aging of the light source.

[0131] Aspect 10. A device for eye tracking, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: obtain an image of an eye, the image of the eye comprising light flickers from a light source; determine information associated with the light flickers based on the image of the eye; obtain one or more target flicker parameter values; compare the information associated with the light flickers with the one or more target flicker parameter values; and adjust an amount of power to the light source based on comparing the information associated with the light flickers with the one or more target flicker parameter values.

[0132] Aspect 11. The apparatus according to aspect 10, wherein the information associated with the light flash comprises a set of parameters including an area of ​​the light flash and a maximum brightness value of the light flash.

[0133] Aspect 12. An apparatus according to any one of Aspects 10 to 11, wherein in order to compare the information associated with the light flash with the one or more target flash parameter values, the at least one processor is configured to compare parameters of the information associated with the light flash with a set of threshold values ​​of corresponding parameter values ​​of the one or more target flash parameter values.

[0134] Aspect 13. An apparatus according to any one of Aspects 10 to 12, wherein, in order to compare the information associated with the light flash with the one or more target flash parameter values, the at least one processor is configured to compare parameters of the information associated with the light flash with corresponding target parameter values ​​of the one or more target flash parameter values.

[0135] Aspect 14. The apparatus according to any one of aspects 10 to 13, wherein to adjust the amount of power, the at least one processor is configured to increase or decrease the amount of power provided to the light source.

[0136] Aspect 15. The device according to any one of aspects 10 to 14, wherein the light source comprises one or more infrared light sources.

[0137] Aspect 16. The device of any one of Aspects 10 to 15, wherein the light glint comprises light from the light source that is scattered or reflected from the eye.

[0138] Aspect 17. An apparatus according to any one of aspects 10 to 16, wherein the one or more target flicker parameter values ​​are predetermined based on eye tracker flicker characteristics associated with an eye tracker.

[0139] Aspect 18. The apparatus of any one of aspects 10 to 17, wherein adjusting the amount of power directed to the light source compensates for aging of the light source.

[0140] Aspect 19. A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one processor, cause the at least one processor to: obtain an image of an eye, the image of the eye including light flicker from a light source; determine information associated with the light flicker based on the image of the eye; obtain one or more target flicker parameter values; compare the information associated with the light flicker with the one or more target flicker parameter values; and adjust the amount of power for the light source based on comparing the information associated with the light flicker with the one or more target flicker parameter values.

[0141] Aspect 20. The non-transitory computer-readable medium of aspect 19, wherein the information associated with the light flash comprises a set of parameters including an area of ​​the light flash and a maximum brightness value of the light flash.

[0142] Aspect 21. A non-transitory computer-readable medium according to any one of Aspects 19 to 20, wherein, in order to compare the information associated with the light flash with the one or more target flash parameter values, the instructions cause the at least one processor to compare parameters of the information associated with the light flash with a set of threshold values ​​of corresponding parameter values ​​of the one or more target flash parameter values.

[0143] Aspect 22. A non-transitory computer-readable medium according to any one of Aspects 19 to 21, wherein in order to compare the information associated with the light flash with the one or more target flash parameter values, the instructions cause the at least one processor to compare parameters of the information associated with the light flash with corresponding target parameter values ​​of the one or more target flash parameter values.

[0144] Aspect 23. The non-transitory computer-readable medium of any one of aspects 19 to 22, wherein to adjust the amount of power, the instructions cause the at least one processor to increase or decrease the amount of power provided to the light source.

[0145] Aspect 24. The non-transitory computer-readable medium of any one of aspects 19 to 23, wherein the light source comprises one or more infrared light sources.

[0146] Aspect 25. The non-transitory computer-readable medium of any one of Aspects 19 to 24, wherein the light glint comprises light from the light source that is scattered or reflected from the eye.

[0147] Aspect 26. The non-transitory computer-readable medium of any one of aspects 19 to 25, wherein the one or more target flicker parameter values ​​are predetermined based on eye tracker flicker characteristics associated with an eye tracker.

[0148] Aspect 27. The non-transitory computer-readable medium of any one of aspects 19 to 26, wherein adjusting the amount of power directed to the light source compensates for aging of the light source.

[0149] Aspect 28. An apparatus for eye tracking, the apparatus comprising: a component for obtaining an image of an eye, the image of the eye including light flickers from a light source; a component for determining information associated with the light flickers based on the image of the eye; a component for obtaining one or more target flicker parameter values; a component for comparing the information associated with the light flickers with the one or more target flicker parameter values; and a component for adjusting an amount of power to the light source based on comparing the information associated with the light flickers with the one or more target flicker parameter values.

[0150] Aspect 29. The apparatus according to aspect 28, wherein the information associated with the light flash comprises a set of parameters including an area of ​​the light flash and a maximum brightness value of the light flash.

[0151] Aspect 30. An apparatus according to any one of Aspects 28 to 29, wherein comparing the information associated with the light flicker with the one or more target flicker parameter values ​​includes comparing parameters of the information associated with the light flicker with a set of threshold values ​​of corresponding parameter values ​​of the one or more target flicker parameter values.

[0152] Aspect 34: An apparatus for image generation, the apparatus comprising means for performing one or more of the operations according to any one of aspects 1 to 9.

Claims

1. A method for eye tracking, the method comprising: obtaining an image of an eye, said image of the eye including a glint of light from a light source; determining information associated with the light glint based on the image of the eye; obtaining one or more target flicker parameter values; comparing the information associated with the light flicker to the one or more target flicker parameter values; as well as An amount of power to the light source is adjusted based on comparing the information associated with the light flicker to the one or more target flicker parameter values. 2 . The method of claim 1 , wherein the information associated with the light flash comprises a set of parameters including an area of ​​the light flash and a maximum brightness value of the light flash.

3. The method of claim 1 , wherein comparing the information associated with the light flicker with the one or more target flicker parameter values ​​comprises comparing parameters of the information associated with the light flicker with a set of threshold values ​​of corresponding parameter values ​​of the one or more target flicker parameter values.

4. The method of claim 1 , wherein comparing the information associated with the light flicker with the one or more target flicker parameter values ​​comprises comparing a parameter of the information associated with the light flicker with a corresponding target parameter value of the one or more target flicker parameter values. The method of claim 1 , wherein adjusting the amount of power comprises increasing or decreasing the amount of power provided to the light source. The method of claim 1 , wherein the light source comprises one or more infrared light sources.

7. The method of claim 1, wherein the light glint comprises light from the light source that is scattered or reflected from the eye.

8. The method of claim 1, wherein the one or more target flicker parameter values ​​are predetermined based on eye tracker flicker characteristics associated with an eye tracker.

9. The method of claim 1, wherein adjusting the amount of power to the light source compensates for aging of the light source.

10. An apparatus for eye tracking, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtain an image of an eye, the image of the eye including a glint of light from a light source; determining information associated with the light glint based on the image of the eye; obtaining one or more target flicker parameter values; comparing the information associated with the light flicker to the one or more target flicker parameter values; as well as An amount of power to the light source is adjusted based on comparing the information associated with the light flicker to the one or more target flicker parameter values. 11 . The apparatus of claim 10 , wherein the information associated with the light flash comprises a set of parameters including an area of ​​the light flash and a maximum brightness value of the light flash.

12. An apparatus according to claim 10, wherein, in order to compare the information associated with the light flicker with the one or more target flicker parameter values, the at least one processor is configured to compare parameters of the information associated with the light flicker with a set of threshold values ​​of corresponding parameter values ​​of the one or more target flicker parameter values.

13. An apparatus according to claim 10, wherein, in order to compare the information associated with the light flash with the one or more target flash parameter values, the at least one processor is configured to compare parameters of the information associated with the light flash with corresponding target parameter values ​​of the one or more target flash parameter values.

14. The apparatus of claim 10, wherein to adjust the amount of power, the at least one processor is configured to increase or decrease the amount of power provided to the light source.

15. The apparatus of claim 10, wherein the light source comprises one or more infrared light sources.

16. The device of claim 10, wherein the light glint comprises light from the light source that is scattered or reflected from the eye.

17. The apparatus of claim 10, wherein the one or more target flicker parameter values ​​are predetermined based on eye tracker flicker characteristics associated with an eye tracker.

18. The apparatus of claim 10, wherein adjusting the amount of power to the light source compensates for aging of the light source.

19. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: obtaining an image of an eye, said image of the eye including a glint of light from a light source; determining information associated with the light glint based on the image of the eye; obtaining one or more target flicker parameter values; comparing the information associated with the light flicker to the one or more target flicker parameter values; as well as An amount of power to the light source is adjusted based on comparing the information associated with the light flicker to the one or more target flicker parameter values.

20. The non-transitory computer-readable medium of claim 19, wherein the information associated with the light flash comprises a set of parameters including an area of ​​the light flash and a maximum brightness value of the light flash.

21. The non-transitory computer-readable medium of claim 19, wherein, in order to compare the information associated with the light flash with the one or more target flash parameter values, the instructions cause the at least one processor to compare parameters of the information associated with the light flash with a set of threshold values ​​of corresponding parameter values ​​of the one or more target flash parameter values.

22. The non-transitory computer-readable medium of claim 19, wherein, in order to compare the information associated with the light flash with the one or more target flash parameter values, the instructions cause the at least one processor to compare parameters of the information associated with the light flash with corresponding target parameter values ​​of the one or more target flash parameter values.

23. The non-transitory computer readable medium of claim 19, wherein to adjust the amount of power, the instructions cause the at least one processor to increase or decrease the amount of power provided to the light source.

24. The non-transitory computer readable medium of claim 19, wherein the light source comprises one or more infrared light sources.

25. The non-transitory computer readable medium of claim 19, wherein the light glint comprises light from the light source that is scattered or reflected from the eye.

26. The non-transitory computer readable medium of claim 19, wherein the one or more target flicker parameter values ​​are predetermined based on eye tracker flicker characteristics associated with an eye tracker.

27. The non-transitory computer readable medium of claim 19, wherein adjusting the amount of power to the light source compensates for aging of the light source.

28. An apparatus for eye tracking, the apparatus comprising: means for obtaining an image of an eye, said image of the eye comprising glints of light from a light source; means for determining information associated with the light glint based on the image of the eye; means for obtaining one or more target flicker parameter values; means for comparing said information associated with said light flicker with said one or more target flicker parameter values; and Means for adjusting an amount of power to the light source based on comparing the information associated with the light flicker to the one or more target flicker parameter values.

29. The apparatus of claim 28, wherein the information associated with the light flash comprises a set of parameters including an area of ​​the light flash and a maximum brightness value of the light flash.

30. The apparatus of claim 28, wherein comparing the information associated with the light flicker with the one or more target flicker parameter values ​​comprises comparing parameters of the information associated with the light flicker with a set of threshold values ​​of corresponding parameter values ​​of the one or more target flicker parameter values.

Images