Eye tracking data filtering

Abstract

The present disclosure relates to eye tracking data filtering. Eye tracking is performed by determining an initial pupil position of a user relative to a lens positioned in front of the user, detecting a change in the pupil position relative to the lens to an updated pupil position relative to the lens, and determining that the updated pupil position is outside a bounding box associated with the lens. The updated pupil position is a replacement pupil position having a replacement pupil position within the bounding box associated with the lens, and the updated pupil position is used for eye tracking functionality. Eye tracking is also performed by determining that a first pixel associated with a gaze direction is outside a visibility region, identifying a replacement pixel within the visibility region, determining an updated gaze angle based on the replacement pixel, and performing eye tracking using the updated gaze angle.

Description

Background Technology

[0001] This disclosure relates generally to image processing. More specifically, but not as a limitation, this disclosure relates to techniques and systems for generating and managing eye-tracking data for improved eye-tracking technologies.

[0002] Eye tracking is a technology used in many fields such as gaze detection, pose estimation, facial analysis, and recognition. Eye tracking often forms the basis of these operations and can be considered as the process of electronically locating a person's gaze point or tracking and recording the movement of that gaze point. In practice, eye tracking is provided by locating and tracking the pupil position and gaze direction. However, the sensors used to track the eyes often shift or jitter, meaning that eye tracking data is not always perfectly calibrated. Therefore, eye tracking data may be jittery or invalid. An improved technique for managing eye tracking data is needed. Attached Figure Description

[0003] Figure 1A and Figure 1B An example diagram of the setup for eye tracking and pupil localization is shown.

[0004] Figure 2 A flowchart is shown of a technique for managing pupil position information for eye-tracking technology, according to one or more embodiments.

[0005] Figure 3 A flowchart is shown of a technique for optimizing pupil position for eye tracking according to one or more embodiments.

[0006] Figure 4A and Figure 4B An example diagram showing the gaze direction relative to the visible area according to one or more implementation schemes is shown.

[0007] Figure 5A and Figure 5B A flowchart is shown of a technique for optimizing the gaze angle for eye-tracking technology according to one or more embodiments.

[0008] Figure 6 A flowchart is shown of a technique for identifying replacement pixels to optimize the gaze angle, according to one or more embodiments.

[0009] Figure 7 A multifunctional electronic device according to one or more embodiments is illustrated in block diagram form.

[0010] Figure 8 A computer system according to one or more implementation schemes is shown in block diagram form. Detailed Implementation

[0011] This disclosure relates to systems, methods, and computer-readable media for optimizing and smoothing eye-tracking data to enhance the performance of eye-tracking technologies.

[0012] In one or more embodiments, the pupil position is held within a predetermined region. A lens passing through which it monitors the pupil may have a predetermined region known as a calibration region, while other regions of the lens may be less calibrated. Thus, the pupil position should be maintained within the calibration region. In one or more embodiments, an initial pupil position relative to a lens located in front of the eye is determined. A change from the pupil position relative to the lens to an updated pupil position relative to the lens can be detected. If the updated pupil position is outside the determined bounding box (e.g., the bounding box associated with the calibration region), an alternative pupil position within that bounding box is selected. The updated pupil position is then used for eye-tracking techniques. In some embodiments, to avoid jitter, the eye-tracking system may change the pupil position over a series of frames, such that the pupil position transitions from the initial pupil position to the alternative pupil position, for example, based on a time-varying easing function.

[0013] In one or more embodiments, the gaze direction can be optimized for eye-tracking technology. Specifically, the gaze direction is optimized so that the user is gazing at a visible area. A first pixel can be determined and associated with a gaze direction outside the visible area. Replacement pixels within the visible area are identified. Replacement pixels can be selected along a vector from the center of the field of view to the first pixel. A gaze angle is determined based on the position of the replacement pixels, and this gaze angle is used to perform eye-tracking functionality.

[0014] For the purposes of this disclosure, the term "gaze origin" refers to the center of the eye that determines the gaze.

[0015] For the purposes of this disclosure, the term "pupil position" refers to the location of the pupil on the surface of the eye.

[0016] For the purposes of this disclosure, the term "gaze direction" is the gaze direction derived from the gaze origin and passing through the pupil position.

[0017] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, as part of this specification, some of the drawings of this disclosure are provided in the form of flowcharts. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is only for illustrative purposes of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, thereby resorting to the necessary claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with that implementation is included in at least one implementation of the disclosed subject matter, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily referring to all of the same implementation.

[0018] It should be understood that in the development of any practical implementation (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement graphical modeling systems benefiting from this disclosure.

[0019] refer to Figure 1A An example diagram of an eye 100 is presented. For the purposes of this disclosure, the eye 100 includes a gaze origin 105, which refers to the center of the eye in 3D space. The eye 100 also includes a pupil position 110. The pupil position 110 refers to the location of the pupil on the surface of the eye 100. In one or more embodiments, the pupil position 110 may refer to the center position of the pupil on the surface of the eye. The gaze origin 105 and the pupil position 110 can be used to determine a gaze direction 115. The gaze direction 115 may refer to a gaze direction originating from the gaze origin 105 and passing through the pupil position 110. In one or more embodiments, the gaze direction 115 may be defined as a vector generated from the gaze origin 105 and passing through the pupil position 110.

[0020] Go to Figure 1BThis figure is an example diagram of an eye 100 undergoing eye tracking. Eye tracking can be performed by a device 125 including a lens 120. When the eye 100 views an object through the lens 120, only some of the pupil data in the pupil data is valid based on the system's calibration. The pupil position 110 can affect the system's calibration. For example, hardware limitations may cause only a portion of the lens 120 to be calibrated, although the pupil position 110 may still reach an uncalibrated portion. Therefore, the pupil boundary 130 can define the portion of the pupil data in the lens 120 that is determined to be valid. In some embodiments, the intersection of the pupil boundary 130 and the lens 120 can identify a bounding box on the lens.

[0021] Figure 2 A flowchart is shown illustrating a technique for managing pupil position information for eye-tracking technology, according to one or more embodiments. Specifically, Figure 2 A technique for clamping the pupil position within a lens's bounding box is described. By determining that clamping the pupil position within the lens is part of providing valid pupil data, the system avoids having to extrapolate data from an uncalibrated portion of the lens, which would cause the image to appear more distorted than it would within the bounding box. For example, if the display device moves or shifts in front of the eye, the pupil position may frequently shift, causing flickering on the screen. Although Figure 2 The various processes shown are presented in a specific order, but it should be understood that the various processes may be executed in different orders. Furthermore, it is not necessary to execute all of the various processes.

[0022] Flowchart 200 begins at 205, where an initial pupil position of the user relative to a lens located in front of the user is determined. For example, the lens may be located in front of the user's eye where the initial pupil position is to be determined. Flowchart 200 continues at 210, where a change in pupil position relative to the lens is detected. As a result, an updated pupil position relative to the lens is determined.

[0023] At box 215, the updated pupil position is compared with the bounding box associated with the lens. As mentioned above, the bounding box refers to the portion of the lens that is determined to provide valid pupil data. The bounding box can be determined, for example, by the intersection of the valid pupil boundary and the lens, as referenced above. Figure 1B As stated above.

[0024] The flowchart continues at 220, where it is determined whether the updated pupil position is outside the bounding box. If it is determined at 220 that the updated pupil position is not outside the bounding box, the flowchart proceeds to 225, and the updated pupil position is used as is for the eye-tracking function.

[0025] Returning to 220, if the pupil position is determined to be outside the bounding box, the flowchart continues to 230. At 230, a replacement pupil position within the bounding box is identified. The replacement pupil position can be determined in several ways. For example, the nearest pupil position within the bounding box can be used. For example, a position on the boundary of the bounding box can be used. The flowchart then ends at 235, where the replacement pupil position is used for eye-tracking functionality. Specifically, in some implementations, the replacement pupil position is used to calculate a warping function for rendering an image on the screen, as referenced above. Figure 1B As stated above.

[0026] In some implementations, using a different pupil position may cause data to jump as the eye moves. For example, data jumping can lead to distortion jumps. In some implementations, distortion jumps can be avoided by applying an easing function to the pupil position. Figure 3 A flowchart of a technique for optimizing pupil position for eye tracking, according to one or more embodiments, is shown, such as above. Figure 2 As described in box 230. Although Figure 3 The various processes shown are presented in a specific order, but it should be understood that the various processes may be executed in different orders. Furthermore, it is not necessary to execute all of the various processes.

[0027] Flowchart 300 begins at 305, where a replacement pupil location is identified within a bounding box. According to one or more embodiments, the replacement pupil location can be selected based on the location within the bounding box closest to the actual pupil location. In some embodiments, the replacement pupil location can be selected as a location on the boundary of the bounding box.

[0028] The flowchart continues at box 310, where an easing function is applied to the pupil position based on the actual pupil position and the alternative pupil position. In some implementations, the easing function may be a time-based function. Therefore, at box 315, a set of intermediate pupil positions can be determined based on the easing function, rather than simply replacing the actual pupil position with the alternative pupil position. According to one or more implementations, the easing function may be a linear function, a cubic easing function, etc. In this way, the set of intermediate pupil positions can easing the user's pupil position to the alternative pupil position.

[0029] Flowchart 300 continues at block 320, where the set of intermediate pupil positions is applied to a series of frames. For example, the set of intermediate pupil positions may include a predetermined number of intermediate pupil positions, which may correspond to the number of frames to which the intermediate pupil positions are applied. Alternatively, in some embodiments, the number of intermediate pupil positions may be determined by the difference between the actual pupil position and the replacement pupil position. For example, if the linear difference between the actual pupil position and the replacement pupil position is small, fewer frames may be required, and therefore fewer intermediate values ​​are needed to reach the replacement pixel position. Conversely, if the linear difference between the actual pupil position and the replacement pupil position is large, more frames may be required, and therefore more intermediate values ​​are needed to reach the replacement pupil position in order to effectively ease the user's movement. Applying this set of intermediate pupil positions allows a subsequent series of frames presented to the user to utilize a series of intermediate pupil positions for that series of frames.

[0030] Flowchart 300 ends at box 325, where the replacement pupil position is used for the next frame. That is, after using the set of intermediate pupil positions for a series of frames to be rendered based on the replacement pupil position rather than the actual pupil position. In some implementations, the replacement pupil position may then be used to render content to be presented to the user until a new pupil position is detected, at which point the process restarts. Figure 2 The process described.

[0031] According to some implementations, other problems may arise, such as those based on the user's gaze. For example, the display may have a region associated with an area that includes valid pixel information. Therefore, the gaze direction can be optimized to ensure that the user is looking at pixels within the valid visibility area. (Reference) Figure 4A This diagram presents a simplified block diagram of a screen, such as screen 425. Screen 425 can be any type of display located in front of a user, such as a display in a mobile device (such as a tablet or mobile phone) or a head-mounted display device. A portion of the display may be visible and is therefore referred to as the visible area. In some embodiments, the visible area is the portion of the screen that can be seen through lens 430. Therefore, the lens cannot “see” every part of the display. Furthermore, not all of screen 425 visible through lens 430 may include valid data. For example, the outer portion of lens 430 may be difficult to calibrate or may not be able to be calibrated otherwise. Thus, screen 425 may include a visible area 435 determined to contain valid pixel data. Additionally or alternatively, the determination of the visible area may be based on hardware, the technical specifications of the eye-tracking system, or software defining what the visible area is.

[0032] Go to Figure 4BThis figure depicts an example view of the screen from the eye. According to some embodiments, the visible area 435 may be defined by a visibility mask based on a portion of the screen determined to provide valid pixel data. Thus, if the user's gaze is directed outside the visible area 435, an alternative gaze position can be utilized. As shown, the user's gaze 460 is directed outside the visible area 435 but within the screen 440. The screen 440 may be the screen of a mobile device such as a tablet, mobile phone, wearable device, etc., located in front of the user. To provide valid data, an alternative gaze position can be selected from within the visible area. An alternative pixel may be selected, for example, based on the nearest pixel within the visible area 435. In some embodiments, an alternative pixel may be selected by identifying the center of the field of view 450. The center of the field of view 450 may be based on the position of the eye relative to the screen. The center of the field of view 450 may be based on the position of the user's eye center relative to the screen. In some embodiments, an alternative pixel 455 may be selected for the gaze target 460 such that the alternative pixel 455 is within the visible area. A replacement pixel can be determined using a vector 470 originating from the center of the field of view 450 and pointing towards the target location of the gaze 460. According to one or more embodiments, the replacement pixel 455 can then be used in eye-tracking technology. Furthermore, in some embodiments, the eye-tracking technology may also rely on a gaze angle that can be determined based on the replacement pixel 455.

[0033] Figure 5A A flowchart is shown illustrating a technique for managing gaze information for eye-tracking technology, according to one or more embodiments. Specifically, Figure 5A A technique for recalculating the user's gaze direction to utilize effective pixel data is described. For clarity, reference is made to Figures 1 and 4 as described above. Figure 3 The description of the processes in Figure 5 is as follows. Although the various processes shown in Figure 5 are presented in a specific order, it should be understood that the processes can be executed in different orders. Furthermore, it is not necessary to execute all of the processes.

[0034] Flowchart 500 begins at box 505, where a detected pixel position is associated with the gaze direction. For example, the pixel position may be associated with the target location the user is gazing at. The pixel position can be determined relative to the screen located in front of the user's eyes 100. At box 510, it is determined whether the detected pixel position is outside the visible area. (Refer to the above...) Figure 4B The visible area 435 may be a subset of the screen 425 that includes effective pixel data. The effective pixel data may be based on a subset of the lens 430 through which the screen 425 can be seen.

[0035] If it is determined at box 510 that the pixel position is not outside the visibility area, flowchart 500 continues to box 515. At box 515, a gaze angle value is determined based on the detected pixel position. The gaze angle describes the angle between the user's eye and the screen based on the determined pixel position. The flowchart ends at box 520, where the determined gaze angle and pixel position are used to process the next frame.

[0036] Returning to box 510 of flowchart 500, if it is determined that the detected pixel location is outside the visibility area, flowchart 500 continues at box 525. At box 525, a replacement pixel location within the visibility area is determined. In one or more embodiments, the pixel closest to the detected pixel within the visibility area may be selected as the replacement pixel. Furthermore, in some embodiments, the replacement pixel location may be selected based on the center point of the field of view, as will be referred to below. Figure 6 As detailed above.

[0037] Flowchart 500 continues at box 530, where the gaze angle is determined based on the position of the replacement pixel. The gaze angle describes the angle between the user's eye and the screen, based on the determined replacement pixel position. The flowchart ends at box 535, where the determined gaze angle and replacement pixel position are used to process the next frame.

[0038] According to some implementations, the client and compositor can use eye-tracking data to render frames in a two-stage process. In some implementations, eye-tracking data can be passed between the client and compositor, and / or historical eye-tracking data can be used to improve visual quality and / or meet hardware limitations. Specifically, consistency between the client and compositor can be improved by sharing at least some eye-tracking parameters. Figure 5B A flowchart illustrating a technique for managing eye-tracking data according to one or more embodiments is shown. Specifically, Figure 5B A technique for sharing eye-tracking data between the client and the synthesizer is described. For clarity, reference is made to Figures 1 and 4 as described above. Figure 3 The description content. Although Figure 5BThe various processes shown are illustrated in a specific order, but it should be understood that these processes can be executed in different orders. Furthermore, not all processes must be executed. The flowchart begins at box 555, where the client processes a frame and obtains eye-tracking parameters. In some embodiments, the client renders the frame for the compositor to render. According to some embodiments, when rendering a frame, the client determines certain eye-tracking parameters, such as the gaze angle. As shown at box 560, in an optional step, the client processes the frame based on historical compositor eye-tracking parameters. That is, the compositor passes back data from the most recently rendered frame that is related to the current gaze angle and / or the visible area to the client for processing subsequent frames. Then, at box 565, the client renders the frame. By utilizing the visible area passed to the client by the compositor, the client can avoid rendering pixels outside the visible area, thereby improving performance. By utilizing the gaze angle passed to the client by the compositor, the client can render the frame more appropriately for the user's gaze, thereby improving image quality.

[0039] Flowchart 550 continues at 570, where the compositor renders the rendered frame, and in the process, obtains the compositor eye-tracking parameters. Specifically, the compositor can perform the above-described steps. Figure 5A The steps described above. For example, based on the visibility region, the synthesizer can obtain a gaze angle value and pixel position or replace the pixel position. According to one or more embodiments, the gaze used to obtain the gaze angle comes from a separate gaze tracker. This gaze angle, pixel position, and visibility region can be considered individually or in some combination as synthesizer eye-tracking parameters. According to some embodiments, the synthesizer uses various eye-tracking parameters to generate frames based on the gaze angle and visibility region.

[0040] According to some implementations, the compositor may optionally process the current frame based on historical client data from client-rendered frames being presented by the compositor. Thus, at box 575, the compositor may optionally process the current frame based on historical compositor eye-tracking parameters. That is, eye-tracking data, such as gaze angle and / or visibility area, may be utilized from one frame to another. In some implementations, this reduces latency when processing consecutive frames.

[0041] Optionally, as shown in box 580, in some implementations, the compositor processes the current frame based on historical client eye-tracking parameters. That is, in some implementations, the client can pass data used during the rendering of the previous frame to the compositor for processing additional frames. For example, the gaze angle used during rendering can be sent to the client along with the rendered frame. Thus, as referenced above... Figure 4B and Figure 5A The aforementioned replacement pixel positions and gaze angles will enable more efficient determination and / or improved latency, depending on the shared data.

[0042] The flowchart continues at box 585. If there are additional frames to process, the flowchart returns to box 555, and the client optionally processes the additional frames based on the eye-tracking parameters passed from the compositor until there are no more additional frames to process.

[0043] Figure 6 A flowchart for selecting an updated pixel position according to one or more embodiments is depicted. Specifically, flowchart 600 illustrates example techniques for determining replacement pixel positions according to some embodiments. For clarity, reference is made to Figures 1 and 4 as described above. Figure 6 The description content. Although Figure 6 The various processes shown are presented in a specific order, but it should be understood that the various processes can be executed in different orders. Furthermore, it is not necessary to execute all of the various processes.

[0044] Flowchart 600 begins at 605, where pixels associated with the center of the field of view are identified within the visible area. The center of the field of view can be a pixel on the screen located in front of the eye and can be determined based on a vector originating from the center of the eye and extending vertically to the screen surface. Therefore, the center of the field of view can be located at or offset from the center of the screen or a pixel associated with the center of the lens between the screen and the eye.

[0045] Flowchart 600 continues at 610, where a vector is determined by the pixel associated with the center of the field of view and the pixel associated with the gaze direction. The pixel associated with the gaze direction can be determined as described above with reference to box 505 of flowchart 500. Flowchart 600 ends at box 615, where a replacement pixel is selected from a set of pixels along the determined vector and within the visibility area.

[0046] According to some implementations, the system settings can be configured to track both eyes. If valid data is unavailable for one eye, the system can be configured to use the other eye to predict gazes with invalid data. For example, gaze data for each eye can be obtained. If it is determined that one set of gaze data includes invalid data, a predictive model can be applied to the other set of gaze data to predict the missing gaze data. This set of gaze data may be determined to be invalid, for example, based on hardware or software feedback, based on checks of data consistency between eyes, based on threshold movement within the expected movement, etc. In some implementations, gaze direction can be determined based on the content being presented. For example, if an object is presented to the user and it is determined that one eye is looking away from the object or within a threshold distance of the object, the gaze can be "clamped" to the object. Similarly, scene depth can be used to determine where the user is looking and to ensure that both eyes are looking at something at the same depth. In some implementations, eye models and pupil diameter can be used to determine the depth at which the eye is looking.

[0047] Go to Figure 7 This document presents an electronic device 700 according to one or more embodiments of the present disclosure. The electronic device 700 may be part of a multifunctional device, such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, base station, laptop computer, desktop computer, network device, or any other electronic device. The electronic device 700 is typically used, for example, for eye tracking in XR applications. It should be understood that the various components of the electronic device 700 may be distributed differently within the device or on additional devices such as additional client devices, network devices, etc.

[0048] Electronic device 700 may include processor 710, such as a central processing unit (CPU). Processor 710 may be a system-on-a-chip (SoC) such as those present in mobile devices, and may include one or more dedicated graphics processing units (GPUs). Additionally, processor 710 may include multiple processors of the same or different types. Electronic device 700 may also include memory 720. Memory 720 may include one or more different types of memory that can be used in conjunction with processor 710 to perform device functions. For example, memory 720 may include cache, read-only memory (ROM), random access memory (RAM), or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 720 may store various programming modules executed by processor 710, including an eye-tracking module 722 and other applications 724. Electronic device 700 may also include storage device 730. Storage device 730 may include one or more non-transitory computer-readable media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). Storage device 730 may include, for example, data for performing eye-tracking technology or data for presenting information to a user utilizing eye-tracking technology.

[0049] Electronic device 700 may also include one or more cameras 712 or other sensors, such as depth sensors that can determine the depth of a scene. In one or more embodiments, each of the one or more cameras 712 may be a conventional RGB camera or a depth camera. Additionally, the cameras 712 may include stereo cameras or other multi-camera systems, time-of-flight camera systems, etc., which capture images from which depth information of a scene can be determined. In some embodiments, one or more cameras 712 may be cameras for eye tracking. Electronic device 700 allows a user to interact with an XR environment. There are many different types of electronic systems that enable people to sense and / or interact with a variety of XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, the head-mounted system may be configured to receive an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or semi-transparent display, rather than an opaque display. The transparent or semi-transparent display may have a medium through which light representing the image is directed to the human eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, the transparent or semi-transparent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection techniques that project graphic images onto the human retina. Projection systems may also be configured to project virtual objects onto the physical environment, such as as holograms or on a physical surface.

[0050] Now for reference Figure 8This document illustrates a simplified functional block diagram of an exemplary multi-functional electronic device 800 according to one embodiment. Each of the electronic devices may be a multi-functional electronic device, or may have some or all of the components described herein. The multi-functional electronic device 800 may include a processor 805, a display 810, a user interface 815, graphics hardware 820, device sensors 825 (e.g., proximity sensor / ambient light sensor, accelerometer and / or gyroscope), a microphone 830, an audio codec 835, a speaker 840, communication circuitry 845, digital image capture circuitry 850 (e.g., including a camera system), a video codec 855 (e.g., supporting a digital image capture unit), a memory 860, a storage device 865, and a communication bus 870. The multi-functional electronic device 800 may be, for example, a digital camera or a personal electronic device, such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0051] Processor 805 may execute instructions necessary for the operation of implementing or controlling many functions performed by device 800 (e.g., the generation and / or processing of images as disclosed herein). Processor 805 may, for example, drive display 810 and may receive user input from user interface 815. User interface 815 may allow a user to interact with device 800. For example, user interface 815 may take various forms, such as buttons, keypad, dial pad, click wheel, keyboard, display screen, and / or touchscreen. Processor 805 may also be, for example, a system-on-a-chip, such as those found in mobile devices, and may include a dedicated GPU. Processor 805 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. Graphics hardware 820 may be dedicated computing hardware for processing graphics and / or assisting processor 805 in processing graphics information. In one embodiment, graphics hardware 820 may include a programmable GPU.

[0052] Image capture circuitry 850 may include two (or more) lens assemblies 880A and 880B, each lens assembly having a separate focal length. For example, lens assembly 880A may have a shorter focal length than lens assembly 880B. Each lens assembly may have a separate associated sensor element 890A or 890B. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuitry 850 may capture still images and / or video images. The output from image capture circuitry 850 may be processed at least in part by a video codec 855 and / or a processor 805 and / or graphics hardware 820, and / or a dedicated image processing unit or pipeline incorporated within circuitry 850. The captured images may be stored in memory 860 and / or storage device 865.

[0053] Sensor and camera circuitry 850 can capture still images and video images that can be processed at least in part by devices according to this disclosure: video codec 855 and / or processor 805 and / or graphics hardware 820, and / or a dedicated image processing unit incorporated within circuitry 850. Images thus captured can be stored in memory 860 and / or storage device 865. Memory 860 may include one or more different types of media used by processor 805 and graphics hardware 820 to perform device functions. For example, memory 860 may include memory cache, ROM, and / or RAM. Storage device 865 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device configuration file information, and any other suitable data. Storage device 865 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tape, optical media (such as CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). Memory 860 and storage device 865 can be used to tangibly hold computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 805, such computer program code can implement one or more of the methods described herein.

[0054] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. For example, an XR system can detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. For example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) that present the XR environment, and in response, adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust the characteristics of the graphical content in the XR environment in response to representations of physical motion (e.g., voice commands).

[0055] It should be understood that the above description is intended to be illustrative and not restrictive. Material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and to provide that material in the context of a particular embodiment, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). Therefore, Figures 2 to 3 and Figure 5 to Figure 6 The specific arrangement of the steps or actions shown, or Figures 1 and 4, and Figures 7 to 8 The arrangement of the elements shown should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents. In the appended claims, the terms "comprising" and "therein" are used as the common Chinese equivalents of the corresponding terms "including" and "wherein".

Claims

1. A method for optimizing pupil position, the method comprising: Determine the initial pupil position of the user relative to a lens located in front of the user; A change in pupil position relative to the lens is detected from an updated pupil position relative to the lens; The updated pupil position is determined to be outside the calibration area of ​​the lens for eye tracking; Replace the updated pupil position with a replacement pupil position within the calibration area of ​​the lens; as well as The updated pupil position is used for the eye-tracking function.

2. The method of claim 1, wherein the initial pupil position is associated with a first frame presented to the user, the method further comprising: The second frame is presented to the user based on the updated pupil position.

3. The method of claim 2, wherein the second frame is distorted according to the updated pupil position.

4. The method of claim 2, wherein replacing the updated pupil position with a replacement pupil position is determined by applying an easing function from the initial pupil position to the replacement pupil position.

5. The method of claim 4, wherein the easing function comprises a time-based easing function, and wherein the alternative pupil position is reached during a series of frames including the second frame.

6. The method of claim 1, wherein the calibration region of the lens is determined based on calibration information for the lens.

7. The method of claim 1, wherein the calibration area of ​​the lens is a predetermined area of ​​the lens.

8. A non-transitory computer-readable medium comprising computer-readable code, said computer-readable code being executable by one or more processors to: Determine the initial pupil position of the user relative to a lens located in front of the user; A change in pupil position relative to the lens is detected from an updated pupil position relative to the lens; The updated pupil position is determined to be outside the calibration area of ​​the lens for eye tracking; Replace the updated pupil position with a replacement pupil position located within the calibration area of ​​the lens; as well as The updated pupil position is used for the eye-tracking function.

9. The non-transitory computer-readable medium of claim 8, wherein the initial pupil position is associated with a first frame presented to the user, and further comprises computer-readable code to: The second frame is presented to the user based on the updated pupil position.

10. The non-transitory computer-readable medium of claim 9, wherein the second frame is distorted according to the updated pupil position.

11. The non-transitory computer-readable medium of claim 9, wherein the computer-readable code for replacing the updated pupil position with a replacement pupil position is determined by applying an easing function from the initial pupil position to the replacement pupil position.

12. The non-transient computer-readable medium of claim 11, wherein the easing function comprises a time-based easing function, and wherein the alternative pupil position is reached during a series of frames including the second frame.

13. The non-transitory computer-readable medium of claim 8, wherein the calibration region of the lens is determined based on calibration information for the lens.

14. The non-transitory computer-readable medium of claim 8, wherein the calibration region of the lens is a predetermined region of the lens.

15. A system for optimizing pupil position, the system comprising: One or more processors; as well as One or more computer-readable media comprising computer-readable code, said computer-readable code being executable by said one or more processors to: Determine the initial pupil position of the user relative to a lens located in front of the user; A change in pupil position relative to the lens is detected from an updated pupil position relative to the lens; The updated pupil position is determined to be outside the calibration area of ​​the lens for eye tracking; Replace the updated pupil position with a replacement pupil position within the calibration area of ​​the lens; as well as The updated pupil position is used for the eye-tracking function.

16. The system of claim 15, wherein the initial pupil position is associated with a first frame presented to the user, and further comprises computer-readable code to: The second frame is presented to the user based on the updated pupil position.

17. The system of claim 16, wherein the second frame is distorted according to the updated pupil position.

18. The system of claim 16, wherein the computer-readable code for replacing the updated pupil position with a replacement pupil position is determined by applying an easing function from the initial pupil position to the replacement pupil position.

19. The system of claim 18, wherein the easing function comprises a time-based easing function, and wherein the alternative pupil position is reached during a series of frames including the second frame.

20. The system of claim 15, wherein the calibration region of the lens is determined based on calibration information for the lens.

Images