Filtering eye tracking data

Abstract

To generate and manage eye tracking data for an improved eye tracking technique.SOLUTION: Eye tracking is executed by: determining an initial pupil position of a user with respect to a lens located in front of the user; detecting a change in the pupil position with respect to the lens to an updated pupil position with respect to the lens; and determining that the updated pupil position is outside a bounding box associated with the lens. The updated pupil position is an alternative pupil position having an alternative pupil position within the bounding box associated with the lens. The updated pupil position is used for an eye tracking function. The eye tracking is also executed by: determining that a first pixel associated with a line-of-sight direction is outside a visible region; identifying an alternative pixel inside the visible region; determining a line-of-sight angle updated on the basis of the alternative pixel; and executing the eye tracking using the updated line-of-sight angle.SELECTED DRAWING: Figure 2

Description

Background Art

[0001] The present disclosure generally relates to image processing. More specifically, and not by way of limitation, the present disclosure relates to techniques and systems for generating and managing eye-tracking data for improved eye-tracking techniques.

[0002] Eye-tracking is a technique used in many fields such as gaze detection, pose estimation, face analysis and recognition. Eye-tracking can often be considered as a process that forms the basis of these operations and electronically determines the location of a person's point of fixation or records the movement of a person's point of fixation. In practice, eye-tracking is provided by identifying and tracking the location of the pupil and the direction of the line of sight. However, since shifts or jitters often occur in the sensors used to track the eyes, eye-tracking data is not always perfectly calibrated. Therefore, the eye-tracking data can be jittery or invalid. Thus, improved techniques for managing eye-tracking data are needed.

Brief Description of the Drawings

[0003] [Figure 1A] An exemplary diagram of a setup for performing eye-tracking and pupil location is shown. [Figure 1B] An exemplary diagram of a setup for performing eye-tracking and pupil location is shown.

[0004] [Figure 2] A flowchart of a technique for managing pupil position information for an eye-tracking technique according to one or more embodiments is shown.

[0005] [Figure 3] A flowchart of a technique for narrowing down the pupil position for eye-tracking according to one or more embodiments is shown.

[0006] [Figure 4A] An exemplary diagram of the line of sight with respect to the visible region according to one or more embodiments is shown. [Figure 4B] The following are illustrative diagrams of the line of sight relative to the visible area according to one or more embodiments.

[0007] [Figure 5A] A flowchart of a technique for narrowing the line of sight angle for eye tracking techniques, according to one or more embodiments, is shown. [Figure 5B] A flowchart of a technique for narrowing the line of sight angle for eye tracking techniques, according to one or more embodiments, is shown.

[0008] [Figure 6] A flowchart of a technique for identifying alternative pixels and narrowing the viewing angle, according to one or more embodiments, is shown.

[0009] [Figure 7] A multifunctional electronic device according to one or more embodiments is shown in the form of a block diagram.

[0010] [Figure 8] A computer system according to one or more embodiments is shown in the form of a block diagram. [Modes for carrying out the invention]

[0011] This disclosure relates to a system, method, and computer-readable medium for filtering and smoothing eye tracking data for enhanced performance in eye tracking techniques.

[0012] In one or more embodiments, the pupil position is clamped within a predetermined region. A lens for monitoring the pupil may have a predetermined region that is a known calibrated region, while other regions of the lens may not be as calibrated. Therefore, the pupil position should be maintained within the calibrated region. In one or more embodiments, an initial pupil position relative to the lens positioned in front of the eye is determined. Changes in the pupil position relative to the lens to an updated pupil position relative to the lens can be detected. For example, if the updated pupil position is outside a determined bounding box associated with the calibrated region, an alternative pupil position is selected within the bounding box. The updated pupil position is then used in the eye-tracking technique. In some embodiments, to avoid jitter, the eye-tracking system may change the pupil position over a series of frames, for example, based on a time-based easing function, so that the pupil position transitions from the original pupil position to the alternative pupil position.

[0013] In one or more embodiments, the gaze direction may be narrowed for use in eye-tracking techniques. In particular, the gaze direction may be narrowed so that the user is fixating toward the visible area. A first pixel may be determined to be associated with a gaze direction outside the visible area. Alternate pixels are identified within the visible area. Alternate pixels may be selected along a vector from the center of the field of view to the first pixel. The gaze angle is determined based on the location of the alternate pixels, and the gaze angle is used to perform the eye-tracking function.

[0014] For the purposes of this disclosure, the term “line of sight origin” refers to the center of the eye from which the line of sight is determined.

[0015] For the purposes of this disclosure, the term “pupil position” refers to the location on the surface of the eye where the pupil is located.

[0016] For the purposes of this disclosure, the term “direction of line of sight” refers to the direction of the line of sight that originates from the line of sight origin and passes through the pupil position.

[0017] The following description provides numerous specific details for illustrative purposes to enhance understanding of the disclosed concepts. As part of this description, some of the drawings in this disclosure represent structures and devices in block diagram form to avoid obscuring novel aspects of the disclosed concepts. Also, for clarity, not all features of actual implementations are described herein. Furthermore, as part of this description, some of the drawings in this disclosure may be provided in flowchart form. Boxes in any particular flowchart may be presented in a specific order. However, it should be understood that any particular sequence in any flowchart is used only to illustrate one embodiment. In other embodiments, any of the various components shown in the flowchart may be omitted, or the illustrated sequence of operations may be performed in a different order or simultaneously. In addition, other embodiments may include additional steps not shown as part of the flowchart. Furthermore, the language used in this disclosure has been chosen primarily for readability and explanatory purposes, and not to limit or restrict the subject matter of the invention, and it is necessary to rely on the claims to determine such subject matter of the invention. In this disclosure, any reference to “one embodiment” or “one embodiment” means that a particular feature, structure, or characteristic described in relation to the embodiment is included in at least one embodiment of the disclosed subject matter, and any multiple references to “one embodiment” or “one embodiment” should not be understood as all referring to the same embodiment.

[0018] It should be understood that in the development of actual implementations (such as software and / or hardware development projects), numerous decisions must be made to achieve the developer's specific objectives (e.g., compliance with system and business-related constraints), and these objectives may vary from implementation to implementation. It should also be understood that while such development efforts can be complex and time-consuming, they are nevertheless routine work for those skilled in the art who are engaged in the design and implementation of graphics modeling systems that are of interest to this disclosure.

[0019] Referring to FIG. 1A, an exemplary diagram of an eye 100 is shown. For the purposes of the present disclosure, the eye 100 includes a line-of-sight origin 105 that refers to the center of the eye in 3D space. The eye 100 also includes a pupil position 110. The pupil position 110 refers to a position on the surface of the eye 100 where the pupil is located. In one or more embodiments, the pupil position 110 may refer to the central location of the pupil on the surface of the eye. The line-of-sight origin 105 and the pupil position 110 can be used to determine a line-of-sight direction 115. The line-of-sight direction 115 may refer to the direction of a line of sight that originates from the line-of-sight origin 105 and passes through the pupil position 110. In one or more embodiments, the line-of-sight direction 115 can be defined as a vector that originates from the line-of-sight origin 105 and passes through the pupil position 110.

[0020] Referring to FIG. 1B, an exemplary diagram of an eye 100 in which eye tracking is being performed is shown. This eye tracking can be performed by a device 125 that includes a lens 120. When the eye 100 is looking at an object through the lens 120, only a portion of the pupil data is valid based on the calibration of the system. The pupil position 110 can affect the calibration of the system. For example, due to hardware constraints, only a portion of the lens 120 may be calibrated, but the pupil position 110 can still reach the uncalibrated portion. Thus, the pupil boundary 130 can define a portion of the lens 120 where the pupil data is determined to be valid. In some embodiments, the intersection of the pupil boundary 130 and the lens 120 can identify a boundary box on the lens.

[0021] Figure 2 shows a flowchart of a technique for managing pupil position information for an eye tracking technique according to one or more embodiments. Specifically, Figure 2 shows a technique for clamping the pupil position inside the boundary box of the lens. By clamping the pupil position to a portion of the lens determined to provide valid pupil data, the system can extrapolate data from the uncalibrated portion of the lens that would cause the image to appear distorted from inside the boundary box and avoid causing the image to appear distorted from inside the boundary box. For example, when the display device moves or shifts while in front of the eye, the pupil position can shift frequently, which causes the presentation of jitter on the screen. Although the various processes shown in Figure 2 are illustrated in a particular order, it should be understood that the various processes described can be executed in a different order. Further, it is not necessary to execute all of the various processes.

[0022] Flowchart 200 starts at 205 by determining an initial pupil position of the user with respect to a lens located in front of the user. For example, the lens can be located in front of the user's eye for which the initial pupil position is determined. Flowchart 200 continues at 210 by detecting a change in the pupil position with respect to the lens. As a result, an updated pupil position with respect to the lens is determined.

[0023] In block 215, the updated pupil position is compared with a boundary box associated with the lens. As described above, the boundary box can refer to a portion of the lens determined to provide valid pupil data. The boundary box can be determined, for example, by the intersection of the valid pupil boundary and the lens as described above with respect to Figure 1B.

[0024] The flowchart continues at 220 with a determination as to whether the updated pupil location is outside the boundary box. If, at 220, it is determined that the updated pupil location is not outside the boundary box, the flowchart proceeds to 225 and this updated pupil location is used as is for the eye tracking function.

[0025] Returning to 220, if it is determined that the pupil location is outside the bounding box, the flowchart proceeds to block 230. In block 230, an alternative pupil location inside the bounding box is identified. The alternative pupil location can be determined in several ways. For example, the nearest pupil location inside the bounding box can be used. As an example, a location on the boundary of the bounding box can be used. The flowchart then concludes in block 235, where the alternative pupil location is used for the eye tracking function. Specifically, in some embodiments, the alternative pupil location is used to calculate a warping function for presenting an image on the screen, as described above with respect to Figure 1B.

[0026] In some embodiments, using alternative pupil positions can result in data jumps as the eye moves. For example, data jumps can lead to distortion jumps. In some embodiments, distortion jumps can be avoided by applying an easing function to the pupil position. Figure 3 shows a flowchart of a technique for narrowing down the pupil position for eye tracking, as described in block 230 of Figure 2 above, according to one or more embodiments. The various processes shown in Figure 3 are illustrated in a specific order, but it should be understood that the processes described may be performed in a different order. Furthermore, it is not always necessary to perform all of the various processes.

[0027] Flowchart 300 begins at 305, where an alternative pupil position is identified inside the bounding box. According to one or more embodiments, the alternative pupil position may be selected based on the location closest to the actual pupil position inside the bounding box. In some embodiments, the alternative pupil position may be selected as a location on the boundary of the bounding box.

[0028] This flowchart continues to block 310, where an easing function is applied to the pupil position based on the actual pupil position and alternative pupil position. In some embodiments, the easing function may be a time-based function. Thus, in block 315, instead of simply replacing the actual pupil position with an alternative pupil position, a set of intermediate pupil positions may be determined based on the easing function. According to one or more embodiments, the easing function may be a linear function, a cubic easing function, and so on. Thus, this set of intermediate pupil positions may ease the user's pupil position to the alternative pupil position.

[0029] Flowchart 300 is followed by block 320, which applies a set of intermediate pupil positions over a series of frames. For example, the set of intermediate pupil positions may include a predetermined number of intermediate pupil positions that correspond to the number of frames over which the intermediate pupil positions are applied. Alternatively, in some embodiments, some intermediate pupil positions may be driven by the difference between the actual pupil position and the alternative pupil position. For example, if the linear difference between the actual pupil position and the alternative pupil position is small, fewer frames are required, and therefore fewer intermediate values ​​may be needed to reach the alternative pixel location. In contrast, if the linear difference between the actual pupil position and the alternative pupil position is large, more frames are required, and therefore more intermediate values ​​may be needed to reach the alternative pixel location to more effectively ease the user. The set of intermediate pupil positions is applied such that a subsequent series of frames presented to the user can utilize the progression of intermediate pupil positions over the series of frames.

[0030] Flowchart 300 concludes in block 325, where the alternative pupil position is used in the next frame; that is, after utilizing a set of intermediate pupil positions over a series of frames rendered based on the alternative pupil position rather than the actual pupil position. In some embodiments, the alternative pupil position can then be used to render content for display to the user until a new pupil position is detected and the process described in Figure 2 begins again at that point.

[0031] According to some embodiments, other issues may arise, for example, based on the user's line of sight. For example, a display may have areas associated with areas containing valid pixel information. Thus, the line of sight can be narrowed to ensure that the user fixates on pixels within the valid visible area. Referring to Figure 4A, a simplified block diagram of a screen, such as screen 425, is shown. Screen 245 can be any type of display positioned in front of the user, such as a mobile device like a tablet or mobile phone, or a head-mounted display device. A portion of the display is visible and may be called the visible area. In some embodiments, the visible area is visible through lens 430. 、 It is only a portion of the screen. Therefore, the lens cannot "see" the entirety of the display. In addition, not all of the screen 425 visible through the lens 430 may contain valid data. For example, the outer portion of the lens 430 may be difficult to calibrate, or may not be calibrated at all. Therefore, the screen 425 may include a visible area 435 that is determined to contain valid pixel data. In addition or alternatively, the determination of the visible area may be based on what is visible by the hardware, the technical specifications of the eye-tracking system, or the area defined by the software.

[0032] Referring to Figure 4B, this figure shows an exemplary view of the screen from the eye. According to some embodiments, the visible area 345 may be defined by a visible mask based on a portion of the screen determined to provide valid pixel data. Thus, if the user's line of sight is directed outside the visible area 435, an alternative line of sight location may be utilized. As illustrated, the user's line of sight 460 is directed inside the screen 440, although it is outside the visible area 435. The screen 440 may be the screen of a mobile device positioned in front of the user, such as a tablet, mobile phone, or wearable device. To provide valid data, an alternative line of sight location may be selected from inside the visible area. The alternative pixel may be selected, for example, based on the nearest pixel inside the visible area 435. In some embodiments, the alternative pixel may be selected by identifying the center of the field of view 450. The center of the field of view 450 may be based on the location of the eye relative to the screen. The center of the field of view 450 may be based on the location of the center of the user's eye relative to the screen. In some embodiments, the alternate pixels 455 of the line-of-sight target 460 may be selected such that the alternate pixels 455 are located within the visible area. The alternate pixels may be determined along a vector 470 originating from the center of the field of view 450 and directed toward the target location of the line of sight 460. According to one or more embodiments, the alternate pixels 455 may then be used in an eye-tracking technique. Furthermore, in some embodiments, the eye-tracking technique may also rely on a line-of-sight angle which may be determined based on the alternate pixels 455.

[0033] Figure 5A shows a flowchart of a technique for managing gaze information for eye-tracking techniques according to one or more embodiments. Specifically, Figure 5A shows a technique for recalculating the user's gaze direction in order to utilize valid pixel data. For clarity, the explanation of Figure 3 will be explained in relation to Figures 1 and 4 as described above. The various processes shown in Figure 5 are illustrated in a specific order, but it should be understood that the processes described may be performed in a different order. Furthermore, it is not always necessary to perform all of the various processes.

[0034] Flowchart 500 begins in block 505, where pixel locations are detected as being associated with the direction of gaze. For example, a pixel location may be associated with the target location of the user's gaze. This pixel location may be determined relative to a screen located in front of the user's eye 100. In block 510, a determination is made as to whether the detected pixel location is outside the visible area. As described above with respect to Figure 4B, the visible area 435 may be a subset of the screen 425 containing valid pixel data. The valid pixel data may be based on a subset of the lens 430 from which the screen 425 is visible.

[0035] If it is determined in block 510 that the pixel location is not outside the visible area, flowchart 500 proceeds to block 515. In block 515, the value of the viewing angle is determined based on the detected pixel location. The viewing angle may represent the angle between the user's eye and the screen, depending on the determined pixel location. This flowchart concludes in block 520, which uses the determined viewing angle and pixel location to process the next frame.

[0036] Returning to block 510 of flowchart 500, if it is determined that the detected pixel location is outside the visible area, flowchart 500 proceeds to block 525. In block 525, an alternative pixel location is determined within the visible area. In one or more embodiments, the alternative pixel may be selected, for example, as the pixel closest to the detected pixel within the visible area. Furthermore, in some embodiments, the alternative pixel location may be selected based on the center point of the field of view, as will be described in detail below with respect to Figure 6.

[0037] Flowchart 500 is followed by block 530, which determines the viewing angle based on the location of the alternate pixel. The viewing angle may represent the angle between the user's eye and the screen, depending on the determined alternate pixel location. This flowchart concludes in block 535, which utilizes the determined viewing angle and alternate pixel location to process the next frame.

[0038] According to some embodiments, eye-tracking data may be used to render frames in a two-phase process by the client and the compositor. In some embodiments, eye-tracking data can be passed between the client and the compositor and / or historical eye-tracking data can be used to improve visual quality and / or meet hardware constraints. In particular, consistency between the client and the compositor can be improved by sharing at least some eye-tracking parameters between them. Figure 5B shows a flowchart of a technique for managing eye-tracking data according to one or more embodiments. Specifically, Figure 5B shows a technique for sharing eye-tracking data between the client and the compositor. For clarity, the description of Figure 3 will be explained in relation to Figures 1 and 4 as described above. The various processes shown in Figure 5B are illustrated in a particular order, but it should be understood that the processes described may be performed in a different order. Furthermore, it is not always necessary to perform all of the various processes. The flowchart begins in block 555, where the client processes frames and obtains eye-tracking parameters. In some embodiments, the client renders frames for display by the compositor. According to some embodiments, the client determines certain eye-tracking parameters, such as the viewing angle, when rendering a frame. In an optional step, as shown in block 560, the client processes the frame based on past compositor eye-tracking parameters. That is, the compositor passes to the client data related to the current viewing angle and / or the visible area from the most recently presented frame in order to process future frames. Then, in block 565, the client renders the frame. By utilizing the visible area passed to the client by the compositor, the client can avoid rendering pixels outside the visible area, and thus improve performance. By utilizing the viewing angle passed by the compositor, the client can render the frame more appropriately to the user's line of sight, thereby improving image quality.

[0039] Flowchart 550 is followed by 570, in which the compositor presents rendered frames and, in the process, acquires compositor eye-tracking parameters. In particular, the compositor can perform the steps described with respect to Figure 5A above. For example, the compositor can acquire the line of sight angle value and pixel location or alternative pixel location based on the visible region. According to one or more embodiments, the line of sight from which the line of sight angle is acquired is from a separate eye-tracking device. This line of sight angle, pixel location, and visible region can be considered compositor eye-tracking parameters, either individually or in some combination. According to some embodiments, the compositor uses various eye-tracking parameters to generate frames according to the line of sight angle and visible region.

[0040] According to some embodiments, the compositor may optionally process the current frame based on past client data from client rendering frames presented by the compositor. Thus, in block 575, the compositor may optionally process the current frame based on past compositor eye-tracking parameters. That is, eye-tracking data such as line-of-sight angle and / or visible region can be used from one frame to another. In some embodiments, this can reduce latency when processing consecutive frames.

[0041] Optionally, as shown in block 580, in some embodiments, the compositor processes the current frame based on past client eye-tracking parameters. That is, in some embodiments, the client can pass data used during rendering of previous frames to the compositor in order to process additional frames. For example, the viewing angle used during rendering may be sent to the client along with the rendered frame. Thus, alternative pixel locations and viewing angles are determined more efficiently and / or latency is improved by relying on shared data, as described above with respect to Figures 4B and 5A.

[0042] The flowchart continues to block 585. If there are additional frames to process, the flowchart returns to block 555, and the client optionally processes the additional frames based on the eye tracking parameters passed from the compositor until it no longer needs to process them.

[0043] Figure 6 shows a flowchart for selecting an updated pixel location according to one or more embodiments. In particular, flowchart 600 shows an exemplary technique for determining an alternative pixel location according to several embodiments. For clarity, the description of Figure 6 will be explained in relation to Figures 1 and 4 as described above. The various processes shown in Figure 6 are illustrated in a specific order, but it should be understood that the processes described may be performed in a different order. Furthermore, it is not always necessary to perform all of the various processes.

[0044] Flowchart 600 begins in step 605, identifying a pixel associated with the center of the field of view within the visible area. The center of the field of view may be a pixel on the screen located in front of the eye and may be determined based on a vector originating from the center of the eye and extending perpendicularly to the surface of the screen. Thus, the center of the field of view may be located at the center of the screen, or offset, or located at a pixel associated with the center of the lens between the screen and the eye, or offset.

[0045] Flowchart 600 continues in 610, where a vector is determined from a pixel associated with the center of the field of view and a pixel associated with the line of sight. The pixel associated with the line of sight may be determined as described above with respect to block 505 of flowchart 500. Flowchart 600 concludes in block 615, where an alternative pixel is selected from a set of pixels along the determined vector and within the visible region.

[0046] According to some embodiments, the system setup may be configured to track both eyes. If valid data is unavailable for one eye, the system may be configured to predict the line of sight of that eye using invalid data from the other eye. For example, line of sight data may be acquired for each eye. If one of the sets of line of sight data is determined to contain invalid data, the prediction model can be applied to other sets of the line of sight dataset to predict the missing line of sight data. For example, a set of line of sight data may be determined to be invalid based on a review of data integrity between the two eyes, based on hardware or software feedback, or based on threshold motion within assumed motion. In some embodiments, the direction of gaze may be determined based on the presented content. For example, if an object is presented to the user and one eye is determined to be looking just outside the object, or within the object's threshold distance, the line of sight may be "clamped" to the object. Similarly, scene depth can be used to determine where the user is looking and to ensure that both eyes are looking at something at the same depth. In some embodiments, an eye model and pupil diameter can be used to determine the depth at which the eye is looking.

[0047] Referring to Figure 7, an electronic device 700 is presented according to one or more embodiments of the present disclosure. The electronic device 700 may be part of a multifunction device such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, base station, laptop computer, desktop computer, network device, or any other electronic device. The electronic device 700 is commonly used, for example, in the context of XR applications, for eye tracking. It should be understood that the various components of the electronic device 700 may be distributed differently within the device or across additional devices such as additional client devices, network devices, etc.

[0048] The electronic device 700 may include a processor 710, such as a central processing unit (CPU). The processor 710 may be a system-on-a-chip, such as those found in mobile devices, and may include one or more dedicated graphics processing units (GPUs). Furthermore, the processor 710 may include multiple processors of the same or different types. The electronic device 700 may also include memory 720. The memory 720 may include one or more different types of memory that can be used in conjunction with the processor 710 to perform device functions. For example, the memory 720 may include a cache, read-only memory (ROM), random access memory (RAM), or any kind of temporary or non-temporary computer-readable storage medium capable of storing computer-readable code. The memory 720 may store various programming modules for execution by the processor 710, including an eye-tracking module 722 and other applications 724. The electronic device 700 may also include a storage device 730. The storage device 730 may include one or more non-temporary computer-readable media, such as magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as electrically programmable read-only memory (EPROMs) and electrically erasable programmable read-only memory (EEPROMs). The storage device 730 may include, for example, data used to perform eye-tracking techniques, or data to present information to users on whom eye-tracking techniques are used.

[0049] The electronic device 700 may also include one or more cameras 712 capable of determining the depth of a scene, or other sensors such as depth sensors. In one or more embodiments, each of the one or more cameras 712 may be a conventional RGB camera or a depth camera. Furthermore, the cameras 712 may include stereo cameras or other multi-camera systems, time-of-flight camera systems, etc., that capture images capable of determining the depth information of a scene. In some embodiments, one or more cameras 712 may be cameras used for eye tracking. The electronic device 700 may enable a user to interact with the XR environment. The existence of a wide variety of electronic systems allows a person to perceive and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned in the human eye (similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mountable system may have one or more speakers and an integrated opaque display. Alternatively, a head-mountable system may be configured to accept an external opaque display (e.g., a smartphone). A head-mountable system may incorporate one or more imaging sensors for capturing images or videos of the physical environment and / or one or more microphones for capturing sounds of the physical environment. A head-mountable system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to the human eye. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies.The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology to project graphical images onto the human retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface.

[0050] Referring next to Figure 8, a simplified functional block diagram of an exemplary multifunctional electronic device 800 according to one embodiment is shown. Each of the electronic devices may be a multifunctional electronic device, or may have some or all of the described components of the multifunctional electronic device described herein. The multifunctional electronic device 800 may include a processor 805, a display 810, a user interface 815, graphics hardware 820, device sensors 825 (e.g., proximity sensor / ambient light sensor, accelerometer, and / or gyroscope), a microphone 830, one or more audio codecs 835, one or more speakers 840, a communication circuit 845, a digital image capture circuit 850 (e.g., including a camera system), one or more video codecs 855 (e.g., supporting a digital image capture unit), a memory 860, a storage device 865, and a communication bus 870. The multifunctional electronic device 800 may be, for example, a digital camera, or a personal electronic device such as a personal digital assistant (PDA), personal music player, mobile phone, or tablet computer.

[0051] The processor 805 can execute instructions necessary to perform or control the operation of numerous functions performed by the device 800 (e.g., image generation and / or processing as disclosed herein). The processor 805 can, for example, drive the display 810 and receive user input from the user interface 815. The user interface 815 may enable a user to interact with the device 800. For example, the user interface 815 can take various forms such as buttons, keypads, dials, click wheels, keyboards, display screens, and / or touchscreens. The processor 805 may also be a system-on-a-chip, such as those found in mobile devices, and may include a dedicated graphics processing unit (GPU). The processor 805 may be based on a reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architecture, or any other preferred architecture, and may include one or more processing cores. The graphics hardware 820 may be dedicated computing hardware for processing graphics and / or for assisting the processor 805 in processing graphics information. In one embodiment, the graphics hardware 820 may include a programmable GPU.

[0052] The image capture circuit 850 may include two (or more) lens assemblies 880A and 880B, each having a distinct focal length. For example, lens assembly 880A may have a shorter focal length than lens assembly 880B. Each lens assembly may have a distinct associated sensor element 890A or 890B. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 850 can capture still images and / or video images. The output from the image capture circuit 850 may be processed at least in part by a video codec(s) 855 and / or a processor 805 and / or graphics hardware 820, and / or a dedicated image processing unit or pipeline incorporated within the circuit 850. The captured images may be stored in memory 860 and / or storage device 865.

[0053] The sensor and camera circuit 850 can capture still images and video images, which, in accordance with this disclosure, may be processed, at least in part, by a video codec(single or multiple) 855 and / or a processor 805 and / or graphics hardware 820, and / or a dedicated image processing unit incorporated within the circuit 850. Images thus captured may be stored in memory 860 and / or storage device 865. Memory 860 may include one or more different types of media used by the processor 805 and graphics hardware 820 to perform the functions of the device. For example, memory 860 may include a memory cache, ROM, and / or RAM. Storage device 865 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and other appropriate data. The storage device 865 may include one or more non-temporary computer-readable storage devices, including, for example, magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and digital video discs (DVDs), and semiconductor memory devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). The memory 860 and storage device 865 can be organized into one or more modules and used to tangibly hold computer program instructions or code written in any desired computer programming language. For example, when executed by the processor 805, such computer program code can perform one or more of the methods described herein.

[0054] A physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic devices. A physical environment may include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through their senses such as sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people perceive and / or interact with through electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, and virtual reality (VR) content. In XR, a subset of a person's body movements or their representation is tracked, and in response, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. As an example, an XR system can detect the rotation of a person's head and, in response, adjust the graphic content and sound field presented to that person in a manner similar to how such views and sounds would change in the physical environment. As another example, an XR system can detect the movement of an electronic device presenting an XR environment (e.g., a mobile phone, tablet, laptop) and, in response, adjust the graphic content and sound field presented to that person in a manner similar to how such views and sounds would change in the physical environment. In some situations (e.g., for accessibility reasons), an XR system can adjust the characteristics of the graphic content within the XR environment in response to a representation of bodily movement (e.g., a voice command).

[0055] It should be understood that the above description is illustrative and not limiting. The material is presented in the content of specific embodiments so that a person skilled in the art can manufacture and use the disclosed subject matter as claimed, and variations of those embodiments will be readily apparent to a person skilled in the art (for example, some of the disclosed embodiments may be used in combination with one another). Accordingly, the specific configurations of steps or actions shown in Figures 2-3 and 5-6, or the configurations of elements shown in Figures 1, 4 and 7-8, should not be construed as limiting the scope of the disclosed subject matter. Accordingly, the scope of the invention should be determined by referring to the appended claims and the entire scope of equivalents given to such claims. In the appended claims, the terms “including” and “in which” are used as plain English equivalents of the terms “comprising” and “wherein,” respectively.

Claims

1. It is a method, This involves determining that the line of sight of the eye is directed toward a first pixel located outside the visible area, wherein the display and the lens are arranged such that a portion of the display is not visible through the lens from the eye's viewpoint, and the visible area is the portion of the display that is visible through the lens. Identifying alternative pixels having pixel locations inside the visible region, The updated viewing angle is determined based on the pixel location of the substitute pixel, wherein the updated viewing angle represents the angle between the eye's viewing direction corresponding to the pixel location of the substitute pixel and the display. The eye-tracking function is performed using the updated gaze angle, A method that includes [a certain feature].

2. A method according to claim 1, wherein identifying the alternative pixel further includes identifying the alternative pixel as a pixel located inside the visible region closest to the first pixel, method.

3. The method according to claim 1, wherein identifying the alternative pixel further means Identifying a pixel related to the center of the visible region, Determining the vector from the pixel related to the center of the visible region to the first pixel, Select the alternative pixel from the set of pixels that lies along the determined vector and inside the visible region. Methods that include...

4. A non-temporary computer-readable medium containing computer-readable code, wherein the computer-readable code is This involves determining that the line of sight of the eye is directed toward a first pixel located outside the visible area, wherein the display and the lens are arranged such that a portion of the display is not visible through the lens from the eye's viewpoint, and the visible area is the portion of the display that is visible through the lens. Identifying alternative pixels having pixel locations inside the visible region, The updated viewing angle is determined based on the pixel location of the substitute pixel, wherein the updated viewing angle represents the angle between the eye's viewing direction corresponding to the pixel location of the substitute pixel and the display. The eye-tracking function is performed using the updated gaze angle, A non-temporary computer-readable medium that can be executed by one or more processors to perform the following actions.

5. A non-temporary computer-readable medium according to claim 4, wherein the computer-readable code for identifying the alternate pixel further includes a computer-readable code for identifying the alternate pixel as a pixel located inside the visible region closest to the first pixel. Non-temporary computer-readable media.

6. The non-temporary computer-readable medium according to claim 4, wherein the computer-readable code for identifying the substitute pixel further comprises: Identifying a pixel related to the center of the visible region, Determining the vector from the pixel related to the center of the visible region to the first pixel, Select the alternative pixel from the set of pixels that lies along the determined vector and inside the visible region. A non-temporary computer-readable medium containing computer-readable code that performs the following actions.

7. It is a system, One or more processors, One or more computer-readable media containing computer-readable code, wherein the computer-readable code is This involves determining that the line of sight of the eye is directed toward a first pixel located outside the visible area, wherein the display and the lens are arranged such that a portion of the display is not visible through the lens from the eye's viewpoint, and the visible area is the portion of the display that is visible through the lens. Identifying alternative pixels having pixel locations inside the visible region, The updated viewing angle is determined based on the pixel location of the substitute pixel, wherein the updated viewing angle represents the angle between the eye's viewing direction corresponding to the pixel location of the substitute pixel and the display. The eye-tracking function is performed using the updated gaze angle, One or more computer-readable media that can be executed by one or more processors to perform the following: A system equipped with these features.

8. The system according to claim 7, wherein the computer-readable code for identifying the substitute pixel further includes a computer-readable code for identifying the substitute pixel as a pixel located inside the visible region closest to the first pixel. system.

9. The system according to claim 7, wherein the computer-readable code for identifying the alternative pixel further comprises: Identifying a pixel related to the center of the visible region, Determining the vector from the pixel related to the center of the visible region to the first pixel, Select the alternative pixel from the set of pixels that lies along the determined vector and inside the visible region. A system that includes computer-readable code to perform a certain action.

Images