Synthetic gaze registration

Abstract

A synthetic gaze feature is generated at the real eye pose Gg using the personalized eye model. A corresponding synthetic gaze pose Gs is estimated from the synthetic gaze feature using an average eye model. A linear regression is applied between Gg and Gs to generate a gaze correction function. The gaze correction function represents a difference between a synthetic gaze Gs of a subject eye at the display and a real gaze Gg of an average eye model at the display, but does not contain security-sensitive information or privacy-sensitive information. Further, the personalized eye model cannot be recovered from the gaze correction function, and thus the gaze correction function can be stored unencrypted and available for use during cold boot of the device prior to login. Upon cold boot of the device, a gaze correction function can be accessed and used with an average eye model to improve gaze-based interaction.

Description

Background Art

[0001] Extended reality (XR) systems, such as mixed reality (MR) or augmented reality (AR) systems, combine computer-generated information (referred to as virtual content) with real-world images or views of the real world to enhance or increase the user's perception of the world. Therefore, XR systems can be used to provide interactive user experiences for a variety of applications, such as applications that add virtual content to a live view of the viewer's environment, applications that interact with virtual training environments, gaming applications, applications that remotely control drones or other mechanical systems, applications that view digital media content, applications that interact with the Internet, and so on. Summary of the invention

[0002] Various embodiments of methods and apparatus for synthetic gaze registration for gaze interactions after a cold start of a device (e.g., a head mounted device (HMD), including but not limited to HMDs used in extended reality (XR) applications and systems) are described. The HMD may include a wearable device such as a headset, a helmet, goggles, or glasses. An XR system may include an HMD that may include one or more cameras that may be used to capture still images or video frames of a user's environment. The HMD may include lenses positioned in front of the eyes through which the wearer may view the environment. In an XR system, virtual content may be displayed on or projected onto the lenses so that the virtual content is visible to the wearer while still being able to view the real environment through the lenses.

[0003] In at least some systems, the HMD may include gaze tracking technology. In an example gaze tracking system, one or more infrared (IR) light sources emit IR light toward the user's eyes. A portion of the IR light is reflected from the eyes and captured by an eye tracking camera. The image captured by the eye tracking camera may be input to a flash and pupil detection process, which is implemented, for example, by one or more processors of a controller of the HMD. The results of this process are passed to a gaze estimation process (which is implemented, for example, by one or more processors of a controller) to estimate the user's current gaze point. This gaze tracking method may be referred to as PCCR (Pupillary Center Corneal Reflection) tracking.

[0004] In such a system, during an initial calibration or registration process, a multi-dimensional personalized model of the eye may be generated based on one or more images of the user's eye captured as described above. During use of the device, the personalized eye model may be used in various algorithms (e.g., in a gaze estimation process). The personalized eye model may include information such as a corneal surface model, an iris and pupil model, an eye center, an entrance pupil, a pupil axis or an optical axis (a vector passing through the geometric eye center and the entrance pupil), and a kappa angle between the optical axis and the visual axis.

[0005] However, the personalized eye model can be privacy-sensitive information and security-sensitive information, and therefore can be stored as encrypted data when the user is not logged into the device. Therefore, after a cold start of the device, due to security and privacy considerations, the gaze registration data including the personalized eye model is inaccessible until the user logs in using a password or other secure login method, after which the secure data can be encrypted. However, entering a password to log in can be performed via a gaze-based interface. Therefore, in order to calibrate the device for gaze interaction before logging in, since the personalized eye model is not available, a privacy-insensitive gaze calibration model is required to enable or improve gaze-based password interaction for user login. This gaze calibration model, which can be referred to as a screen space model, can be estimated during the gaze registration process using synthetic gaze.

[0006] In an embodiment of the registration method for generating a screen space model, an N-dimensional personalized eye model is used to generate a screen space model based on the real eye pose G. g A synthetic gaze feature is then generated at . The corresponding synthetic gaze pose G is then estimated from the synthetic gaze feature using an average eye model (e.g., an eye model generated from features of multiple examples of eyes, which can be stored unencrypted on the device as it is not privacy-sensitive or security-sensitive). s Then in G g and G s Linear regression is applied between to generate a gaze correction function, which can be a two-dimensional screen space model. The gaze correction function basically contains a synthetic gaze G representing the measured eye at the display. s The actual gaze G with the average eye model at the display g The difference between the data.

[0007] The gaze correction function does not contain security-sensitive information and privacy-sensitive information of the user. In addition, since the method collapses the hyperdimensional space (N-dimensional personalized eye model) into two dimensions (gaze correction function), the personalized eye model cannot be recovered from the gaze correction function, and thus the gaze correction function can be stored unencrypted and can be used during a device cold boot before login.

[0008] To generate the gaze correction function, the user does not have to look at the real eye point to generate the eye features required for the generative model. The personalized eye model can be used to generate the eye pose G mapped to the display screen. g The average eye model can be used to generate the corresponding eye pose G from the synthesized gaze features s . Linear regression is then used to determine G g and G s This generates a regression-based model (gaze correction function).

[0009] After the gaze correction function is generated during the enrollment process, upon a device cold boot, the gaze correction function can be accessed (since it is stored unencrypted) and used together with the average eye model to improve gaze-based interactions, such as password entry, prior to logging in. After a successful login, the personalized eye model can be decrypted and used for gaze-based interactions and other gaze-based functions.

[0010] In some embodiments, the system may detect that the gaze correction function may need to be updated or supplemented. For example, the system may detect that the user is wearing the device differently, or is wearing contact lenses, or has added additional lenses (e.g., prescription lenses) to the device. In these cases, a new gaze correction function may be generated (note that the personalized eye model may also be updated accordingly, and the updated eye model may be used to generate the gaze correction function). The old gaze correction function may be replaced by the new model, or alternatively, two models may be stored. Therefore, in some embodiments, multiple gaze correction functions may be stored for selection at cold start, and the system may select a best model from among them for use based on one or more criteria. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An N-dimensional model of an eye according to some embodiments is graphically illustrated.

[0012] Figure 2 A method for generating a two-dimensional gaze correction function for an eye during a registration process, which can then be used to calibrate gaze interactions on a cold start of a device, is graphically illustrated in accordance with some embodiments.

[0013] Figure 3A and Figure 3B is a flowchart of a method for generating a gaze correction function for an eye according to some embodiments.

[0014] Figure 4 is a high-level flowchart of a method for using a gaze correction function during a cold start of a device according to some embodiments.

[0015] Figure 5 is a high-level flow chart of a method for generating and using multiple gaze correction functions for an eye, according to some embodiments.

[0016] FIG. 6A to FIG. 6C Illustrated are some embodiments in which Figures 1 to 5 An example device for the method.

[0017] Figure 7is a block diagram illustrating an example device according to some embodiments, which example device may include Figures 1 to 5 The components illustrated in the figures are implemented and the methods are implemented as shown in the figures.

[0018] This specification includes references to "one embodiment" or "an embodiment." The appearance of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. The particular features, structures or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0019] The term "comprising" is open ended. As used in the claims, the term does not exclude additional structures or steps. Consider the following cited claim: "an apparatus comprising one or more processor units..." Such a claim does not exclude the apparatus from including additional components (e.g., a network interface unit, a graphics circuit, etc.).

[0020] "Configured to", various units, circuits or other components may be described or described as "configured to" perform one or more tasks. In such contexts, "configured to" is used to imply a structure (e.g., a circuit) that performs the one or more tasks during operation by indicating that the unit / circuit / component includes the structure. In this way, the unit / circuit / component is said to be configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not turned on). The units / circuits / components used with the "configured to" language include hardware-such as circuits, memories storing executable program instructions to implement operations, etc. The reference to a unit / circuit / component "configured to" perform one or more tasks is explicitly intended not to invoke the sixth paragraph of 35 U.S.C. §112 for the unit / circuit / component. In addition, "configured to" may include a general structure (e.g., a general circuit) manipulated by software or firmware (e.g., an FPGA or a general processor executing software) to operate in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor fabrication facility) to produce a device (eg, an integrated circuit) suitable for implementing or performing one or more tasks.

[0021] "First," "second," etc. As used herein, these terms act as labels for the nouns that precede them, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing a write operation of a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0022] "Based on" or "depending on", as used herein, these terms are used to describe one or more factors that influence a determination. These terms do not exclude additional factors that may influence the determination. That is, the determination may be based solely on these factors or at least in part on these factors. Consider the phrase "A is determined based on B." In this case, B is a factor that influences the determination of A, and such phrases do not exclude that the determination of A may also be based on C. In other examples, A may be determined based solely on B.

[0023] "Or", when used in the claims, the term "or" is used as an inclusive or rather than an exclusive or. For example, the phrase "at least one of x, y or z" means any one of x, y and z and any combination thereof. DETAILED DESCRIPTION

[0024] Various embodiments of methods and apparatus for synthetic gaze registration for gaze interactions after a cold start of a device (e.g., a head mounted device (HMD), including but not limited to HMDs used in extended reality (XR) applications and systems) are described. The HMD may include a wearable device such as a headset, a helmet, goggles, or glasses. An XR system may include an HMD that may include one or more cameras that may be used to capture still images or video frames of a user's environment. The HMD may include lenses positioned in front of the eyes through which the wearer may view the environment. In an XR system, virtual content may be displayed on or projected onto the lenses so that the virtual content is visible to the wearer while still being able to view the real environment through the lenses.

[0025] In at least some systems, the HMD may include gaze tracking technology. In an example gaze tracking system, one or more infrared (IR) light sources emit IR light toward the user's eyes. A portion of the IR light is reflected from the eyes and captured by an eye tracking camera. The image captured by the eye tracking camera may be input to a flash and pupil detection process, which is implemented, for example, by one or more processors of a controller of the HMD. The results of this process are passed to a gaze estimation process (which is implemented, for example, by one or more processors of a controller) to estimate the user's current gaze point. This gaze tracking method may be referred to as PCCR (Pupillary Center Corneal Reflection) tracking.

[0026] In such a system, during an initial calibration or enrollment process, a multi-dimensional personalized model of the eye may be generated based on one or more images of the user's eye captured as described above. Figure 1An N-dimensional model 100 of an eye according to some embodiments is graphically illustrated. The physical components of the eye may include a sclera 102, a cornea 104, an iris 106, and a pupil 108. In some embodiments, during an initial calibration or registration process, an N-dimensional model of a user's eye 100 may be generated based on one or more images of the eye 100. In an example method, one or more infrared (IR) light sources emit IR light toward the user's eye. A portion of the IR light is reflected from the eye and captured by an eye tracking camera. Two or more images captured by the eye tracking camera may be input to an eye model generation process, which is implemented, for example, by one or more processors of a controller of an HMD. The process may determine the shape and relationship of the components of the eye based at least in part on the position of the flash (reflection of a point light source) in the two or more captured images. This information can then be used to generate a personalized eye model. The personalized eye model may include information such as a corneal surface model, an iris and pupil model, an eye center 112, an entrance pupil 110, a pupil axis or optical axis 120 (a vector passing through the eye center 112 and the entrance pupil 110), and a kappa angle between the optical axis 120 and the visual axis 122 of the eye. The personalized eye model may be used in various algorithms (e.g., in a gaze estimation process) during use of the device. The personalized eye model may then be used in various algorithms (e.g., in a gaze estimation process) during use of the device.

[0027] However, the personalized eye model can be privacy-sensitive information and security-sensitive information, and therefore can be stored as encrypted data when the user is not logged into the device. Therefore, after a cold start of the device, due to security and privacy considerations, the gaze registration data including the personalized eye model is inaccessible until the user logs in using a password or other secure login method, after which the secure data can be encrypted. However, entering a password to log in can be performed via a gaze-based interface. Therefore, in order to calibrate the device for gaze interaction before logging in, since the personalized eye model is not available, a privacy-insensitive gaze calibration model is required to enable or improve gaze-based password interaction for user login. This gaze calibration model, which can be referred to as a screen space model, can be estimated during the gaze registration process using synthetic gaze.

[0028] Figure 2 10 is a graphical illustration of a method for generating a two-dimensional gaze correction function for an eye during a registration process according to some embodiments, which can then be used to calibrate gaze interactions on a cold start of a device. In an embodiment of a registration method for generating a screen space model, an N-dimensional personalized eye model 202 is generated 200 and used to calibrate the gaze interaction on a real eye pose G gA synthetic gaze feature 214 is synthesized 210 at 212. A corresponding synthetic gaze pose G is then estimated 220 from the synthetic gaze feature 214 using an average eye model 222 (e.g., an eye model generated from features of multiple examples of eyes, which may be stored unencrypted on the device as it is not privacy-sensitive or security-sensitive). s 224. Then in G g and G s Linear regression 230 is applied between the two to generate a two-dimensional gaze correction function 232. The gaze correction function 232 basically contains a function representing the gaze G of the tested eye at the display. s 224 The real gaze G with the average eye model at the display g 212 of the difference between the data.

[0029] The gaze correction function 232 does not contain security-sensitive information and privacy-sensitive information of the user. In addition, since the method collapses the hyperdimensional space (N-dimensional personalized eye model 202) into two dimensions (gaze correction function 232), the personalized eye model 202 cannot be recovered from the gaze correction function 202, and thus the gaze correction function 232 can be stored unencrypted and can be used during a device cold boot before login.

[0030] Figure 3A and Figure 3B is a flow chart of a method for generating a gaze correction function for an eye according to some embodiments. Figure 3A As indicated at 310, an N-dimensional personalized eye model may be generated. As indicated at 320, a two-dimensional gaze correction function for the eye may be generated. As indicated at 330, the personalized eye model may be encrypted and stored. As indicated at 340, the unencrypted gaze correction function may be stored.

[0031] Figure 3B is Figure 3A Flowchart of a process executed at element 320 of FIG. 1 to generate a two-dimensional gaze correction function. As indicated at 322, the personalized eye model is used to generate a two-dimensional gaze correction function in the real eye pose G. g As indicated at 324, a gaze tracking algorithm is applied using the synthetic gaze feature and the average eye model to determine a synthetic gaze G s As indicated at 326, for G g and G s Linear regression is performed to generate a two-dimensional gaze correction function. The gaze correction function basically contains the synthetic gaze G representing the measured eye at the display. s The actual gaze G with the average eye model at the display g The difference between the data.

[0032] After the gaze correction function is generated during the enrollment process, upon a device cold boot, the gaze correction function can be accessed (since it is stored unencrypted) and used together with the average eye model to improve gaze-based interactions, such as password entry, prior to logging in. After a successful login, the personalized eye model can be decrypted and used for gaze-based interactions and other gaze-based functions.

[0033] Figure 4 is a high-level flow chart of a method for using a gaze correction function during a cold start of a device according to some embodiments. As indicated at 400, on a cold start, a user may log in; an average eye model with corrections applied by an unencrypted gaze correction function is used for gaze interactions. For example, a user may log in by entering a password using a gaze-based interface. At 410, if the login is successful, then as indicated at 420, the personalized eye model may be unencrypted and used for gaze tracking while the system is in use.

[0034] In some embodiments, if necessary, the gaze correction function can be updated, as indicated at 430. In some embodiments, the system may detect that the gaze correction function may need to be updated or supplemented. For example, the system may detect that the user is wearing the device differently, or is wearing contact lenses, or has added additional lenses (e.g., prescription lenses) to the device. In these cases, a new gaze correction function can be generated (note that the personalized eye model can also be updated accordingly, and the updated eye model can be used to generate the gaze correction function). The old gaze correction function can be replaced by the new model, or alternatively, two models can be stored. Therefore, in some embodiments, multiple gaze correction functions can be stored for selection at cold start, and the system can select a best model from them based on one or more criteria for use.

[0035] Figure 5 is a high-level flow chart of a method for generating and using multiple gaze correction functions for an eye according to some embodiments. As indicated at 500, two or more gaze correction functions may be generated and stored, as shown in reference to Figure 4 430 of . At a cold start, as indicated at 510, an appropriate one of the stored gaze correction functions may be selected, for example, based on a detected position of the device on the user's head, a detected contact lens, or a detected supplemental lens in the device. As indicated at 520, the user may log in; the average eye model with the correction applied by the selected gaze correction function is used for gaze interaction. For example, the user may log in by entering a password using a gaze-based interface.

[0036] Although embodiments are generally described and illustrated with reference to one eye, there may be eye tracking cameras for both eyes, and gaze tracking may be performed for both eyes, and thus the techniques described herein may be implemented in an HMD for both the left and right eyes.

[0037] FIG. 6A to FIG. 6C Illustrated are some embodiments in which Figures 1 to 5 Example device of the method. Note that FIG. 6A to FIG. 6C The illustrated HMD 1000 is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the HMD 1000 may vary, and the location, number, type, and other features of the components of the HMD 1000 and the eye imaging system may vary. Fig. 6A A side view of an example HMD 1000 is shown, and Figure 6B and Figure 6C An alternative front view of an example HMD 1000 is shown, wherein Fig. 6A A device is shown having one lens 1030 covering both eyes, and Figure 6B A device is shown having a right lens 1030A and a left lens 1030B.

[0038] The HMD 1000 may include a lens 1030 mounted in a wearable housing or frame 1010. The HMD 1000 may be worn on the head of a user ("wearer") such that the lens is disposed in front of the wearer's eyes. In some embodiments, the HMD 1000 may implement any of various types of display technologies or display systems. For example, the HMD 1000 may include a display system that directs light that forms an image (virtual content) through one or more waveguide layers in the lens 1020; the output coupler of the waveguide (e.g., an embossed grating or a volume holographic grating) may output light toward the wearer to form an image at or near the wearer's eyes. As another example, the HMD 1000 may include a direct retinal projection system that directs light to a reflective component of a lens; the reflective lens is configured to redirect the light to form an image at the wearer's eyes.

[0039] In some embodiments, the HMD 1000 may also include one or more sensors (e.g., eye or gaze tracking sensors) that collect information about the wearer's environment (video, depth information, lighting information, etc.) and information about the wearer. The sensors may include one or more of the following, but are not limited to: one or more eye tracking cameras 1020 (e.g., infrared (IR) cameras) that capture the view of the user's eyes, one or more world-facing or PoV cameras 1050 (e.g., RGB cameras) that can capture images or videos of the real-world environment in the field of view in front of the user, and one or more ambient light sensors that capture lighting information of the environment. The cameras 1020 and 1050 may be integrated into or attached to the frame 1010. The HMD 1000 may also include one or more light sources 1080, such as LEDs or infrared point light sources, that emit light (e.g., light in the IR portion of the spectrum) toward one or more eyes of the user.

[0040] The controller 1060 of the XR system may be implemented in the HMD 1000, or alternatively may be implemented at least in part by an external device (e.g., a computing system or a portable device) that is communicatively coupled to the HMD 1000 via a wired interface or a wireless interface. The controller 1060 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), systems on chips (SOCs), CPUs, and / or other components for processing and rendering video and / or images. In some embodiments, the controller 1060 may render frames (each frame including a left image and a right image) including virtual content based at least in part on input obtained from sensors and from an eye tracking system, and may provide the frames to a display system.

[0041] The memory 1070 for the XR system may be implemented in the HMD 1000, or alternatively may be implemented at least in part by an external device (e.g., a computing system) communicatively coupled to the HMD 1000 via a wired interface or a wireless interface. The memory 1070 may be used, for example, to record video or images captured by one or more cameras 1050 integrated in or attached to the frame 1010. The memory 1070 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, etc., or low power versions of SDRAM, such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled to a circuit board to form a memory module, such as a single inline memory module (SIMM), a dual inline memory module (DIMM), etc. Alternatively, these devices may be mounted in a chip stack configuration, a package stack configuration, or a multi-chip module configuration with an integrated circuit implementing the system. In some embodiments, DRAM may be used as temporary storage for images or video for processing, but other storage options may also be used in the HMD to store processed data, such as flash memory or other "hard drive" technology. This other storage device may be separate from the externally coupled storage device mentioned below.

[0042] Although FIG. 6A to FIG. 6C Only the light source 1080 and cameras 1020 and 1050 for one eye are shown, but embodiments may include a light source 1080 and cameras 1020 and 1050 for each eye, and gaze tracking may be performed for both eyes. In addition, the light source 1080, eye tracking camera 1020, and PoV camera 1050 may be located elsewhere than shown.

[0043] like FIG. 6A to FIG. 6CThe embodiments of the HMD 1000 illustrated in the example may be used, for example, for augmented or mixed (AR) applications to provide an augmented or mixed reality view to a user. The HMD 1000 may include, for example, one or more sensors positioned on an outer surface of the HMD 1000 that collect information about the wearer's external environment (video, depth information, lighting information, etc.); the sensors may provide the collected information to a controller 1060 of the XR system. The sensors may include one or more visible light cameras 1050 (e.g., RGB cameras) that capture video of the wearer's environment, which, in some embodiments, may be used to provide the wearer with a virtual view of his or her real environment. In some embodiments, the video stream of the real environment captured by the visible light camera 1050 may be processed by the controller 1060 of the HMD 1000 to render an augmented or mixed reality frame including virtual content overlaid on a view of the real environment, and the rendered frame may be provided to a display system. In some embodiments, input from eye tracking camera 1020 may be used in a PCCR gaze tracking process performed by controller 1060 to track the gaze / pose of the user's eyes for use in rendering augmented or mixed reality content for display. Figures 1 to 5 One or more of the methods illustrated in can be implemented in an HMD to provide synthetic gaze registration for the HMD 1000.

[0044] Figure 7 is a block diagram illustrating an example device according to some embodiments, which example device may include Figures 1 to 5 The components illustrated in the figures are implemented and the methods are implemented as shown in the figures.

[0045] In some embodiments, the XR system may include a device 2000, such as a headset, a helmet, goggles, or glasses. The device 2000 may implement any of various types of display technologies. For example, the device 2000 may include: a transparent or translucent display 2060 (glasses lens) through which a user can view the real environment; and a medium integrated with the display 2060, through which light representing a virtual image is guided to the wearer's eyes to provide the wearer with an augmented reality view.

[0046] In some embodiments, the device 2000 may include a controller 2060 configured to implement the functions of the XR system and generate frames (each frame including a left image and a right image) provided to the display 2030. In some embodiments, the device 2000 may also include a memory 2070 configured to store software (code 2074) for the XR system that can be executed by the controller 2060, and data 2078 that can be used by the XR system when executed on the controller 2060. In some embodiments, the memory 2070 can also be used to store video captured by the camera 2050. In some embodiments, the device 2000 may also include one or more interfaces (e.g., Bluetooth technology interface, USB interface, etc.), and one or more interfaces (e.g., Bluetooth technology interface, USB interface, etc.) are configured to communicate with an external device (not shown) via a wired or wireless connection. In some embodiments, at least a portion of the functions described for the controller 2060 can be implemented by an external device. The external device may be or include any type of computing system or computing device, such as a desktop computer, notebook or laptop computer, tablet or tablet device, smart phone, handheld computing device, game controller, gaming system, and the like.

[0047] In various embodiments, the controller 2060 may be a single processor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight or another suitable number). The controller 2060 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture, and may be configured to execute instructions defined in the instruction set architecture. For example, in various embodiments, the controller 2060 may include a general-purpose processor or embedded processor that implements any instruction set architecture in a variety of instruction set architectures (ISAs) such as x86, PowerPC, SPARC, RISC or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each processor may implement the same ISA together, but it is not necessary. The controller 2060 may adopt any microarchitecture, including scalar, superscalar, pipeline, superpipeline, out of order, in order, speculative, non-speculative, etc., or a combination thereof. The controller 2060 may include a circuit that implements microcode technology. The controller 2060 may include one or more processing cores that are each configured to execute instructions. Controller 2060 may include one or more levels of cache, which may be of any size and any configuration (set associative, direct mapping, etc.). In some embodiments, controller 2060 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuit. Typically, a GPU may be configured to render an object to be displayed into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). The GPU may include one or more graphics processors that may execute graphics software to perform some or all of the graphics operations or hardware acceleration of certain graphics operations. In some embodiments, controller 2060 may include one or more other components for processing and rendering video and / or images, such as an image signal processor (ISP), an encoder / decoder (codec), etc.

[0048] The memory 2070 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, etc., or low power versions of SDRAM, such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled to a circuit board to form a memory module, such as a single inline memory module (SIMM), a dual inline memory module (DIMM), etc. Alternatively, these devices may be installed in a chip stack configuration, a package stack configuration, or a multi-chip module configuration with an integrated circuit implementing the system. In some embodiments, DRAM may be used as temporary storage for images or video for processing, but other storage options may also be used to store processed data, such as flash memory or other "hard disk" technology.

[0049] In some embodiments, device 2000 may include one or more sensors that collect information about the user's environment (video, depth information, lighting information, etc.). The sensors may provide the information to a controller 2060 of the XR system. In some embodiments, the sensors may include, but are not limited to, at least one visible light camera (e.g., an RGB camera) 2050, an ambient light sensor, and at least one eye tracking camera 2020. In some embodiments, device 2000 may also include one or more IR light sources; light from the light sources reflected from the eyes may be captured by eye tracking camera 2020. A gaze tracking algorithm implemented by controller 2060 may process images or videos of the eyes captured by camera 2020 to determine eye posture and gaze direction. Additionally, as Figures 1 to 5 One or more of the methods illustrated in can be implemented in device 2000 to provide synthetic gaze registration for device 2000.

[0050] In some embodiments, the device 2000 may be configured to render and display frames to provide an augmented or mixed reality (MR) view for the user based at least in part on sensor input, including input from the eye tracking camera 2020. The MR view may include rendering the user's environment, including rendering real objects in the user's environment based on video captured by one or more cameras, the one or more cameras capturing high-quality, high-resolution video of the user's environment for display. The MR view may also include virtual content (e.g., virtual objects, virtual labels for real objects, an avatar of the user, etc.) generated by the XR system and synthesized with the displayed view of the user's real environment.

[0051] Extended Reality

[0052] A real environment is an environment that a person can perceive (e.g., see, hear, feel) without using a device. For example, an office environment may include furniture such as tables, chairs, and filing cabinets; structural members such as doors, windows, and walls; and objects such as electronic devices, books, and writing instruments. A person in a real environment can perceive various aspects of the environment and may be able to interact with objects in the environment.

[0053] On the other hand, an extended reality (XR) environment is partially or fully simulated using electronic devices. For example, in an XR environment, a user can see or hear computer-generated content that partially or completely replaces the user's perception of the real environment. In addition, the user can interact with the XR environment. For example, the user's movements can be tracked, and virtual objects in the XR environment can change in response to the user's movements. As another example, a device that presents the XR environment to the user can determine that the user is moving their hand toward the virtual location of a virtual object, and can move the virtual object in response. In addition, the user's head position and / or eye gaze can be tracked, and the virtual object can move to stay in the user's line of sight.

[0054] Examples of XR include augmented reality (AR), virtual reality (VR), and mixed reality (MR). XR can be considered a spectrum of realities, where VR, on the one hand, allows users to fully immerse themselves in the real environment, replacing it with virtual content, and on the other hand, users can experience the real environment without the aid of equipment. In between are AR and MR, which mix virtual content with the real environment.

[0055] VR generally refers to a type of XR that fully immerses the user and replaces the user's real environment. For example, VR can be presented to the user using a head-mounted device (HMD), which may include a near-eye display for presenting a virtual visual environment to the user and headphones for presenting a virtual audible environment. In a VR environment, the user's movements can be tracked and cause the user's observation of the environment to change. For example, a user wearing an HMD can walk in a real environment, and the user will appear to be walking in the virtual environment they are experiencing. In addition, the user can be represented by an avatar in the virtual environment, and the HMD can use various sensors to track the user's movements to animate the user's avatar.

[0056] AR and MR refer to a class of XR that includes some mix of real environment and virtual content. For example, a user might hold a tablet computer that includes a camera that captures images of the user's real environment. The tablet computer may have a display that displays images of the real environment mixed with images of virtual objects. AR or MR may also be presented to the user through an HMD. The HMD may have an opaque display, or may use a transparent display, which allows the user to see the real environment through the display while displaying virtual content overlaid on the real environment.

[0057] In different embodiments, the method described herein can be implemented in software, hardware or a combination thereof. In addition, the order of the frames of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For those skilled in the art who benefit from the present disclosure, it is obvious that various modifications and changes can be made. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions and improvements are possible. Therefore, multiple examples can be provided for the components described as a single example in this article. The boundaries between various components, operations and data repositories are arbitrary to a certain extent, and specific operations are illustrated in the context of a specific illustrative configuration. Other allocations of functions are expected, and they may fall within the scope of the appended claims. Finally, the structure and function presented as discrete components in the example configuration may be implemented as a combined structure or component. These and other variations, modifications, additions and improvements may fall within the scope of the embodiments defined in the following claims.

Claims

1. A device, comprising: a display configured to be positioned in front of an eye; an eye-facing camera configured to capture an image of the eye; and A controller comprising one or more processors configured to: generating a multi-dimensional personalized model of the eye based on images captured by the eye-facing camera; generating a synthetic gaze feature for the personalized model at a real eye pose; estimating a synthetic eye pose from the synthetic gaze features using an average eye model; as well as A gaze correction function is generated from the real eye gesture and the synthesized eye gesture, wherein the gaze correction function represents a difference between the real eye gesture at the display and the synthesized eye gesture at the display. 2 . The apparatus of claim 1 , wherein to generate a gaze correction function from the real eye gesture and the synthesized eye gesture, a linear regression technique is applied to the real eye gesture and the synthesized eye gesture.

3. The apparatus of claim 1, wherein the gaze correction function is a two-dimensional screen space model.

4. The device of claim 1, wherein the controller is further configured to: encrypting and storing the personalized eye model; and The gaze correction function is stored in an unencrypted format. 5 . The device of claim 4 , wherein the controller is further configured to read the stored gaze correction function and apply the gaze correction function to the average eye model to perform gaze-based interaction upon cold boot of the device. The device of claim 5 , wherein the gaze-based interaction comprises entering a password to log into the device.

7. The device of claim 6, wherein the controller is further configured to, after successful login: decrypting the stored personalized eye model; and The personalized eye model is used for gaze-based interaction.

8. The device of claim 6, wherein the controller is further configured to, after successful login: detecting that a stored gaze correction function needs to be updated or supplemented; and In response to the detecting, a new gaze correction function is generated and stored.

9. The device of claim 1, wherein the controller is further configured to: generating and storing two or more gaze correction functions; and Upon cold start of the device, one of the two or more stored gaze correction functions is selected and the selected gaze correction function is applied to the average eye model to perform gaze-based interaction.

10. The device of claim 1, wherein the device is a head mounted device (HMD) of an extended reality (XR) system.

11. A method comprising: The following operations are performed by a controller including one or more processors: generating a multi-dimensional personalized model of the eye based on images captured by a camera facing the eye; generating a synthetic gaze feature for the personalized model at a real eye pose; estimating a synthetic eye pose from the synthetic gaze features using an average eye model; as well as A gaze correction function is generated from the real eye gesture and the synthesized eye gesture, wherein the gaze correction function represents a difference between the real eye gesture at the display and the synthesized eye gesture at the display.

12. The method of claim 11, wherein generating a gaze correction function from the real eye gesture and the synthesized eye gesture comprises applying a linear regression technique to the real eye gesture and the synthesized eye gesture.

13. The method according to claim 12, further comprising: Encrypting and storing the personalized eye model; as well as The gaze correction function is stored as an unencrypted screen-space model.

14. The method of claim 13, further comprising applying the gaze correction function to an average eye model to perform gaze-based interaction upon cold boot of the device, wherein the gaze-based interaction comprises entering a password to log into the device.

15. The method of claim 14, further comprising, after successful login: decrypting the stored personalized eye model; and The personalized eye model is used for gaze-based interaction.

16. The method of claim 14, further comprising, after successful login: detecting that a stored gaze correction function needs to be updated or supplemented; and In response to the detecting, a new gaze correction function is generated and stored.

17. The method according to claim 11, further comprising: generating and storing two or more gaze correction functions; as well as Upon cold start of the device, one of the two or more stored gaze correction functions is selected and the selected gaze correction function is applied to the average eye model to perform gaze-based interaction.

18. The method of claim 11, wherein the controller, the display, and the eye-facing camera are components of a head mounted device (HMD) of an extended reality (XR) system.

19. A system, comprising: A head mounted device (HMD), the head mounted device (HMD) comprising: a display configured to be positioned in front of an eye; an eye-facing camera configured to capture an image of the eye; and A controller comprising one or more processors configured to: generating a multi-dimensional personalized model of the eye based on images captured by the eye-facing camera; generating a synthetic gaze feature for the personalized model at a real eye pose; estimating a synthetic eye pose from the synthetic gaze features using an average eye model; applying a linear regression technique to the real eye gesture and the synthesized eye gesture to generate a gaze correction function, wherein the gaze correction function represents a difference between the real eye gesture at the display and the synthesized eye gesture at the display; encrypting and storing the personalized eye model; and The gaze correction function is stored as an unencrypted screen-space model.

20. The system of claim 19, wherein the controller is further configured to, upon cold boot of the HMD, read the stored gaze correction function and apply the gaze correction function to an average eye model to perform gaze-based interaction, wherein the gaze-based interaction comprises entering a password to log into the device.

Images