Methods and systems for real-time calibration of eye gaze for a head mounted display device

Abstract

A method (700) and a system (102) for real-time calibration of eye gaze for HMD device are disclosed. The method (700) comprises identifying display content rendered on the HMD device into a plurality of regions. Further, the method (700) comprises capturing eye images within the plurality of regions. The method (700) comprises estimating an optical center of eye using the captured eye images. The method (700) comprises identifying at least one salient feature in each region. The method (700) comprises estimating gaze correction parameters based on the at least one salient feature. The method (700) comprises correlating the estimated optical center of eye and the estimated gaze correction parameters with a pre-calibrated optical center of eye and pre-calibrated gaze correction parameters. Further, the method (700) comprises updating the pre-calibrated optical center of eye and the pre-calibrated gaze correction parameters based on the correlation.

Description

METHODS AND SYSTEMS FOR REAL-TIME CALIBRATION OF EYE GAZE FOR A HEAD MOUNTED DISPLAY DEVICE

[0001] The present disclosure relates to Head Mounted Display (HMD) devices, and in particular, relates to a method and a system for real-time calibration of eye gaze for a HMD device.

[0002] With advancements in technology, various devices have been developed with enhanced capabilities. Eye tracking, in particular, is utilized to facilitate a more natural and intuitive interaction with virtual objects, particularly in Virtual Reality (VR), Augmented Reality (AR), and Extended Reality (XR) environments. For instance, users can focus on a virtual object and select it by simply directing their gaze, akin to how they would interact with a physical object. By precisely tracking eye movements, a more immersive and engaging experience can be achieved. Such devices required accurate user calibration in order to function effectively and accurately for prolonged periods.

[0003] Generally, the user calibration in eye tracking refers to the process of adjusting the eye tracker to an individual's unique eye characteristics, such as the location of the fovea. Calibration is essential to ensure the accuracy of gaze tracking, as variations in eye anatomy among individuals can impact gaze precision if not properly accounted for. However, the need for frequent calibration, particularly after each device remount, can be cumbersome and lead to a tedious user experience during interactions.

[0004] Referring to Figure 1a, eye tracking is a challenging problem in head-mounted devices, especially with prolonged use which causes a slippage of the device due to many reasons like fitting, movement of the device by the user, and remount of the device. The slippage refers to the gradual displacement or movement of the device from the initial position over extended periods of use. This can significantly impact the precision and accuracy of the gaze prediction algorithm. Also remounting the device can introduce alignment discrepancies between the eye camera and the user's eye.

[0005] Further challenges related to the eye tracking include errors due to slippage and errors on remount. For example, the prolonged usage of the device causes the device to be displaced from the initial position, thereby resulting in the displacement of the eye and the device. In addition, the user calibration at the initial position will cause errors in the gaze output.

[0006] Further, the errors due to remount result in misalignment between the eye and the camera. This causes the eye appearance to change in the eye camera images. Also, the initial user calibration learned for a configuration of points will not be usable as the target points undergo significant displacement with reference to the eye position making the eye tracking results inaccurate.

[0007] Further, in Figure 1b, a graphical visualization of the calibration distribution and the remount distribution is shown. The eye-camera location with respect to the eyeball centre affects the shape of the captured eye features and hence the estimated gaze. This effect occurs due to corneal refraction and the curvature of the eyeball. The grids shown in Figure 1b represent a set of 16 x 16 simulated gaze directions for the same eye centred at 2 different positions with respect to the camera. The grids clearly depict the nonlinear change between the gaze directions between respective points during calibration and the remount phases.

[0008] Therefore, it is desirable to provide a system and a method that can eliminate one or more of the above-mentioned problems associated with the calibration process during slippage and remounting of the HMD devices.

[0009] This summary is provided to introduce a selection of concepts, in a simplified format, that are further described in the detailed description of the invention. This summary is neither intended to identify key or essential inventive concepts of the invention and nor is it intended for determining the scope of the invention.

[0010] In an embodiment, a method for real-time calibration of eye gaze for a Head Mounted Display (HMD) device is disclosed. The method comprises identifying display content rendered on the HMD device into a plurality of regions. Further, the method comprises capturing eye images within the plurality of regions. The method comprises estimating an optical centre of eye using the captured eye images. Further, the method comprises identifying at least one salient feature in at least one of the plurality of regions. Furthermore, the method comprises estimating gaze correction parameters based on the at least one salient feature. The method comprises correlating the estimated optical centre of eye and the estimated gaze correction parameters with a pre-calibrated optical centre of eye and pre-calibrated gaze correction parameters. Further, the method comprises updating the pre-calibrated optical centre of eye and the pre-calibrated gaze correction parameters based on the correlation.

[0011] In another embodiment, a system for real-time calibration of user gaze for a Head Mounted Display (HMD) device is disclosed. The system comprises at least one processor configured to identify display content rendered on the HMD device into a plurality of regions. Further, the at least one processor is configured to capture eye images within the plurality of divided regions. The at least one processor is configured to estimate an optical centre of eye using the captured eye images. The at least one processor is configured to identify at least one salient feature in at least one of the plurality of regions. Further, the at least one processor is configured to estimate gaze correction parameters based on the at least one salient feature. Furthermore, the at least one processor is configured to correlate the estimated optical centre of eye and the estimated gaze correction parameters with a pre-calibrated optical centre of eye and pre-calibrated gaze correction parameters. The at least one processor is configured to update the pre-calibrated optical centre of eye and the pre-calibrated gaze correction parameters based on the correlation.

[0012] To further clarify the advantages and features of the methods, systems, and apparatuses / devices, a more particular description of the methods, systems, and apparatuses / devices will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting of its scope. The disclosure will be described and explained with additional specificity and detail with the accompanying drawings.

[0013] These and other features, aspects, and advantages of the present invention will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

[0014] Figure 1a illustrates slippage of a Head Mounting Display (HMD) device due to movement user's head;

[0015] Figure 1b illustrates a graphical representation of calibration distribution and the remount distribution;

[0016] Figure 2 illustrates a block diagram depicting a Head Mounting Display (HMD) device 100, according to an embodiment of the present disclosure;

[0017] Figure 3 illustrates a block diagram depicting dividing display content rendered on the HMD device 100, according to an embodiment of the present disclosure;

[0018] Figure 4 illustrates a block diagram depicting identifying at least one salient feature and estimating gaze correction parameters, according to one embodiment of the present disclosure;

[0019] Figure 5 illustrates a block diagram depicting identifying at least one salient feature and estimating gaze correction parameters, according to another embodiment of the present disclosure;

[0020] Figure 6 illustrates a graphical representation of an optical axis and a visual axis, according to an embodiment of the present disclosure; and

[0021] Figure 7 illustrates a flowchart depicting a method for real-time calibration of eye gaze for the HMD device, according to an embodiment of the present disclosure.

[0022] Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the disclosure. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the disclosure so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0023] Terms used in the specification are briefly described, and the disclosure is then described in detail.

[0024] General terms currently widely used are selected as terms used in embodiments of the disclosure in consideration of their functions in the disclosure, and may be changed based on the intentions of those skilled in the art or a judicial precedent, the emergence of a new technique, or the like. In addition, in a specific case, terms arbitrarily chosen by an applicant may exist. In this case, the meanings of such terms are mentioned in detail in corresponding description portions of the disclosure. Therefore, the terms used in the disclosure need to be defined on the basis of the meanings of the terms and the contents throughout the disclosure rather than simple names of the terms.

[0025] In the disclosure, an expression “have”, “may have”, “include”, “may include”, or the like, indicates the existence of a corresponding feature (for example, a numerical value, a function, an operation, or a component such as a part), and does not exclude the existence of an additional feature.

[0026] In the disclosure, an expression “A or B”, “at least one of A and / or B”, “one or more of A and / or B”, or the like, may include all possible combinations of items enumerated together. For example, “A or B”, “at least one of A and B” or “at least one of A or B” may indicate all of 1) a case where at least one A is included, 2) a case where at least one B is included, or 3) a case where both of at least one A and at least one B are included.

[0027] Expressions “first”, “second”, and the like, used in the specification may qualify various components regardless of the sequence or importance of the components. The expression is used only to distinguish one component from another component, and does not limit the corresponding component.

[0028] In a case that any component (for example, a first component) is mentioned to be “(operatively or communicatively) coupled with / to” or “connected to” another component (for example, a second component), it is to be understood that any component may be directly coupled to another component or may be coupled to another component through still another component (for example, a third component).

[0029] An expression “configured (or set) to” used in the disclosure may be replaced by an expression “suitable for”, “having the capacity to”, “designed to”, “adapted to”, “made to” or “capable of” based on a situation. The expression “configured (or set) to” may not necessarily indicate “specifically designed to” in hardware.

[0030] Instead, an expression “a device configured to” in a certain situation may indicate that the device may “perform~” together with another device or component. For example, “a processor configured (or set) to perform A, B and C” may indicate a dedicated processor (for example, an embedded processor) that may perform the corresponding operations or a generic-purpose processor (for example, a central processing unit (CPU) or an application processor) that may perform the corresponding operations by executing one or more software programs stored in a memory device.

[0031] A term of a single number may include its plural number unless explicitly indicated otherwise in the context. It is to be understood that a term “include”, “formed of”, or the like used in the application specifies the presence of features, numerals, steps, operations, components, parts, or combinations thereof, mentioned in the specification, and does not preclude the presence or addition of one or more other features, numerals, steps, operations, components, parts, or combinations thereof.

[0032] In the embodiments, a “module” or a “~er / or” may perform at least one function or operation, and be implemented in hardware or software, or be implemented by a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “~ers / ~ors” may be integrated in at least one module to be implemented by at least one processor except for a “module” or a “~er / or” that needs to be implemented by specific hardware.

[0033] Various elements and regions in the drawings are schematically shown. Therefore, the spirit of the disclosure is not limited by relative sizes or intervals shown in the accompanying drawings.

[0034] For the purpose of promoting an understanding of the principles of the invention, reference will now be made to the embodiment illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur to one skilled in the art to which the invention relates. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skilled in the art to which this invention belongs. The system, methods, and examples provided herein are illustrative only and not intended to be limiting.

[0035] In Head Mounting Display (HMD) devices, eye calibration may be required to ensure that eye gaze interaction is properly captured by the HMD. Initially, such eye calibration may be performed by displaying an animated object to attract the user's attention. The animated object may reach a location of a calibration point. The user may have a few seconds to focus their gaze on the animated object located at the calibration point. In an embodiment, the animated object may initiate shrinking to assist the user in focusing on a centre of the calibration point. Further, the HMD device may start collecting data for the specific calibration point when the user focuses their gaze on the calibration point. The HMD device may send a notification to a client application when the data collection is completed for the calibration point. Further, the animated object may then be moved to a next location of another calibration point using an animation. The aforementioned process may be repeated for all desired calibration points. In an embodiment, if data is missing, low in accuracy, or has low precision for one or more calibration points, then the calibration process is repeated. Once the calibration result is satisfactory, the calibration procedure is concluded. However, the HMD device may tend to undergo slippage due to various factors during prolonged usage of the HMD device. Due to such slippage, eye calibration may be required to be performed again in order to ensure that the HMD device efficiently operates.

[0036] In an embodiment of the present disclosure, the HMD device may be deployed with a system configured to perform real-time calibration of eye gaze for the HMD device. The system may be configured to continuously update an optical centre of eye and a visual axis corresponding to the user using the HMD device. Therefore, the system ensures that even during slippage the HMD device is re-calibrated and performs its operation optimally.

[0037] Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

[0038] Figure 2 illustrates a block diagram depicting a HMD device 100, according to an embodiment of the present disclosure. As explained earlier, the HMD device, such as 100, may tend to undergo slippage during a prolonged usage of such device. The slippage may occur due to various factors, such as fitting of the HMD device on user's head, movement of the HMD device by the user, and remounting of the HMD. Owing to such slippage, the HMD device 100 may be required to be calibrated / re-calibrated to ensure that the HMD device 100 functions accurately and effectively. In the illustrated embodiment, the HMD device 100 may include, but is not limited to a system 102 and an eye tracker 104.

[0039] The system 102 may be configured to automatically calibrate / re-calibrate the HMD device 100. Further, the system 102 may also be configured to perform calibration / re-calibrations at predefined time intervals, over the prolonged usage of the HMD device 100. The system 102 may be configured to operate the eye tracker 104 to capture eye images of the user during calibration / re-calibrations of the HMD device 100. In an exemplary embodiment, the eye tracker 104 may be embodied as a camera or any other device capable of capturing images, without departing from the scope of the present disclosure. Operational and constructional aspects of the system 102 are explained in detail in the subsequent paragraphs of the present disclosure.

[0040] Referring to Figure 2, the system 102 may include a processor 202, memory 204, module(s) 206, and data 208. The module(s) 206 and the memory 204 are coupled to the processor 202. The processor 202 can be a single processing unit or a number of units, all of which could include multiple computing units. The processor 202 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor 202 is configured to fetch and execute computer-readable instructions and data stored in the memory 204.

[0041] The processor 202 may include one or a plurality of processors. At this time, one or a plurality of processors may be a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning.

[0042] Here, being provided through learning means that, by applying a learning technique to a plurality of learning data, a predefined operating rule or AI model of a desired characteristic is made. The learning may be performed in a device itself in which AI according to an embodiment is performed, and / or may be implemented through a separate server / system. The AI model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[0043] In case the method according to an embodiment of the disclosure includes a plurality of operations, the plurality of operations may be performed by one processor, or performed by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by the method according to an embodiment, all of the first operation, the second operation, and the third operation may be performed by a first processor, or the first operation and the second operation may be performed by the first processor (e.g., a generic-purpose processor), and the third operation may be performed by a second processor (e.g., an artificial intelligence-dedicated processor).

[0044] The processor 202 may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including a plurality of cores (e.g., multicores of the same kind or multicores of different kinds). In case the processor 202 is implemented as multicore processors, each of the plurality of cores included in the multicore processors may include internal memory of the processor such as cache memory, on-chip memory, etc., and common cache shared by the plurality of cores may be included in the multicore processors. Also, each of the plurality of cores (or some of the plurality of cores) included in the multicore processors may independently read a program instruction for implementing the method according to an embodiment of the disclosure and perform the instruction, or the plurality of entire cores (or some of the cores) may be linked with one another, and read a program instruction for implementing the method according to an embodiment of the disclosure and perform the instruction.

[0045] In case the method according to an embodiment of the disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in the multicore processors, or they may be performed by the plurality of cores. For example, when the first operation, the second operation, and the third operation are performed by the method according to an embodiment, all of the first operation, the second operation, and the third operation may be performed by a first core included in the multicore processors, or the first operation and the second operation may be performed by the first core included in the multicore processors, and the third operation may be performed by a second core included in the multicore processors.

[0046] In the embodiments of the disclosure, the processor 202 may mean a system on chip (SoC) wherein at least one processor and other electronic components are integrated, a single core processor, a multicore processor, or a core included in the single core processor or the multicore processor. Also, here, the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, etc., but the embodiments of the disclosure are not limited thereto.

[0047] The memory 204 may include any non-transitory computer-readable medium known in the art including, for example, volatile memory, such as static random-access memory (SRAM) and dynamic random-access memory (DRAM), and / or non-volatile memory, such as read-only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes.

[0048] The module(s) 206, amongst other things, include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement data types. The module(s) 206 may also be implemented as, signal processor(s), state machine(s), logic circuitries, and / or any other device or component that manipulate signals based on operational instructions.

[0049] Further, the module(s) 206 may be implemented in hardware, instructions executed by at least one processing unit, for e.g., the processor 202, or by a combination thereof. The processing unit may comprise a computer, a processor, a state machine, a logic array and / or any other suitable devices capable of processing instructions. The processing unit may be a general-purpose processor which executes instructions to cause the general-purpose processor to perform operations or, the processing unit may be dedicated to performing the required functions. In some example embodiments, the module(s) 206 may be machine-readable instructions (software, such as web-application, mobile application, program, etc.) which, when executed by a processor / processing unit, perform any of the described functionalities.

[0050] In an implementation, the module(s) 206 may include a discrete binning module 210, an eye tracking module 212, an incremental calibration module 216, and a gaze correction module 218. The discrete binning module 210, the eye tracking module 212, the incremental calibration module 216, and the gaze correction module 218 are in communication with each other. At least one of the modules 206 may be implemented through an AI model. A function associated with AI may be performed through the non-volatile memory, the volatile memory, and the processor. The data 208 serves, amongst other things, as a repository for storing data processed, received, and generated by one or more of the modules 212.

[0051] In an embodiment of the present disclosure, the module(s) 206 may be implemented as part of the processor 202. In another embodiment of the present disclosure, the module(s) 206 may be external to the processor 202. In yet another embodiment of the present disclosure, the module(s) 206 may be part of the memory 204. In another embodiment of the present disclosure, the module(s) 206 may be part of hardware, separate from the processor 202.

[0052] Figure 3 illustrates a block diagram depicting dividing display content rendered on the HMD device 100, according to an embodiment of the present disclosure. The processor 202 of the system 102 may be configured to divide display content rendered on the HMD device 100 into a plurality of regions 302-1, 302-2...302-n. In the illustrated embodiment, the discrete binning module 210 may be configured to divide display content rendered on the HMD device 100 into the plurality of regions 302-1, 302-2...302-n. The plurality of regions 302-1, 302-2...302-n may interchangeably be referred to as bins, without departing from the scope of the present disclosure. Further, the plurality of regions 302-1, 302-2...302-n may interchangeably be referred to as the regions 302. In an embodiment, the discrete binning module 302 may be configured to initialize queues corresponding to each divided region 302.

[0053] Further, the processor 202 may be configured to capture eye images 304-1, 304-2...304-n corresponding to eye gaze interactions within one or more divided regions 302. In the illustrated embodiment, the eye tracking module 212 may be configured to capture a plurality of eye images 304-1, 304-2...304-n corresponding to the eye gaze interactions within each divided region 302. In an embodiment, the plurality of eye images 304-1, 304-2...304-n may interchangeably be referred to as the eye images 304 or images 304, without departing from the scope of the present disclosure. Each image 304 may be indicative of a position of a pupil and an orientation of the pupil corresponding to the eye gaze interactions. In an embodiment, the eye tracking module 212 may be configured to operate the eye tracker 104 to capture the plurality of eye images 304 corresponding to the eye gaze interactions within each divided region 302.

[0054] In an embodiment, the discrete binning module 210 may be configured to identify the divided regions 302 based on each captured eye image 304. Further, the discrete binning module 210 may be configured to map the captured eye images 304 corresponding to the eye gaze interactions to one of the initialized queues of the respective divided region 302. In particular, the plurality of eye images 304 of the user may be captured corresponding to each divided region 302 and subsequently stored in the memory for further processing. In particular, the discrete binning module 210 may be configured to accumulate the gaze information, i.e., the captured eye images 304, obtained from the eye tracking module 212 and store the gaze information into discrete bins spread over the entire spatial display size of the HMD device 100. The discrete binning module 210 may help in updating the eye parameters of an eye model typically used in geometric-based eye trackers. The discrete binning module 210 may help in reducing the gaze errors caused due to device slippage over the prolonged period.

[0055] The processor 202 may be configured to estimate an optical centre of eye using the captured eye images 304. In an embodiment, the eye tracking module 212 may be configured to estimate the optical centre of eye using the captured eye images 304 mapped to the initialized queues corresponding to the divided regions 302. In an embodiment, the eye tracking module 212 may be configured to predict a geometrical model of eye based on the plurality of eye images 304, and subsequently estimate the optical centre of eye using the geometrical model of eye, without departing from the scope of the present disclosure. In one exemplary embodiment, a pupil-based method may be implemented to estimate the optical centre of eye. In the pupil-based method, pupils of eyes may be unprojected from a two-dimensional (2D) ellipse on an image space to a three-dimensional (3D) circle in a world space which provides a ray on which the eye centre may lie on. Hence, a ray candidate per frame may be obtained. Thereafter, the optical centre may be estimated by combining (intersecting) multiple such candidates (rays). In various other embodiments, the eye tracking module 212 may be configured to implemented different methods for estimating the optical centre of eye, without departing from the scope of the present disclosure.

[0056] The processor 202 may be configured to determine whether a predefined time interval is lapsed upon estimating the optical centre of eye. In an embodiment, the discrete binning module 210 may be configured to determine whether the predefined time interval is lapsed upon estimating the optical centre of eye. If the predefined time interval has lapsed, then the discrete binning module 210 may be configured to remove the captured eye images 304 from each of the initialized queues corresponding to each divided region 302.

[0057] Further, the discrete binning module 210 may be configured to re-capture the plurality of eye images 304 corresponding to the eye gaze interactions within each divided region 302. The discrete binning module 210 may be configured to identify the divided region 302 based on each re-captured eye image 304. Further, the discrete binning module 210 may be configured to re-map the captured eye images 304 corresponding to the eye gaze interactions to one of the initialized queues of the respective divided region 302. Furthermore, the eye tracking module 212 may be configured to re-estimate the optical centre of eye using the re-captured eye images 304 mapped to the initialized queues corresponding to the divided regions 302.

[0058] In an embodiment, the processor 202 may be configured to identify at least one salient feature in at least one of the plurality of regions 302. Further, the processor 202 may be configured to estimate gaze correction parameters using the eye gaze interactions with the at least one salient feature.

[0059] Figure 4 illustrates a block diagram depicting identifying at least one salient feature and estimating gaze correction parameters, according to one embodiment of the present disclosure. Referring to Figure 4, in one embodiment, the incremental calibration module 216 may be configured to incrementally calibrate eye tracking of the user's eye during the operation of the HMD device 100. The incremental calibration module 216 may include, but is not limited to, a feature identification module 402 and a gaze estimation module 404. The feature identification module 402 may be configured to identify one or more User Interface (UI) elements 406 rendered on the HMD device 100. The identified UI elements 406 may be representative of calibration points.

[0060] In such an embodiment, the eye tracking module 212 may be configured to track the eye gaze interactions with the at least one salient feature, i.e., the UI elements 406. In one or more examples, the UI elements 406 may be embodied as notifications, icons, navigation bars, tabs, images, labels, etc. Further, the gaze estimation module 404 may be configured to determine a gaze prediction value based on the tracking of the eye gaze interactions. In an embodiment, the gaze prediction value may indicate a value associated with a direction vector originating from the eyes of the user or a midpoint between the eye(s) of the user, or any other defined point where user's gaze is directed. In one or more embodiments, the gaze prediction value may indicate a value associated with a point where the user is looking at in either a three-dimensional space or a two-dimensional space. In an embodiment, the gaze estimation module 404 may be configured to continuously determine the gaze prediction value corresponding to one of the UI elements 406 until the gaze interaction is tracked with respect to such UI element 406. Once the gaze interaction is interrupted for one of the UI elements 406, then the gaze estimation module 404 may halt the operation of determining the gaze prediction value for such UI element 406.

[0061] The gaze estimation module 404 may be configured to map the determined gaze prediction value with a position of the respective salient feature, i.e., the UI elements 406. The incremental calibration module 216 may have a predefined position of each UI element 406 being displayed on the HMD device 100. The UI elements 406 may be rendered in specific locations in the displayed content on the HMD device 100, ensuring that their positions are accurately known and consistent throughout the interaction. The spatial arrangement of the UI elements 406 may be dynamically maintained in real-time, allowing for precise alignment with the user's field of view and the device's tracking capabilities. As a result, the system 102 may reliably manage user interactions with the UI elements 406 during calibration.

[0062] In an example, whenever the eye tracker 104 may predict a gaze positioned near a centre of an UI element. In such an example, calibration may be performed using the location of the UI element as the calibration point. Until the user is looking at a particular UI element, the gaze estimation module 404 may keep recording the gaze prediction value and keep mapping it to the position of the respective UI element. In such a manner, the new mapping between the gaze prediction and the position of the UI element improves the gaze degradation which might have happened due to remount or slippage of the HMD device 100. If the UI elements are not displayed, the system 102 may utilize human visual saliency, explained in subsequent paragraphs.

[0063] Figure 5 illustrates a block diagram depicting identifying at least one salient feature and estimating gaze correction parameters, according to another embodiment of the present disclosure. Referring to Figure 5, in another embodiment, the incremental calibration module 216 may be configured to incrementally calibrate eye tracking of the user's eye during the operation of the HMD device 100. The incremental calibration module 216 may include, but is not limited to, the gaze estimation module 404 and a salient object detection module 502. In such an embodiment, the salient object detection module 502 may be configured to predict one or more saliency rings 504 in a scene 506 of the display content rendered on the HMD device 100. Further, the salient object detection module 502 may be configured to identify a region in each predicted saliency ring 504. The identified region may be representative of a calibration point. In one or more examples, movement of cars and the swaying of trees in the scene of the displayed content may be predicted as the salient rings 504.

[0064] In such an embodiment, the salient object detection module 502 may be deployed with one or more deep learning-based saliency prediction models configured to predict one or more saliency rings 504 in the scene of the display content rendered on the HMD device 100. The saliency rings 504 may be predicted in the regions, that have high human attention, of the displayed content. Referring to Figure 5, each region may have multiple saliency rings 504 having an inner ring as a high saliency which gradually decreases towards an outer ring. In an embodiment, a centre of the inner ring may be assumed to be a POR (point of regard) 506 where the user is focused at and identified as the calibration point for incremental updating of user calibration parameters.

[0065] Referring to Figure 5, the eye tracking module 404 may be configured to track the eye gaze interactions, such as 508, with the at least one salient feature, i.e., the salient rings 504. Further, the gaze estimation module 404 may be configured to determine a gaze prediction value based on the tracking of the eye gaze interactions 508. The gaze estimation module 404 may be configured to map the determined gaze prediction value with a position of the respective salient feature, i.e., the saliency rings 504. In an embodiment, the gaze estimation module 404 may be configured to continuously determine the gaze prediction value corresponding to one of the saliency rings 504 until the gaze interaction 508 is tracked with respect to such saliency rings 504. Once the gaze interaction 508 is interrupted for one of the saliency rings 504, then the gaze estimation module 404 may halt the operation of determining the gaze prediction value for such a saliency ring 504.

[0066] Further, the gaze estimation module 404 may be configured to estimate the gaze correction parameters based on the mapping. The gaze correction module 218 may be configured to correlate the estimated optical centre of eye and the estimated gaze correction parameters with a pre-calibrated optical centre of eye and pre-calibrated gaze correction parameters. Further, the gaze correction module 218 may be configured to update the pre-calibrated optical centre of eye and the pre-calibrated gaze correction parameters based on the correlation.

[0067] Figure 6 illustrates a graphical representation of an optical axis and a visual axis, according to an embodiment of the present disclosure. Referring to Figure 6, in an embodiment, the gaze correction module 218 may be configured to determine an optical axis based on the updated optical centre of eye. The optical axis may be denoted as x1, x2,1. The gaze correction module may be deployed with a gaze refiner which is modelled as a multivariate polynomial regression of 2 variables of degree 2. The gaze,1 refiner may be configured to receive the optical axis (x1, x2,1) as an input. Further, the gaze correction module 218 may be configured to estimate a visual axis based on the updated optical axis (x1, x2,1) and the updated gaze correction parameters. In an embodiment, the gaze refiner may be configured to estimate the visual axis based on the updated optical axis (x1, x2,1). The visual axis may be denoted as y1,y2,1 and estimated using the following equations (1) and (2):

[0068] y1= (a1*1) + (a2*x1) + (a3*x2) + (a4*x12) + (a5*x22) + (a6*x1*x2)......(1)

[0069] y2= (b1*1) + (b2*x1) + (b3*x2) + (b4*x12) + (b5*x22) + (b6*x1*x2)......(2)

[0070] Where, a1, a2, a3,a4,a5,a6,...anand b1, b2, b3,b4,b5,b6,...bnare gaze correction parameters estimated by the gaze estimation module. The equations (1) and (2) may represent a function provided to map the updated optical axis (x1, x2,1) to the visual axis (y1,y2,1). The function may be a polynomial regression of degree 2, where the gaze correction parameters are referred to the parameters of the polynomial regression function. In one or more embodiments, the linear regression or polynomial regression of different degrees may be implemented for mapping the updated optical axis (x1, x2,1) to the visual axis (y1,y2,1).

[0071] As explained earlier, the system 102 may be configured to perform real-time recalibration of the HMD device 100 at the predefined time intervals, over the prolonged usage of the HMD device 100. During such recalibrations, the eye tracking module 212 may be configured to re-capture the plurality of eye images 304 corresponding to the eye interactions within each divided region 302 initialized by the discrete binning module 210. Based on the re-captured eye images 304, the eye tracking module 212 may re-estimate the optical centre of eye. Such re-capturing of eye images 304 and the re-estimation of the optical centre of eye may be performed at the predefined time intervals during the real-time usage of the HMD device 100.

[0072] Further, during the real-time recalibration, an updated optical axis (x1, x2,1) may be determined based on the re-estimated optical centre of eye. Furthermore, the gaze correction parameters may be re-estimated by the gaze estimation module 404 and subsequently, the gaze correction module 218 may be configured to re-update the pre-calibrated optical centre of eye, i.e., previously calibrated optical centre, and the pre-calibrated gaze correction parameters, i.e., previously calibrated gaze correction parameters. Based on the re-estimated gaze correction parameters and the re-estimated optical centre, a re-estimated visual axis may be obtained using the following equations (3) and (4):

[0073] y1= ((a1+△a1) *1) + ((a2+△a2)*x1) + ((a3+△a3)*x2) + ((a4+△a4)*x12) + ((a5+△a5)*x22) + ((a6+△a6)*x1*x2)......(3)

[0074] y2= ((b1+△b1) *1) + ((b2+△b2)*x1) + ((b3+△b3)*x2) + ((b4+△b4)*x12) + ((b5+△b5)*x22) + ((b6+△b6)*x1*x2)......(4)

[0075] where, a1, a2, a3, a4, a5, a6,...anand b1, b2, b3, b4, b5, b6,...bnare previously estimated gaze correction parameters, and △a1, △a2, △a3, △a4, △a5, △a6,...△an, and △b1, △b2, △b3, △b4, △b5, △b6,...△bnindicates difference between previously estimated gaze correction parameters and the re-estimated gaze correction parameters.

[0076] Figure 7 illustrates a flowchart depicting a method 700 for real-time calibration of eye gaze for the HMD device 100, according to an embodiment of the present disclosure. The method 700 may be implemented in the system 102 using components thereof, as described above. In an embodiment, the method 700 may be executed by the processor 202 of the system 102, as described above. Further, for the sake of brevity, details of the present disclosure that are explained in detail in the description of Figures 1-6 are not explained in detail in the description of Figure 7.

[0077] At step 702, the method 700 may include dividing display content rendered on the HMD device 100 into the plurality of regions 302. At step 704, the method 700 may include capturing the eye images 304 corresponding to the eye gaze interactions within the plurality of regions 302. At step 706, the method 700 may include estimating the optical centre of eye using the captured eye images 304.

[0078] In an embodiment, the method 700 may include initializing the queues corresponding to each divided region 302. Further, the method 700 may include capturing the plurality of user eye images 304 corresponding to the eye gaze interactions within each divided region 302. Each image 304 may be indicative of the position of the pupil and the orientation of the pupil corresponding to the eye gaze interactions. The method 700 may include identifying the divided region based on each captured eye image 304. Further, the method 700 may include mapping the captured eye images 304 corresponding to the eye gaze interactions to one of the initialized queues of the respective divided region 302. Furthermore, the method 700 may include estimating the optical centre of eye using the captured eye images 304 mapped to the initialized queues corresponding to the divided regions 302.

[0079] In an embodiment, the method 700 may include determining whether the predefined time interval is lapsed upon estimating the optical centre of eye. The method 700 may include removing the captured eye images 304 from each of the initialized queues corresponding to each divided region 302 if the predefined time interval is lapsed. Further, the method 700 may include re-capturing the plurality of eye images 304 corresponding to the eye gaze interactions within each divided region 302. The method 700 may include identifying the divided region 302 based on each re-captured eye image 304. Further, the method 700 may include re-mapping the captured eye images 304 corresponding to the eye gaze interactions to one of the initialized queues of the respective divided region 302. Furthermore, the method 700 may include re-estimating the optical centre of eye using the captured eye images 304 mapped to the initialized queues corresponding to the divided regions 302.

[0080] At step 708, the method 700 may include identifying at least one salient feature in at least one of the plurality of regions 302. In one embodiment, the method 700 may include identifying the one or more User Interface (UI) elements 406 rendered on the HMD device 100. The identified UI elements 406 may be representative of the calibration points. In another embodiment, the method 700 may include predicting the one or more saliency rings 504 in the scene 506 of the display content rendered on the HMD device 100. In such an embodiment, the method 700 may include identifying the region in each predicted saliency ring 504. The identified region may be representative of the calibration point.

[0081] At step 710, the method 700 may include estimating the gaze correction parameters using the eye gaze interactions with the at least one salient feature. In an embodiment, the method 700 may include tracking the eye gaze interactions with the at least one salient feature. Further, the method 700 may include determining the gaze prediction value based on the tracking of the eye gaze interactions. Furthermore, the method 700 may include mapping the determined gaze prediction value with the position of the respective salient feature. The method 700 may include estimating the gaze correction parameters based on the mapping.

[0082] At step 712, the method 700 may include correlating the estimated optical centre of eye and the estimated gaze correction parameters with a pre-calibrated optical centre of eye and pre-calibrated gaze correction parameters. At step 714, the method 700 may include updating the pre-calibrated optical centre of eye and the pre-calibrated gaze correction parameters based on the correlation.

[0083] In an embodiment, the method 700 may include determining the optical axis based on the updated optical centre of eye. Further, the method 700 may include estimating the visual axis based on the updated optical axis and the updated gaze correction parameters.

[0084] The system and method of the present disclosure can be deployed for eye tracking in various applications as explained in the subsequent paragraphs. One common use case for eye tracking in Virtual Reality (VR) / Augmented Reality (AR) is to provide a natural way for users to navigate through virtual spaces. By tracking user's gaze, viewpoint is adjusted to follow the user's eyes, making it easier for them to explore the virtual environment. Another advantage of eye tracking is its ability to provide contextual feedback to users. For example, when a user looks at a specific object in the virtual environment, the eye tracking can display additional information or instructions about its functionality. The eye tracking helps users better understand how to interact with the virtual world and makes it more engaging.

[0085] Further, the eye tracking is used to improve the ways of teaching and learning. The eye tracking provides personalized feedback and insights into student's understanding, by tracking the movement and focus of student's eyes. The educators may identify areas where guidance is needed, and tailor their teaching strategies accordingly. In an example, the eye tracking is used in language learning by analysing student's eye movements and the gaze patterns, teachers can identify areas where the students struggle with certain vocabulary or grammar concepts. The information can then be used to create personalized learning materials and provide targeted support, ultimately improving student's overall language skills. In an example, the eye tracking is used in cognitive training by tracking students eye movements and gaze patterns, the educators can identify areas where they may be struggling with cognitive processes such as attention, memory, or problem-solving. The information may be used to create customized training programs and exercises that specifically target these areas, leading to improved cognitive function and overall performance.

[0086] Furthermore, the eye tracking enhances the overall engagement and immersion of users in VR and AR games by providing more natural and intuitive ways to interact with the virtual environment. In an example, the eye tracking is used in First-person shooters (FPS) by analysing player's eye movement and gaze patterns. The developers may create a more immersive and engaging gameplay experience. For example, when a player looks at a specific enemy, the eye tracking may automatically aim their weapon at that target, providing a more realistic and immersive gaming experience. In an example, the eye tracking is used in virtual reality experiences by tracking player's eye movements and gaze patterns. The developers can create more immersive and engaging virtual environments that truly reflect the real world. For example, when a player looks at a specific object in the virtual environment, the eye tracking can automatically adjust the viewpoint to follow the player's eyes, making it easier for them to see and explore the virtual environment. In this manner, the eye tracking enhances the overall engagement and immersion of users in VR and AR games by providing natural and intuitive interactions.

[0087] As would be gathered, the present disclosure offers a comprehensive approach for the calibration of eye gaze for the HMD device 100. The system 102 and the method 700 of the present disclosure efficiently and effectively perform re-calibration in real-time and in regular intervals during the operation of the HMD device 100. As explained earlier, the system 102 and the method 700 may divide the display content into the plurality of regions 302 in order to capture eye images 304 corresponding to each region 302. Further, the system 102 and the method 700 may estimate the optical centre of eye based on the captured eye images 304. The system 102 and the method 700 may perform the aforesaid operations in real-time and in regular intervals to ensure that the optical centre of eye is updated even if the HMD device 100 undergoes slippage. In particular, in order to counter the remounting and slippage problem, three-dimensional (3D) eye centres are required to be re-estimated continually at certain intervals of time. Therefore, the system 102 and the method 700 consider pupil-based geometric models and the re-estimation of the eyeball centres is performed to capture variations in the position of the pupil and the orientation of the pupil.

[0088] Further, in one embodiment, the system 102 and the method 700 may consider the UI elements 406 as the calibration point in order to estimate the gaze correction parameters. In another embodiment, the system 102 and the method 700 may consider the saliency rings 504 as the calibration points in order to estimate the gaze correction parameters. Such estimation of the gaze correction parameters is performed in real-time and in regular intervals to ensure that the gaze correction parameters are updated to counter the remounting and slippage condition of the HMD device. Further, the system 102 and the method 700 may estimate the gaze correction parameters without a requirement of conscious input from the user to perform such calibration. In particular, the system 102 and the method 700 may automatically track the gaze interaction with the UI elements or the saliency rings in the displayed content and subsequently, estimate the gaze correction parameters. This reduces the overall cognitive load of the user while using the HMD device 100 and also enhances the overall user experience.

[0089] Therefore, the system 102 and the method 700 may frequently update the optical centre and the gaze correction parameters to ensure that the optimal estimation of the visual axis for the operation of the HMD device 100. This helps ensure accurate eye tracking, maintain precise measurements, and enhance the overall accuracy of the system 102. In this manner, the method 700 and the system 102 provide substantially improved gaze tracking accuracy even when the HMD device 100 undergoes slippage. Also, the system 102 and the method 700 avoid user calibration every time a user removes and wears the HMD device 100.

[0090] The methods according to the one or more embodiments of the disclosure described above may be implemented only by software upgrade or hardware upgrade of the conventional electronic apparatus.

[0091] In addition, the one or more embodiments of the disclosure described above may be performed through an embedded server disposed in the electronic apparatus, or a server disposed outside the electronic apparatus.

[0092] According to an embodiment of the disclosure, the one or more embodiments described above may be implemented in software including an instruction stored in a machine-readable storage medium (for example, a computer-readable storage medium). A machine may be an apparatus that invokes the stored instruction from the storage medium, may be operated based on the invoked instruction, and may include the electronic apparatus (e.g., electronic apparatus A) according to the disclosed embodiments. In a case that the instruction is executed by the processor, the processor may perform a function corresponding to the instruction directly or by using other components under control of the processor. The instruction may include a code provided or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a term “non-transitory” may only indicate that the storage medium is tangible without including a signal, and does not distinguish whether data are semi-permanently or temporarily stored in the storage medium.

[0093] In addition, according to an embodiment of the disclosure, the methods in the one or more embodiments described above may be provided by being included in a computer program product. The computer program product may be traded as a product between a seller and a purchaser. The computer program product may be distributed in a form of the machine-readable storage medium (for example, a compact disc read only memory (CD-ROM)), or may be distributed online through an application store (for example, PlayStoreTM). In case of the online distribution, at least some of the computer program products may be at least temporarily stored in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server, or be temporarily generated.

[0094] In addition, each of the components (for example, modules or programs) according to the one or more embodiments described above may include one entity or a plurality of entities, and some of the corresponding sub-components described above may be omitted or other sub-components may be further included in the one or more embodiments. Alternatively or additionally, some of the components (e.g., modules or programs) may be integrated into one entity, and may perform functions performed by the respective corresponding components before being integrated in the same or similar manner. Operations performed by the modules, the programs, or other components according to the one or more embodiments may be executed in a sequential manner, a parallel manner, an iterative manner, or a heuristic manner, at least some of the operations may be performed in a different order or be omitted, or other operations may be added.

[0095] Although the embodiments are shown and described in the disclosure as above, the disclosure is not limited to the above-mentioned specific embodiments, and may be variously modified by those skilled in the art to which the disclosure pertains without departing from the gist of the disclosure of the accompanying claims. These modifications should also be understood to fall within the scope and spirit of the disclosure.

[0096] While specific language has been used to describe the present disclosure, any limitations arising on account thereto, are not intended. As would be apparent to a person in the art, various working modifications may be made to the method in order to implement the inventive concept as taught herein. The drawings and the foregoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment.

Claims

1.A method (700) for real-time calibration of eye gaze for a Head Mounted Display (HMD) device, the method (700) comprising:identifying display content rendered on the HMD device into a plurality of regions;capturing eye images within the plurality of regions;estimating an optical centre of eye using the captured eye images;identifying at least one salient feature in at least one of the plurality of regions;estimating gaze correction parameters based on the at least one salient feature;correlating the estimated optical centre of eye and the estimated gaze correction parameters with a pre-calibrated optical centre of eye and pre-calibrated gaze correction parameters; andupdating the pre-calibrated optical centre of eye and the pre-calibrated gaze correction parameters based on the correlation.2.The method (700) as claimed in claim 1, further comprising:initializing queues corresponding to each of the plurality of regions; andcapturing a plurality of user eye images corresponding to eye gaze interactions within each of the plurality of regions, wherein each image is indicative of a position of a pupil and an orientation of the pupil corresponding to the eye gaze interactions.3.The method (700) as claimed in claim 2, further comprising:identifying the plurality of regions based on each captured eye image;mapping the captured eye images corresponding to the eye gaze interactions to one of the initialized queues of the respective plurality of regions; andestimating the optical centre of eye using the captured eye images mapped to the initialized queues corresponding to the plurality of regions.4.The method (700) as claimed in claim 3, further comprising:determining whether a predefined time interval is lapsed upon estimating the optical centre of eye.5.The method (700) as claimed in claim 4, further comprising:removing the captured eye images from each of the initialized queues corresponding to each of the plurality of regions if the predefined time interval is lapsed; andre-capturing the plurality of eye images corresponding to the eye gaze interactions within each of the plurality of regions.6.The method (700) as claimed in claim 5, further comprising:identifying the plurality of regions based on each re-captured eye image;re-mapping the captured eye images corresponding to the eye gaze interactions to one of the initialized queues of the respective plurality of regions; andre-estimating the optical centre of eye using the captured eye images mapped to the initialized queues corresponding to the plurality of regions.7.The method (700) as claimed in claim 1, wherein identifying the at least one salient feature in the at least one of the plurality of regions comprises:identifying one or more User Interface (UI) elements rendered on the HMD device, wherein the identified UI elements are representative of calibration points.8.The method (700) as claimed in claim 1, wherein identifying the at least one salient feature in the at least one of the plurality of regions comprises:predicting one or more saliency rings in a scene of the display content rendered on the HMD device; andidentifying a region in each predicted saliency ring, wherein the identified region is representative of a calibration point.9.The method (700) as claimed in claim 1, wherein estimating the gaze correction parameters using the eye gaze interactions comprises:tracking the eye gaze interactions with the at least one salient feature;determining a gaze prediction value based on the tracking of the eye gaze interactions;mapping the determined gaze prediction value with a position of the respective salient feature; andestimating the gaze correction parameters based on the mapping.10.The method (700) as claimed in claim 1, further comprising:determining an optical axis based on the updated optical centre of eye; andestimating a visual axis based on the updated optical axis and the updated gaze correction parameters.11.A Head Mounted Display (HMD) device for real-time calibration of user gaze, the HMD device comprising:at least one processor configured to:identify display content rendered on the HMD device into a plurality of regions;capture eye images within the plurality of regions;estimate an optical centre of eye using the captured eye images;identify at least one salient feature in at least one of the plurality of regions;estimate gaze correction parameters based on the at least one salient feature;correlate the estimated optical centre of eye and the estimated gaze correction parameters with a pre-calibrated optical centre of eye and pre-calibrated gaze correction parameters; andupdate the pre-calibrated optical centre of eye and the pre-calibrated gaze correction parameters based on the correlation.12.The HMD device as claimed in claim 11, wherein the at least one processor is configured to:initialize queues corresponding to each of the plurality of regions;capture a plurality of eye images corresponding to eye gaze interactions within each of the plurality of regions, wherein each image is indicative of a position of a pupil and an orientation of the pupil corresponding to the eye gaze interactions.13.The HMD device as claimed in claim 12, wherein the at least one processor is configured to:identify the plurality of regions based on each captured eye image;map the captured eye images corresponding to the eye gaze interactions to one of the initialized queues of the respective plurality of regions; andestimate the optical centre of eye using the captured eye images mapped to the initialized queues corresponding to the plurality of regions.14.The HMD device as claimed in claim 13, wherein the at least one processor is configured to:determine whether a predefined time interval is lapsed upon estimating the optical centre of eye.15.A non-transitory computer readable medium, having instructions stored therein, which when executed by a processor of a HMD device, cause the processor to execute a method comprising:identifying display content rendered on the HMD device into a plurality of regions;capturing eye images within the plurality of regions;estimating an optical centre of eye using the captured eye images;identifying at least one salient feature in at least one of the plurality of regions;estimating gaze correction parameters based on the at least one salient feature;correlating the estimated optical centre of eye and the estimated gaze correction parameters with a pre-calibrated optical centre of eye and pre-calibrated gaze correction parameters; andupdating the pre-calibrated optical centre of eye and the pre-calibrated gaze correction parameters based on the correlation.

Images