Content rendering method and apparatus

Abstract

According to an embodiment of the disclosure, a media content rendering method may be provided. The method may include determining a plurality of eye attributes of a user, based on eye position for each of a left eye and a right eye. The method may include identifying eye movement for each of the left eye and right eye, based on the plurality of eye attributes. The method may include determining a constraint for a movement of each of the left eye and right eye, based on the eye movement. The method may include determining a binocular field-of-view (FOV) of the user, based on the constraint for each of the left eye and right eye. The method may include determining transformation values, based on the binocular FOV of the user. The method may include providing a transformed media content based on the transformation values to render the transformed media content.

Description

CONTENT RENDERING METHOD AND APPARATUS

[0001] The present disclosure generally relates to content rendering techniques. Particularly, the present disclosure relates to media content rendering techniques for a user having an eye alignment disorder.

[0002] Eye alignment and eye teaming disorders are caused when the one eye of a person fails or struggles to maintain an alignment with the other eye. Strabismus or crossed-eyed condition is a commonly used term for such disorders. The condition arises due convergence of the lines of sight of the two eyes, either insufficiently or in excess. The line of sight is the direction of sight of an eye. A normal eyed person is able to view an image clearly because the line of sight of both eyes converge precisely thereby causing the eyes to send clear images to brain for blending both visions into a single 3D image. However, persons with eye alignment disorders are unable to send a single vision to the brain for blending of the two images.

[0003] Advancements in augmented or virtual reality (AR / VR) technology have resulted in the development of VR devices capable of rendering an immersive virtual environment, however the conventional AR / VR devices still does not take into consideration the above-mentioned condition of eyes thereby causing discomfort and difficulty to a person with misaligned eyes to view an immersive content in the VR device. Further a strabismus eyed person interacting with a VR device may view double images, blurred images, or cropped images with reduced depth perception, as the line of sight of both eyes point to different objects displayed on the VR device.

[0004] Thus, there exists a need to overcome the above-mentioned limitations and provide for further improvements in the AR / VR devices for enhanced vision generation for persons with misaligned eyes.

[0005] The information disclosed in this background section is only for enhancement of understanding of the general background of the disclosure and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

[0006] According to an embodiment of the disclosure, a media content rendering method may be provided. The method may include determining a plurality of eye attributes of a user, based on eye position for each of a left eye and a right eye. The method may include identifying eye movement for each of the left eye and right eye, based on the plurality of eye attributes. The method may include determining a constraint for each of the left eye and right eye, based on the eye movement. The method may include determining a binocular field-of-view (FOV) of the user, based on the constraint for each of the left eye and right eye. The method may include determining transformation values, based on the binocular FOV of the user. The method may include providing a transformed media content based on the transformation values to render the transformed media content.

[0007] According to an embodiment of the disclosure, an electronic apparatus for rendering media content may be provided. The electronic apparatus may include memory storing one or more instructions. and at least one processor. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a plurality of eye attributes of a user, based on eye position for each of a left eye and a right eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify eye movement for each of the left eye and right eye, based on the plurality of eye attributes. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a constraint for a movement of each of the left eye and right eye, based on the eye movement. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a binocular field-of-view (FOV) of the user, based on the constraint for each of the left eye and right eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine transformation values, based on the binocular FOV of the user. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to provide a transformed media content based on the transformation values to render the transformed media content.

[0008] According to an embodiment of the disclosure, a computer-readable medium storing one or more instructions may be provided. The one or more instructions, when executed by at least one processor, may cause the at least one processor of an electronic apparatus to perform operation corresponding to the method.

[0009] The embodiments of the disclosure are illustrated in the accompanying drawings, throughout which like reference letters indicate corresponding parts in the various figures. The embodiments herein will be better understood from the following description with reference to the drawings, in which:

[0010] Figure 1 shows an exemplary illustration 100 of a line of sight of a person with an eye alignment disorder, in accordance with an exemplary embodiment of the present disclosure.

[0011] Figure 2A illustrates a block diagram 200A of a content rendering system for a user having an eye alignment disorder, in accordance with an exemplary embodiment of the present disclosure.

[0012] Figure 2B illustrates a detailed block diagram 200B showing multiple modules involved in rendering enhanced content on a Virtual Reality (VR) device, in accordance with an exemplary embodiment of the present disclosure.

[0013] Figure 3A shows an exemplary illustration 300A of an eye image captured by a VR device of a user, in accordance with an exemplary embodiment of the present disclosure.

[0014] Figure 3B shows a detailed block diagram 300B of an eye position sensing module 202 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0015] Figure 3C shows a detailed block diagram 300C of the eye localization module 202B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0016] Figures 3D-3F depict illustrations 300D, 300E, 300F showing processing of an eye image data by the eye localization module 202B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0017] Figure 3G shows an exemplary block diagram 300G of the eye localization module 202B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0018] Figure 4A shows an exemplary illustration 400A of an interpupillary distance, an interpupillary angle and an inter pupillary plane derived from an eye image of the user, in accordance with an exemplary embodiment of the present disclosure.

[0019] Figure 4B shows a detailed block diagram 400B of interpupillary distance calculation module 408, interpupillary angle calculation module 410, and a pupillary plane determination module 412 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0020] Figure 5A depicts exemplary illustration 500A of the process of eye calibration of the user by a lens calibration module 204 of the present system, in accordance with an exemplary embodiment of the present disclosure.

[0021] Figure 5B shows an exemplary block diagram 500B of the lens lateral shifting module 204A of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0022] Figures 6A-6C depicts three scenarios 600A, 600B and 600C when calculating relative eye velocity by an eye movement observation module 206 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0023] Figure 6D depicts exemplary illustration 600D of the process of calculating relative eye velocity by the eye movement observation module 206 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0024] Figure 6E shows an exemplary block diagram 600E of an eye movement constraint calculation module 206B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0025] Figure 6F depicts exemplary illustration 600F of the process of calculating boundary points for each eye of the user by the eye movement constraint calculation module 206B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0026] Figure 7A shows an exemplary illustration 700A of an interpupillary plane with respect to Headset plane, in accordance with an exemplary embodiment of the present disclosure.

[0027] Figure 7B shows exemplary block diagram 700B of a FOV determination module 208 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0028] Figure 7C shows an exemplary illustration 700C of calculation of a monocular FOV of an eye of the user, in accordance with an exemplary embodiment of the present disclosure.

[0029] Figure 7D shows an exemplary block diagram 700D of FOV determination module 208 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0030] Figure 7E shows an exemplary illustration 700E of an interpupillary plane with respect to the monocular FOVs of the eye of the user, in accordance with an exemplary embodiment of the present disclosure.

[0031] Figure 7F shows an exemplary illustration 700F of calculation of a binocular FOV of both eyes of the user, in accordance with an exemplary embodiment of the present disclosure.

[0032] Figure 7G shows an exemplary block diagram 700G of a FOV determination module 208 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0033] Figure 7H shows an exemplary block diagram 700H of a FOV determination module 208, for determination of a resize ratio, in accordance with an exemplary embodiment of the present disclosure.

[0034] Figure 8 shows an exemplary illustration 800 of training of a neural network 210B of the system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0035] Figure 9A shows a block diagram 900A of a correlation generation engine 210 of the system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0036] Figure 9B shows an exemplary block diagram 900B of the correlation generation engine 210 of the system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0037] Figure 10 shows a block diagram 1000 of a coordinate shifting module of the system 200B for display of a shifted image on the VR device, in accordance with an exemplary embodiment of the present disclosure.

[0038] Figure 11 illustrates a flow chart of a method 1100 for media content rendering for a user having an eye alignment disorder, in accordance with an exemplary embodiment of the present disclosure.

[0039] Figure 12 illustrates a flow chart of a method for media content rendering, in accordance with an exemplary embodiment of the present disclosure.

[0040] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described in detail below. It should be understood, however that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0041] The term "some" or "one or more" as used herein may refer to "one", "more than one", or "all." Accordingly, the terms "more than one," "one or more" or "all" may all fall under the definition of "some" or "one or more". The terms "an embodiment", "another embodiment", "some embodiments", or "in one or more embodiments" may refer to one embodiment or several embodiments, or all embodiments. Accordingly, the term "some embodiments" may refer to one embodiment, or more than one embodiment, or all embodiments.

[0042] The terminology and structure employed herein are for describing, teaching, and illuminating some embodiments and their specific features and elements and may not limit, restrict, or reduce the spirit and scope of the claims or their equivalents. The phrase "exemplary" may refer to an example.

[0043] That is, any terms used herein such as, but not limited to, "includes," "comprises," "has," "consists," "have" and grammatical variants thereof may not specify an exact limitation or restriction and may not exclude the possible addition of one or more features or elements, unless otherwise stated, and may not be taken to exclude the possible removal of one or more of the listed features and elements, unless otherwise stated with the limiting language "must comprise" or "needs to include".

[0044] Whether or not a certain feature or element was limited to being used only once, either way, the feature or element may still be referred to as "one or more features", "one or more elements", "at least one feature", or "at least one element." Furthermore, the use of the terms "one or more" or "at least one" feature or element may not preclude there being none of that feature or element unless otherwise specified by limiting language such as "there needs to be one or more" or "one or more element is required."

[0045] Unless otherwise defined, all terms, and especially any technical and / or scientific terms, used herein may be taken to have the same meaning as commonly understood by one having ordinary skill in the art.

[0046] As used herein, each of such phrases as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C," may include any one of, or all possible combinations of the items enumerated together in a corresponding one of the phrases. As used herein, such terms as "1st" and "2nd," or "first" and "second" may be used to simply distinguish a corresponding component from another, and does not limit the components in other aspect (e.g., importance or order). It is to be understood that if an element (e.g., a first element) is referred to, with or without the term "operatively" or "communicatively", as "coupled with," "coupled to," "connected with," or "connected to" another element (e.g., a second element), it means that the element may be coupled with the other element directly (e.g., wired), wirelessly, or via a third element.

[0047] It is to be understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed are an illustration of exemplary approaches. Based on design preferences, it is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying claims present elements of the various blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0048] The embodiments herein may be described and illustrated in terms of blocks, as shown in the drawings, which carry out a described function or functions. These blocks, which may be referred to herein as units or modules, or the like, or by names such as device, logic, circuit, controller, counter, comparator, generator, converter, or the like, may be physically implemented by analog and / or digital circuits including one or more of a logic gate, an integrated circuit, a microprocessor, a microcontroller, a memory circuit, a passive electronic component, an active electronic component, an optical component, or the like.

[0049] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings.

[0050] The terms like "augmented reality" and "virtual reality" and "extended reality" and "immersive experience" have been used interchangeably throughout the disclosure. Further, the terms like "VR headset" and "VR device" and "head mounted display (HMD)" have been used interchangeably throughout the disclosure. The terms like "control unit / module" and "processor" have been used interchangeably throughout the disclosure.

[0051] In the present disclosure, Artificial Intelligence (AI) model (also referred to as "Machine Learning" models) may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[0052] It may be noted that visual understanding is a technique for recognizing and processing things as does human vision and includes, e.g., object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0053] As discussed in the background section, the strabismus condition of eyes is when one eye (e.g., the fixing eye) is fixed on what the person intends to look at and the other eye (e.g., the deviated eye) is looking in a different direction. The deviated eye may be pointed towards inwards (e.g., esotropia), outwards (e.g., exotropia) or vertically upwards (e.g., hypertropia) or downwards (e.g., Hypotropia). Thus, causing each eye to send a different image to the brain. The brain upon receiving the two images merge them into one image, thereby generating a distorted image that creates a visual confusion and discomfort to the users. The conventional AR / VR headsets do not consider this condition of eyes and thus are unable to provide an enhanced or improved vision to such users.

[0054] In case of a normal eyed person, when a content is supplied to a left and right eye through a VR headset, the line of sight for a particular object displayed on the VR device falls on the exact set of coordinates, thereby providing a better-quality image compared strabismus person.

[0055] Figure 1 shows an exemplary illustration 100 of a line of sight of a person with an eye alignment impairment (e.g., disorder). As shown in the Figure, the line of sight of both eyes converge insufficiently, thereby causing double vision. Such person when interact with a VR device may view two different regions of an image in VR Headset even though same image is displayed to the left and the right eye. Therefore, current VR headsets are unable to adjust the image that is being shown to the user based on the user's condition. Another issue that arises with such condition is reduced depth perception in the image due to inaccurate generation of binocular vision of the eyes. On other occasions, the image projected to the one eye may appear cropped because of the limitation of the eye to view beyond a certain direction. The conventional VR headsets do not consider the restriction of movement of strabismus eye and thus restricting the FOV. This may be due to the line of sight of one eye being too far away from the binocular FOV.

[0056] Nystagmus is an example of the eye alignment impairment (e.g., disorder), where the speed or movement of eyes is not synchronized with each other. Due slow or fast movement of a single eye relative to other, the line-of-sight changes more frequently for the eyes, thus creating a visual confusion.

[0057] In an example, such as amblyopia, the vision in the deviated eye is permanently reduced. In such scenarios, the conventional VR headsets only uses the fixed eye for tracking and significantly reduces the FOV.

[0058] Thus, the conventional VR headsets do not solve the above limitations and significantly impact the VR experience. The conventional AR / VR headsets do not take into account above condition of eyes and the eye gazing information to accurately identify the eye positions and project an enhanced vision that is adapted as per user's eye condition.

[0059] The present disclosure provides techniques to overcome the above limitations and provide an enhanced content for person with visual impairment (e.g., strabismus or nystagmus eyes. The present disclosure utilizes an artificial neural network-based regression approach for generating a correlation between various eye parameters such as, but not limited to, eyes position, interpupillary distance, interpupillary angle, interpupillary plane, distance between lens and eyes, relative speed between eyes movement, monocular and binocular FOV to ensure the user receives an improved image with better depth perception causing no visual confusion and less visual fatigue.

[0060] Figure 2A illustrates a block diagram of a content rendering system, in accordance with an exemplary embodiment of the present disclosure.

[0061] The electronic apparatus 200A may include at least one of one or more sensors 201, processor 203, memory 205, an input / output (I / O) interface, a or a communication interface.

[0062] The electronic apparatus 200A may include one or more sensors 201, to capture eye gazing information of the user. For example, the one or more sensors may include camera sensors, accelerometer, gyroscope, infrared sensors, but not limited thereto.

[0063] The electronic apparatus 200A may include at least one processor 203 and memory 205 that are coupled together and are configured to perform content rendering techniques disclosed in the present disclosure, utilizing the information captured from the one or more sensors 201.

[0064] At least part of the functions in a device or electronic apparatus provided in the embodiments of the disclosure may be implemented through an AI model, such as, at least one of a plurality of modules of the device or electronic apparatus may be implemented through the AI model. A function associated with AI may be performed through the non-volatile memory, the volatile memory, and the processor.

[0065] The processor 203 may include one or more processors. At this time, the one or more processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, or may be a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-dedicated processor such as a neural processing unit (NPU).

[0066] The one or more processors 203 control processing of input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning.

[0067] The processor 203 may include various processing circuitry and / or multiple processors. For example, as used herein, including the claims, the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of at least one processor, individually and / or collectively in a distributed manner, may be configured to perform various functions described herein. As used herein, when "a processor", "at least one processor", and "one or more processors" are described as being configured to perform numerous functions, these terms cover situations, for example and without limitation, in which one processor performs some of recited functions and another processor(s) performs other of recited functions, and also situations in which a single processor may perform all recited functions. Additionally, the at least one processor may include a combination of processors performing various of the recited / disclosed functions, e.g., in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0068] The memory 205 can include a volatile and / or non-volatile memory. For example, the memory 205 can store commands or data related to at least one other component of the electronic apparatus. According to embodiments of this disclosure, the memory 205 can store software and / or a program. The program includes, for example, a kernel, middleware, an application programming interface (API), and / or an application program (or "application"). At least a portion of the kernel, middleware, or API may be denoted an operating system (OS).

[0069] The kernel can control or manage system resources (such as a bus, processor 203, or memory 205) used to perform operations or functions implemented in other programs (such as the middleware, API, or application). The kernel provides an interface that allows the middleware, the API, or the application to access the individual components of the electronic apparatus to control or manage the system resources. The application may support various functions related to generation of dynamic and adaptive soundscapes for sound masking. For example, the application includes one or more applications supporting the receipt of audio from the environment external to the user. The Application further includes one or more applications supporting analysis of the current sound environment properties of an environment external to a user, current device properties of a sound playback device, and current media properties of media that is played through the sound playback device. These functions can be performed by a single application or by multiple applications that each carries out one or more of these functions. The middleware can function as a relay to allow the API or the application to communicate data with the kernel, for instance. A plurality of applications can be provided. The middleware is able to control work requests received from the applications, such as by allocating the priority of using the system resources of the electronic apparatus (like the bus, the at least one processor 203, or the memory 205) to at least one of the plurality of applications. The API is an interface allowing the application to control functions provided from the kernel or the middleware. For example, the API includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.

[0070] The I / O interface serves as an interface that can, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic apparatus. The I / O interface can also output commands or data received from other component(s) of the electronic apparatus to the user or the other external device.

[0071] In an embodiment of the disclosure, the electronic apparatus 200A may receive the eye gazing information or eye image through the I / O interface.

[0072] The communication interface, for example, is able to set up communication between the electronic apparatus and an external electronic device (such as a first electronic device, a second electronic device, or a server 207). For example, the communication interface can be connected with a network or through wireless or wired communication to communicate with the external electronic device. The communication interface can be a wired or wireless transceiver or any other component for transmitting and receiving signals.

[0073] The wireless communication is able to use at least one of, for example, WiFi, long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (5G), millimeter-wave or 60 GHz wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a communication protocol. The wired connection can include, for example, at least one of a universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). The network or includes at least one communication network, such as a computer network (like a local area network (LAN) or wide area network (WAN)), Internet, or a telephone network. The one or more servers 207 may include one or more databases to store data required by the processor in order to perform the techniques of the present disclosure.

[0074] In an embodiment, the electronic apparatus may be an AR / VR device. The device may include a plurality of displays (e.g., a display for each eye of a user) and one or more camera sensors. The electronic display displays media content to the user. In an embodiment the VR device is wearable headset such as a head-mounted display (HMD). In the present disclosure, the electronic apparatus may be referred to as a VR headset, but the electronic apparatus is not limited to a VR headset.

[0075] The display may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display can also be a depth-aware display, such as a multi-focal display. The display is able to display, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display can include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.

[0076] Figure 2B illustrates a detailed block diagram 200B of a system showing multiple modules involved in rendering enhanced content on a VR device, in accordance with an exemplary embodiment of the present disclosure. The present disclosure describes six modules explained in the forthcoming paragraphs. The first module is referred as eye position sensing module 202 and is configured to determine by use of a sensor unit 202A, a position of both left and right eye, a distance and angle between them associated with a user. The eye position sensing module 202 further comprises an eye localization module 202B, an interpupillary distance calculation module 202C, an interpupillary angle calculation module 202D and a pupillary plane determination module 202E.

[0077] The eye information from the eye position sensing module 202 may be then transmitted to the second module referred as "lens calibration module" 204. The lens calibration module 204 may be configured to determine the position of lens with respect to eyes and adjust or calibrate the position of lens according to eyes in a limited space. The lens calibration module 204 may comprise a lens lateral shifting module 204A.

[0078] Once the lens are adjusted with respect to the eyes, the eye movement observation module 206 may calculate a speed of both eyes relative to each other and the movement of eyes with respect to head position thus aiding in determining the exact frame of reference to be taken into account The eye observation module 206 may comprise an eye relative velocity calculation module 206A and an eye movement constraint calculation module 206B.

[0079] The fourth module is the field-of-view (FOV) determination module 208 that may be configured to calculate a monocular and binocular FOV of user, considering multiple scenarios depending on the movements of head and eyes of the user. The FOV determination module 208 may comprise a head position determination module 208A, a viewing plane calculation module 208B, FOV calculation module 208C and a resize ratio determination module 208D. The FOV for the person with eye alignment disorder may get tilted as eyes are not aligned with each other. Therefore, the FOV determination module 208 may be configured to calculate extreme coordinates of image with respect to correct FOV of the user.

[0080] Once the monocular and binocular FOV are determined, a correlation generation engine 210 may generate a mapping of eye gazing information of the user with the coordinates of images displayed on the VR headset, using an artificial neural network 210B. The correlation output may contain coordinates for both eyes and ratio of the image to be resized to fit into frame of reference of the user for enabling enhanced depth vision. The correlation generation engine 210 may comprise an information collection module 210A and a coordinate and resize ratio prediction module 210C.

[0081] Upon prediction of the image coordinates and a resize ratio, a coordinate shifting module 212 is configured to shift the coordinates of images rendered on the VR headset to the frame of reference so obtained from the above modules. The shifted image may be then displayed on the VR headset.

[0082] The various data from the above-mentioned modules are then stored in a database 216 and retrieved at various stages of the content rendering process of the present disclosure.

[0083] Figure 3A shows an exemplary illustration 300A of an eye image captured by a VR device of a user, in accordance with an exemplary embodiment of the present disclosure.

[0084] Once the eye image of a user is captured by the sensor unit 202A, the eye localization module 202B, processes the eye image to obtain a plurality of points to identify exact pupil location, outer eye line of the user and other eye related information. These points or coordinates are used in tracking the movement of eyes from one point to another.

[0085] As illustrated in Figure 3A, the various (X,Y) coordinates (e.g., {X1,Y1}, {X2,Y2}, {X3,Y3}, {A, B}, ..., {Xn, Yn}) of the plurality of points may be determined. For example, the coordinates corresponding to plurality of points of pupil location may be identified. The coordinates corresponding to the plurality of the outer eye line may be identified. The coordinates of the points may be stored in a database. The database may comprise a plurality of left and right eye images each associated with a plurality of relevant coordinates. The relevant coordinates may include a PupilBoundaryPointsX, PupilBoundaryPointsY, PupilEllipseAxis_X, PupilEllipseAxis_Y, PupilCenter_PupilX and, PupilCenter_PupilY. For instance, PupilCenter_PupilX denotes the x coordinate of the pupil centre and PupilCenter_PupilY denotes the y coordinate of the pupil centre.

[0086] Figure 3B shows a detailed block diagram 300B of an eye position sensing module 202 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0087] Fig. 3B depicts the eye localization module 304. As shown in Figure 3B, the sensor unit 302 (may also be referred as 202A of Figure 2B) may comprise at least one of an accelerometer, a gyroscope, a camera, or infrared light source. The sensor unit 302 is configured to capture various attributes of the eye and transmit to the eye localization module 304 (may also be referred as 202B of Figure 2B). The eye localization module 304 is configured to perform pre-processing of the image such as adjusting the image illumination characteristics e.g., brightness, contrast for generating an image with reduced intensity. The eye localization module 304 may employ pre-processing module 306 for generating an image histogram 306A and applying a histogram equalization 306B technique for adjusting the intensity of the eye image captured by the sensor unit 302. The adjusted image is then fed to a ResNet neural network 308.

[0088] Figure 3C shows a detailed block diagram 300C of an eye localization module 202B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0089] The ResNet model 308 of the present disclosure uses a residual learning approach, which enables the model 308 to learn more complex features and avoid "degradation problem" when number of layers increases. The ResNet model 308 may be a Convolutional Neural Network (CNN) model that may have 50 layers, for example. The architecture of the ResNet model 308 used in the present disclosure is conventional, comprising a plurality of known layers such as convolution layer and a batch normalization layer, a ReLU activation function and a max pooling layer.

[0090] The convolution block and identity block of the model 308 may be used for input and output activation.

[0091] In the convolution block, the input activation may have different dimension as output activation. In an embodiment of the disclosure, the convolution block may have a plurality set of convolution layer, batch normalization layer, and ReLU activation layer. Each convolution layer of the set may have different filter size and different number of channels, for example, 1x1 convolution with 64 channels, 3x3 convolution with 64 channels, and 1x1 convolution with 256 channels.

[0092] For example, the convolution block may reduce the amount of computation by reducing the number of channels to 64 channels through 1x1 convolution. The convolution block may perform batch normalization and ReLu activation on the 64-channel output of the 1x1 convolution layer. Subsequently, the convolution block may extract features through 3x3 convolution. The convolution block may perform batch normalization and ReLu activation on the 64-channel output of the 3x3 convolution layer. The convolution block may perform batch normalization and ReLu activation on the 64-channel output of the 3x3 convolution layer. In addition, subsequently, the convolution block may increase the number of channels to 256 through 1x1 convolution to match the dimension with the residual connection.

[0093] Since the dimensions of the input of the convolution block and output of the last convolution layers are different, additional set of convolution layer, batch normalization layer, and ReLU activation layer may be included for the convolution block. The additional set may be a bypass set to perform the addition operation. The additional set may adjust the number of the channels to be able to perform addition operation. The convolution layer for the additional set may perform 1x1 convolution and adjust the number of the channels to be same with output of the convolution. The addition operation may be performed using an addition layer. The convolution block may finally perform ReLu activation on the addition result.

[0094] In the identity block, the input activation may have same dimension as output activation. In an embodiment of the disclosure, the identity block may have a plurality set of convolution layer, batch normalization layer, and ReLU activation layer. Each convolution layer of the set may have different filter size with different number of channels, for example, 1x1 convolution with 64 channels, 3x3 convolution with 64 channels, and 1x1 convolution with 256 channels.

[0095] For example, the identity block may reduce the amount of computation by reducing the number of channels to 64 channels through 1x1 convolution. The identity block may perform batch normalization and ReLu activation on the 64-channel output of the 1x1 convolution layer. Subsequently, the identity block may extract features through 3x3 convolution. The identity block may perform batch normalization and ReLu activation on the 64-channel output of the 3x3 convolution layer. The identity block may perform batch normalization and ReLu activation on the 64-channel output of the 3x3 convolution layer. In addition, subsequently, the identity block may increase the number of channels to 256 through 1x1 convolution to match the dimension with the residual connection. Since the dimensions of the input and output of the identity block are the same, an addition operation may be performed using an addition layer. The identity block may finally perform ReLu activation on the addition result.

[0096] In one embodiment of the present disclosure, the convolution blocks and the identity blocks are repeated for a predefined number of times in the model 308.

[0097] The ResNet model 308 may perform eye detection using a Regional Proposal Network (RPN) 312 and a Region-based Convolutional Neural network (RCNN) 314. The regional Proposal Network 314 may take the convolutional feature map from the last layer of the model 308 and then may generate the region proposals over the eye image and predict the object's probability. Further, the Non-Maximum Suppression (NMS) function of the regional Proposal Network 312 is used to remove the redundancy and select the most accurate proposals, based on the object's scores of overlapping proposals. The proposal contains the adjustments to anchor's coordinates and the probability of the anchor containing the object. ROI Pooling may be used to transform the RPN's variable sized region proposals into fixed sized cells that may be fed into subsequent layers. SoftMax Object Classification of the RCNN 314 may predict class probabilities for each region proposal, indicating the possibility that the proposal contains an object of a specific class. SoftMax gives the Probabilistic scores for each object. Bounding Box Regressor of the RCNN 314 redefines the bounding box on an object e.g., position and scale. Regression layer tightens the center and size of the bounding box around the target. The SoftMax Object classification and Bounding Box regressor generates the eye detection and localization coordinates. Further the eye localization module 304 comprises an eye center localization module 316 that receives the eye detection and localization coordinates for further processing.

[0098] Figures 3D-3F depict illustrations 300D, 300E, 300F showing processing of an eye image data by the eye localization module 202B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure. As depicted in Figure 3D and 3E, the left and right eye image of the user are divided into multiple regions of interests (ROIs) such as r0, r1, r2, ..., r7. The ROIs are then selected for determining the exact coordinates of the eye as discussed in Figure 3A and illustrated in Figure 3F.

[0099] Figure 3G shows an exemplary block diagram 300G of the eye localization module 202B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure. The eye center localization module 316 of the eye localization module 304, then sub-divides the iris shape features 316A and derives intensity information 316B from the iris features (e.g., in form of pixel values). This is based on the fact that average intensity of the iris region is less than surrounding regions of an eye, thereby having less more pixel value. Further the intensity of the pupil region is less than the intensity of the iris region. The eye center localization module 316 then determines gradient 316C upon calculating the intensity difference between neighbouring pixels and generating a gradient vector or slope vector. Based on the pixel location calculated, a displacement vector 316D is calculated from the center of the eye position using bound box regression technique. The angular distribution of the gradient vector 316E provides an angle between the axis of the eye and center of eye, resulting in a gaze direction and finally, a pupil location is determined 316F. This eye pupil information is then transmitted to the interpupillary distance calculation module 202C, interpupillary angle calculation module 202D and the pupillary plane determination module 202E for determination of an interpupillary distance, an interpupillary angle and an inter pupillary plane between the left and the right eye of the user.

[0100] Figure 4A shows an exemplary illustration 400A of an interpupillary distance, an interpupillary angle and an inter pupillary plane derived from an eye image of the user, in accordance with an exemplary embodiment of the present disclosure.

[0101] The electronic apparatus 200A may identify the interpupillary distance. The electronic apparatus 200A may identify the pupil coordinate of left eye. The electronic apparatus 200A may identify the pupil coordinate of right eye. Figure 4A illustrates the interpupillary distance 402, which is the actual distance between the pupil coordinates of the left and the right eye.

[0102] In one non-limiting embodiment, the interpupillary distance 402 is computed as:

[0103] (equation 1),

[0104] where C1 and C2 are the coordinates of the two-camera present in the VR device and

[0105] Left PupilCenter_PupilX is the x coordinate of the pupil of left eye and

[0106] Left PupilCenter_PupilX is the x coordinate of the pupil of right eye.

[0107] The electronic apparatus 200A may identify the interpupillary angle. Interpupillary Angle 404 is the angle between two eye pupil's axes with respect to VR headset camera reference and is computed as:

[0108] (equation 2)

[0109] Interpupillary Plane 406 is the Plane between two eye pupil's axis planes with respect to VR headset plane reference when eyes are at fixed reference position.

[0110] Figure 4B shows a detailed block diagram 400B of interpupillary distance calculation module 408, interpupillary angle calculation module 410, and a pupillary plane determination module 412 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0111] The interpupillary distance calculation module 408 (also referred as 202C), interpupillary angle calculation module 410 (also referred as 202D) and the pupillary plane determination module 412 (also referred as 202E) receives the eye center and pupil coordinates from the eye localization module 304.

[0112] The interpupillary distance calculation module 408 may receive the left eye pupil coordinates 408A, right eye pupil coordinates 408B and VR headset camera distance 408C. The VR headset camera distance may be a predefined value obtained from the VR headset specifications for the calculation 408D of interpupillary distance 402 using equation 1.

[0113] The interpupillary angle calculation module 410 may receive the left eye pupil coordinates 410A, and right eye pupil coordinates 410B. The interpupillary angle calculation module 410 may receive VR headset camera distance as reference 410C for the calculation 410D of interpupillary angle 404 using equation 2.

[0114] The interpupillary plane determination module 412 may receive the left eye pupil coordinates 412A, and right eye pupil coordinates 412B. The interpupillary angle calculation module 410 may receive VR headset plane as reference 412C for the determination 412D of interpupillary plane 406. The data from the 3 modules- 408, 410 and 412 are then transmitted to the lens calibration module 414. The data from the interpupillary distance calculation module 408, the interpupillary angle calculation module 410, the interpupillary plane determination module 412 may be stored in the database 216.

[0115] Figure 5A depicts exemplary illustration 500A of the process of eye calibration of the user by a lens calibration module 204 of the present system, in accordance with an exemplary embodiment of the present disclosure. In conventional VR headsets, there exists an eye tracking module comprising camera sensors to capture the position of the eye in real-time for different features and for recording the movement of eyes based on the media content viewable to the user. The eye tracking module helps in adjusting the eyes of the user with respect to the lens of the VR headset.

[0116] The X-Y coordinates of left lens (Left LensX, Left LensY) and the X-Y coordinates of right lens (Right LensX, Right LensY ) are adjusted according to the X-Y coordinates of left pupil center ( denoted by Left PupilCenter_PupilX, Left PupilCenter_PupilY ) and the X-Y coordinates of right pupil center (denoted by Right PupilCenter_PupilX, Right PupilCenter_PupilY). The coordinates: Left LensX, Left LensY, Left PupilCenter_PupilX, Left PupilCenter_PupilY and Right PupilCenter_PupilX, Right PupilCenter_PupilY are obtained from the database 216 and the eye position sensing module 202.

[0117] Figure 5B shows an exemplary block diagram 500B of the lens lateral shifting module 204A of the present system 200B, in accordance with an exemplary embodiment of the present disclosure. Figure 5B illustrates a lens lateral shifting module 502 ( also referred as 204A of Figure 2B)

[0118] The lens lateral shifting module 502 shifts the left and right eye lens according to the obtained pupillary plane and principal axis of left and right eyes within the limited space available to the lenses. In an embodiment, the appropriate scale is determined before from image of eye with respect to lens dimensions and then principal axis of lens is aligned to that of eye.

[0119] The lens lateral shifting module 502 obtains pupillary plane information 504, lens movement extremes information 506 and align principal axis on lens to that of eyes 508. Further below equations (3)-(6) are utilized to compute the error after alignment 510 and the error data is updated 512 in the database 216.

[0120] (equation 3)

[0121] (equation 4)

[0122] (equation 5)

[0123] (equation 6)

[0124] In an embodiment of the HMD display the lens calibration module 204 calibrates a position of a user's eye relative to a single HMD display screen.

[0125] Figures 6A, 6B and 6C depicts three examples 600A, 600B and 600C when calculating relative eye velocity by an eye movement observation module 206 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0126] The eye movement observation module 206 comprises eyes relative velocity calculation module 606 (also referred as 206A of Figure 2B). The eyes relative velocity calculation module 606 is configured to calculate the speed of both eye pupils relative to each other. The eyes relative velocity calculation module 606 is configured to calculate movement of eyes with respect to head position in order to determine the exact frame of reference to be considered for the user with the eye condition. Data from the eye localization module 602 (also referred as 304 of Figures 3B, 3C and 3G) and a predetermined camera image for VR screen Resolution 604 is obtained from the database 216 and transmitted to the eyes relative velocity calculation module 606.

[0127] A displacement calculation, an angle deviation calculation and finally a relative velocity is calculated. The above-mentioned processes are described in the forthcoming paragraphs.

[0128] The displacement Calculation process provides displacement between Initial pixel and Final Pixel for an object shown at VR device ( unScaled Pixels ) and also provides a displacement between for both eye pupils.

[0129] The following equations (7)-(10) are used in displacement calculation of both the eye pupils and the object.

[0130] (equation 7)

[0131] (equation 8)

[0132] (equation 9)

[0133] (equation 10)

[0134] The angle deviation calculation process measures an angle between Initial pixel and Final Pixel for both eyes and also measures an angle for an object from an initial to final position, shown at the VR device. In one embodiment, for the calculation of angle for an object from an initial to final position, the VR headset is taken as reference. In an embodiment, for the calculation of angle between Initial pixel and Final Pixel for both eyes, the Pupillary plane for both eyes is taken as reference.

[0135] The following equations (11)-(13) are used in angle deviation calculation for object and left and right Pupils.

[0136] (equation 11)

[0137] (equation 12)

[0138] (equation 13)

[0139] The following equations (14)-(15) are used in relative velocity calculation.

[0140] (equation 14)

[0141] (equation 15)

[0142] Each of the above equations are utilized in the calculation of displacement of object and left and right pupils; angle deviation of object and left and right pupils and relative velocity.

[0143] Examples are considered to obtain the various values from the above-mentioned calculation processes. The case scenario considers when left eye is normal and for the left Eye, the object displayed on the VR device screen aligns with left pupil center; and right eye is not Normal and for the right Eye, the object displayed on the VR device screen does not align with right pupil center.

[0144] Figure 6A explains the example 600A when the object is at center of VR Screen for both eyes, but object movement is tracked for both the eyes. The object may be a virtual object rendered in content displayed on the VR device.

[0145] According to an embodiment of the disclosure, the electronic apparatus 200A may obtain the eye image while tracking the movement of an object being moved till extremity for both eyes 606A.

[0146] According to an embodiment of the disclosure, the electronic apparatus 200A may obtain the real-time eye image for the object is moving. For example, the electronic apparatus 200A may capture a plurality of the eye images 606B. For example, the electronic apparatus 200A may receive the real-time eye image or data comprising the information of the eye.

[0147] According to an embodiment of the disclosure, the electronic apparatus 200A may localize the eye position 606C. The electronic apparatus 200A may calculate the displacement of each eye pupil 606D. For example, the electronic apparatus 200A may identify the first eye coordinate corresponding to the eye position from the first eye image. The electronic apparatus 200A may identify the second eye coordinate corresponding to the eye position from the second eye image. The electronic apparatus 200A may calculate the displacement by using the first eye coordinate from the first eye image, and the second eye coordinate from the second eye image.

[0148] According to an embodiment of the disclosure, the electronic apparatus 200A may calculate the displacement of object. For example, the electronic apparatus 200A may identify the first object coordinate corresponding to the object position from the first object image. The electronic apparatus 200A may identify the second object coordinate corresponding to the object position from the second object image. The electronic apparatus 200A may calculate the displacement by using the first object coordinate from the first object image, and the second object coordinate from the second object image.

[0149] According to an embodiment of the disclosure, the electronic apparatus 200A may calculate the angle to the eye pupil 606E. The electronic apparatus 200A may identify the angle using the displacement of each eye. The electronic apparatus 200A may calculate the angle to the object. The electronic apparatus 200A may identify the angle using the displacement of the object.

[0150] According to an embodiment of the disclosure, the electronic apparatus 200A may calculate the velocity corresponding to movement of each eye 606F. The electronic apparatus 200A may identify the velocity using the displacement of the eye pupil and the time taken for the displacement. The electronic apparatus 200A may calculate the angle deviation 606F using the angle to the eye pupil, and angle to the object.

[0151] Figure 6B discusses the example 600B the object is at center of VR Screen for both eyes, but object movement is tracked for one normal eye (for example, the left eye) only.

[0152] According to an embodiment of the disclosure, the electronic apparatus 200A may obtain the eye image while tracking the movement of an object being moved till extremity for the left eye. The electronic apparatus 200A may obtain the eye image while keeping the object still for the right eye.

[0153] According to an embodiment of the disclosure, for at least one of the eyes, the operations already described 606B to 606F in Figure 6A can be performed.

[0154] Figure 6C discusses the example 600C the object is at center of VR Screen for both eyes, but object movement is tracked for visually impaired eye or misaligned eye (for example, right eye) only.

[0155] According to an embodiment of the disclosure, the electronic apparatus 200A may obtain the eye image while tracking the movement of an object being moved till extremity for the right eye. The electronic apparatus 200A may obtain the eye image while keeping the object still for the left eye.

[0156] According to an embodiment of the disclosure, for at least one of the eyes, the operations already described 606B to 606F in Figure 6A can be performed.

[0157] Figure 6D depicts exemplary illustration 600D of the process of calculating relative eye velocity by the eye movement observation module 206 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0158] Upon tracking of eye movement, the following are determined:

[0159] An initial position of an object in X-Y direction is determined (e.g., Left unScaledObjectInitialX, Left unScaledObjectInitialY are determined).

[0160] A final position of an object in X-Y direction is determined (e.g., a final position of an object in X-Y direction is determined).

[0161] An initial position of the left pupil in X-Y direction is determined (e.g., Left InitialPupilX, Left InitialPupilY are determined).

[0162] An initial position of the left pupil in X-Y direction is determined (e.g., Left FinalPupilX, Left FinalPupilY are determined).

[0163] An initial position of the right pupil in X-Y direction is determined (e.g., Right InitialPupilX, Right InitialPupilY are determined).

[0164] An initial position of the right pupil in X-Y direction is determined (e.g., Right FinalPupilX, Right FinalPupilY are determined).

[0165] From equations (7)-(10) displacement is calculated for both the left and right eye pupils and the object.

[0166] From the above displacement values, using equations (11)-(15), the velocity corresponding to the movement of left and right pupil in X direction, represented as VelocityInitialToFinalPupilX and the velocity corresponding to the movement of pupil in Y direction, represented as VelocityInitialToFinalPupilY are computed. Further, angle of deviation for object and the pupils are computed (represented by AngleInitialToFinalObject and AngleInitialToFinalPupil).

[0167] Figure 6E shows an exemplary block diagram of an eye movement constraint calculation module 206B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0168] The eyes movement constraint calculation module 612 (also referred as 206B) collates the data 612A obtained from the case scenarios described in Figures 6A-6C and determines observations related to movement constraints of both the eyes. The differences in AngleInitialToFinalObject and AngleInitialToFinalPupil indicates that for movement of object in a particular direction and with particular velocity, a certain eye is not able to follow with the same speed and angle e.g., a deviation is observed. The eyes movement constraint calculation module 612 determines that from one point to another point the eye movement may vary in terms of velocity and direction according to level of strabismus condition in the user.

[0169] The eyes movement constraint calculation module 612 is configured to determine rate of Change of Angle of Pupil 612B and Velocity in order to determine whether the movement of Pupil from initial position to Final Position in a particular direction diminishes 612D. The rate of change may be also referred to as the change rate in the disclosure. The eyes movement constraint calculation module 612 may identify the boundary of eye movement 612E. Further, a Boundary point or Restriction point is the point when no further movement of eye is possible or user not able to move eyes even though the object is still moving. Further, boundary points with respect to every PupilCenterX, PupilCenterY taken as first starting point and accordingly all the intermediate initial and final points are calculated for all the restricted regions of the left and right eye pupils.

[0170] The eyes movement constraint calculation module 612 may identify whether the difference in angle from initial point to final point remains. For example, if the difference is zero, the movement direction may be identified as unchanged (e.g. in the same direction). If the difference is variable, the movement direction may be identified as changed.

[0171] The eyes movement constraint calculation module 612 is configured to perform numerical method analysis 612C for determining the differences between initial and final points and to check whether the differences are negligible when compared to both the Angle of Pupil and Velocity of movement of pupil.

[0172] The eyes movement constraint calculation module 612 may calculate rate of change corresponding to angle deviation for the object 612F. The eyes movement constraint calculation module 612 is configured to perform numerical method analysis 612G for determining the differences between initial and final points of the object. The eyes movement constraint calculation module 612 is configured to determine rate of Change of Angle of object 612F and Velocity in order to determine whether the movement of the object from initial position to Final Position in a particular direction diminishes 612H. The eyes movement constraint calculation module 612 may identify the boundary of the object movement 612I.

[0173] The eyes movement constraint calculation module 612 may identify available range of movements of eye when movement of the object is restricted 612K. The eyes movement constraint calculation module 612 may identify a plurality of boundary regions from the pupil center coordinates 612L. The eyes movement constraint calculation module 612 may calculate the restricted regions for the initial and final points by using the pupil center coordinate.

[0174] Figure 6F depicts exemplary illustration 600F of the process of calculating boundary points for each eye of the user by the eye movement constraint calculation module 206B of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0175] The boundary points for each eye of the user are calculated by starting from multiple pupil centers and reaching to maximum possible points where eye movement is possible. The generation of a region from the boundary points is known as a boundary region. As illustrated in Figure 6F, the Left BoundaryPointX, Left BoundaryPointY are the BoundaryPoint (X,Y) for a left eye upto which the eye movement can happen.

[0176] Figure 7A shows an exemplary illustration 700A of an interpupillary plane with respect to Headset plane, in accordance with an exemplary embodiment of the present disclosure. When a user wears a VR headset, it may not be necessary that the interpupillary plane of both the eyes is aligned with the headset plane taken in respect to the ground. As illustrated in Figure 7A, when the eyes of the user are misaligned, the interpupillary plane between the two eyes may not be normal with respect to ground. The FOV determination module 208 is configured to align the interpupillary plane with the headset plane before generating an optimal binocular FOV for the user.

[0177] Figure 7B shows exemplary block diagram 700B of a FOV determination module 208 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0178] The FOV determination module 208 comprises a head position determination module 702 (also referred as 208A of Figure 2B) that receives data 702A from the sensor module comprising an accelerometer and a gyroscope. The Head Position Determination module 702 is configured to fuse the data 702B from the sensor module. Upon fusing data from sensor module, an orientation of the headset is generated. This fused data is then estimated to determine a headset plane 702E based on the roll, orientation, yaw, and pitch of the headset 702D. In one embodiment, the estimation is done using a predetermined technique such as Kalman Filter Estimation technique 702C. Kalman Filter Estimation uses a nine-element state vector to track error in the orientation estimate, gyroscope bias estimate and a linear acceleration estimate. Specifically, the head position determination module 702 is configured to generate the Headset Plane with reference to ground.

[0179] The FOV determination module 208 comprises a viewing plane determination module 704 (also referred as 208B of Figure 2B) that is configured to obtain the pupillary plane information 704B from the pupillary plane determination module 412 and obtain a normal to the pupillary plane 704C. Further upon determination of the headset plane 704A from the head position determination module 702, a normal is obtained to the headset plane. The viewing plane determination module 704 is configured to obtain the viewing plane with respect to headset plane 704E by determining 704D the angle between the normal of the pupillary plane and normal of the headset plane and sends this information to the FOV calculation module 708 (also referred as 208C of Figure 2B).

[0180] Figure 7C shows an exemplary illustration 700C of calculation of a monocular FOV of an eye of the user, in accordance with an exemplary embodiment of the present disclosure. A monocular FOV comprises boundary Points for a single eye (e.g., Left FOVBoundaryPointX1 ... Left FOVBoundaryPointXn; and Left FOVBoundaryPointY1 ... Left FOVBoundaryPointYn).

[0181] Figure 7D shows an exemplary block diagram 700D of FOV determination module 208 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure. The FOV calculation module 708 (also referred as 208C of Figure 2B) generates left and right monocular FOV by first obtaining a normal eyed person's monocular Field-of-view (FOV) for each of the left and the right eye 708A. The normal eyed person FOV data contains FOV boundary coordinates for all points starting from the Pupil Center up to which a person can see objects clearly. Further, from the eye movement observation module 206, all the Pupil Center coordinates nearest to Final Pupil points for Left and right Eye are determined 708B. In other words, a set of coordinates are determined starting from the Pupil Center to the nearest possible Final Pupil position of the normal eyed person data. For all such determined set of coordinates, a determination is made of whether a boundary point has been reached, e.g., whether the Coordinate (X,Y) and FinalPupil (X,Y) are the same is checked. If the boundary point is reached, the FOV of that eye is limited to that boundary point 708C. In case the boundary point is not reached, FOV is same for FinalPupil (X,Y) point of Left and right Eye 708D. The FOV calculation module 708 may retain the FOV points 708E. If the FOV Coordinates (X,Y) of Normal eyed person is bigger than BoundaryPoint (X,Y ) of strabismus eyed person (e.g. if the FOV region of normal eyed person is larger than the FOV region of strabismus eyed person), the monocular FOV is restricted for the strabismus eyed person. If the BoundaryPoints (X,Y) is bigger than Coodinates (X,Y) of normal eyed person, e.g., a strabismus eyed person has a greater FOV than normal eyed person in this case.

[0182] Figure 7E shows an exemplary illustration 700E of an interpupillary plane with respect to the monocular FOVs of the eye of the user, in accordance with an exemplary embodiment of the present disclosure. Considering the left monocular FOV of the left eye and the right monocular FOV of the right eye are determined from the FOV calculation module 708, an interpupillary plane information may then be represented by a plane existing between the Left FinalPupilX, Left FinalPupilY coordinates and Right FinalPupilX, Right FinalPupilY coordinates.

[0183] Figure 7F shows an exemplary illustration 700F of calculation of a binocular FOV of both eyes of the user, in accordance with an exemplary embodiment of the present disclosure.

[0184] An overlapping region between the two monocular FOVs are computed as illustrated in Figure 7F. The overlapping region so computed is used in the determination of the binocular FOV.

[0185] Figure 7G shows an exemplary block diagram 700G of a FOV determination module 208 of the present system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0186] In an embodiment of the disclosure, the monocular FOV of the left and right eye are obtained from the FOV calculation module 708 and the viewing plane is obtained from the viewing plane determination module 704. Based on the data obtained from the modules- 704 and 708, the FOVs of the left and the right eye angles are transformed 710A as per viewing plane angle. In an embodiment, when the Headset Plane and Pupillary Plane are not aligned, the viewing Plane is also therefore not aligned. So, an angle is calculated from FinalPupil(X,Y) point to every Boundary Point and the computed angle is transformed with Viewing Plane angle in order to correct the alignment. Upon obtaining the transformed FOV, the overlapping region between the transformed left FOV and transformed right FOV is obtained 710B. In case the overlapping region is not found, the boundary of the non-overlapping region is determined 710C. In case, the overlapping region is found, the boundary of the overlapping region is determined 710D.

[0187] Figure 7H shows an exemplary block diagram 700H of the FOV determination module 208, for determination of a resize ratio, in accordance with an exemplary embodiment of the present disclosure.

[0188] Once the monocular and binocular FOV are computed, these are adjusted according to user with respect to movements of head. The resize coordinates are generated only for cases when the Eye is able to move but the screen dimensions extremities have been reached, thus causing the object to not be displayed on the VR headset screen. In an embodiment, the object may appear cropped if it is still present in FOV of the eye. The data from the eye movement constraint module 206B and FOV calculation module 710 (also referred as 208C of Figure 2B; and 708 of Figure 7D) is transmitted to the resize ratio determination module 712 (also referred as 208D of Figure 2B). The eye movement constraint module 206B may identify data related to available eye movement ranges, when the movement of object is restricted 712A. While FOV calculation module 710 provides boundary points of the overlapping region of the left and the right monocular FOV 712B. This data is then combined with the coordinates of the pupil center PupilCenter (X,Y) and the maximum pupil position in X and Y directions, e.g., FinalPupil (X,Y) 712C. Further, the Resize Ratio Determination module 712 is configured to scale the object the ratio of the eye image 712D to obtain scaled coordinates of the object. Thereafter, resize ratio is obtained 712E and 712F, the resize ratio is computed as a percentage of object that need to be resized, to prevent the object from being cropped on VR headset screen. The following equations (16-17) define the Resize Ratio coordinates for X and Y directions:

[0189] (equation 16)

[0190] ( equation 17)

[0191] Figure 8 shows an exemplary illustration 800 of training of a neural network 210B of the system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0192] In one non-limiting embodiment, the correlation generation engine 210 may be a Machine Learning (ML) model that is iteratively trained until a loss function is minimized (e.g., using backward propagation and forward propagation). The prediction values generated by the trained ML model are compared with true values pre-stored in the database. In one embodiment, the true values are the coordinates obtained for a normal eyed person. Upon comparison of the prediction values with the true values, a loss score is generated based on a loss function. Further, the loss value is back propagated and optimized using an optimizer to generate a plurality of weights based on the loss score. The generated weights are assigned to the ML model. This process is repeated (e.g., using forward propagation) until the loss score is minimized.

[0193] Figure 9A shows a block diagram 900A of a correlation generation engine 210 of the system 200B, in accordance with an exemplary embodiment of the present disclosure. The correlation generation 900A is illustrated in Figure 9A. The correlation engine is configured to utilize a neural network to generate a mapping of eye gazing to the coordinates of images. The correlation engine is configured to utilize a neural network to resize ratio of the images is to be displayed onto the VR headset. Parameters from the modules- 202, 206 and 208 are obtained. The eye position sensing module 202 transmits PupilCenter_Pupil (X, Y), InterpupillaryDistance, InterPupillaryPlaneAngle to the preprocessing module 904. The eye movement observation module 206 transmits :ScaledObjectInitial (X,Y), scaledObjectFinal (X,Y), VelocityInitialToFinalObject(X,Y), VelocityInitialToFinalPupil (X,Y), AngleInitialToFinalObject (X,Y) and AngleInitialToFinalPupil (X,Y) to the preprocessing module 904. The FOV determination module 208 transmits : InitialPupil (X,Y), FinalPupil (X,Y) TxFOVBoundaryPoint (X,Y) HeadsetPlaneAngle (X,Y) ViewingPlaneAngle (X,Y) OverlappingRegion (X,Y) ResizeRatio (X,Y) ) to the preprocessing module 904. The preprocessing module 904 is configured to form an input array 904A comprising the transmitted parameters and thereafter normalizes the input array 904B using a Standard Scaler routine. The normalized data are then augmented 904C and combined to form an output array 904D. Due to the augmentation of data, missing values from the normalized data are substituted with valid values.

[0194] Figure 9B shows an exemplary block diagram 900B of the correlation generation engine 210 of the system 200B, in accordance with an exemplary embodiment of the present disclosure.

[0195] Figure 9B depicts two modules - a preprocessing module 904 and a feed forward neural network 906. The feed forward neural network 906 (also referred as 210B of Figure 2B) is configured to obtain a plurality of processed parameters as input from the preprocessing module 904 and outputs an array comprising coordinates of the object (CoordinatesX, CoordinatesY) and resize ratio (ResizeX, ResizeY) 906A. The input parameters may be integer or float values. The feedforward neural network further includes a sequential regressor 906B, an L2 regularization layers 906D and multiple dense layers 906C. These multiple dense layers 906E increase the complexity of the feed forward neural network 906 and provide predicted values in the output array. The feed forward neural network 906 is trained to prevent the issue of overfitting. For example, the feed forward neural network 906 may compile regressor with Adam optimizer and mean squared error loss 906F. The feed forward neural network 906 may fit regressor 906G.

[0196] Figure 10 shows a block diagram 1000 of a coordinate shifting module 1006 of the system 200B for display of a shifted image on the VR device, in accordance with an exemplary embodiment of the present disclosure.

[0197] The coordinate shifting module 1006 (also referred as 212 of Figure 2B) is configured to receive real-time eye information of the user from the pre-stored database 216, the eye localization module 202B, the lens calibration module 204 and the eye movement observation module 206, that can be collectively referred as original image information 1002. The coordinate shifting module 1006 also receives information from the correlation generation engine 1004. Further, the coordinate shifting module 1006 then resizes the image or real-time content visible to the user based on the resize coordinates received from correlation generation engine and also aligns the pupils of the user's eyes with the centers of the resized image or content. Thus, a shifted image 1008 is reproduced on the VR headset exhibiting enhanced depth properties.

[0198] In one embodiment, before receiving the real-time information, original image information is received from the database, where the original information comprises image center coordinates for the left and right eye of the user and image dimensions for the images displayed on the left and right display screens of the VR device. The correlation generation engine 1000 then correlates this original image information with the eye information of the user obtained from the pre-stored database 216, the eye localization module 202B, the lens calibration module 204, eye movement observation module 206 and generates resize coordinates and dimensions to the coordinate shifting module 1006 for further processing.

[0199] Referring now to Figure 11 which illustrates a flow chart of a media content rendering method 1100 for a user having an eye alignment disorder, in accordance with an exemplary embodiment of the present disclosure. The various operations of the method 1100 may be performed by the apparatus 209 of Figure 2BA (particularly, using the processor 203 of the apparatus 209).

[0200] At block 1102, the method 1100 comprising determining a plurality of static eye attributes and a plurality of dynamic eye attributes for the user. In a non-limiting embodiment, the plurality of static eye attributes are determined based at least on a captured eye image of the user. The plurality of dynamic eye attributes are determined based on detecting eye movement of the user interacting with the VR device. In a non-limiting embodiment, the plurality of static eye attributes comprises at least one of: pupil center coordinates of the left eye, pupil center coordinates of the right eye, an interpupillary distance, an inter pupillary angle, and an interpupillary plane and wherein the plurality of dynamic eye attributes comprises at least one of a pupil displacement, an object displacement, an angle of pupil movement, an angle of object movement, a relative speed of eye pupils, a rate of change of angle of pupil movement, and a rate of change of an angle of object movement.

[0201] At block 1104, the method 1100 comprising determining a binocular field-of-view (FOV) of the user based on a left monocular FOV and a right monocular FOV corresponding to a left eye and a right eye of the user. The left monocular FOV and the right monocular FOV are generated based on the plurality of dynamic eye attributes. In a non-limiting embodiment, the left monocular FOV and the right monocular FOV are generated by adjusting the left monocular FOV based on a predefined left FOV, wherein the predefined left FOV is the FOV of a left eye of a normal user without having the eye alignment disorder and adjusting the right monocular FOV based on a predefined right FOV, wherein the predefined right FOV is the FOV of a right eye of the normal user without having the eye alignment disorder. In a non-limiting embodiment, the binocular FOV of the user is determined by determining an overlapping region between the left monocular FOV and the right monocular FOV after being adjusted.

[0202] At block 1106, the method 1100 comprises determining transformation values by a correlation generation engine based on the binocular FOV, while displaying a real-time media content on a VR device. In a non-limiting embodiment, the determining transformation values by the correlation generation engine further comprises generating a mapping of the plurality of static eye attributes, the plurality of dynamic eye attributes, and the binocular FOV to obtain at least a scaling ratio and a plurality of translation coordinates for transforming the real-time media content.

[0203] At block 1108, the method 1100 comprises transforming the real-time media content based on the transformation values to render transformed real-time media content, while displaying a real-time media content on the VR device.

[0204] One of the technical advantages of the techniques described in the present disclosure is enhanced vision. For instance, Strabismus eyed person has FOV different from normal eyed person which results sometimes the image on screen on the VR device to be cropped or being partially visible due to screen dimension restriction. The techniques facilitate full image display without getting cropped or going out of FOV of user. The techniques also facilitate zooming in or out the image according to headset position, eye position & head Movement etc, thereby providing improved user experience. Furthermore, the problem of Double Vision faced by a strabismus eyed person is Resolved using Intelligent Correlation & Calibration.

[0205] The techniques also aid a Person with Nystagmus Eyes that face issue of correct aligning of visual content and as a result causing several problems such as Shaky or Blurry vision, Balance Problem, and Oscillopsia. Oscillopsia is a visual condition where objects appear to move or oscillate involuntarily. With present technique of accurate eye Tracking, user can view the content without Shaky or Blurry Vision. Further, a Strabismus eyed User need not wear Prism Glasses inside the VR headset for vision correction. The proposed techniques dynamically correct the vision of the user based enhanced Correlation & Calibration techniques as mentioned in the present disclosure.

[0206] The above method 1100 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, and functions, which perform specific functions or implement specific abstract data types.

[0207] The order in which the various operations of the methods are described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method. Additionally, individual blocks may be deleted from the methods without departing from the scope of the subject matter described herein. Furthermore, the methods can be implemented in any suitable hardware, software, firmware, or combination thereof.

[0208] The various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software component(s) and / or module(s) shown in Figures 2A-2B. Generally, where there are operations illustrated in Figures, those operations may have corresponding counterpart means-plus-function components. It may be noted here that the subject matter of some or all embodiments described with reference Figures 2A-2B may be relevant for the method 1100 and the same is not repeated for the sake of brevity.

[0209] Figure 12 illustrates a flow chart of a method 1200 for media content rendering, in accordance with an exemplary embodiment of the present disclosure.

[0210] The method may include determining 1202 a plurality of eye attributes of a user, based on eye position for each of a left eye and a right eye. The method may include identifying 1204 eye movement for each of the left eye and right eye, based on the plurality of eye attributes. The method may include determining 1206 a constraint for a movement of each of the left eye and right eye, based on the eye movement. The method may include determining 1208 a binocular field-of-view (FOV) of the user, based on the constraint for each of the left eye and right eye. The method may include determining 1210 transformation values, based on the binocular FOV of the user. The method may include providing 1212 a transformed media content based on the transformation values to render the transformed media content.

[0211] In a non-limiting embodiment of the present disclosure, one or more non-transitory computer-readable media may be utilized for implementing the embodiments consistent with the present disclosure. Certain aspects may comprise a computer program product for performing the operations presented herein. For example, such a computer program product may comprise a computer readable media having instructions stored (and / or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein. For certain aspects, the computer program product may include packaging material.

[0212] In the present disclosure, at least one of the plurality of modules may be implemented through an AI model. A function associated with AI may be performed through the non-volatile memory, the volatile memory, and the processor.

[0213] The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-dedicated processor such as a neural processing unit (NPU).

[0214] The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning.

[0215] Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or AI model of a desired characteristic is made. The learning may be performed in a device itself in which AI according to an embodiment is performed, and / o may be implemented through a separate server / system.

[0216] The AI model may consist of a plurality of neural network layers. Each layer has a plurality of weight values and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[0217] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0218] Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0219] According to an embodiment of the disclosure, a media content rendering method may be provided. The method may include determining a plurality of eye attributes of a user, based on eye position for each of a left eye and a right eye. The method may include identifying eye movement for each of the left eye and right eye, based on the plurality of eye attributes. The method may include determining a constraint for a movement of each of the left eye and right eye, based on the eye movement. The method may include determining a binocular field-of-view (FOV) of the user, based on the constraint for each of the left eye and right eye. The method may include determining transformation values, based on the binocular FOV of the user. The method may include providing a transformed media content based on the transformation values to render the transformed media content.

[0220] According to an embodiment of the disclosure, the method may include obtaining an eye image. The eye image may comprise a first eye image corresponding to the left eye and a second eye image corresponding to the right eye. The method may include determining a first coordinate corresponding to the left eye. The method may include determining a second coordinate corresponding to the right eye.

[0221] According to an embodiment of the disclosure, the method may include identifying an interpupillary distance between the left eye and the right eye based on the eye position. The method may include identifying an interpupillary angle between the left eye and the right eye based on the eye position. The method may include determining an interpupillary plane between the left eye and the right eye based on the eye position.

[0222] According to an embodiment of the disclosure, the method may include identifying a displacement of the eye movement for each of the left eye and the right eye. The method may include identifying an angle deviation of the eye movement for each of the left eye and the right eye, based on the displacement of the eye movement. The method may include identifying a velocity of the eye movement for each of the left eye and the right eye, based on the displacement of the eye movement.

[0223] According to an embodiment of the disclosure, the method may include identifying a change rate of an angle corresponding to the eye movement for each of the left eye and the right eye. The method may include identifying a change rate of a velocity corresponding to the eye movement for each of the left eye and the right eye. The method may include determining a boundary of eye movement for each of the left eye and the right eye.

[0224] According to an embodiment of the disclosure, the method may include determining a first monocular field-of-view (FOV) corresponding to the left eye by adjusting a predefined left FOV. The method may include determining a second monocular field-of-view (FOV) corresponding to the right eye by adjusting a predefined right FOV. The method may include determining an overlapping region between the first monocular FOV and the second monocular FOV.

[0225] According to an embodiment of the disclosure, the method may include determining a scaling ratio of the media content for each of the left eye and the right eye, based on the constraint and the binocular FOV of the user.

[0226] According to an embodiment of the disclosure, an electronic apparatus for rendering media content may be provided. The electronic apparatus may include memory storing one or more instructions. and at least one processor. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a plurality of eye attributes of a user, based on eye position for each of a left eye and a right eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify eye movement for each of the left eye and right eye, based on the plurality of eye attributes. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a constraint for a movement of each of the left eye and right eye, based on the eye movement. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a binocular field-of-view (FOV) of the user, based on the constraint for each of the left eye and right eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine transformation values, based on the binocular FOV of the user. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to provide a transformed media content based on the transformation values to render the transformed media content.

[0227] According to an embodiment of the disclosure, the instructions when executed by the at least one processor individually or collectively, cause the apparatus to obtain an eye image, wherein the eye image comprises a first eye image corresponding to the left eye and a second eye image corresponding to the right eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a first coordinate corresponding to the left eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a second coordinate corresponding to the right eye.

[0228] According to an embodiment of the disclosure, the instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify an interpupillary distance between the left eye and the right eye based on the eye position. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify an interpupillary angle between the left eye and the right eye based on the eye position. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine an interpupillary plane between the left eye and the right eye based on the eye position.

[0229] According to an embodiment of the disclosure, the instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify a displacement of the eye movement for each of the left eye and the right eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify an angle deviation of the eye movement for each of the left eye and the right eye, based on the displacement of the eye movement. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify a velocity of the eye movement for each of the left eye and the right eye, based on the displacement of the eye movement.

[0230] According to an embodiment of the disclosure, the instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify a change rate of an angle corresponding to the eye movement for each of the left eye and the right eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to identify a change rate of a velocity corresponding to the eye movement for each of the left eye and the right eye. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a boundary of eye movement for each of the left eye and the right eye.

[0231] According to an embodiment of the disclosure, the instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a first monocular field-of-view (FOV) corresponding to the left eye by adjusting a predefined left FOV. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a second monocular field-of-view (FOV) corresponding to the right eye by adjusting a predefined right FOV. The instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine an overlapping region between the first monocular FOV and the second monocular FOV.

[0232] According to an embodiment of the disclosure, the instructions when executed by the at least one processor individually or collectively, cause the apparatus to determine a scaling ratio of the media content for each of the left eye and the right eye, based on the constraint and the binocular FOV of the user.

[0233] According to an embodiment of the disclosure, a computer-readable medium storing one or more instructions may be provided. The one or more instructions, when executed by at least one processor, may cause the at least one processor of an electronic apparatus to perform operation corresponding to the method.

[0234] One or more shortcomings discussed above may be overcome and additional advantages may be provided by the present disclosure.

[0235] The present disclosure provides techniques for generating a vision with enhanced depth perception for the user with misaligned / strabismus eyes. The techniques discloses determining, by a sensor module of Virtual Reality (VR) device a position of both left and right eye of the user, a distance, and an angle between the eyes. The techniques further disclose determining the position of lens with respect to eyes and adjusting the position of lens according to eyes in a limited space. The techniques further disclose calculating the speed of both eyes relative to each other, movement of eyes with respect to head position and thus finding the exact frame of reference to be considered. The techniques further disclose calculating the monocular and binocular FOV of user for multiple scenarios with respect to movements of head and eyes. Further, a correlation generation engine is provided to generate a mapping of eye gazing to the coordinates of images displayed on VR device. Here, resizing such as zooming and contracting of the content is facilitated by the present disclosure through analysis of the image depth and field-of-view (FOV). Finally, the coordinates of displayed images are shifted to the frame of reference obtained in above steps and to provide an enhanced vision to the user.

[0236] According to an aspect of the present disclosure, methods, systems, and devices are provided for media content rendering for a user having an eye alignment disorder such as strabismus eyes.

[0237] In a non-limiting embodiment of the present disclosure, the present disclosure discloses a media content rendering method for a user having an eye alignment disorder. The method comprising determining a plurality of static eye attributes and a plurality of dynamic eye attributes for the user. The method comprising determining a binocular field-of-view (FOV) of the user based on a left monocular FOV and a right monocular FOV corresponding to a left eye and a right eye of the user. The left monocular FOV and the right monocular FOV are generated based on the plurality of dynamic eye attributes. The method comprises, while displaying a real-time media content on the VR device, determining transformation values by a correlation generation engine based on the binocular FOV. The method comprises, while displaying a real-time media content on a virtual reality (VR) device, transforming the real-time media content based on the transformation values to render transformed real-time media content.

[0238] In a non-limiting embodiment, the left monocular FOV and the right monocular FOV are generated by adjusting the left monocular FOV based on a predefined left FOV, wherein the predefined left FOV is the FOV of a left eye of a normal user without having the eye alignment disorder and adjusting the right monocular FOV based on a predefined right FOV, wherein the predefined right FOV is the FOV of a right eye of the normal user without having the eye alignment disorder.

[0239] In a non-limiting embodiment, the binocular FOV of the user is determined by determining an overlapping region between the left monocular FOV and the right monocular FOV after being adjusted.

[0240] In a non-limiting embodiment, the plurality of static eye attributes are determined based at least on a captured eye image of the user. The plurality of dynamic eye attributes are determined based on detecting eye movement of the user interacting with the VR device.

[0241] In a non-limiting embodiment, the plurality of static eye attributes comprises at least one of: pupil center coordinates of the left eye, pupil center coordinates of the right eye, an interpupillary distance, an inter pupillary angle, and an interpupillary plane and wherein the plurality of dynamic eye attributes comprises at least one of a pupil displacement, an object displacement, an angle of pupil movement, an angle of object movement, a relative speed of eye pupils, a rate of change of angle of pupil movement, and a rate of change of an angle of object movement.

[0242] In a non-limiting embodiment, the determining transformation values by the correlation generation engine further comprises generating a mapping of the plurality of static eye attributes, the plurality of dynamic eye attributes, and the binocular FOV to obtain at least a scaling ratio and a plurality of translation coordinates for transforming the real-time media content.

[0243] In a non-limiting embodiment of the present disclosure, the present application discloses a media content rendering apparatus for a user having an eye alignment disorder. The apparatus comprises a memory; and at least one processor coupled to the memory. The processor is configured to determine a plurality of static eye attributes and a plurality of dynamic eye attributes for the user. The processor is configured to determine a binocular field-of-view (FOV) of the user based on a left monocular FOV and a right monocular FOV corresponding to a left eye and a right eye of the user. The left monocular FOV and the right monocular FOV are generated based on the plurality of dynamic eye attributes. The processor is configured to perform, while displaying a real-time media content on a virtual reality (VR) device, operations of determining transformation values by a correlation generation engine based on the binocular FOV and transforming the real-time media content based on the transformation values to render transformed real-time media content.

[0244] In a non-limiting embodiment, the at least one processor is configured to generate the left monocular FOV and the right monocular FOV by adjusting the left monocular FOV based on a predefined left FOV, wherein the predefined left FOV is the FOV of a left eye of a normal user without having the eye alignment disorder and adjusting the right monocular FOV based on a predefined right FOV, wherein the predefined right FOV is the FOV of a right eye of the normal user without having the eye alignment disorder.

[0245] In a non-limiting embodiment, the at least one processor is configured to determine the binocular FOV by determining an overlapping region between the left monocular FOV and the right monocular FOV after being adjusted.

[0246] In a non-limiting embodiment, the plurality of static eye attributes are determined based at least on a captured eye image of the user. The plurality of dynamic eye attributes are determined based on detecting eye movement of the user interacting with the VR device.

[0247] In a non-limiting embodiment, the at least one processor is configured to the plurality of static eye attributes comprises at least one of: pupil center coordinates of the left eye, pupil center coordinates of the right eye, an interpupillary distance, an inter pupillary angle, and an interpupillary plane. The plurality of dynamic eye attributes comprises at least one of a pupil displacement, an object displacement, an angle of pupil movement, an angle of object movement, a relative speed of eye pupils, a rate of change of angle of pupil movement, and a rate of change of an angle of object movement.

[0248] In a non-limiting embodiment, for determining the transformation values by the correlation generation engine, the at least one processor is further configured to generate a mapping of the plurality of static eye attributes, the plurality of dynamic eye attributes, and the binocular FOV to obtain at least a scaling ratio and a plurality of translation coordinates for transforming the real-time media content.

[0249] The techniques of the present disclosure provide various technical advantages. For example, the techniques of the present disclosure provide enhanced vision with improved depth perception to a user of a VR device. In an example, the techniques of the present disclosure reduce discomfort and unease suffered by a person with an eye alignment disorder, when interacting with an AR / VR device. In an example, the techniques of the present disclosure eliminate the challenges such as double vision, blurred vision, reduced depth, visual confusion, and cropped viewpoint that are faced by VR device users having misaligned eyes. In an example, the techniques of the present disclosure consider the changes associated with a headset position, an interpupillary plane angle difference, movement of eyes in different directions and FOV adjustments for providing an enhanced vision to the user with misaligned eyes.

Claims

1.A media content rendering method (1200), the method comprising:determining (1202) a plurality of eye attributes of a user, based on eye position for each of a left eye and a right eye;identifying (1204) eye movement for each of the left eye and right eye, based on the plurality of eye attributes;determining (1206) a constraint for a movement of each of the left eye and right eye, based on the eye movement;determining (1208) a binocular field-of-view (FOV) of the user, based on the constraint for each of the left eye and right eye;determining (1210) transformation values, based on the binocular FOV of the user; andproviding (1212) a transformed media content, based on the transformation values to render the transformed media content.2.The method of claim 1,wherein the determining the plurality of eye attributes comprises:obtaining an eye image, wherein the eye image comprises a first eye image corresponding to the left eye and a second eye image corresponding to the right eye;determining a first coordinate corresponding to the left eye; anddetermining a second coordinate corresponding to the right eye.3.The method any one of claims 1 and 2,wherein the determining the plurality of eye attributes comprises:identifying an interpupillary distance (402) between the left eye and the right eye based on the eye position;identifying an interpupillary angle (404) between the left eye and the right eye based on the eye position; anddetermining an interpupillary plane (406) between the left eye and the right eye based on the eye position.4.The method any one of claims 1 to 3,wherein the identifying eye movement for each of the left eye and right eye comprises:identifying a displacement of the eye movement for each of the left eye and the right eye;identifying an angle deviation of the eye movement for each of the left eye and the right eye, based on the displacement of the eye movement; andidentifying a velocity of the eye movement for each of the left eye and the right eye, based on the displacement of the eye movement.5.The method any one of claims 1 to 4,wherein the determining a constraint for a movement of each of the left eye and right eye comprises:identifying a change rate of an angle corresponding to the eye movement for each of the left eye and the right eye;identifying a change rate of a velocity corresponding to the eye movement for each of the left eye and the right eye; anddetermining a boundary of eye movement for each of the left eye and the right eye.6.The method any one of claims 1 to 5,wherein the determining a binocular field-of-view (FOV) based on the eye constraint comprises:determining a first monocular field-of-view (FOV) (708A) corresponding to the left eye by adjusting a predefined left FOV;determining a second monocular field-of-view (FOV) (708B) corresponding to the right eye by adjusting a predefined right FOV; anddetermining an overlapping region (708C) between the first monocular FOV and the second monocular FOV.7.The method any one of claims 1 to 6,wherein the determining transformation values comprises:determining a scaling ratio of the media content for each of the left eye and the right eye, based on the constraint and the binocular FOV of the user.8.An electronic apparatus (200A) for rendering media content comprising:memory storing one or more instructions; andat least one processor,wherein the instructions, when executed by the at least one processor individually or collectively, cause the apparatus (200A) to:determine a plurality of eye attributes of a user, based on eye position for each of a left eye and a right eye;identify eye movement for each of the left eye and right eye, based on the plurality of eye attributes;determine a constraint for a movement of each of the left eye and right eye, based on the eye movement;determine a binocular field-of-view (FOV) of the user, based on the constraint for each of the left eye and right eye;determine transformation values, based on the binocular FOV of the user; andprovide a transformed media content, based on the transformation values to render the transformed media content.9.The electronic apparatus of claim 8,wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the apparatus to:obtain an eye image, wherein the eye image comprises a first eye image corresponding to the left eye and a second eye image corresponding to the right eye;determine a first coordinate corresponding to the left eye; anddetermine a second coordinate corresponding to the right eye.10.The electronic apparatus any one of claims 8 and 9,wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the apparatus to:identify an interpupillary distance (402) between the left eye and the right eye based on the eye position;identify an interpupillary angle (404) between the left eye and the right eye based on the eye position; anddetermine an interpupillary plane (406) between the left eye and the right eye based on the eye position.11.The electronic apparatus any one of claims 8 to 10,wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the apparatus to:identify a displacement of the eye movement for each of the left eye and the right eye;identify an angle deviation of the eye movement for each of the left eye and the right eye, based on the displacement of the eye movement; andidentify a velocity of the eye movement for each of the left eye and the right eye, based on the displacement of the eye movement.12.The electronic apparatus any one of claims 8 to 11,wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the apparatus to:identify a change rate of an angle corresponding to the eye movement for each of the left eye and the right eye;identify a change rate of a velocity corresponding to the eye movement for each of the left eye and the right eye; anddetermine a boundary of eye movement for each of the left eye and the right eye.13.The electronic apparatus any one of claims 8 to 12,wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the apparatus to:determine a first monocular field-of-view (FOV) (708A) corresponding to the left eye by adjusting a predefined left FOV;determine a second monocular field-of-view (FOV) (708B) corresponding to the right eye by adjusting a predefined right FOV; anddetermine an overlapping region (708C) between the first monocular FOV and the second monocular FOV.14.The electronic apparatus any one of claims 8 to 13,wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the apparatus to:determine a scaling ratio of the media content for each of the left eye and the right eye, based on the constraint and the binocular FOV of the user.15.A computer-readable medium storing one or more instructions, wherein the one or more instructions, when executed by at least one processor, cause the at least one processor of an electronic apparatus to perform the method of any one the claims 1 to 7.

Images