Method and system of extended reality environment interaction based on eye motion

Abstract

To provide systems and methods for extended reality environment interaction based on eye motions.SOLUTION: Systems and methods are described for extended reality environment interaction. An extended reality environment including an object is generated for display, and a first sensor is used to detect that a gaze has shifted from a first portion of the extended reality environment to a second portion of the extended reality environment, the object being excluded from the first portion of the extended reality environment and included in the second portion of the extended reality environment. An indicator of the shift in the gaze is generated for display within the extended reality environment in response to detecting the gaze shift, and a voice command is detected by a second sensor while the indicator is in a vicinity of the object.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to improved extended reality environment interaction, and in particular, a system and method for detecting eye movement and performing actions within an extended reality environment based on the detected eye movement is disclosed. Summary of the Invention [Means for solving the problem]

[0002] Advances in media technology have led to the development of extended reality (XR) technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. VR systems may fully immerse a user (e.g., giving the user the sense of being in the environment) or partially immerse (e.g., giving the user the sense of viewing the environment) in a three-dimensional computer-generated environment. The environment may include objects or items with which the user can interact. AR systems may provide a modified version of reality, such as augmented information overlaid over real-world objects. MR systems map interactive virtual objects onto the real world. Such systems may utilize wearables, such as head-mounted devices with stereoscopic displays or smart glasses.

[0003] XR systems introduce many challenges. For example, because the wearable device being used to view the environment may not include external devices (e.g., lenses), it may be difficult for the XR system to detect when a user alters their field of view or focus within the XR environment. As another example, pupil dilation and constriction may vary depending on what the user is viewing within the XR environment or the amount of light entering the user's eyes, but the user may not have control over their pupils, and thus monitoring the user's pupils may not be a reliable way to determine a user's gaze or field of view within an XR environment. Worse yet, even if the user's field of view is accurately ascertained, it may be difficult to determine which object the user wishes to interact with if multiple objects are present within the user's field of view.

[0004] Additionally, current approaches to XR suffer from certain drawbacks. In one approach, a user employs hand gestures or a joystick to navigate an XR environment. However, requiring such user input to interact with an XR environment can be cumbersome or inconvenient for the user, not to mention detracting from the XR experience (i.e., reminding the user that the XR environment is not real). Additionally, current approaches to XR may not allow a user to conveniently obtain information about objects within their field of view or to interact within the XR environment.

[0005] To overcome these problems, systems and methods are provided herein for identifying an object within a user's field of view, detecting the user's eyelid movement, and regenerating the object for display within an extended reality environment with a modified level of detail based on such detection. The systems and methods described herein also provide for matching detected eyelid movement with a stored eyelid movement identifier and performing an action on the object based on such matching. Additionally, systems and methods are provided for generating an indicator to reflect a user's gaze shift to a new portion of the extended reality environment that includes the object, and for performing the action when a voice command is received while the indicator is near the object. The systems and methods described herein also provide for generating an opacity-based indicator for display within the extended reality environment near a portion of the extended reality environment that includes the object, and varying the opacity of such indicator based on the identified boundary of the object. Additionally, systems and methods are provided for enabling a user to conveniently obtain additional information about items within an extended reality environment.

[0006] In some aspects of the present disclosure, an extended reality system generates an extended reality environment for display, including a first object, and receives input from one or more sensors. Based on the received input, the system identifies the first object within a field of view, detects eyelid movement, and, in response to detecting the eyelid movement, regenerates the first object for display with a modified level of detail. Thus, eyelid movement can be monitored to overcome challenges associated with determining an object within a user's field of view that is of interest to the user. Additionally, detecting such eyelid movement of the user may allow the user to view finer details of objects that appear to be far away from the user within the extended reality environment, for example, which may improve the user experience in the extended reality system, particularly for users with visual impairments.

[0007] The extended reality environment may comprise a plurality of objects, including a first object and a second object, within a field of view, and the system may re-generate for display the first object with a modified level of detail in response to determining that the detected eyelid movement is associated with the first object. If the system determines that the detected eyelid movement is associated with a second object, the system may re-generate for display the second object with a modified level of detail. The first object may be in one of the foreground or background within the field of view within the extended reality environment, and the second object may be in the other of the foreground or background within the field of view within the extended reality environment.

[0008] In some embodiments, reproducing the first object for display with the modified level of detail includes presenting the object at a higher resolution. Additionally, or alternatively, one or more actions may be performed on the first object based on the one or more detected eyelid movements.

[0009] In some aspects of the present disclosure, the system calculates individual virtual distances of multiple objects relative to the user, and identifying a first object within the field of view includes determining that the first object is at a virtual distance closest to the user of the individual virtual distances.

[0010] In some embodiments, detecting eyelid movement includes determining an amount of eyelid movement and / or detecting eyelid movement includes determining one or more eyelid levels. The system may detect that a user is navigating from a first location to a new location within the extended reality environment while the first object remains within the field of view, and generate an updated version of the first object for display based on the user's viewpoint at the new location.

[0011] In some aspects of the present disclosure, an extended reality system generates an extended reality environment for display, including an object, and stores in memory a table of eyelid movement identifiers and corresponding actions that can be performed on the object in the extended reality environment. Using a sensor, the system detects eyelid movement and matches the detected eyelid movement to one of the stored eyelid movement identifiers. In response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, the system generates an updated version of the extended reality environment for display based on the action corresponding to the matched eyelid movement. Thus, eyelid movement can be monitored to overcome challenges associated with determining an object in a field of view with which a user wishes to interact. Additionally, detecting such eyelid movement of a user allows the user to interact with objects that appear to be far away from the user in the extended reality environment, which may improve the user experience in the extended reality system, particularly for users with visual impairments.

[0012] An object may be selected from a plurality of objects in the extended reality environment by detecting that a user's gaze is directed toward the object. The system may generate for display a subset of eyelid movement identifiers operable for the object to which the user's gaze is directed (e.g., to remind or guide the user regarding an action that a certain eyelid movement causes to be performed). An action of the plurality of actions may correspond to manipulating the object and / or altering the appearance of the object. (For example, if the object is a book, the action may be turning a page of the book, tilting the book, tearing a page of the book, etc.) The system may detect that the user navigates from a first location to a new location in the extended reality environment while the user's gaze remains on the object, and generate for display an updated version of the first object, the updated version having an altered appearance, based on the user's viewpoint at the new location.

[0013] In some embodiments, a user may be associated with a user profile that defines a relationship between eyelid movement identifiers and corresponding actions that can be performed on objects in the extended reality environment. The actions that can be performed on an object may vary based on the type of object. To detect eyelid movement, the system may determine whether the eyelid remains closed for a predetermined period of time and, in response to determining that the eyelid remains closed for the predetermined period of time (e.g., to ensure that the eyelid movement is not an involuntary blink), match the detected eyelid movement to one of the stored eyelid movement identifiers.

[0014] In some aspects of the present disclosure, an extended reality system generates an extended reality environment for display, the extended reality environment comprising an object, and detects using a first sensor a shift in gaze from a first portion of the extended reality environment to a second portion of the extended reality environment, where the object is excluded from the first portion of the extended reality environment and included within the second portion of the extended reality environment. In response to detecting the gaze shift, the system generates an indicator of the shift in gaze for display within the extended reality environment, and using a second sensor, detects a voice command while the indicator is in the vicinity of the object. In response to detecting the voice command, the extended reality system performs an action corresponding to the voice command. Thus, extended reality may be utilized in combination with voice to improve the user experience. More specifically, a user may conveniently use their eyes to navigate the extended reality environment (e.g., as a surrogate for how a mouse or trackpad is used with a desktop, laptop, or mobile device), receive real-time confirmation regarding the location of their gaze, and perform desired actions within the environment via voice commands when an indicator of the user's gaze is in the vicinity of an object of interest within the extended reality environment.

[0015] An interactive media guide may be provided on the display, and the actions referred to above may be instructions related to media assets accessible via the interactive media guide. The voice commands may include commands to identify and perform actions on media assets, and / or instructions to present new media assets on the display and / or instructions to retrieve content related to an entity, and the object is associated with the entity.

[0016] In some embodiments, the extended reality system may determine whether the rate of retinal movement exceeds a predetermined value, and in response to determining that the rate of retinal movement exceeds the predetermined value, normalize the retinal movement when translating the retinal movement into movement of an indicator on a display. In response to determining that gaze has been directed at the object for at least a predetermined threshold period, the system may detect a voice command while the indicator is near (e.g., overlapping) the object. The display is presented via a virtual reality head-mounted device.

[0017] In some aspects of the present disclosure, an extended reality system may generate an extended reality environment for display that includes an object and detect, using a sensor, that a gaze is directed toward a first portion of the extended reality environment, the object being contained within the first portion of the extended reality environment. The extended reality system may generate a plurality of opacity-based indicators for display within the extended reality environment near the first portion of the extended reality environment, identify boundaries of the object, and vary the opacity of at least one of the plurality of opacity-based indicators based on the identified boundaries of the object. Thus, a user may conveniently use their eyes to navigate the extended reality environment (e.g., as a surrogate for the way a mouse or trackpad is used with a desktop, laptop, or mobile device) and receive real-time confirmation regarding the location of their gaze, with the opacity of such real-time gaze indicators conveniently adjusted to avoid obscuring the user's field of view and degrading the user's experience.

[0018] The extended reality system may determine whether at least one of the opacity-based indicators overlaps a boundary of the object and vary an individual opacity of the opacity-based indicator that overlaps the boundary. The multiple opacity-based indicators are arrows directed toward the object. The extended reality system may detect whether gaze has shifted to a second portion of the extended reality environment by using a sensor, and in response to determining that gaze has shifted to the second portion, overlay the multiple opacity-based indicators near the second portion of the display.

[0019] In some embodiments, the individual opacities are varied based on the distance from the object. For example, the individual opacities of the indicators may increase as the distance between the indicator and the object decreases (e.g., to emphasize the object the user is gazing at), or may increase as the distance between the indicator and the object decreases (e.g., to avoid obscuring the object the user is gazing at).

[0020] In some embodiments, an interactive media guide may be provided on a display, and actions related to media assets accessible through the interactive media guide may be received based, at least in part, on the detected gaze. Such a display may be presented via a virtual reality head-mounted device or without the use of a virtual reality head-mounted device.

[0021] In some aspects of the present disclosure, an extended reality system generates an extended reality environment for display including an object, detects eye movement, and determines, based on the detection, whether the object is within the field of view for at least a predetermined period of time. In response to determining that the object is within the field of view for at least the predetermined period of time, the system generates one or more items related to the object for display within the extended reality environment. Thus, information related to an object of interest may be conveniently displayed to a user based on detecting the user's eye movement related to the object of interest.

[0022] The one or more items associated with the object may comprise text information, images, videos, or any combination thereof. The system may further determine that at least a second predetermined period of time has elapsed since the start of display of the one or more items without the object being present in the field of view for a first predetermined period of time, and in response to such a determination, cease displaying the one or more items. The extended reality environment may be presented via a virtual reality head-mounted device. In some embodiments, detecting eye movement includes monitoring eyelid movement or monitoring gaze.

[0023] The system may determine whether an object is in the field of view for a predetermined period of time in response to determining that the field of view is continuously (or non-continuously) on the object for a predetermined period of time during the virtual reality session.

[0024] In some embodiments, the system may determine that a new object is within the field of view for at least a predetermined amount of time, and in response to such determination, generate one or more items related to the new object for display within the extended reality environment while continuing to generate one or more items related to the object for display within the extended reality environment. The present invention provides, for example, the following. (Item 1) 1. A method for extended reality environment interaction, comprising: generating an extended reality environment comprising the object for display; detecting, using a first sensor, a shift in gaze from a first portion of the extended reality environment to a second portion of the extended reality environment, wherein the object is excluded from the first portion of the extended reality environment and included within the second portion of the extended reality environment; in response to detecting the gaze shift, generating an indicator of the shift in gaze for display within the extended reality environment; detecting a voice command while the indicator is in proximity to the object by using a second sensor; In response to detecting the voice command, performing an action corresponding to the voice command. A method comprising: (Item 2) Item 10. The method of item 1, wherein an interactive media guide is provided on the display and the actions are instructions related to media assets accessible via the interactive media guide. (Item 3) The gaze is detected based on retinal movement of the eye, and the method further comprises: 3. The method of claim 1 or 2, comprising translating the retinal movement into a movement of the indicator on the display. (Item 4) determining whether the rate of retinal movement exceeds a predetermined value; normalizing the retinal movement when converting the retinal movement to a movement of the indicator on the display in response to determining that the rate of the retinal movement exceeds the predetermined value; Item 4. The method of item 3, further comprising: (Item 5) 5. The method of any of items 2-4, wherein the voice command includes an identification of the media asset and a command to perform the action. (Item 6) 6. The method of any of items 2-5, wherein the voice command includes an instruction to present a new media asset on the display. (Item 7) 7. The method of any of items 2-6, wherein the object is associated with an entity and the voice command comprises an instruction to retrieve content related to the entity. (Item 8) 8. The method of any of items 1-7, wherein detecting the voice command while the indicator is in the vicinity of the object includes determining that the gaze is directed toward the object for at least a predetermined threshold period. (Item 9) 9. The method of any of items 1-8, wherein the indicator being in the vicinity of the object includes the indicator overlapping the object. (Item 10) 10. The method of any of items 1-9, wherein the display is presented via an extended reality head-mounted device. (Item 11) 11. A system for extended reality environment interaction, the system comprising control circuitry configured to perform the method of any of items 1-10. (Item 12) 1. A system for extended reality environment interaction, comprising: The display and A control circuitry comprising: generating an extended reality environment comprising the object for display; detecting, using a first sensor, a shift in gaze from a first portion of the extended reality environment to a second portion of the extended reality environment, wherein the object is excluded from the first portion of the extended reality environment and included within the second portion of the extended reality environment; in response to detecting the gaze shift, generating an indicator of the shift in gaze for display within the extended reality environment; detecting a voice command while the indicator is in proximity to the object by using a second sensor; In response to detecting the voice command, performing an action corresponding to the voice command. control circuitry configured to A system comprising: (Item 13) 13. The system of claim 12, wherein an interactive media guide is provided on the display, and the actions are instructions related to media assets accessible via the interactive media guide. (Item 14) The control circuitry further comprises: detecting said gaze based on retinal movement of the eye; converting said retinal movement into a movement of said indicator on said display; 14. The system according to item 12 or 13, configured to perform the following: (Item 15) 11. A non-transitory computer-readable medium having instructions encoded thereon that, when executed by control circuitry, cause the control circuitry to perform the method of any of items 1-10. [Brief explanation of the drawings]

[0025] These and other objects and advantages of the present disclosure will become apparent upon consideration of the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout.

[0026] [Figure 1] FIG. 1 illustrates an example of regenerating an object for display within an extended reality environment, according to some embodiments of the present disclosure.

[0027] [Figure 2] FIG. 2 illustrates an example of regenerating an object for display within an extended reality environment, according to some embodiments of the present disclosure.

[0028] [Figure 3] FIG. 3 illustrates an example of performing an action on an object in an extended reality environment, according to some embodiments of the present disclosure.

[0029] [Figure 4] 4A-4B show examples of receiving a voice command while an indicator is in the vicinity of an object in an extended reality environment, according to some embodiments of the present disclosure.

[0030] [Figure 5] FIG. 5 illustrates an example of receiving a voice command while an indicator is in the vicinity of an object in an extended reality environment, according to some embodiments of the present disclosure.

[0031] [Figure 6] FIG. 6 illustrates an example of receiving a voice command while an indicator is in the vicinity of an object in an extended reality environment, according to some embodiments of the present disclosure.

[0032] [Figure 7] FIG. 7 illustrates an example of presenting information related to an item in an extended reality environment, according to some embodiments of the present disclosure.

[0033] [Figure 8] FIG. 8 is a block diagram of an illustrative device in an extended reality system according to some embodiments of the present disclosure.

[0034] [Figure 9] FIG. 9 is a block diagram of an illustrative extended reality system according to some embodiments of the present disclosure.

[0035] [Figure 10] FIG. 10 is a flowchart of a detailed illustrative process for regenerating objects for display within an extended reality environment, according to some embodiments of the present disclosure.

[0036] [Figure 11] FIG. 11 is a flowchart of a detailed illustrative process for regenerating objects for display within an extended reality environment, according to some embodiments of the present disclosure.

[0037] [Figure 12] FIG. 12 is a flowchart of a detailed illustrative process for performing an action on an object in an extended reality environment, according to some embodiments of the present disclosure.

[0038] [Figure 13] FIG. 13 is a flowchart of a detailed illustrative process for receiving a voice command while an indicator is in the vicinity of an object in an extended reality environment, according to some embodiments of the present disclosure.

[0039] [Figure 14]FIG. 14 is a flowchart of a detailed illustrative process for varying the opacity of an indicator in an extended reality environment, according to some embodiments of the present disclosure.

[0040] [Figure 15] FIG. 15 is a flowchart of a detailed illustrative process for presenting additional information related to an item within an extended reality environment, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0041] Detailed Description FIG. 1 illustrates an exemplary process for regenerating objects for display within an extended reality (XR) environment according to some embodiments of the present disclosure. A head-mounted display 102 may project images and generate a three-dimensional XR environment 100 for immersing a user therein. A user may be fully or partially immersed in the XR environment 100, which may be a completely virtual environment. The head-mounted display 102 may alternatively be a wearable device (e.g., smart glasses) or a computer or mobile device equipped with a camera and an XR application to facilitate the generation of the environment 100. The environment 100 may alternatively be an augmented reality (AR) environment, in which real-world objects are supplemented with computer-generated objects or information, or a mixed reality (MR), in which, for example, virtual objects interact with or are otherwise connected to the real world. In some embodiments, a user's field of view or perspective of the environment 100 changes as the user moves their head, and other features (e.g., audio) are suitably modified to simulate the physical world. The environment 100 may be for entertainment purposes (e.g., video games, movies, videos, sports, etc.), communication (e.g., social media), educational purposes (e.g., virtual classrooms), professional purposes (e.g., training simulations), medical purposes, etc.

[0042] The XR system may identify one or more objects within the user's field of view. The field of view is the portion of the XR environment 100 (e.g., an angle in a 360-degree spherical environment) presented to the user at a given time by the display 102. In the case of a VR device, the field of view may comprise a pair of 2D images to generate a stereoscopic view, and in the case of an AR device (e.g., smart glasses), the field of view may comprise a 3D or 2D image that may include a mixture of real objects and virtual objects overlaid on top of the real objects using the AR device (e.g., for smart glasses, photos captured with a camera and content added by the smart glasses). If the XR environment has a single degree of freedom, e.g., 360 degrees of rotation, any field of view may be defined either by an edge angle coordinate (e.g., +135 degrees, +225 degrees) or by a single angle coordinate (e.g., -55 degrees) combined with a known angular opening of the field of view. If an XR environment has six degrees of freedom, i.e., three rotations of 360 degrees and three spatial positions, any field of view may be defined by three angular coordinates and three spatial coordinates. The field of view may therefore be understood as the part of the XR environment that is displayed when the user is in a particular location within the XR environment and has the XR set oriented in a particular direction.

[0043] An XR system (e.g., system 900 of FIG. 9 ) may generate a data structure related to the field of view, including object identifiers associated with virtual objects in the field of view, and such data structure may include coordinates representing the location of the field of view within the XR environment. The system may determine the current field of view and identify objects in the user's field of view based on the data structure and / or images captured by the XR device 102. As shown in the example of FIG. 1 , the user's detected field of view within the environment 100 includes an object 104, depicted as a car, although those skilled in the art will understand that any number or combination of different types of objects may be included within the environment 100. The XR system may generate the object 104 for display at a default level of detail (e.g., a default resolution or number of displayed pixels, or a default size or appearance). For example, objects in the environment 100 may be presented by default at 4K resolution (3,840 x 2,160) or any other suitable resolution. The resolution of objects in the environment 100 may be the same for each eye of the user or may vary. In some embodiments, the level of detail may refer to the size or appearance of an object; for example, an object may be generated at a default size or a default color.

[0044] In some embodiments, in response to determining one or more objects 104 within the user's field of view, the XR system may generate for display an identifier 106 (e.g., "blink once to modify car details") that may indicate or otherwise provide guidance to the user regarding how a particular eyelid movement will cause an action to be performed on the object 104. In some embodiments, the XR system may reference a table (e.g., table 316 of FIG. 3, which may be stored in storage device 808 of FIG. 8) that includes multiple eyelid movement identifiers and corresponding actions that can be performed on the object 104 within the XR environment. For example, the table may additionally store an identifier (e.g., blink twice) that may correspond to increasing or decreasing the size of the object displayed to the user in response to detecting the indicated eyelid movement.

[0045] Once an object of interest within the field of view is identified, the XR system may detect the user's eyelid movement 108 by using a sensor (e.g., a camera). In some embodiments, the XR system may detect whether the eyelid movement exceeds a predetermined period (e.g., 0.5 seconds or 1 second) to avoid performing an action based on an involuntary blink (e.g., if such an action is undesired by the user). In response to detecting the user's eyelid movement 108 (e.g., a single blink corresponding to an action modifying details of the object of interest), the XR regenerates the object 104 for display, which is provided to the user via the head-mounted display 102. For example, the object 104 may be presented to the user at a higher resolution (e.g., 8K resolution, 7,680 x 4,320) than that initially provided (e.g., 4K resolution, 3,840 x 2,160). In some embodiments, the detected eyelid movement may cause the XR system to modify details of the object 104 in different ways (e.g., increasing or decreasing the size of the object compared to the initial presentation of the object, changing the color or texture of the object compared to the initial appearance of the object, etc.).

[0046] In some embodiments, detecting eyelid movement includes determining an amount of eyelid movement, or detecting eyelid movement includes determining one or more eyelid levels. For example, the XR system may use a sensor (e.g., a camera) to detect the amount of eyelid movement and compare the detected amount to a threshold amount of movement over a predetermined period of time (e.g., five eyelid movements detected over a three-second period), and an image may be modified or selected when the detected amount of eyelid movement exceeds the threshold amount of movement over the predetermined period of time. As another example, the XR system may use a sensor (e.g., a camera) to detect one or more eyelid levels (e.g., distinct eyelid levels) over a predetermined period of time and compare the detected amount to a threshold number of eyelid levels over the predetermined period of time (e.g., five distinct eyelid levels detected over a three-second period), and an image may be modified or selected when the detected number of eyelid levels exceeds the threshold number of eyelid levels over the predetermined period of time.

[0047] FIG. 2 illustrates an exemplary process in which multiple objects are detected within a user's field of view. The XR system may detect that objects 204 and 208 are each within the field of view of a user who is viewing an XR environment 200 via a head-mounted display 202. As shown in the environment 200 in the top portion of FIG. 2, the objects 204, 208 detected as being within the user's field of view may initially be presented at a default level of detail (e.g., default resolution or default size). In response to detecting eyelid movement 210 (e.g., corresponding to "blink once to modify details of the car" indicated in identifier 206), the XR system may re-generate the object 204 within the field of view for display with a modified level of detail (e.g., increasing the resolution of the object 204). On the other hand, in response to detecting eyelid movement 212 (e.g., corresponding to "blink twice to modify details of the airplane" indicated in identifier 206), the XR system may re-generate the object 208 within the field of view for display with a modified level of detail (e.g., increasing the resolution of the object 208).

[0048] In some embodiments, detecting a further eyelid movement 212 may subsequently reverse the modification implemented in response to detecting eyelid movement 210 (e.g., object 204 may revert to the default resolution initially presented to the user, while object 208 is presented with modified detail). Alternatively, the object may remain in the modified state throughout the XR session and / or in future sessions. In some embodiments, detecting eyelid movement being re-implemented may reverse the action (e.g., detecting eyelid movement 210 a second time may cause object 204 to revert to the default resolution initially presented to the user). In some embodiments, of multiple objects that may be within the user's field of view, one such object (e.g., object 204) may be in the foreground of a display in XR environment 200, and another such object may be in the background of a display (e.g., object 208) in XR environment 200. Additionally, one or more actions may be performed on the modified objects in the user's field of view (e.g., a particular eyelid movement may correspond to opening a door on car object 204 or interacting with airplane object 208). The most recently modified object in the field of view may be the "selected" object so that actions may be performed on such object. Such an aspect may allow objects that are far from the user or otherwise too small to be seen in detail to be re-generated for display in modified detail to allow the user to clearly view the object.

[0049] In some aspects of the present disclosure, the XR system may detect the user's eyelid movements consistent with objects in the user's field of view, calculate the relative degree to which the eyelids are closed, and determine an initial focused object in the user's field of view. In some embodiments, when a user enters an XR environment, the XR system may set a default field of view, detect the number of objects in the environment and / or field of view, and calculate a respective virtual distance or focal distance for each detected object relative to the user. The objects may be at different virtual distances from the user. In some embodiments, identifying the objects in the field of view includes determining that the objects are at a virtual distance closest to the user of the respective virtual distance or focal distance. The virtual distance may be, for example, the perceived distance at which an object in the XR environment is located from the user and may be calculated based on the coordinates of the object in the XR environment. The user's eyelid level may be calculated, at least in part, based on such virtual distance, and in response to detecting a change in eyelid level, the object at the virtual distance closest to the user may be detected and selected as an object of interest, to which modifications may be implemented.

[0050] In some embodiments, the XR system may detect the user's movement around the XR environment 200 and detect that the user's field of view changes as the user moves around. In such a situation, the user's field of view may be reset to determine the number of objects in the user's new field of view. On the other hand, if the XR system detects the user's movement around the XR environment 200 but detects that the user's gaze remains fixed on a particular object, the display 200 may generate for display such object from a varied perspective consistent with the user's movement to maintain the simulated environment. In some embodiments, any changes in eyelid level detected by the virtual reality system may be used to determine an object in the user's field of view whose details should be modified. The XR system may track the user's movement within the XR environment by using sensors (e.g., gyroscopes, accelerometers, cameras, etc. in combination with control circuitry).

[0051] 3 illustrates an exemplary process for performing actions on objects in an XR environment, according to some embodiments of the present disclosure. A head-mounted display 302 may generate an XR environment 300 for display, including objects 304, 310, 312, and 314. Four objects (a book object 304, a lamp object 310, a desk object 312, and a chair object 314) are shown in the environment 300, but it should be understood that any number and type of objects may be generated for display. An XR system (e.g., system 900 of FIG. 9 ) may store (e.g., in storage device 808 of FIG. 8 ) a table 316 of eyelid movement identifiers and corresponding actions that can be performed on objects in the XR environment. For example, table 316 may store multiple associations, including an association for book object 304 as indicated by identifier 306, that may be displayed to the user to facilitate the desired action: "blink once to turn the page of the book, blink twice to tilt the book, blink three times to tear the page from the book." It should be understood that any number of actions and any type of identifier may be stored in table 316, and the action may depend on the type of object (e.g., table 316 may store an identifier associated with the action of turning on the virtual light bulb associated with lamp object 310). In cases where an environment includes multiple objects, an object may be selected from among the multiple by detecting (e.g., using a sensor) that the user's gaze is directed toward the object.

[0052] The XR system detects the user's eyelid movement by using a sensor (e.g., a camera). The system may determine whether the detected eyelid movement matches any of the identifiers in table 316, for example, by analyzing the sensor output and comparing such output with stored identifiers (e.g., a predetermined number of blinks, blink pattern, amount of eyelid movement, eyelid level, etc.). In some embodiments, the stored identifiers may include eyelid movement in combination with a voice command or other input. In some embodiments, the XR system may detect whether eyelid movement exceeds a predetermined period (e.g., 0.5 seconds) to avoid taking action based on an involuntary blink (e.g., if such action is not desired by the user). The system may detect the user's eyelid movement based on the degree of opening and closing of the user's eyelids over time.

[0053] In response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, the XR system generates an updated version of the XR environment 300 for display based on an action corresponding to the matched eyelid movement. In the example of FIG. 3 , the system detects 318 an eyelid movement associated with turning a page of a book and performs such an action, as shown in the environment 300 in the lower portion of FIG. 3 , which depicts the turned page of the book object 304 compared to the book object 304 in the upper portion of the environment 300, which depicts a closed book. Those skilled in the art will understand that objects in the environment 300 can be manipulated in various ways, for example, the chair object 314 can be moved adjacent to a different portion of the desk object 312, or altered in various ways, for example, removing a cushion from the chair object 314.

[0054] In some embodiments, a subset of the identifiers 306 that are suitable for a selected object of interest may be displayed to the user for the user's convenience in determining available actions to be performed based on a particular eyelid movement. In some embodiments, the XR system may store one or more user profiles that define relationships between eyelid movement identifiers and corresponding actions that can be performed on objects within the XR environment 300. For example, the user profile may include, for example, actions tailored to user preferences, the user's favorite actions, the user's most recently performed actions, the user's most commonly performed actions, the user's purchasing actions, etc., that may be displayed in association with the identifiers 306 for the user's convenience.

[0055] In some embodiments, the XR system may detect the user's movement around the XR environment 200 and may detect that the user's gaze changes as the user moves around. In such a situation, the system may select a new object of interest. Alternatively, the system may detect that the user is navigating from a first location to a new location within the XR environment while the user's gaze remains on the object, and in response to such a determination, generate an updated version of the object for display (e.g., modifying the size or angle of the object presented to the user) based on the user's viewpoint. The updated version of the object may include presenting the object to the user with an altered appearance (e.g., a book with a torn page if the user previously performed an eyelid movement associated with such an action in the table 316).

[0056] In some embodiments, the aspects discussed in FIGS. 1-2 may be combined with the embodiment of FIG. 3 (e.g., in a single user session within an XR environment, objects within the XR environment may be reproduced in more detail and various actions may be performed on such objects).

[0057] 4A-4B show examples of receiving a voice command while an indicator is near an object in an XR environment, according to some embodiments of the present disclosure. An XR system (e.g., system 900 of FIG. 9 ) may generate XR environment 400 for display to a user via head-mounted display 402. In some embodiments, XR environment 400 may include an interactive media guidance application to facilitate media content selection and consumption. XR environment 400 may include one or more objects 408, 410 that may correspond to identifiers for selectable media content. The system uses sensors (e.g., cameras) to detect when a user's gaze shifts from one part of the XR environment (e.g., near object 408 ( FIG. 4A )) to another part of the XR environment (e.g., near object 410 ( FIG. 4B )). It should be understood that Figures 4A-4B are illustrative and show a user's gaze shifting from one part of the XR environment 400, which may contain no objects or may contain multiple objects, to another part of the XR environment 400.

[0058] In response to detecting a gaze shift, the XR system may generate for display within the XR environment 400 an indicator 406 indicating the shift in gaze. For example, in FIG. 4A , indicator 406 reflects that the user's gaze is on object 408 (e.g., an identifier for the movie “The Dark Knight”), while in FIG. 4B , indicator 406 reflects that the user's gaze has shifted to object 410 (e.g., an identifier for the movie “American Psycho”). In some embodiments, a single indicator may be generated for display, or alternatively, multiple indicators may be generated for display. In some embodiments, the indicator may vary in translucency based on proximity to the object of interest. In the examples of FIGS. 4A-4B , indicator 406 is shown as an arrow pointed toward the object of interest, but it will be understood by those skilled in the art that the indicator may comprise any suitable indicia or markings that highlight or prominently display the associated object to the user. For example, the indicator may be a certain color or shape to highlight the object of interest, an image or pictogram of an eyeball, a magnification of the object in the vicinity of the indicator, an animation of the object in the vicinity of the indicator, etc.

[0059] The system may detect a voice command while indicator 406 is in the vicinity of object 410 (e.g., if the indicator overlaps or otherwise is within a predetermined distance of the object of interest) by using a sensor (e.g., a microphone). The XR system may process the voice command and perform an action (e.g., provided there is a match between the object included in the voice command and the object toward which the user's gaze is directed as indicated by indicator 406). For example, in response to receiving voice command 404 in the example of FIG. 4A , the system may begin presenting the media asset “The Dark Knight” associated with object 408, and in response to receiving voice command 405 in the example of FIG. 4B , the system may begin presenting the media asset “American Psycho” associated with object 410. In some embodiments, if the user determines that the indicator does not accurately reflect the user's gaze, the system may receive a suitable voice command from the user requesting that the system recalibrate the user's gaze and / or indicating the portion of the display to which the user believes they are gazed.

[0060] In some embodiments, the user's gaze is detected based on retinal movement of the eye (tracked by a sensor, e.g., a camera measuring the reflection of a light source from the retina, eye-tracking glasses, or screen-based eye tracking). The user's retinal movement may be plotted or converted to a display of the XR environment as movement of the indicator 406 on the display. In some aspects of the present disclosure, the system may determine whether the rate of retinal movement exceeds a predetermined value and, in response to such a determination, perform normalization when converting the retinal movement to movement of the indicator 406 on the display of the XR environment. For example, if the rate of gaze shift exceeds a predetermined value, normalization may be performed to slow down the movement of the indicator 406 on the display (e.g., to allow the user to more easily track the movement of the indicator 406 on the display). The entire cluster of indicators may move to such new portion of the display.

[0061] In some embodiments, the system includes an electronic voice recognition (or voice assistance) device (e.g., a television, computer, voice assistant) that responds to user voice commands, where the voice input may be in the form of an audio or digital signal (or audio or digital input). The system may implement natural language understanding (NLU) techniques and may include natural language understanding and / or speech-to-text circuitry to transcribe the voice command into text, and may analyze the voice command and identify and extract keywords from the voice input. The system may compare the extracted keywords with metadata associated with the object of interest and determine whether a match exists, e.g., whether to execute the voice command. In some embodiments, if a received voice command does not match an object in the vicinity of the indicator, the system may notify the user of the mismatch and refrain from performing the associated action or prompt the user for a new voice command.

[0062] In some embodiments, the voice command includes an identification of a media asset and a command to perform an action (e.g., play, fast forward, rewind, etc.) or an instruction to present a new media asset on the display (e.g., scroll through other media assets or move to a new page of media assets in a carousel). In some aspects of the present disclosure, determining that the indicator is in the vicinity of the object includes determining that the user's gaze has been directed at the object for at least a predetermined threshold period (e.g., 5 seconds).

[0063] FIG. 5 illustrates an example of receiving a voice command while an indicator is near an object in an XR environment, according to some embodiments of the present disclosure. An XR system (e.g., system 900 of FIG. 9 ) may include a head-mounted display 502 and may use the head-mounted display 502 to generate an XR environment 500 for display, including objects 508 and 510. As shown in the upper environment 500 of FIG. 5 , the system may use a sensor (e.g., a camera) to detect that a user's gaze is directed toward a portion of the XR environment (e.g., including object 508). The system may generate multiple opacity-based indicators 506 for display within the XR environment 500 near the portion of the XR environment 500 that includes object 508. In the example of FIG. 5 , the indicators 506 are shown as arrows pointed toward the object of interest, but it will be understood by those skilled in the art that the indicators may comprise any suitable indicia or markings that highlight or prominently display the associated object to the user. For example, the indicator may be a certain color or shape to highlight the object of interest, an image or pictogram of an eyeball, a magnification of the object in the vicinity of the indicator, an animation of the object in the vicinity of the indicator, etc.

[0064] The system may identify a boundary (e.g., an edge, shape contour, edge) of the object 508, for example, by edge detection techniques, reading the coordinates of the object 508, analyzing pixel values ​​of an area surrounding the object 508, etc. Based on the identified boundary of the object, the XR system may vary the opacity of at least one of the plurality of opacity-based indicators 506. In some embodiments, the system may determine whether at least one of the plurality of opacity-based indicators overlaps or is within a predetermined distance of the boundary of the object, and in response to such a determination, may vary the individual opacity of one or more indicators 506 that overlap the boundary of the object 508. For example, the system may compare the coordinates of the indicators 506 with the coordinates of the object of interest in the XR system 900 of FIG. 9 . In some embodiments, if the system detects that a user's gaze is shifting from a portion of the display (e.g., including object 508) to a portion of the display that includes another object (e.g., object 510), the system overlays multiple transparency-based indicators near the portion of the display that includes object 510. The entire cluster of indicators may move to such new portion of the display.

[0065] The system may vary the individual opacity based on the distance from the object. As shown in the example of FIG. 5, the individual opacity of indicator 506 may increase as the distance between indicator 506 and object 508 decreases. This may be desirable to emphasize to the user the portion of the display to which the user's gaze is directed. Alternatively, as shown in the example of FIG. 6, the individual opacity of indicator 606 may decrease as the distance between indicator 606 and object 608 decreases. This may be desirable to minimize obscuring portions of the object of interest. In some embodiments, the system may determine whether any of the indicators overlap or are otherwise near another object that may not be of interest, and in such a situation, the indicator may be set to be semi-transparent to avoid either obscuring portions of such uninterested object or falsely indicating to the user that such object is of interest.

[0066] As shown in the examples of Figures 5 and 6, the XR environments 500, 600, respectively, may include an interactive media guide. In response to receiving a voice command to perform an action associated with an object in the vicinity of the indicator 506, 606, the system may perform the action (e.g., generate for presentation an identifier for a movie having metadata associated with "Christian Bale," the actor associated with the object 508, 608). While the examples of Figures 5-6 show objects related to the "Actors" category, various other categories may be generated and viewed for display (e.g., genre, cast, director, etc.). In some embodiments, the environments 500, 600 may be presented with or without an XR head-mounted device. For example, the XR environments may be presented as fully immersive VR or in AR or MR overlaid on a television screen as shown in Figures 5-6.

[0067] In some embodiments, features of the examples of Figures 1-4 may be used in conjunction with features of the examples of Figures 5-6. For example, indicators (such as, e.g., indicator 106 in Figure 1) may be displayed to a user to guide the user in performing eyelid movements or gazes, which may be associated with possible actions (e.g., in table 316 in Figure 3). The indicators of Figures 5 and 6 may also be used in connection with other environments (e.g., video games, sports, communications, social media, etc.).

[0068] 7 shows an example of presenting additional information related to an item in an XR environment, according to some embodiments of the present disclosure. A head-mounted display 702 presents an XR environment 700 to a user. The XR environment 700 may include one or more objects, such as a washing machine object 702, a refrigerator object 704, a wall object 708, etc. In some embodiments, the XR environment 700 includes an identifier 706 that may indicate to the user actions that may be taken within the environment 700. Although the environment 700 shows a room within a building (e.g., a house or apartment), the environment 700 may comprise any virtual or augmented environment, such as a mountain range, a scenic location, a video game being played by a user, etc.

[0069] The system detects the user's eye movement (e.g., via a sensor) and determines, based on the detected eye movement, whether an object (e.g., washing machine object 702, refrigerator object 704, and / or wall object 708) is within the user's field of view for at least a predetermined period 710 (e.g., 5 seconds). In response to determining that the object (e.g., wall object) is within the field of view for at least the predetermined period, the system generates one or more items 714 related to the object 708 for display within the XR environment. The one or more items 714 related to the object may comprise text information, images, videos, or any combination thereof, and may be retrieved from an item information database 712 (e.g., stored in storage device 808 of FIG. 8 ). For example, item 714 may indicate the thickness of the wall object 708, and such additional information may be provided when the user gazes at the wall object 708 via the head-mounted display 702 within the XR environment 700. In some embodiments, detecting eye movement includes monitoring eyelid movement. In some aspects of the disclosure, generating one or more items 714 related to the object 708 for display within the XR environment is performed in response to detecting that the object 708 is both within the field of view and within the user's gaze for at least a predetermined period of time. In some embodiments, the system allows for modification of the predetermined period of time (e.g., by the user or recommended by the system based on historical data of the user).

[0070] As another example, the environment 700 may be a mountain range, and in response to determining, based on detected eye movement of the user (e.g., detected via a sensor), that an object (e.g., a mountain) is within the user's field of view for at least a predetermined period of time, information about the mountain object may be generated for display to the user (e.g., the name of the mountain, the elevation of the mountain, the location of the mountain, etc.).

[0071] In some aspects of the present disclosure, determining whether an object is in the field of view for a predetermined period of time may include determining that the field of view is continuously on the object for a predetermined period of time (e.g., 5 seconds) during the XR session. For example, if the system detects that the user's field of view has shifted during a timer countdown to the predetermined period of time, the countdown may be reset. Alternatively, determining whether an object is in the field of view for a predetermined period of time may include determining that the field of view is non-continuously on the object for a predetermined period of time (e.g., 5 seconds) during the XR session. For example, if the system detects that the user's field of view has shifted during a timer countdown to the predetermined period of time, the countdown may be paused and resumed in response to detecting that the user's gaze has returned to the object.

[0072] In some embodiments, the system may determine whether a new object is within the field of view for at least a predetermined amount of time. For example, in response to the system making such a determination, the system may generate one or more items related to the new object for display within the XR environment while continuing to generate one or more items related to the object (e.g., object 708 and item 714) for display within the XR environment. This allows a user to view additional information about multiple items for at least the duration of the XR session. In some embodiments, the additional information items may be maintained in subsequent user sessions within the XR environment.

[0073] In some embodiments, the system may determine that at least a predetermined period of time (e.g., 10 seconds) has elapsed since the initiation of display of one or more items 714 without the object 708 being present in the field of view for a first predetermined period of time. For example, if at least a predetermined period of time (e.g., 10 seconds) has elapsed since the item 714 was provided to the user and the object 708 associated with the item 714 has not been in the field of view during such period of time, the display of the item 714 may be stopped (e.g., because the user may no longer be focusing on the object 708).

[0074] In some embodiments, additional input (e.g., voice commands, hand gestures, touch) may be utilized in combination with detecting the user's eye movements. For example, the system may receive a voice command indicating that the user no longer wishes to view additional information related to the item of interest, or may receive a voice command to display additional information, and may retrieve additional information related to the item of interest (e.g., item 708) from item information database 712.

[0075] 8 is a block diagram of an illustrative device in an XR system according to some embodiments of the present disclosure. Device 800 in the XR system may include one or more servers for generating, displaying, and / or managing an XR environment, which are transmitted to XR devices (e.g., devices 102, 202, 302, 402, 502, 602, 702, respectively, of FIGS. 1-7 ) via a computer network. Device 800 may include control circuitry 804 and / or I / O paths 810. Control circuitry 804 may be based on any suitable processing circuitry, such as processing circuitry 806. As referred to herein, processing circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include multi-core processors (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputers.

[0076] In some embodiments, processing circuitry 806 may be distributed across multiple separate processors or processing units, for example, multiple processing units of the same type (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). I / O paths 810 may provide device information or other data and / or other content and data to control circuitry 804, including processing circuitry 806 and storage device 808, via a local area network (LAN) or a wide area network (WAN). Control circuitry 804 may be used to send and receive commands, requests, signals (digital and analog), and other suitable data using I / O paths 810. I / O paths 810 may connect control circuitry 804 (and specifically processing circuitry 806) to one or more communication paths. In some embodiments, storage device 808 may include table 316 of FIG. 3 and / or item information database 712 of FIG. 7.

[0077] In some embodiments, the control circuitry 804 executes instructions for performing XR content generation and / or processing functions that are stored in memory (e.g., storage device 808). The instructions may be stored in either non-volatile memory 814 and / or volatile memory 812 and loaded into the processing circuitry 806 at runtime. A system for generating and processing XR content (e.g., the system described with reference to FIGS. 1-7) may be a standalone application implemented on a user device (e.g., end-user device 920) and / or a server (e.g., server 900), or may be distributed across multiple devices according to device 800. The system may be implemented as software or a set of executable instructions. Instructions for performing any of the embodiments of XR processing discussed herein may be encoded on a non-transitory computer-readable medium (e.g., a hard drive, random access memory on a DRAM integrated circuit, read-only memory on a Bluray disc, etc.) or a transitory computer-readable medium (e.g., a propagated signal carrying data and / or instructions). For example, instructions according to the processes described herein may be stored in memory 808 and executed by control circuitry 804 of device 800 .

[0078] Control circuitry 804 may include video generation and tuning circuitry, such as one or more analog tuners, one or more MPEG-2 decoders or other digital decoding circuitry, a high-definition tuner, or any other suitable tuning or video circuitry or combination of such circuits. Encoding circuitry (e.g., for converting over-the-air, analog, or digital signals to MPEG signals for storage) may also be included. Control circuitry 804 may also include scaler circuitry for upconverting and downconverting content to a preferred output format of end-user devices 920, 925, 930 of FIG. 9. Circuitry 804 may also include digital-to-analog converter circuitry and analog-to-digital converter circuitry for converting between digital and analog signals. The tuning and encoding circuitry may be used to receive, display, play, or record content. The tuning and encoding circuitry may also be used to receive guidance data. For example, the circuitry described herein, including tuning, video generation, encoding, decoding, encryption, decryption, scaler, and analog / digital circuitry, may be implemented using software running on one or more general-purpose or specialized processors. Multiple tuners may be provided to handle simultaneous tuning functions (e.g., viewing and recording functions, functions for incorporating video or other recordings of the physical environment, multi-tuner recording, etc.). If storage 808 is provided as a device separate from device 800, the tuning and encoding circuitry (including multiple tuners) may be associated with storage 808. Device 800 may be a central device that communicates with each device (e.g., 102, 202, 302, 402, 502, 602, 702). Alternatively, device 800 may correspond to device 104, i.e., each system (e.g., of FIGS. 1-7) may not have its own device 920; rather, a single central device 800 may perform XR environment generation and processing for each system.

[0079] FIG. 9 shows an illustrative block diagram of an XR content system according to some embodiments of the present disclosure. At least one XR content server 900 generates XR content, such as that described herein. The XR content server 900 may transmit content via the Internet 910 to multiple end-user devices, including devices 920, 925, and 930. The end-user XR devices may include, for example, personal computers, mobile devices (e.g., smartphones), and / or wearable devices, including XR headsets, goggles, suits, gloves, etc., configured to present and enable interaction with an XR environment. These devices are configured to enable operators / users to view and interact with the multi-user XR content (e.g., via a display screen). These devices may provide visual, audio, and haptic feedback, for example, presenting perspective and attention-directing cues as described herein. The end user devices 920, 925, 930 also transmit data over the Internet 910 to the server 900, such data including orientation information indicating the direction the devices 920, 925, 930 are facing (and therefore the view the server 900 must generate for display on each device 920, 925, 930), audio signals detected by each device 920, 925, 930, and user input such as selection of an XR object. The server 900 may then generate a view of the XR environment for each device 920, 925, 930.

[0080] 10 is a flowchart of a detailed illustrative process for regenerating objects for display within an XR environment according to some embodiments of the present disclosure. Note that process 1000, or any steps thereof, may be performed on or provided by any of the devices shown in FIGS. 8-9. For example, process 1300 may be performed by control circuitry of network device 800 (e.g., via control circuitry 804), and / or server 900, and / or devices 920, 925, 930, as instructed by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. In addition, one or more steps of process 1000 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14, process 1500 of FIG. 15).

[0081] At 1002, input / output (I / O) circuitry (e.g., I / O circuitry of head-mounted display 102) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such user selection may be received via input (e.g., a voice command or a touchscreen).

[0082] At 1004, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head-mounted display 102 of FIG. 1) may generate an XR environment for display (e.g., via head-mounted device 102 of FIG. 1) including the object (e.g., automobile object 104 of FIG. 1) in response to receiving the user selection.

[0083] At 1006, I / O circuitry (e.g., I / O circuitry of head mounted display 102) may receive input from a sensor (e.g., a camera). The sensor may detect various attributes of the user's eye (e.g., eyelid movement, gaze).

[0084] At 1008, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head-mounted display 102 of FIG. 1) may identify an object in the user's field of view (e.g., automobile object 104 of FIG. 1) based on the received input. In some embodiments, the control circuitry may generate a data structure related to the field of view that includes object identifiers associated with virtual objects in the field of view, and such data structure may include coordinates representing the location of the field of view within the XR environment. The control circuitry may determine the current field of view and identify objects in the user's field of view based on the data structure and / or images captured by XR device 102.

[0085] At 1010, the control circuitry may detect the user's eyelid movement (e.g., amount of eyelid movement, blink pattern, eyelid level, etc.) based on input received from a sensor (e.g., a camera). In some embodiments, such eyelid movement may be distinguished from involuntary blinks to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore blink patterns in which the eyes remain closed for less than a predetermined amount of time (e.g., 0.5 seconds).

[0086] At 1012, the control circuitry may regenerate for display an object (e.g., automobile object 104 of FIG. 1 ) with a modified level of detail in response to detecting eyelid movement. For example, the control circuitry may regenerate for display an object at a higher resolution in response to detecting a particular eyeblink pattern. In some embodiments, the control circuitry may reference a table (e.g., table 316 of FIG. 3 ) that stores associations between eyelid movement identifiers and possible actions to determine whether the detected eyelid movement matches an eyelid movement identifier for modifying the level of detail of the object.

[0087] 11 is a flowchart of a detailed illustrative process for regenerating objects for display within an XR environment according to some embodiments of the present disclosure. For example, process 1100 may be performed by network device 800 (e.g., via control circuitry 804), and / or control circuitry of server 900, and / or control circuitry of devices 920, 925, 930, as directed by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. Additionally, one or more steps of process 1100 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10 , process 1200 of FIG. 12 , process 1300 of FIG. 13 , process 1400 of FIG. 14 , process 1500 of FIG. 15 ).

[0088] At 1102, I / O circuitry (e.g., I / O circuitry of head mounted display 202) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such user selection may be received via input (e.g., a voice command or a touchscreen).

[0089] At 1104, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head-mounted display 202 of FIG. 2) may generate for display (e.g., via head-mounted device 202 of FIG. 2) an XR environment including a plurality of objects (e.g., car object 204 of FIG. 2 and airplane object 208 of FIG. 2) in response to receiving the user selection.

[0090] At 1106, I / O circuitry (e.g., I / O circuitry of head mounted display 102) may receive input from a sensor (e.g., a camera). The sensor may detect various attributes of the user's eye (e.g., eyelid movement, gaze).

[0091] At 1108, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head-mounted display 202 of FIG. 1) may identify objects in the user's field of view (e.g., car object 204 of FIG. 2 and airplane object 208 of FIG. 2) based on the received input. In some embodiments, the control circuitry may generate a data structure related to the field of view that includes object identifiers associated with virtual objects in the field of view, and such data structure may include coordinates representing the location of the field of view within the XR environment. The control circuitry may determine the current field of view and identify objects in the user's field of view based on the data structure and / or images captured by XR device 202.

[0092] At 1110, the control circuitry may detect the user's eyelid movement (e.g., amount of eyelid movement, blink pattern, eyelid level, etc.) based on input received from a sensor (e.g., a camera). In some embodiments, such eyelid movement may be distinguished from involuntary blinks to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore blink patterns in which the eyes remain closed for less than a predetermined amount of time (e.g., 0.5 seconds).

[0093] At 1112, the control circuitry may determine whether the eyelid movement is associated with a first object (e.g., car object 204 of FIG. 2). In the example of FIG. 2, the control circuitry may determine whether the detected eyelid movement matches the eyelid movement identifier "blink once to modify car details" (e.g., stored in table 316 of FIG. 3).

[0094] At 1114, in response to determining that the detected eyelid movement matches an eyelid movement identifier for a first object (e.g., automobile object 204 of FIG. 2), the control circuitry may regenerate such object for display with a modified level of detail. For example, the control circuitry may regenerate such object for display at a higher resolution than initially presented in response to detecting a particular eyeblink pattern, or at a larger size than initially presented in response to detecting a particular eyeblink pattern.

[0095] At 1116, if the control circuitry determines that the eyelid movement is not associated with the first object (e.g., car object 204 of FIG. 2), the control circuitry may determine whether the eyelid movement is associated with another object (e.g., airplane object 208 of FIG. 2). In the example of FIG. 2, the control circuitry may determine whether the detected eyelid movement matches the eyelid movement identifier "blink twice to correct airplane details" (e.g., stored in table 316 of FIG. 3).

[0096] At 1118, in response to determining that the detected eyelid movement matches an eyelid movement identifier for a second object (e.g., airplane object 208 of FIG. 2), the control circuitry may regenerate such object for display with a modified level of detail. For example, the control circuitry may regenerate the object for display at a higher resolution than initially presented in response to detecting a particular eyeblink pattern, or at a larger size than initially presented in response to detecting a particular eyeblink pattern. Thus, a user may selectively shift their gaze or focus within the XR environment by actuating their eyelids.

[0097] 12 is a flowchart of a detailed illustrative process for performing actions on objects in an XR environment according to some embodiments of the present disclosure. For example, process 1200 may be performed by network device 800 (e.g., via control circuitry 804), and / or control circuitry of server 900, and / or control circuitry of devices 920, 925, 930, as directed by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. Additionally, one or more steps of process 1200 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10 , process 1100 of FIG. 11 , process 1300 of FIG. 13 , process 1400 of FIG. 14 , process 1500 of FIG. 15 ).

[0098] At 1202, I / O circuitry (e.g., I / O circuitry of head mounted display 302) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such user selection may be received via input (e.g., a voice command or a touchscreen).

[0099] At 1204, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head mounted display 302 of FIG. 3) may generate for display (e.g., via head mounted device 302 of FIG. 3) an XR environment including an object (e.g., book object 304 of FIG. 3) in response to receiving the user selection. In some embodiments, the control circuitry may generate multiple objects (e.g., book object 304, lamp object 310, desk 312, chair 314 of FIG. 3).

[0100] At 1206, the control circuitry may store or access in memory (e.g., storage device 808 of FIG. 8 and / or storage device of head-mounted display 302 of FIG. 3) a table of eyelid movement identifiers and corresponding actions that can be performed on the object (e.g., table 316 of FIG. 3).

[0101] At 1208, the control circuitry may detect the user's eyelid movement (e.g., amount of eyelid movement, blink pattern, eyelid level, etc.) based on input received from a sensor (e.g., a camera). In some embodiments, such eyelid movement may be distinguished from involuntary blinks to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore blink patterns in which the eyes remain closed for less than a predetermined amount of time (e.g., 0.5 seconds).

[0102] At 1210, the control circuitry determines whether the detected eyelid movement matches any of the identifiers in a table (e.g., table 316 of FIG. 3). In some embodiments, the control circuitry may analyze the sensor output and compare such output to stored identifiers (e.g., a predetermined number of blinks, a blink pattern, an amount of eyelid movement, an eyelid level, etc.). In some embodiments, the stored identifiers may include eyelid movement in combination with a voice command or other input.

[0103] At 1212, the control circuitry determines whether the detected eyelid movement matches a stored eyelid movement identifier. In some embodiments, the control circuitry may calculate a match score and determine that a match exists between the detected eyelid movement and the stored eyelid movement identifier when the calculated match score exceeds a predetermined threshold.

[0104] At 1214, in response to determining that the detected eyelid movement matches the stored eyelid movement identifier, the control circuitry generates for display an updated version of the XR environment based on the action corresponding to the matched eyelid movement. In the example of Figure 3, the updated version of the XR environment 300 includes the book object 304 having a page turned (e.g., in the environment 300 illustrated in the lower portion of Figure 3 compared to the environment 300 illustrated in the upper portion of Figure 3, where the book object 304 is closed).

[0105] 13 is a flowchart of a detailed illustrative process for receiving a voice command while an indicator is in the vicinity of an object in an XR environment, according to some embodiments of the present disclosure. For example, process 1300 may be performed by control circuitry of network device 800 (e.g., via control circuitry 804), and / or server 900, and / or devices 920, 925, 930, as directed by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. Additionally, one or more steps of process 1300 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1400 of FIG. 14, process 1500 of FIG. 15).

[0106] At 1302, I / O circuitry (e.g., the I / O circuitry of head-mounted display 402 of FIGS. 4A-4B ) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such user selection may be received via input (e.g., a voice command or a touchscreen).

[0107] At 1304, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head mounted display 402 of FIG. 4A) may generate for display (e.g., via head mounted device 402 of FIG. 4) an XR environment including an object (e.g., object 408 of FIG. 4) in response to receiving the user selection. In some embodiments, the control circuitry may generate multiple objects (e.g., objects 408 and 410 of FIG. 4).

[0108] At 1306, the control circuitry may detect the user's gaze. For example, a sensor (e.g., a camera) may be used to track the user's retinal movements, and such retinal movements of the user may be plotted on a display of the XR environment (e.g., environment 400 of FIGS. 4A-4B).

[0109] At 1308, the control circuitry may determine whether the user's gaze has shifted to a second portion of the XR environment that includes an object (e.g., object 410 in FIGS. 4A-4B). In some embodiments, the user's gaze may have shifted from a first portion of the display that does not include any object. Alternatively, such first portion of the display may include an object (e.g., object 408 in FIGS. 4A-4B).

[0110] At 1310, in response to determining that the user's gaze has shifted to such a second portion of the XR environment, including an object (e.g., object 410 of FIGS. 4A-4B), the control circuitry may generate for display an indicator of the shift in gaze (e.g., indicator 406 of FIG. 4). For example, such an indicator allows the user to confirm that the system is accurately tracking their gaze.

[0111] At 1312, in response to determining that the user's gaze has not shifted to such a second portion of the XR environment, the control circuitry may determine that the user's gaze has not shifted or has shifted to a different portion of the display and may generate for display an indicator (e.g., indicator 406 of FIG. 4 ) in the portion of the display toward which the user's gaze is directed.

[0112] At 1314, I / O circuitry (e.g., I / O circuitry of head mounted display 402 of FIGS. 4A-4B) may receive a voice command while an indicator (e.g., indicator 406 of FIG. 4) is in proximity to an object of interest (e.g., object 408 of FIG. 4A or object 410 of FIG. 4B). In some embodiments, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head mounted display 402 of FIG. 4A) may determine whether the voice command is associated with such object of interest. In the example of FIG. 4A, the control circuitry may extract keywords from the voice command "Play 'The Dark Knight'" and compare the extracted keywords with metadata associated with the object to which the user's gaze is directed (e.g., object 408) to determine whether the voice command is associated with the object of interest.

[0113] At 1316, the control circuitry may perform an action corresponding to the voice command associated with the object. For example, in the example of FIG. 4B, object 410, i.e., an identifier for media content item "American Psycho," is in proximity to indicator 406, and in response to receiving voice command 405 ("Play "American Psycho"), presentation of such media asset to the user may be initiated.

[0114] 14 is a flowchart of a detailed illustrative process for varying the opacity of an indicator in an XR environment, according to some embodiments of the present disclosure. Note that process 1400, or any steps thereof, may be performed on or provided by any of the devices shown in FIGS. 8-9 . For example, process 1400 may be performed by control circuitry of network device 800 (e.g., via control circuitry 804), and / or server 900, and / or devices 920, 925, 930, as instructed by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. Additionally, one or more steps of process 1400 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1500 of FIG. 15).

[0115] At 1402, I / O circuitry (e.g., I / O circuitry of head-mounted display 502 of FIG. 5) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such user selection may be received via input (e.g., a voice command or a touchscreen).

[0116] At 1404, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head mounted display 502 of FIG. 5) may generate for display (e.g., via head mounted device 502 of FIG. 5) an XR environment including an object (e.g., object 508 of FIG. 5) in response to receiving the user selection. In some embodiments, the control circuitry may generate multiple objects (e.g., objects 508 and 510 of FIG. 5).

[0117] At 1406, the control circuitry may detect the user's gaze. For example, a sensor (e.g., a camera) may be used to track the user's retinal movements, and such retinal movements of the user may be plotted on a display of the XR environment (e.g., environment 500 of FIG. 5). The control circuitry may determine whether the user's gaze is directed at a portion of the XR environment that includes an object (e.g., object 508 of FIG. 5).

[0118] At 1410, in response to determining that the user's gaze is not directed to a portion of the XR environment that includes an object (e.g., object 508 of FIG. 5 ), the control circuitry may determine whether the user's gaze is directed to a different portion of the XR environment that includes a different object (e.g., object 510 of FIG. 5 ).

[0119] At 1410, in response to determining that the user's gaze is directed at a portion of the XR environment that includes an object (e.g., object 508 of FIG. 5 ), the control circuitry may generate an opacity-based indicator (e.g., indicator 506 of FIG. 6 ) near such portion of the XR environment for display within the XR environment. On the other hand, in response to determining that the user's gaze is directed at a different portion of the XR environment that includes a different object (e.g., object 510 of FIG. 5 ), the control circuitry may generate an opacity-based indicator (e.g., indicator 506 of FIG. 6 ) near such different portion of the XR environment for display within the XR environment.

[0120] At 1412, the control circuitry may determine the boundary of the object of interest (e.g., object 508 in FIG. 5). For example, the control circuitry may perform edge detection techniques and / or pixel comparisons to determine the boundary of the object.

[0121] At 1414, the control circuitry varies the opacity of at least one opacity-based indicator (e.g., indicator 506 of FIG. 5) based on the identified boundary of the object. In some embodiments, the opacity of an indicator that overlaps a boundary of the object of interest is varied. For example, the opacity of an indicator that overlaps a boundary of the object of interest may be decreased to avoid obscuring the object of interest (FIG. 6), or increased to emphasize that the user's gaze is directed toward the object of interest (FIG. 5).

[0122] 15 is a flowchart of a detailed illustrative process for presenting additional information related to an item within an XR environment, according to some embodiments of the present disclosure. Note that process 1500, or any steps thereof, may be performed on or provided by any of the devices shown in FIGS. 8-9 . For example, process 1500 may be performed by network device 800 (e.g., via control circuitry 804), and / or control circuitry of server 900, and / or devices 920, 925, 930, as instructed by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as to distribute control of an extended reality environment among multiple devices. Additionally, one or more steps of process 1500 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14).

[0123] At 1502, I / O circuitry (e.g., I / O circuitry of head-mounted display 702 of FIG. 7) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such user selection may be received via input (e.g., a voice command or a touchscreen).

[0124] At 1504, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head mounted display 702 of FIG. 7) may generate for display (e.g., via head mounted device 702 of FIG. 7) an XR environment including an object (e.g., object 708 of FIG. 7) in response to receiving the user selection. In some embodiments, the control circuitry may generate multiple objects (e.g., objects 702, 704, 708 of FIG. 7).

[0125] At 1506, control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of head-mounted display 702 of FIG. 7) may detect the user's eye movement (e.g., using a sensor).

[0126] At 1508, the control circuitry may determine whether the object has been in the user's field of view for at least a predetermined period of time. In some embodiments, the control circuitry may generate a data structure related to the field of view, including object identifiers associated with virtual objects in the field of view, and such data structure may include coordinates representing the location of the field of view within the VR environment. The control circuitry may determine the current field of view and identify objects in the user's field of view based on the data structure and / or images captured by the head-mounted display 702. In an embodiment, the control circuitry may additionally detect whether the user's gaze is directed toward an object in the field of view.

[0127] At 1510, in response to the control circuitry determining that the object is within the field of view for at least a predetermined period of time (e.g., 5 seconds), the control circuitry generates one or more items (e.g., item 714 in FIG. 7) associated with the object (e.g., object 708 in FIG. 7) for display within the XR environment. In some embodiments, in determining whether the object is within the field of view for at least a predetermined period of time, the control circuitry may determine whether the object remains within the user's field of view for continuous (or non-continuous) periods during the XR session.

[0128] The processes discussed above are intended to be illustrative, not limiting. Those skilled in the art will understand that steps of the processes discussed herein may be omitted, modified, combined, and / or rearranged, and that any additional steps may be implemented without departing from the scope of the present invention. More generally, the above disclosure is intended to be illustrative, not limiting. Only the following claims are intended to set boundaries regarding the content covered by the present invention. Furthermore, it should be noted that features and limitations described in any one embodiment may be applied to any other embodiment herein, and that flowcharts or examples related to one embodiment may be combined with, performed in a different order, or performed in parallel with, any other embodiment in a suitable manner. In addition, the systems and methods described herein may be implemented in real time. It should also be noted that the systems and / or methods described above may be applied to or used in accordance with other systems and / or methods. This specification discloses embodiments, including, but not limited to, the following. 1. A method for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment comprising the object; detecting, using a first sensor, a shift in gaze from a first portion of the extended reality environment to a second portion of the extended reality environment, wherein the object is excluded from the first portion of the extended reality environment and included within the second portion of the extended reality environment; in response to detecting a gaze shift, generating an indicator of the shift in gaze for display within the extended reality environment; detecting a voice command while the indicator is in the vicinity of the object by using a second sensor; In response to detecting the voice command, performing an action corresponding to the voice command; A method comprising: 2. The method of item 1, wherein an interactive media guide is provided on the display and the actions are instructions related to media assets accessible via the interactive media guide. 3. The gaze is detected based on retinal movement of the eye, and the method further comprises: 3. The method of claim 1 or 2, comprising converting retinal movement into movement of an indicator on the display. 4. Determining whether the rate of retinal movement exceeds a predetermined value; normalizing the retinal movement when converting the retinal movement into a movement of an indicator on a display in response to determining that the rate of the retinal movement exceeds a predetermined value; Item 4. The method of item 3, further comprising: 5. The method of any of items 2-4, wherein the voice command includes an identification of a media asset and a command to perform an action. 6. The method of any of items 2-5, wherein the voice command includes an instruction to present a new media asset on the display. 7. The method of any of items 2-6, wherein the object is associated with an entity and the voice command comprises an instruction to retrieve content associated with the entity. 8. A method according to any of items 1-7, wherein the step of detecting a voice command while the indicator is in the vicinity of the object includes the step of determining that gaze is directed at the object for at least a predetermined threshold period of time. 9. The method of any of items 1-8, wherein the indicator being in the vicinity of the object includes the indicator overlapping the object. 10. The method of any of items 1-9, wherein the display is presented via an extended reality head-mounted device. 11. A system for extended reality environment interaction, comprising: The display and generating an extended reality environment for display, the extended reality environment comprising the object; detecting, using a first sensor, a shift in gaze from a first portion of the extended reality environment to a second portion of the extended reality environment, wherein the object is excluded from the first portion of the extended reality environment and included within the second portion of the extended reality environment; In response to detecting the gaze shift, generating an indicator of the shift in gaze for display within the extended reality environment; Detecting a voice command while the indicator is in the vicinity of the object by using a second sensor; In response to detecting the voice command, performing an action corresponding to the voice command; a control circuitry configured to A system comprising: 12. The system of item 11, wherein an interactive media guide is provided on the display and the actions are instructions related to media assets accessible via the interactive media guide. 13. The control circuitry further comprises: Detecting gaze based on retinal movement of the eye; Converting retinal movement into movement of an indicator on the display, 13. The system according to item 11 or 12, configured as follows: 14. The control circuitry further comprises: determining whether the rate of retinal movement exceeds a predetermined value; normalizing the retinal movement when converting the retinal movement into a movement of an indicator on the display in response to determining that the rate of the retinal movement exceeds a predetermined value; Item 14. The system according to item 13, configured as follows: 15. The system of any of items 12-14, wherein the voice command includes an identification of a media asset and a command to perform an action. 16. The system of any of items 12-15, wherein the voice command includes an instruction to present a new media asset on the display. 17. A system according to any of items 12-16, wherein the object is associated with an entity and the voice command comprises an instruction to retrieve content associated with the entity. 18. The system of any of items 11-17, wherein the control circuitry is further configured to determine that gaze is directed toward the object for at least a predetermined threshold period of time when detecting a voice command while the indicator is in the vicinity of the object. 19. The system of any of items 11-18, wherein the indicator being in the vicinity of the object includes the indicator overlapping the object. 20. The system of any of items 11-19, wherein the display is presented via an extended reality head-mounted device. 21. A method for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment including the first object; receiving input from one or more sensors; identifying a first object within the field of view based on the received input; detecting eyelid movement based on the received input; regenerating the first object for display with a modified level of detail in response to detecting eyelid movement; A method comprising: 22. The extended reality environment comprises a plurality of objects, the plurality of objects including a first object and a second object; The identifying step further includes identifying a second object within the field of view; regenerating the first object for display with the modified level of detail is performed in response to determining that the detected eyelid movement is associated with the first object. Item 21. The method according to item 21. 23. The method of item 22, further comprising, in response to determining that the detected eyelid movement is associated with the second object, regenerating the second object for display with a modified level of detail. 24. The method of item 22 or 23, wherein the first object is in one of the foreground or background within the field of view of the extended reality environment, and the second object is in the other of the foreground or background within the field of view of the extended reality environment. 25. The method of any of items 21-24, wherein regenerating the first object for display using a modified level of detail includes presenting the object at a higher resolution. 26. The method of any of items 21-25, further comprising the step of performing one or more actions on the identified object based on the one or more detected eyelid movements. 27. The method further includes calculating individual virtual distances of the plurality of objects relative to the user; and identifying the first object within the field of view includes determining that the first object is at a virtual distance closest to the user at a discrete virtual distance. 27. The method according to any one of items 22-26. 28. The method of any of items 21-27, wherein the step of detecting eyelid movement includes the step of determining the amount of eyelid movement. 29. The method of any of items 21-28, wherein the step of detecting eyelid movement includes the step of determining one or more eyelid levels. 30. Detecting that a user is navigating from a first location to a new location within the extended reality environment while the first object remains within the user's field of view; generating an updated version of the first object for display based on the user's viewpoint at the new location; 30. The method of any of items 21-29, further comprising: 31. A system for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment including the first object; a control circuitry configured to receiving input from one or more sensors; an input / output (I / O) circuitry configured to Equipped with The control circuitry further comprises: Identifying a first object in the field of view based on the received input; Detecting eyelid movement based on the received input; regenerating the first object for display with a modified level of detail in response to detecting eyelid movement; The system is configured as follows: 32. An extended reality environment comprising a plurality of objects, the plurality of objects including a first object and a second object; the control circuitry is further configured to identify a second object within the field of view; The control circuitry is further configured to perform, in response to determining that the detected eyelid movement is associated with the first object, regenerating the first object for display with a modified level of detail. Item 32. The system according to item 31. 33. The control circuitry further comprises: Item 33. The system of item 32, configured to, in response to determining that the detected eyelid movement is associated with the second object, regenerate the second object for display with a modified level of detail. 34. A system described in item 31 or 33, wherein the first object is in one of the foreground or background within the field of view of the extended reality environment, and the second object is in the other of the foreground or background within the field of view of the extended reality environment. 35. The system of any of items 31-34, wherein the control circuitry is further configured to present the object at a higher resolution when regenerating the first object for display using the modified level of detail. 36. The control circuitry further comprises: 36. The system of any of items 31-35, configured to perform one or more actions on the identified object based on the one or more detected eyelid movements. 37. The control circuitry further comprises: Calculating individual virtual distances of a plurality of objects relative to a user; Upon identifying a first object within the field of view, determining that the first object is at a virtual distance closest to the user at a discrete virtual distance; 37. The system according to any one of items 32 to 36, configured as follows: 38. The system of any of items 31-37, wherein the control circuitry is further configured to, upon detecting eyelid movement, determine an amount of eyelid movement. 39. The system of any of items 31-38, wherein the control circuitry is further configured to determine one or more eyelid levels upon detecting eyelid movement. 40. The control circuitry further comprises: Detecting that a user is navigating from a first location to a new location within the extended reality environment while the first object remains within the field of view; generating an updated version of the first object for display based on the user's viewpoint at the new location; 39. The system according to any one of items 31-38, configured as follows: 41. A method for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment including the object; storing in memory a table of eyelid movement identifiers and corresponding actions that can be performed on objects in the extended reality environment; Detecting eyelid movement by using a sensor; matching the detected eyelid movement with one of the stored eyelid movement identifiers; In response to matching the detected eyelid movement with one of the stored eyelid movement identifiers, generating an updated version of the extended reality environment for display based on an action corresponding to the matched eyelid movement; A method comprising: 42. The method of item 41, further comprising the step of selecting an object from a plurality of objects in the extended reality environment by detecting that the user's gaze is directed toward the object. 43. The method of claim 42, further comprising generating for display a subset of eyelid movement identifiers operable for an object to which the user's gaze is directed. 44. The method of any of items 41-43, wherein an action of the plurality of actions corresponds to manipulating an object. 45. The method of any of items 41-44, wherein the action corresponds to modifying the appearance of the object. 46. ​​A method according to any of items 41-45, wherein a user is associated with a user profile that defines relationships between eyelid movement identifiers and corresponding actions that can be performed on objects within the extended reality environment. 47. Detecting that a user is navigating from a first location to a new location within the extended reality environment while the user's gaze remains on the object; generating an updated version of the first object for display based on the user's viewpoint at the new location, the updated version of the object having a modified appearance; 47. The method according to items 45-46, further comprising: 48. The method of any of items 41-47, wherein the actions that can be performed on an object vary based on the type of object. 49. The step of detecting eyelid movement includes: 49. The method of any of items 41-48, comprising determining whether the eyelids remain closed for a predetermined period of time. 50. The method of item 49, wherein the step of matching the detected eyelid movement to one of the stored eyelid movement identifiers is performed in response to determining that the eyelid remains closed for a predetermined period of time. 51. A system for extended reality environment interaction, comprising: generating an extended reality environment for display, including the object; a control circuitry configured to storing a table of eyelid movement identifiers and corresponding actions that can be performed on objects within the extended reality environment; a memory configured to Equipped with The control circuitry further comprises: By using a sensor, eyelid movement is detected, matching the detected eyelid movement to one of the stored eyelid movement identifiers; in response to matching the detected eyelid movement with one of the stored eyelid movement identifiers, generating for display an updated version of the extended reality environment based on an action corresponding to the matched eyelid movement; The system is configured as follows: 52. The control circuitry further comprises: Item 52. The system of item 51, configured to select an object from a plurality of objects in an extended reality environment by detecting that a user's gaze is directed toward the object. 53. The control circuitry further comprises: Item 53. The system of item 51 or 52, configured to generate for display a subset of eyelid movement identifiers that are operable with respect to an object to which the user's gaze is directed. 54. A system according to any of items 51-53, wherein an action of the plurality of actions corresponds to manipulating an object. 55. A system according to any of items 51-54, wherein the action corresponds to modifying the appearance of the object. 56. A system according to any of items 51-55, wherein a user is associated with a user profile that defines relationships between eyelid movement identifiers and corresponding actions that can be performed on objects within the extended reality environment. 57. The control circuitry further comprises: Detecting that a user is navigating from a first location to a new location within the extended reality environment while the user's gaze remains on the object; generating an updated version of the first object for display based on the user's viewpoint at the new location, the updated version of the first object having an altered appearance; 57. The system according to item 55 or 56, configured as follows: 58. A system according to any of items 51-57, wherein the actions that can be performed on an object vary based on the type of object. 59. The control circuitry further comprises, upon detecting eyelid movement: 59. The system of any of items 51-58, configured to determine whether the eyelid remains closed for a predetermined period of time. 60. The system of item 59, wherein the control circuitry is further configured to perform a step of matching the detected eyelid movement to one of the stored eyelid movement identifiers in response to determining that the eyelid remains closed for a predetermined period of time. 61. A method for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment comprising the object; detecting, using a sensor, that a gaze is directed toward a first portion of the extended reality environment, the object being contained within the first portion of the extended reality environment; generating a plurality of opacity-based indicators for display within the extended reality environment proximate a first portion of the extended reality environment; identifying a boundary of the object; Varying the opacity of at least one of the plurality of opacity-based indicators based on the identified boundary of the object; A method comprising: 62. Further comprising determining whether at least one of the plurality of opacity-based indicators overlaps a boundary of the object; Item 62. The method of item 61, wherein varying the opacity of at least one of the plurality of opacity-based indicators based on a boundary of the object includes varying a respective opacity of at least one of the plurality of opacity-based indicators that overlaps the boundary. 63. The method of item 61 or 62, wherein the plurality of opacity-based indicators are arrows directed toward the object. 64. Detecting whether gaze has shifted to a second portion of the extended reality environment by using a sensor; in response to determining that gaze has shifted to the second portion, overlaying a plurality of opacity-based indicators adjacent the second portion of the display; 64. The method according to any one of items 61-63, further comprising: 65. The method of any of items 61-64, wherein the individual opacities are varied based on the distance from the object. 66. The method according to item 65, wherein the individual opacity of the indicators increases as the distance between the individual indicators and the object decreases. 67. The method according to item 65 or 66, wherein the individual opacity of the indicator increases as the distance between the individual indicator and the object increases. 68. The method of any of items 61-67, wherein an interactive media guide is provided on a display and actions related to media assets accessible via the interactive media guide are received based, at least in part, on detected gaze. 69. The method of any of items 61-68, wherein the display is presented via an extended reality head-mounted device. 70. The method of any of items 61-69, wherein the display is presented without the use of an extended reality head-mounted device. 71. A system for extended reality environment interaction, comprising: The display and generating an extended reality environment for display, the extended reality environment comprising the object; Detecting, using a sensor, that a gaze is directed toward a first portion of the extended reality environment, the object being contained within the first portion of the extended reality environment; generating a plurality of opacity-based indicators for display within the extended reality environment proximate a first portion of the extended reality environment; Identify the boundaries of the object, Varying the opacity of at least one of the plurality of opacity-based indicators based on the identified boundary of the object. a control circuitry configured to A system comprising: 72. The control circuitry further comprises: determining whether at least one of the plurality of opacity-based indicators overlaps a boundary of the object; Varying the opacity of at least one of the plurality of opacity-based indicators based on a boundary of the object comprises varying a respective opacity of at least one of the plurality of opacity-based indicators that overlaps the boundary; Item 72. The system of item 71, configured as follows: 73. The system of item 71 or 72, wherein the plurality of opacity-based indicators are arrows directed toward the object. 74. The control circuitry further comprises: Detecting whether gaze has shifted to a second portion of the extended reality environment using the sensor; in response to determining that gaze has shifted to the second portion, overlaying a plurality of opacity-based indicators adjacent the second portion of the display; 74. The system according to any one of items 71-73, configured to: 75. A system according to any of items 71-74, wherein the individual opacities are varied based on distance from the object. 76. The system of item 75, wherein the individual opacity of the indicators increases as the distance between the individual indicators and the object decreases. 77. The system of item 75 or 76, wherein the individual opacity of the indicators increases as the distance between the individual indicators and the object increases. 78. A system described in any of items 71-77, wherein an interactive media guide is provided on the display and actions related to media assets accessible via the interactive media guide are received based, at least in part, on detected gaze. 79. A system according to any of items 71-78, wherein the display is presented via an extended reality head-mounted device. 80. The system of any of items 71-79, wherein the display is presented without the use of an extended reality head-mounted device. 81. A method for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment including the object; detecting eye movement; determining, based on the detection, whether the object is within the field of view for at least a predetermined period of time; in response to determining that the object is within the field of view for at least a predetermined period of time, generating one or more items related to the object for display within the extended reality environment; A method comprising: 82. The method of item 81, wherein the one or more items associated with the object comprise text information, an image, a video, or any combination thereof. 83. Determining that at least a second predetermined period of time has elapsed since the start of display of the one or more items without the object being present in the field of view for a first predetermined period of time; ceasing the display of the one or more items in response to determining that a second predetermined time period has elapsed from the initiation of the display of the one or more items without the object being present within the field of view; 83. The method of item 81 or 82, further comprising: 84. A method according to any of items 81-83, wherein the step of detecting eye movement includes the step of monitoring eyelid movement. 85. A method according to any of items 81-84, wherein the step of generating one or more items related to the object for display within the extended reality environment is performed in response to determining that the object is within the field of view and gaze for at least a predetermined period of time. 86. A method according to any of items 81-85, wherein the step of determining whether the object is within the field of view for a predetermined period of time based on the detection includes the step of determining that the field of view is continuously on the object for a predetermined period of time during the extended reality session. 87. A method according to any of items 81-86, wherein determining whether the object is within the field of view for a predetermined period based on the detection includes determining that the field of view is non-continuously on the object for a predetermined period during the extended reality session. 88. Determining that a new object is in the field of view for at least a predetermined time; in response to determining that a new object is within the field of view for at least a predetermined period of time, generating one or more items related to the new object for display within the extended reality environment while continuing to generate one or more items related to the object for display within the extended reality environment; 88. The method of any of items 81-87, further comprising: 89. The method of any of items 81-88, further comprising the step of allowing modification of the amount of time for the predetermined period. 90. A method according to any of items 81-89, wherein the extended reality environment is presented via an extended reality head-mounted device. 91. A system for extended reality environment interaction, comprising: The display and generating an extended reality environment for display, the extended reality environment including the object; Detects eye movements, determining, based on the detection, whether the object is within the field of view for at least a predetermined period of time; in response to determining that the object is within the field of view for at least a predetermined period of time, generating one or more items related to the object for display within the extended reality environment; a control circuitry configured to A system comprising: 92. The system of item 91, wherein the one or more items associated with the object comprise text information, an image, a video, or any combination thereof. 93. The control circuitry further comprises: determining that at least a second predetermined period of time has elapsed since the initiation of display of the one or more items without the object being present in the field of view for a first predetermined period of time; ceasing the display of the one or more items in response to determining that a second predetermined time period has elapsed from the initiation of the display of the one or more items without the object being present within the field of view; Item 93. The system according to item 91 or 92, configured as follows: 94. The system of any of items 91-93, wherein the control circuitry is further configured to monitor eyelid movement when eye movement is detected. 95. The system of any of items 91-94, wherein the control circuitry is further configured to, in response to determining that the object is within the field of view and gaze for at least a predetermined period of time, generate one or more items related to the object for display within the extended reality environment. 96. The system of any of items 91-95, wherein the control circuitry is further configured, when determining whether the object is within the field of view for a predetermined period of time, to determine that the field of view is continuously on the object for a predetermined period of time during the extended reality session. 97. The system of any of items 91-96, wherein the control circuitry is further configured, when determining whether the object is within the field of view for a predetermined period of time, to determine that the field of view is on the object non-continuously for a predetermined period of time during the extended reality session. 98. The control circuitry further comprises: determining that the new object is within the field of view for at least a predetermined time; in response to determining that a new object is within the field of view for at least a predetermined period of time, generating one or more items related to the new object for display within the extended reality environment while continuing to generate one or more items related to the object for display within the extended reality environment; 98. The system of any one of items 91-97, configured to: 99. The control circuitry further comprises: 99. The system of any of items 91-98, configured to allow modification of the amount of time for a predetermined period. 100. The system of any of items 91-99, wherein the extended reality environment is presented via an extended reality head-mounted device.

Claims

1. A head-mounted device configured to be worn on a user's head, the head-mounted device comprising: two displays for displaying three-dimensional (3D) images for consumption by said user; at least one camera configured to capture at least one image of at least one eye of the user; first circuitry configured to analyze data associated with the user's gaze based at least on the at least one image of the at least one eye of the user; second circuitry distinct from the first circuitry, the second circuitry configured to execute a media application that causes the two displays to generate for display a virtual 3D environment for consumption of media content by the user, the virtual 3D environment including a plurality of selectable virtual reality (VR) objects; Equipped with The second network comprises: based at least in part on determining, using the first circuitry, that the gaze of the user is directed toward a particular VR object of the plurality of VR objects for at least a first predetermined time period; accessing a data structure that stores individual text data for one or more of the plurality of VR objects; causing the individual text data represented by the data structure associated with the particular VR object to be displayed within the virtual 3D environment; based at least in part on determining, using the first circuitry, that the eye gaze of the user is not directed toward the particular VR object for a second predetermined period of time; causing the display of the text data represented by the data structure associated with the particular VR object within the virtual 3D environment to be stopped; The head-mounted device is further configured to:

2. The head-mounted device described in claim 1, wherein the virtual 3D environment includes an interface for an interactive media guide application for selecting and consuming media content.

3. The head-mounted device of claim 2, wherein the interface of the interactive media guidance application includes a plurality of user interface elements associated with selecting or consuming media content items available for playback using the interactive media guidance application.

4. A head-mounted device as described in claim 3, wherein the particular VR object is one of the plurality of user interface elements associated with selecting or consuming a media content item.

5. A head-mounted device as described in claim 1, wherein the second circuitry is further configured to generate an indicator for display indicating the current position of the user's gaze.

6. A head-mounted device as described in claim 1, wherein determining that the user's eye gaze is not directed toward the specific VR object for a second predetermined period of time includes determining that the user's eye gaze is not directed toward the location of the specific VR object for the second predetermined period of time.

7. A head-mounted device as described in claim 1, wherein determining that the user's eye gaze is not directed toward the specific VR object for a second predetermined period of time includes determining that the user's eye gaze is directed toward a location in the virtual 3D environment that is not the location of the specific VR object for the second predetermined period of time.

8. A head-mounted device as described in claim 1, wherein determining that the user's gaze is directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period includes determining that the gaze has been continuously directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period.

9. A head-mounted device as described in claim 1, wherein determining that the user's gaze is directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period of time includes determining that the gaze has been discontinuously directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period of time.

10. The head-mounted device further comprises at least one additional sensor; The head-mounted device of claim 1 , wherein determining that the user's gaze is directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period of time is further based at least in part on data from the at least one additional sensor.

11. A head-mounted device as described in claim 10, wherein the at least one additional sensor includes at least one of a gyroscope or an accelerometer.

12. A method performed using a head-mounted device configured to be worn on a user's head, the head-mounted device comprising: two displays for displaying three-dimensional (3D) images for consumption by said user; at least one camera configured to capture at least one image of at least one eye of the user; first circuitry configured to analyze data associated with the user's gaze based at least on the at least one image of the at least one eye of the user; second circuitry distinct from the first circuitry, the second circuitry configured to execute a media application that causes the two displays to generate for display a virtual 3D environment for consumption of media content by the user, the virtual 3D environment including a plurality of selectable virtual reality (VR) objects; Equipped with The method comprises: using the second circuitry based at least in part on determining, using the first circuitry, that the user's gaze is directed toward a particular VR object of the plurality of VR objects for at least a first predetermined time period; accessing a data structure that stores individual text data for one or more of the plurality of VR objects; causing the individual text data represented by the data structure associated with the particular VR object to be displayed within the virtual 3D environment; using the second circuitry based at least in part on determining, using the first circuitry, that the eye gaze of the user is not directed toward the particular VR object for a second predetermined period of time; causing the display of the text data represented by the data structure associated with the particular VR object within the virtual 3D environment to be stopped; A method comprising:

13. The method of claim 12, wherein the virtual 3D environment includes an interface of an interactive media guide application for selecting and consuming media content.

14. The method described in claim 13, wherein the interface of the interactive media guidance application includes a plurality of user interface elements associated with selecting or consuming media content items available for playback using the interactive media guidance application.

15. The method of claim 14, wherein the particular VR object is one of the plurality of user interface elements associated with selecting or consuming a media content item.

16. The method of claim 12, wherein the second circuitry is further configured to generate an indicator for display indicating the current position of the user's gaze.

17. The method described in claim 12, wherein determining that the user's eye gaze is not directed toward the specific VR object for a second predetermined period of time includes determining that the user's eye gaze is not directed toward the location of the specific VR object for the second predetermined period of time.

18. The method described in claim 12, wherein determining that the user's eye gaze is not directed toward the particular VR object for a second predetermined period of time includes determining that the user's eye gaze is directed toward a location in the virtual 3D environment that is not the location of the particular VR object for the second predetermined period of time.

19. The method described in claim 12, wherein determining that the user's gaze is directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period of time includes determining that the gaze has been continuously directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period of time.

20. The method described in claim 12, wherein determining that the user's gaze is directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period of time includes determining that the gaze has been discontinuously directed toward the particular VR object of the plurality of VR objects for at least the first predetermined period of time.

Images