Method and system for extended reality environment interaction based on eye movement

Abstract

A system and method for extended reality environment interaction is described. An extended reality environment including an object is generated for display, and a first sensor is used to detect a shift in gaze from a first portion of the extended reality environment to a second portion of the extended reality environment, whereby the object is excluded from the first portion of the extended reality environment and included within the second portion of the extended reality environment. An indicator of the shift in gaze is generated for display within the extended reality environment in response to detecting the gaze shift, and a voice command is detected by the second sensor while the indicator is in proximity to the object. In response to detecting the voice command, an action corresponding to the voice command can be performed.

Description

Technical Field

[0001] The present disclosure relates to improved extended reality environment interactions, and in particular, systems and methods for detecting eye movements and performing operations within an extended reality environment based on the detected eye movements are disclosed.

Summary of the Invention

Means for Solving the Problems

[0002] Advances in media technology have led to the development of extended reality (XR) technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. VR systems can fully immerse a user (e.g., give the user a sense of being inside the environment) or partially immerse (e.g., give the user a sense of looking at the environment) in a three-dimensional computer-generated environment. The environment can include objects or items with which the user can interact. AR systems can provide a modified version of reality, such as extended information overlaid across real-world objects. MR systems map bidirectional virtual objects to the real world. Such systems can utilize wearables such as head-mounted devices that include a stereoscopic display or smart glasses.

[0003] XR systems introduce many challenges. For example, wearable devices used to visualize the environment may not include external devices (e.g., lenses), so it can be difficult for an XR system to detect when a user modifies their field of view or focus within the XR environment. As another example, pupil dilation and constriction can vary depending on what the user is viewing within the XR environment or the amount of light incident on the user's eyes, but the user may not have control over their pupils, and thus monitoring the user's pupils may not be a reliable way to determine the user's gaze or field of view within the XR environment. Even worse, even if the user's field of view is accurately identified, if there are multiple objects within the user's field of view, it can be difficult to determine which object the user desires to interact with.

[0004] In addition, current approaches to XR suffer from certain drawbacks. In one approach, the user employs hand gestures or a joystick to navigate the XR environment. However, requiring such user input to interact with the XR environment undoubtedly detracts from the XR experience (i.e., reminds the user that the XR environment is not real) and can be cumbersome or inconvenient for the user. Additionally, with current approaches to XR, it may not be possible for the user to conveniently obtain information about objects within their field of view or for the user to interact within the XR environment.

[0005] To overcome these problems, a system and method are provided herein for identifying objects within a user's field of view, detecting a user's eyelid movement, and based on such detection, regenerating the objects for display within an extended reality environment using a modified level of detail. The systems and methods described herein also provide a step of matching the detected eyelid movement with a stored eyelid movement identifier and a step of performing an action on the object based on such matching. Additionally, a system and method are provided for generating an indicator for reflecting a user's gaze shift to a new portion of an extended reality environment that includes the object, and for performing an action when an audio command is received while the indicator is near the object. The systems and methods described herein also provide a step of generating an opacity-based indicator for display within the extended reality environment in the vicinity of a portion of the extended reality environment that includes the object, and a step of varying the opacity of such indicator based on the identified boundaries of the object. Additionally, a system and method are provided for enabling a user to conveniently obtain additional information about an item within the extended reality environment.

[0006] In some aspects of the present disclosure, an extended reality system generates an extended reality environment for display that includes a first object and receives input from one or more sensors. Based on the received input, the system identifies the first object within the field of view, detects eyelid movement, and in response to detecting the eyelid movement, regenerates the first object for display using a modified level of detail. Thus, eyelid movement can be monitored to overcome issues associated with determining which objects within the user's field of view are being focused on by the user. Additionally, detecting such eyelid movement by the user enables the user to visually perceive finer details of objects that may appear, for example, far away from the user within the extended reality environment, which can improve the user experience in the extended reality system, particularly for users with visual impairments.

[0007] The extended reality environment may comprise a plurality of objects including a first object and a second object within the field of view, and the system may regenerate the first object for display using a modified level of detail in response to determining that the detected eyelid movement is associated with the first object. If the system determines that the detected eyelid movement is associated with the second object, the system may regenerate the second object for display using a modified level of detail. The first object may be in either the foreground or the background within the field of view within the extended reality environment, and the second object may be in the other of the foreground or the background within the field of view within the extended reality environment.

[0008] In some embodiments, the step of regenerating a first object for display using a modified level of detail includes presenting the object at a higher resolution. Additionally, or alternatively, one or more actions may be performed on the first object based on one or more detected eyelid movements.

[0009] In some aspects of the present disclosure, the system calculates an individual virtual distance of a plurality of objects relative to the user, and the step of identifying a first object within the field of view includes determining that the first object is at a virtual distance that is closest to the user of the individual virtual distances.

[0010] In some embodiments, the step of detecting eyelid movement includes determining the amount of movement of the eyelids, and / or the step of detecting eyelid movement includes determining one or more eyelid levels. The system may detect that the user is navigating from a first position to a new position within the extended reality environment while the first object remains within the field of view, and generate an updated version of the first object for display based on the user's viewpoint at the new position.

[0011] In some aspects of the present disclosure, an extended reality system generates an extended reality environment for display that includes objects, and stores in memory a table of eyelid movement identifiers and corresponding actions that are executable on objects within the extended reality environment. Using sensors, the system detects eyelid movements and matches the detected eyelid movements to one of the stored eyelid movement identifiers. In response to matching the detected eyelid movements to one of the stored eyelid movement identifiers, the system generates an updated version of the extended reality environment for display based on the action corresponding to the matched eyelid movement. Thus, eyelid movements can be monitored to overcome challenges associated with determining objects within the field of view that the user desires to interact with. Additionally, detecting such eyelid movements of the user enables the user to interact with objects that appear to be far away from the user within the extended reality environment, which can improve the user experience in the extended reality system, particularly with respect to users having visual impairments.

[0012] An object may be selected from a plurality of objects within an extended reality environment by detecting that the user's gaze is directed at the object. The system may generate, for display, a subset of eyelid movement identifiers that are applicable to the object at which the user's gaze is directed (e.g., to remind the user or to guide the user regarding an action to be performed by a certain eyelid movement). The actions of the plurality of actions may correspond to manipulating the object and / or modifying the appearance of the object. (For example, if the object is a book, the action may be turning the pages of the book, tilting the book, tearing the pages of the book, etc.). The system may detect that the user is navigating from a first position to a new position within the extended reality environment while the user's gaze remains on the object, and generate, for display, an updated version of the first object, having a modified appearance, based on the user's viewpoint at the new position.

[0013] In some embodiments, the user may be associated with a user profile that defines the relationship between the eyelid movement identifiers applicable to an object within the extended reality environment and the corresponding actions. The actions applicable to an object may vary based on the type of the object. To detect an eyelid movement, the system may determine whether the eyelids remain closed over a predetermined period, and match the detected eyelid movement to one of the eyelid movement identifiers stored in response to determining that the eyelids remain closed over a predetermined period (e.g., to ensure that the eyelid movement is not an involuntary blink).

[0014] In some aspects of the present disclosure, an extended reality system generates an extended reality environment with objects for display, uses a first sensor to detect that a gaze has shifted from a first portion of the extended reality environment to a second portion of the extended reality environment, and the object is excluded from the first portion of the extended reality environment and is included within the second portion of the extended reality environment. In response to detecting the gaze shift, the system generates an indicator of the shift in gaze for display within the extended reality environment and uses a second sensor to detect an audio command while the indicator is near the object. In response to detecting the audio command, the extended reality system performs an action corresponding to the audio command. Thus, extended reality may be utilized in combination with audio to improve the user experience. More specifically, a user may conveniently use their eyes to navigate an extended reality environment (e.g., as a proxy for how a mouse or trackpad is used in conjunction with a desktop, laptop, or mobile device), receive real-time confirmation of where their gaze is directed, and perform desired actions within the environment via an audio command when an indicator of the user's gaze is near a target object within the extended reality environment.

[0015] A two-way media guide may be provided on the display, and the actions referred to above may be instructions related to media assets accessible via the two-way media guide. The audio commands may include commands to identify and perform actions on media assets, and / or commands to present new media assets on the display and / or commands to read out content related to an entity, and the object is associated with the entity.

[0016] In some embodiments, the extended reality system may normalize retinal movement when determining whether the rate of retinal movement exceeds a predetermined value and converting the retinal movement into movement of an indicator on a display in response to determining that the rate of retinal movement exceeds the predetermined value. The system may detect an audio command while the indicator is near (e.g., overlapping) the object in response to determining that the gaze is directed at the object for at least a predetermined threshold period. The display is presented via a virtual reality head-mounted device.

[0017] In some aspects of the present disclosure, the extended reality system generates an extended reality environment comprising an object for display and, by using sensors, may detect that the gaze is directed at a first portion of the extended reality environment, wherein the object is included within the first portion of the extended reality environment. The extended reality system generates, for display within the extended reality environment, a plurality of opacity-based indicators in the vicinity of the first portion of the extended reality environment, identifies the boundaries of the object, and may vary the opacity of at least one of the plurality of opacity-based indicators based on the identified boundaries of the object. Accordingly, the user may conveniently use their eyes to navigate the extended reality environment (e.g., as a proxy for how a mouse or trackpad is used in conjunction with a desktop, laptop, or mobile device) and receive real-time confirmation regarding the location of their gaze, and the opacity of such real-time gaze indicators is conveniently adjusted so as not to obscure the user's field of view and avoid degrading the user's experience.

[0018] The extended reality system may determine whether at least one of the opacity-based indicators overlaps the boundary of an object and vary the individual opacity of the opacity-based indicators that overlap the boundary. The plurality of opacity-based indicators are arrows directed towards the object. The extended reality system may detect, by using a sensor, whether the gaze has shifted to a second portion of the extended reality environment, and in response to determining that the gaze has shifted to the second portion, overlay the plurality of opacity-based indicators in the vicinity of the second portion of the display.

[0019] In some embodiments, the individual opacity is varied based on the distance from the object. For example, the individual opacity of the indicator may increase as the distance between the indicator and the object decreases (e.g., to emphasize the object that the user is gazing at), or may increase as the distance between the indicator and the object decreases (e.g., to avoid obscuring the object that the user is gazing at).

[0020] In some embodiments, a bidirectional media guide may be provided on the display, and actions related to media assets accessible via the bidirectional media guide are received at least in part based on the detected gaze. Such a display may be presented via a virtual reality head-mounted device or presented without the use of a virtual reality head-mounted device.

[0021] In some aspects of the present disclosure, an extended reality system generates an extended reality environment that includes an object for display, detects eye movements, and based on the detection, determines whether the object is within the field of view for at least a predetermined period. In response to determining that the object is within the field of view for at least a predetermined period, the system generates one or more items related to the object for display within the extended reality environment. Thus, information about the object of interest may be conveniently displayed to the user based on detecting the user's eye movements related to the object of interest.

[0022] One or more items related to the object may comprise text information, images, video, or any combination thereof. The system may further determine that the object has not been within the field of view for a first predetermined period and that at least a second predetermined period has elapsed since the start of the display of one or more items, and in response to such a determination, may stop the display of one or more items. The extended reality environment may be presented via a virtual reality head-mounted device. In some embodiments, the step of detecting eye movements includes the step of monitoring eyelid movements or the step of monitoring fixation.

[0023] The system may determine whether the object is within the field of view for a predetermined period in response to determining that the field of view is continuously (or discontinuously) over the object for a predetermined period during a virtual reality session.

[0024] In some embodiments, the system determines that a new object has been in the field of view for at least a predetermined time and, in response to such determination, generates one or more items related to the new object for display within the extended reality environment, while it may also continue to generate one or more items related to the object for display within the extended reality environment. The present invention provides, for example, the following. (Item 1) A method for extended reality environment interaction, comprising: generating an extended reality environment with objects for display; detecting, by using a first sensor, that a gaze has shifted from a first part of the extended reality environment to a second part of the extended reality environment, wherein the object is excluded from the first part of the extended reality environment and is included within the second part of the extended reality environment; in response to detecting the gaze shift, generating an indicator of the shift in the gaze for display within the extended reality environment; detecting, by using a second sensor, a voice command while the indicator is in the vicinity of the object; and in response to detecting the voice command, executing an action corresponding to the voice command . (Item 2) The method according to item 1, wherein a two-way media guide is provided on the display, and the action is a command related to a media asset accessible via the two-way media guide. (Item 3) The gaze is detected based on retinal movement of an eye, and the method further comprises: converting the retinal movement into movement of the indicator on the display, according to the method of item 1 or 2. (Item 4) determining whether a rate of the retinal movement exceeds a predetermined value; and normalizing the retinal movement when converting the retinal movement into movement of the indicator on the display in response to determining that the rate of the retinal movement exceeds the predetermined value, according to the method of item 3. (Item 5) The method according to any one of items 2-4, wherein the voice command includes identification of the media asset and a command for executing the action. (Item 6) The method according to any one of items 2-5, wherein the voice command includes a command for presenting a new media asset on the display. (Item 7) The method according to any one of items 2-6, wherein the object is associated with an entity, and the voice command includes a command for reading out content related to the entity. (Item 8) ​ Detecting the voice command while the indicator is in the vicinity of the object includes determining that the gaze is directed at the object for at least a predetermined threshold period, the method according to any one of items 1-7. (Item 9) The indicator being in the vicinity of the object includes the indicator overlapping the object, the method according to any one of items 1-8. (Item 10) The display is presented via an extended reality head-mounted device, the method according to any one of items 1-9. (Item 11) A system for extended reality environment interaction, the system comprising a control network configured to execute the method according to any one of items 1-10. (Item 12) A system for extended reality environment interaction, A display, A control network, Generating an extended reality environment with an object for display, Detecting, by using a first sensor, that the gaze is shifted from a first part of the extended reality environment to a second part of the extended reality environment, wherein the object is excluded from the first part of the extended reality environment and is included within the second part of the extended reality environment, In response to detecting the gaze shift, generating an indicator of the shift in the gaze for display within the extended reality environment, Detecting a voice command while the indicator is in the vicinity of the object by using a second sensor, In response to detecting the voice command, executing an action corresponding to the voice command A control network configured to perform A system comprising. (Item 13) A two-way media guide is provided on the display, and the action is an instruction related to a media asset accessible via the two-way media guide, the system according to item 12. (Item 14) The control network further Detecting the gaze based on eye retinal movement, Converting the retinal movement into movement of the indicator on the display The system according to item 12 or 13, configured to perform (Item 15) A non-transitory computer-readable medium having instructions encoded thereon, which, when executed by a control circuitry, cause the control circuitry to perform the method according to any one of items 1-10

Brief Description of the Drawings

[0025] The above and other objects and advantages of the present disclosure will become apparent upon consideration of the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout.

[0026]

Figure 1

[0027]

Figure 2

[0028]

Figure 3

[0029]

Figure 4

[0030]

Figure 5

[0031]

Figure 6

[0032]

Figure 7

[0033]

Figure 8

[0034]

Figure 9

[0035]

Figure 10

[0036]

Figure 11

[0037]

Figure 12

[0038]

Figure 13

[0039]

Figure 14

[0040]

Figure 15

[0041] Detailed Description FIG. 1 illustrates an exemplary process for rendering an object for display within an extended reality (XR) environment, according to some embodiments of the present disclosure. A head-mounted display 102 may generate a three-dimensional XR environment 100 for projecting an image and immersing a user therein. The user may be fully or partially immersed within the XR environment 100, and such an environment may be a fully virtual environment. The head-mounted display 102 may alternatively be a wearable device (e.g., smart glasses) or a computer or mobile device equipped with a camera and an XR application to facilitate generation of the environment 100. The environment 100 may alternatively be an augmented reality (AR) environment in which real-world objects are supplemented with computer-generated objects or information, or a mixed reality (MR) in which, for example, virtual objects interact with the real world or the real world is otherwise connected to virtual objects. In some embodiments, the user's field of view or perspective of the environment 100 changes as the user moves their head, and other features (e.g., audio) are preferably modified to simulate the physical world. The environment 100 may be for entertainment purposes (e.g., video games, movies, videos, sports, etc.), communication (e.g., social media), educational purposes (e.g., virtual classrooms), occupational purposes (e.g., training simulations), medical purposes, etc.

[0042] The XR system may identify one or more objects within the user's field of view. The field of view is a part of the XR environment 100 presented to the user at a given time by the display 102 (e.g., a certain angle in a 360-degree spherical environment). The field of view may comprise a pair of 2D images for generating stereoscopic vision in the case of a VR device, and in the case of an AR device (e.g., smart glasses), the field of view may comprise a 3D or 2D image that includes a mixture of real objects and virtual objects overlaid on the real objects using the AR device (e.g., for smart glasses, a photograph captured using a camera and content added by the smart glasses). If the XR environment has a single degree of freedom, e.g., a 360-degree rotation, any field of view may be defined either by edge angle coordinates (e.g., +135 degrees, +225 degrees) or by a single angle coordinate (e.g., -55 degrees) combined with a known angular opening of the field of view. If the XR environment has six degrees of freedom, i.e., three rotations of 360 degrees and three spatial positions, any field of view may be defined by three angle coordinates and three spatial coordinates. The field of view can thus be understood as a part of the XR environment that is displayed when the user is at a particular location within the XR environment and the XR set is oriented in a particular direction.

[0043] An XR system (e.g., system 900 of FIG. 9) may generate a data structure regarding a field of view that includes object identifiers associated with virtual objects within the field of view, and such a data structure may include coordinates representing the position of the field of view within the XR environment. The system may determine the current field of view and identify objects within the user's field of view based on the data structure and / or image captured by the XR device 102. As shown in the embodiment of FIG. 1, the detected field of view of the user within environment 100 includes object 104 depicted as an automobile, but those skilled in the art will understand that any number or combination of different types of objects may be included within environment 100. The XR system may generate object 104 for display at a default level of detail (e.g., default resolution or number of pixels displayed, or default size or appearance). For example, objects within environment 100 may be presented by default at 4K resolution (3,840 × 2,160) or any other suitable resolution. The resolution of objects within environment 100 may be the same or vary for each of the user's eyes. In some embodiments, the level of detail may refer to the size or appearance of the object, e.g., the object may be generated at a default size or default color.

[0044] In some embodiments, in response to determining one or more objects 104 within the user's field of view, the XR system may generate an identifier 106 (e.g., "blink once to modify details of the car") for display that indicates to the user, or otherwise provides guidance, regarding how to perform an action on the object 104 with a particular eyelid movement. In some embodiments, the XR system may refer to a table (e.g., table 316 of FIG. 3, which may be stored in storage device 808 of FIG. 8) that includes a plurality of eyelid movement identifiers and corresponding actions that are performable on object 104 within the XR environment. For example, the table may additionally store an identifier (e.g., "blink twice") that may correspond to increasing or decreasing the size of the object displayed to the user in response to detecting the indicated eyelid movement.

[0045] Once the object of interest within the field of view is identified, the XR system may detect the user's eyelid movement 108 by using a sensor (e.g., a camera). In some embodiments, the XR system may detect whether the eyelid movement exceeds a predetermined period (e.g., 0.5 seconds or 1 second) in order to avoid performing an action based on an involuntary blink (e.g., if such an action is not desired by the user). In response to detecting the user's eyelid movement 108 (e.g., a single blink corresponding to an action to modify details of the object of interest), the XR regenerates the object 104 for display provided to the user via the head-mounted display 102. For example, the object 104 may be presented to the user at a higher resolution (e.g., 8K resolution, 7,680 × 4,320) than the initially provided one (e.g., 4K resolution, 3,840 × 2,160). In some embodiments, the detected eyelid movement may cause the XR system to modify the details of the object 104 in different ways (e.g., increase or decrease the size of the object compared to the initial presentation of the object, change the color or texture of the object compared to the initial appearance of the object, etc.).

[0046] In some embodiments, the step of detecting eyelid movement includes the step of determining the amount of movement of the eyelid, or the step of detecting eyelid movement includes the step of determining one or more eyelid levels. For example, the XR system may use a sensor (e.g., a camera) to detect the amount of movement of the eyelid, compare the detected amount with a threshold amount of movement over a predetermined period (e.g., 5 eyelid movements detected over a 3-second period), and an image may be modified or selected when the detected amount of eyelid movement exceeds the threshold amount of movement over the predetermined period. As another example, the XR system may use a sensor (e.g., a camera) to detect one or more eyelid levels (e.g., distinct eyelid levels) over a predetermined period, compare the detected number with a threshold number of eyelid levels over a predetermined period (e.g., 5 distinct eyelid levels detected over a 3-second period), and an image may be modified or selected when the detected number of eyelid levels exceeds the threshold number of eyelid levels over the predetermined period.

[0047] Figure 2 shows an exemplary process in which a plurality of objects are detected within the user's field of view. The XR system may detect that objects 204 and 208 are within the field of view of a user who is viewing the XR environment 200 via the head-mounted display 202, respectively. As shown in the environment 200 in the uppermost part of FIG. 2, the objects 204, 208 detected as being within the user's field of view may first be presented at a default level of detail (e.g., default resolution or default size). In response to detecting an eyelid movement 210 (e.g., corresponding to "blinking once to modify the details of the car" shown in identifier 206), the XR system may reproduce the object 204 within the field of view using the modified level of detail for display (e.g., improving the resolution of the object 204). On the other hand, in response to detecting an eyelid movement 212 (e.g., corresponding to "blinking twice to modify the details of the airplane" shown in identifier 206), the XR system may reproduce the object 208 within the field of view using the modified level of detail for display (e.g., improving the resolution of the object 208).

[0048] In some embodiments, detecting additional eyelid movement 212 may subsequently reverse a modification that was performed in response to detecting eyelid movement 210 (e.g., object 204 may return to the default resolution first presented to the user, while object 208 is presented with the modified details). Alternatively, the object may be maintained in the modified state throughout the XR session and / or in future sessions. In some embodiments, detecting that the eyelid movement is being re-performed may reverse an action (e.g., detecting eyelid movement 210 a second time may cause object 204 to return to the default resolution first presented to the user). In some embodiments, of the plurality of objects that may be within the user's field of view, one of such objects (e.g., object 204) may be in the foreground of the display of XR environment 200, and another of such objects may be in the background of the display within XR environment 200 (e.g., object 208). Additionally, one or more actions may be performed on the modified object within the user's field of view (e.g., a particular eyelid movement may correspond to opening the door of automobile object 204 or interacting with airplane object 208). The most recently modified object within the field of view may be the "selected" object so that an action can be performed on such object. Such aspects may enable objects that are far from the user or are otherwise too small to be viewed in detail to be re-rendered with modified details to enable the user to clearly visually identify the object.

[0049] In some aspects of the present disclosure, the XR system may detect the user's eyelid movements that match the objects within the user's field of view, calculate the relative degree to which the eyelids are closed, and determine the object to be initially focused on within the user's field of view. In some embodiments, when the user enters the XR environment, the XR system sets a default field of view, detects the number of objects within the environment and / or within the field of view, and may calculate the respective individual virtual distances or focal distances of the detected objects with respect to the user. The objects can be at different virtual distances from the user. In some embodiments, the step of identifying an object within the field of view includes determining that the object is at the virtual distance closest to the user of the individual virtual distances or focal distances. The virtual distance may be, for example, the perceived distance at which an object within the XR environment is located from the user and may be calculated based on the coordinates of the object within the XR environment. The user's eyelid level may be calculated at least in part based on such virtual distances, and in response to detecting a change in the eyelid level, the object at the virtual distance closest to the user may be detected and selected as the object of interest, to which a modification may be applied.

[0050] In some embodiments, the XR system may detect the movement of a user around the XR environment 200 and detect that the user's field of view changes as the user moves around. In such a situation, the user's field of view may be reset to determine the number of objects within the user's new field of view. On the other hand, if the XR system detects the movement of a user around the XR environment 200 but detects that the user's gaze remains fixed on a particular object, the display 200 may generate such an object from a varied viewpoint consistent with the user's movement for display and maintain the simulated environment. In some embodiments, any change in eyelid level detected by the virtual reality system may be used to determine the object whose details within the user's field of view should be modified. The XR system may track the movement of a user within the XR environment by using sensors (e.g., a gyroscope, an accelerometer, a camera, etc. in combination with a control circuitry).

[0051] Figure 3 shows an exemplary process for performing an action on an object within an XR environment, according to some embodiments of the present disclosure. The head-mounted display 302 may generate an XR environment 300 for display that includes objects 304, 310, 312, 314. Four objects (book object 304, lamp object 310, desk object 312, and chair object 314) are shown within the environment 300, but it should be understood that any number and any type of objects may be generated for display. The XR system (e.g., system 900 of FIG. 9) may store a table 316 of eyelid movement identifiers and corresponding actions that can be performed on objects within the XR environment (e.g., within the storage device 808 of FIG. 8). For example, the table 316 may store a plurality of associations, including an association related to the book object 304 as indicated by the identifier 306, which may be displayed to the user to facilitate the desired actions of "blinking once to turn a page of the book, blinking twice to tilt the book, and blinking three times to tear a page from the book." Any number of actions and any type of identifiers may be stored within the table 316, and the actions may depend on the type of object (e.g., the table 316 may store an identifier associated with an action to turn on a virtual light bulb related to the lamp object 310). If the environment includes multiple objects, the objects may be selected from the plurality by detecting (e.g., using sensors) that the user's gaze is directed at the object.

[0052] The XR system detects a user's eyelid movement by using sensors (e.g., cameras). The system may determine whether the detected eyelid movement matches any of the identifiers stored in Table 316, for example, by analyzing the sensor output and comparing such output with the stored identifiers (e.g., a predetermined number of blinks, blink patterns, amount of eyelid movement, eyelid level, etc.). In some embodiments, the stored identifiers may include eyelid movements combined with voice commands or other inputs. In some embodiments, the XR system may detect whether the eyelid movement exceeds a predetermined period (e.g., 0.5 seconds) in order to avoid performing an action based on an involuntary blink (e.g., if such an action is not desired by the user). The system may detect the user's eyelid movement based on the degree of opening and closing of the user's eyelids over time.

[0053] In response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, the XR system causes an updated version of the XR environment 300 to be generated for display based on the action corresponding to the matched eyelid movement. In the example of FIG. 3, the system detects an eyelid movement associated with turning the page of a book (318) and performs such an action as shown in the lower portion of FIG. 3, depicting the turned page of the book object 304 in the environment 300, compared to the book object 304 depicting the closed book in the upper portion of the environment 300. One of ordinary skill in the art will understand that the objects within the environment 300 can be manipulated in various ways, for example, the chair object 314 can be moved adjacent to different portions of the desk object 312 or modified in various ways, such as removing a cushion from the chair object 314.

[0054] In some embodiments, a subset of the identifiers 306 suitable for a selected object of interest may be presented to the user for the user's convenience in determining available actions to be performed based on specific eyelid movements. In some embodiments, the XR system may store one or more user profiles that define the relationship between eyelid movement identifiers executable on objects within the XR environment 300 and corresponding actions. For example, the user profile may include actions adjusted to user preferences, the user's favorite actions, the actions most recently performed by the user, the actions most commonly performed by the user, the user's purchase actions, etc., which may be presented in association with the identifier 306 for the user's convenience.

[0055] In some embodiments, the XR system may detect the movement of the user around the XR environment 200 and may detect that the user's gaze changes as the user moves around. In such a situation, the system may select a new object of interest. Alternatively, the system may detect that the user is navigating from a first position to a new position within the XR environment while the user's gaze remains on an object, and in response to such a determination, generate an updated version of the object for display based on the user's perspective (e.g., modify the size or angle of the object presented to the user). The updated version of the object may include presenting to the user an object with a modified appearance (e.g., a book with a torn page if the user has previously performed an eyelid movement associated with such an action within the table 316).

[0056] In some embodiments, the aspects discussed in FIGS. 1-2 may be combined with the embodiment of FIG. 3 (e.g., within a single user session within the XR environment, objects within the XR environment may be reproduced in more detail and various actions may be performed on such objects).

[0057] Figures 4A-4B show examples of receiving voice commands while an indicator is near an object in an XR environment, according to some embodiments of the present disclosure. The XR system (e.g., system 900 of FIG. 9) may cause the XR environment 400 to be displayed for the user via the head-mounted display 402. In some embodiments, the XR environment 400 may include a two-way media guide application to facilitate the selection and consumption of media content. The XR environment 400 may include one or more objects 408, 410 that may correspond to identifiers for selectable media content. The system may detect that the user's gaze has shifted from a portion of the XR environment (e.g., near object 408 (FIG. 4A)) to another portion of the XR environment (e.g., near object 410 (FIG. 4B)) by using sensors (e.g., cameras). It should be understood that FIGS. 4A-4B are exemplary and that the user's gaze may shift from a portion of the XR environment 400 that does not contain any objects or may contain multiple objects to another portion of the XR environment 400.

[0058] In response to detecting the gaze deviation, the XR system may cause an indicator 406 indicating the deviation in the gaze to occur in the XR environment 400 for display. For example, the indicator 406 in FIG. 4A reflects that the user's gaze is on the object 408 (e.g., an identifier related to the movie "The Dark Knight"), and in FIG. 4B, the indicator 406 reflects that the user's gaze has deviated to the object 410 (e.g., an identifier related to the movie "American Psycho"). In some embodiments, a single indicator may be caused to occur for display, or alternatively, a plurality of indicators may be caused to occur for display. In some embodiments, the indicator may vary in translucency based on proximity to the object of interest. In the examples of FIGS. 4A-4B, the indicator 406 is shown as an arrow directed at the object of interest, but it will be understood by those skilled in the art that the indicator may comprise any suitable mark or marking for emphasizing or prominently displaying the associated object to the user. For example, the indicator may be a certain color or shape for emphasizing the object of interest, an image of an eyeball or an emoji, an enlargement of the object in the vicinity of the indicator, an animation of the object in the vicinity of the indicator, etc.

[0059] This system may detect a voice command while the indicator 406 is in the vicinity of the object 410 by using a sensor (e.g., a microphone), such as when the indicator overlaps the object of interest or is otherwise within a predetermined distance thereof. The XR system may process the voice command and execute an action, such as when there is a match between an object included in the voice command and the object that the user's gaze, as indicated by the indicator 406, is directed to. For example, in response to receiving the voice command 404 in the embodiment of FIG. 4A, the system may initiate presentation of the media asset "The Dark Knight" associated with the object 408, and in response to receiving the voice command 405 in the embodiment of FIG. 4B, the system may initiate presentation of the media asset "American Psycho" associated with the object 410. In some embodiments, if the user determines that the indicator does not accurately reflect the user's gaze, the system may receive from the user a suitable voice command that requires the system to re-calibrate the user's gaze and / or indicates the portion of the display that the user believes they are looking at.

[0060] In some embodiments, the user's gaze is detected based on eye retinal movement (tracked by sensors, e.g., a camera that measures the reflection of a light source from the retina, eye-tracking glasses, screen-based eye tracking). The user's retinal movement may be plotted or transformed into the display of the XR environment as the movement of the indicator 406 on the display. In some aspects of the present disclosure, the system may determine whether the rate of retinal movement exceeds a predetermined value, and in response to such a determination, perform normalization when transforming the retinal movement into the movement of the indicator 406 on the display of the XR environment. For example, if the speed of the gaze shift exceeds a predetermined value, normalization may be performed to slow down the movement of the indicator 406 on the display (e.g., to enable the user to more easily track the movement of the indicator 406 on the display). The entire cluster of indicators may move to such a new portion of the display.

[0061] In some embodiments, the system includes an electronic speech recognition (or speech assistance) device (e.g., a television, computer, voice assistant) that responds to user voice commands, and the voice input may be in the form of an audio or digital signal (or audio or digital input). The system may implement natural language understanding (NLU) techniques and may include a natural language understanding circuit network and / or a speech-to-text circuit network to transcribe the voice command into text, analyze the voice command, and identify and extract keywords from the voice input. The system may compare the extracted keywords with metadata associated with the object of interest and determine whether there is a match, e.g., whether to execute the voice command. In some embodiments, if the received voice command does not match an object in the vicinity of the indicator, the system may notify the user of the mismatch, refrain from performing the associated action, or prompt the user for a new voice command.

[0062] In some embodiments, the voice command includes a command for identifying a media asset and performing an action (e.g., play, fast forward, rewind, etc.) or an instruction for presenting a new media asset on the display (e.g., scrolling through other media assets or moving to a new page of the media assets in the carousel). In some aspects of the present disclosure, the step of determining that the indicator is near the object includes determining that the user's gaze is directed at the object for at least a predetermined threshold period (e.g., 5 seconds).

[0063] FIG. 5 shows an example of receiving a voice command while the indicator is near an object in an XR environment according to some embodiments of the present disclosure. The XR system (e.g., system 900 of FIG. 9) includes a head-mounted display 502, and the head-mounted display 502 may be used to generate an XR environment 500 for display, including objects 508, 510. As shown in the upper environment 500 of FIG. 5, the system may detect that the user's gaze is directed at a part of the XR environment (e.g., including object 508) by using a sensor (e.g., a camera). The system may generate a plurality of opacity-based indicators 506 for display within the XR environment 500 in the vicinity of the part of the XR environment 500 including the object 508. In the example of FIG. 5, the indicator 506 is shown as an arrow directed at the object of interest, but it will be understood by those skilled in the art that the indicator may comprise any suitable mark or marking for emphasizing or prominently displaying the associated object to the user. For example, the indicator may be a certain color or shape for emphasizing the object of interest, an image of an eyeball or an emoji, an enlargement of the object in the vicinity of the indicator, an animation of the object in the vicinity of the indicator, etc.

[0064] This system may identify the boundary of object 508 (e.g., edge, shape contour, end) by, for example, edge detection techniques, reading the coordinates of object 508, analyzing the pixel values of the area surrounding object 508, etc. Based on the identified boundary of the object, the XR system may vary the opacity of at least one of the plurality of opacity-based indicators 506. In some embodiments, the system may determine whether at least one of the plurality of opacity-based indicators overlaps the boundary of the object or is within a predetermined distance thereof, and in response to such determination, may vary the individual opacity of one or more indicators 506 that overlap the boundary of object 508. For example, the system may compare the coordinates of the object of interest in XR system 900 of FIG. 9 with the coordinates of indicator 506. In some embodiments, when the system detects that the user's gaze has shifted from a part of the display (e.g., including object 508) to a part of the display including another object (e.g., object 510), the system overlays a plurality of transparency-based indicators in the vicinity of the part of the display including object 510. The entire cluster of indicators may move to such a new part of the display.

[0065] This system may vary the individual opacity based on the distance from the object. As shown in the embodiment of FIG. 5, the individual opacity of the indicator 506 may increase as the distance between the indicator 506 and the object 508 decreases. This may be desirable to emphasize to the user the portion of the display to which the user's gaze is directed. Alternatively, as shown in the embodiment of FIG. 6, the individual opacity of the indicator 606 may decrease as the distance between the individual indicator 606 and the object 608 decreases. This may be desirable to minimize obscuring the portion of the object of interest. In some embodiments, the system may determine whether any of the indicators overlap or are otherwise in the vicinity of another object that may not be noticed, and in such situations, the indicator may be set to be translucent to avoid either obscuring the portion of such unnoticed object or misleading the user into thinking such object is being noticed.

[0066] As shown in the embodiments of FIGS. 5 and 6, the XR environments 500, 600 may each include a two-way media guide. In response to receiving a voice command to perform an action associated with an object in the vicinity of the indicators 506, 606, the system may perform the action (e.g., generate for presentation an identifier for a movie having metadata associated with "Christian Bale", who is an actor associated with the objects 508, 608). The embodiments of FIGS. 5-6 show objects related to the "actor" category, but various other categories may be generated and browsed for display (e.g., genre, cast, director, etc.). In some embodiments, the environments 500, 600 may be presented regardless of the presence of an XR head-mounted device. For example, the XR environment may be presented as fully immersive VR or as AR or MR overlaid on a TV screen as shown in FIGS. 5-6.

[0067] In some embodiments, the features of the examples of FIGS. 1-4 may be used in conjunction with the features of the examples of FIGS. 5-6. For example, an indicator (e.g., indicator 106 in FIG. 1) may be displayed to a user to guide the user when performing eyelid movement or gaze, which may be associated with an actionable action (e.g., in table 316 of FIG. 3). The indicators of FIGS. 5 and 6 may also be used in connection with other environments (e.g., video games, sports, communications, social media, etc.).

[0068] FIG. 7 shows an example of presenting additional information related to an item within an XR environment according to some embodiments of the present disclosure. A head-mounted display 702 presents an XR environment 700 to a user. The XR environment 700 may include one or more objects, such as a washing machine object 702, a refrigerator object 704, a wall object 708, etc. In some embodiments, the XR environment 700 includes an identifier 706 that may indicate to the user an action that can be taken within the environment 700. The environment 700 represents a room within a building (e.g., a house or an apartment), but the environment 700 may comprise any virtual or augmented environment, such as a mountain range, a scenic location, a video game being played by the user, etc.

[0069] This system detects a user's eye movement (e.g., via a sensor) and, based on the detected eye movement, determines whether an object (e.g., a washing machine object 702, a refrigerator object 704, and / or a wall object 708) has been within the user's field of view for at least a predetermined period 710 (e.g., 5 seconds). In response to determining that an object (e.g., a wall object) has been within the field of view for at least a predetermined period, the system generates one or more items 714 associated with the object 708 for display within the XR environment. The one or more items 714 associated with the object may comprise text information, images, video, or any combination thereof, and may be read from an item information database 712 (e.g., stored within the storage device 808 of FIG. 8). For example, the item 714 may indicate the thickness of the wall object 708, and such additional information may be provided when the user gazes at the wall object 708 via the head-mounted display 702 within the XR environment 700. In some embodiments, the step of detecting eye movement includes the step of monitoring eyelid movement. In some aspects of the present disclosure, the step of generating one or more items 714 associated with the object 708 for display within the XR environment is performed in response to detecting both that the object 708 has been within the field of view for at least a predetermined period and that the object is within the user's gaze. In some embodiments, the system enables modification of the predetermined period (e.g., by the user or as recommended by the system based on the user's historical data).

[0070] As another example, the environment 700 may be a mountain range, and information regarding a mountain object may be generated for display to the user (e.g., the name of the mountain, the elevation of the mountain, the location of the mountain, etc.) in response to determining, based on the user's detected eye movement (e.g., detected via a sensor), that an object (e.g., a mountain) has been within the user's field of view for at least a predetermined period.

[0071] In some aspects of the present disclosure, the step of determining whether an object is within the field of view for a predetermined period may include determining that the field of view is continuously over the object for a predetermined period (e.g., 5 seconds) during the XR session. For example, if the system detects that the user's field of view has deviated during the countdown of a timer for a predetermined period, the countdown may be reset. Alternatively, the step of determining whether an object is within the field of view for a predetermined period may include determining that the field of view is discontinuously over the object for a predetermined period (e.g., 5 seconds) during the XR session. For example, if the system detects that the user's field of view has deviated during the countdown of a timer for a predetermined period, the countdown may be paused and resumed in response to detecting that the user's gaze has returned to the object.

[0072] In some embodiments, the system may determine whether a new object is within the field of view for at least a predetermined time. For example, in response to the system making such a determination, the system may generate one or more items related to the new object for display within the XR environment, while also continuing to generate one or more items related to the object (e.g., object 708 and item 714) for display within the XR environment. This enables the user to visually recognize additional information regarding multiple items for at least the duration of the XR session. In some embodiments, the additional information items may be maintained in a subsequent user session within the XR environment.

[0073] In some embodiments, the system may determine that at least a predetermined period (e.g., 10 seconds) has elapsed since the start of the display of one or more items 714 without the object 708 being present within the field of view for a first predetermined period. For example, after an item 714 is provided to the user, if at least a predetermined period (e.g., 10 seconds) has elapsed and the object 708 associated with the item 714 has not been within the field of view during such period, the display of the item 714 may be stopped (e.g., because the user may no longer be focusing on the object 708).

[0074] In an embodiment, additional inputs (e.g., voice commands, hand gestures, touches) may be used in combination with detecting the user's eye movements. For example, the system may receive a voice command indicating that the user no longer desires to view additional information related to the focused item, or may receive a voice command to display additional information, and may read additional information related to the focused item (e.g., item 708) from the item information database 712.

[0075] FIG. 8 is a block diagram of an exemplary device in an XR system according to some embodiments of the present disclosure. A device 800 in the XR system may include one or more servers for generating, displaying, and / or managing an XR environment that is transmitted via a computer network to XR devices (e.g., devices 102, 202, 302, 402, 502, 602, 702 of FIGS. 1-7, respectively). The device 800 may include a control circuitry 804 and / or an I / O path 810. The control circuitry 804 may be based on any suitable processing circuitry such as a processing circuitry 806. As referred to herein, the processing circuitry is to be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or a supercomputer.

[0076] In some embodiments, processing circuitry 806 may be distributed across a plurality of distinct processors or processing units, such as a plurality of the same type of processing units (e.g., two Intel Core i7 processors) or a plurality of different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). I / O path 810 may provide device information or other data and / or other content and data to control circuitry 804, including processing circuitry 806 and storage device 808, via a local area network (LAN) or a wide area network (WAN). Control circuitry 804 may be used to transmit and receive commands, requests, signals (digital and analog), and other suitable data using I / O path 810. I / O path 810 may connect control circuitry 804 (and specifically, processing circuitry 806) to one or more communication paths. In some embodiments, storage device 808 may include table 316 of FIG. 3 and / or item information database 712 of FIG. 7.

[0077] In some embodiments, the control circuitry 804 executes instructions stored in a memory (e.g., storage device 808) for performing XR content generation and / or processing functions. The instructions may be stored in either non-volatile memory 814 and / or volatile memory 812 and loaded into the processing circuitry 806 at runtime. A system for generating and processing XR content (e.g., the system described with reference to FIGS. 1-7) may be a stand-alone application implemented on a user device (e.g., end-user device 920) and / or a server (e.g., server 900), or may be distributed across multiple devices according to device 800. The system may be implemented as a set of software or executable instructions. Instructions for implementing any of the embodiments discussed herein for XR processing may be encoded on a non-transitory computer-readable medium (e.g., a hard drive, random access memory on a DRAM integrated circuit, read-only memory on a BLURAY disk, etc.) or a transitory computer-readable medium (e.g., a propagation signal that conveys data and / or instructions). For example, instructions according to the processes described herein may be stored in storage device 808 and executed by the control circuitry 804 of device 800.

[0078] The control circuit network 804 may include a video generation circuit network and a tuning circuit network such as one or more analog tuners, one or more MPEG-2 decoders or other digital decoding circuit networks, high-definition tuners, or any other suitable tuning or video circuits or combinations of such circuits. An encoding circuit network (e.g., for converting a radio, analog, or digital signal into an MPEG signal for storage) may also be included. The control circuit network 804 may also include a scaler circuit network for up-converting and down-converting content to the preferred output formats of the end-user devices 920, 925, 930 of FIG. 9. The circuit network 804 may also include a digital / analog converter circuit network and an analog / digital converter circuit network for converting between digital and analog signals. The tuning and encoding circuit networks may be used to receive, display, play, or record content. The tuning and encoding circuit networks may also be used to receive guidance data. For example, the circuit networks described herein, including tuning, video generation, encoding, decoding, encryption, decryption, scaler, and analog / digital circuit networks, may be implemented using software that runs on one or more general-purpose or specialized processors. A plurality of tuners may be provided to handle simultaneous tuning functions (e.g., viewing and recording functions, functions for incorporating video or other recordings of the physical environment, multi-tuner recording, etc.). If the storage device 808 is provided as a separate device from the device 800, the tuning and encoding circuit networks (including multiple tuners) may be associated with the storage device 808. The device 800 may be a central device that communicates with each device (e.g., 102, 202, 302, 402, 502, 602, 702). Alternatively, the device 800 may correspond to the device 104, i.e., each system (e.g., of FIGS. 1-7) may not have its own device 920, but rather a single central device 800 may execute the XR environment generation and processes for each system.

[0079] FIG. 9 shows an illustrative block diagram of an XR content system according to some embodiments of the present disclosure. At least one XR content server 900 generates XR content such as that described herein. The XR content server 900 may transmit content to a plurality of end-user devices, including devices 920, 925, 930, via the Internet 910. The end-user XR devices may include, for example, personal computers, mobile devices (e.g., smartphones), and / or wearable devices, including XR headsets, goggles, suits, gloves, etc., configured to present and enable interaction with an XR environment. These devices are configured to enable an operator / user to view and interact with multi-user XR content (e.g., via a display screen). These devices may provide visual, audio, and haptic feedback, presenting viewpoints and attention-directing cues as described herein, for example. The end-user devices 920, 925, 930 may also transmit data to the server 900 via the Internet 910, such data including orientation information indicating the direction the devices 920, 925, 930 are facing (and thus the view the server 900 must generate for display on each device 920, 925, 930), audio signals detected by each device 920, 925, 930, and user input such as selection of XR objects. The server 900 may then generate a view of the XR environment for each of the devices 920, 925, 930.

[0080] FIG. 10 is a flowchart of a detailed illustrative process for regenerating an object for display within an XR environment, according to some embodiments of the present disclosure. Note that process 1000 or any of its steps may be implemented on or provided by any of the devices shown in FIGS. 8-9. For example, process 1300 may be commanded by one or more programs of computer-executable instructions implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as for distributing control of an extended reality environment among multiple devices, and may be executed by the control circuitry of network device 800 (e.g., via control circuitry 804), and / or the control circuitry of server 900, and / or the control circuitry of devices 920, 925, 930. Additionally, one or more steps of process 1000 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14, process 1500 of FIG. 15).

[0081] At 1002, an input / output (I / O) circuitry (e.g., the I / O circuitry of the head-mounted display 102) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such a user selection may be received via an input (e.g., a voice command or a touch screen).

[0082] At 1004, in response to receiving the user selection, a control circuitry (e.g., control circuitry 804 of FIG. 8 and / or the control circuitry of the head-mounted display 102 of FIG. 1) may generate an XR environment for display (e.g., via the head-mounted device 102 of FIG. 1) that includes an object (e.g., the automobile object 104 of FIG. 1).

[0083] In 1006, an I / O circuit network (e.g., the I / O circuit network of the head-mounted display 102) may receive an input from a sensor (e.g., a camera). The sensor may detect various attributes of the user's eyes (e.g., eyelid movement, fixation).

[0084] In 1008, a control circuit network (e.g., the control circuit network 804 of FIG. 8 and / or the control circuit network of the head-mounted display 102 of FIG. 1) may identify an object (e.g., the vehicle object 104 of FIG. 1) within the user's field of view based on the received input. In some embodiments, the control circuit network may generate a data structure regarding the field of view that includes an object identifier associated with a virtual object within the field of view, and such a data structure may include coordinates representing the position of the field of view within the XR environment. The control circuit network may determine the current field of view and identify an object within the user's field of view based on the data structure and / or image captured by the XR device 102.

[0085] In 1010, the control circuit network may detect the user's eyelid movement (e.g., the amount of eyelid movement, blink pattern, eyelid level, etc.) based on the received input from a sensor (e.g., a camera). In some embodiments, such eyelid movement may be distinguished from involuntary blinks in order to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore a blink pattern in which the eyes remain closed for less than a predetermined amount of time (e.g., 0.5 seconds).

[0086] At 1012, the control circuitry may reproduce an object (e.g., the motor vehicle object 104 of FIG. 1) for display using a modified level of detail in response to detecting eyelid movement. For example, the control circuitry may reproduce the object for display at a higher resolution in response to detecting a particular blink pattern. In some embodiments, the control circuitry may reference a table (e.g., table 316 of FIG. 3) storing an association between an eyelid movement identifier and an actionable action to determine whether the detected eyelid movement matches the eyelid movement identifier for modifying the level of detail of the object.

[0087] FIG. 11 is a flowchart of a detailed illustrative process for reproducing an object for display within an XR environment, according to some embodiments of the present disclosure. For example, process 1100 may be performed by the control circuitry of network device 800 (e.g., via control circuitry 804), and / or the control circuitry of server 900, and / or the control circuitry of devices 920, 925, 930, as commanded by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as to distribute control of an extended reality environment among a plurality of devices. Additionally, one or more steps of process 1100 may be incorporated into, or combined with, one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14, process 1500 of FIG. 15).

[0088] At 1102, an I / O circuitry (e.g., the I / O circuitry of the head-mounted display 202) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such a user selection may be received via an input (e.g., a voice command or a touch screen).

[0089] In 1104, a control circuit network (e.g., the control circuit network 804 of FIG. 8 and / or the control circuit network of the head-mounted display 202 of FIG. 2) may generate, in response to receiving a user selection, an XR environment including a plurality of objects (e.g., the vehicle object 204 and the airplane object 208 of FIG. 2) for display (e.g., via the head-mounted device 202 of FIG. 2).

[0090] In 1106, an I / O circuit network (e.g., the I / O circuit network of the head-mounted display 102) may receive an input from a sensor (e.g., a camera). The sensor may detect various attributes of the user's eyes (e.g., eyelid movement, fixation).

[0091] In 1108, a control circuit network (e.g., the control circuit network 804 of FIG. 8 and / or the control circuit network of the head-mounted display 202 of FIG. 1) may identify objects (e.g., the vehicle object 204 and the airplane object 208 of FIG. 2) within the user's field of view based on the received input. In some embodiments, the control circuit network may generate a data structure regarding the field of view that includes object identifiers associated with virtual objects within the field of view, and such a data structure may include coordinates representing the position of the field of view within the XR environment. The control circuit network may determine the current field of view and identify objects within the user's field of view based on the data structure and / or image captured by the XR device 202.

[0092] In 1110, the control circuit network may detect the user's eyelid movement (e.g., the amount of eyelid movement, blink pattern, eyelid level, etc.) based on the received input from a sensor (e.g., a camera). In some embodiments, such eyelid movement may be distinguished from involuntary blinks to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore a blink pattern where the eyes remain closed for less than a predetermined amount of time (e.g., 0.5 seconds).

[0093] At 1112, the control circuitry may determine whether the eyelid movement is associated with a first object (e.g., the vehicle object 204 of FIG. 2). In the embodiment of FIG. 2, the control circuitry may determine whether the detected eyelid movement matches an eyelid movement identifier "blink once to modify details of the vehicle" (e.g., stored in table 316 of FIG. 3).

[0094] At 1114, in response to determining that the detected eyelid movement matches an eyelid movement identifier for a first object (e.g., the vehicle object 204 of FIG. 2), the control circuitry may reproduce such an object for display using a modified level of detail. For example, the control circuitry may reproduce such an object for display at a higher resolution than that first presented, or at a larger size than that first presented, in response to detecting a particular blink pattern.

[0095] At 1116, if the control circuitry determines that the eyelid movement is not associated with a first object (e.g., the vehicle object 204 of FIG. 2), the control circuitry may determine whether the eyelid movement is associated with another object (e.g., the airplane object 208 of FIG. 2). In the embodiment of FIG. 2, the control circuitry may determine whether the detected eyelid movement matches an eyelid movement identifier "blink twice to modify details of the airplane" (e.g., stored in table 316 of FIG. 3).

[0096] In 1118, in response to determining that the detected eyelid movement matches an eyelid movement identifier for a second object (e.g., the airplane object 208 of FIG. 2), the control circuitry may reproduce such an object for display using a modified level of detail. For example, the control circuitry may reproduce the object for display at a higher resolution than that first presented, or at a larger size than that first presented, in response to detecting a particular blink pattern. Accordingly, the user may selectively shift their gaze or focus within the XR environment by actuating their eyelids.

[0097] FIG. 12 is a flowchart of a detailed illustrative process for performing an action on an object within an XR environment, according to some embodiments of the present disclosure. For example, process 1200 may be performed by the control circuitry of network device 800 (e.g., via control circuitry 804), and / or the control circuitry of server 900, and / or the control circuitry of devices 920, 925, 930, as commanded by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, for example, to distribute control of an extended reality environment among a plurality of devices. Additionally, one or more steps of process 1200 may be incorporated into, or combined with, one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1300 of FIG. 13, process 1400 of FIG. 14, process 1500 of FIG. 15).

[0098] At 1202, an I / O circuit network (e.g., the I / O circuit network of the head-mounted display 302) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such a user selection may be received via an input (e.g., a voice command or a touch screen).

[0099] At 1204, a control circuit network (e.g., the control circuit network 804 of FIG. 8 and / or the control circuit network of the head-mounted display 302 of FIG. 3) may generate, in response to receiving the user selection, an XR environment including an object (e.g., the book object 304 of FIG. 3) for display (e.g., via the head-mounted device 302 of FIG. 3). In some embodiments, the control circuit network may generate a plurality of objects (e.g., the book object 304, the lamp object 310, the desk 312, the chair 314 of FIG. 3).

[0100] At 1206, the control circuit network may store in a memory (e.g., the storage device 808 of FIG. 8 and / or the storage device of the head-mounted display 302 of FIG. 3) or access in the memory a table of eyelid movement identifiers and corresponding actions executable on an object (e.g., the table 316 of FIG. 3).

[0101] At 1208, the control circuit network may detect a user's eyelid movement (e.g., the amount of eyelid movement, the blink pattern, the eyelid level, etc.) based on an input received from a sensor (e.g., a camera). In some embodiments, such eyelid movement may be distinguished from involuntary blinks to avoid interpreting such involuntary blinks of the user as commands. For example, the system may ignore a blink pattern where the eyes remain closed for less than a predetermined amount of time (e.g., 0.5 seconds).

[0102] At 1210, the control circuitry determines whether the detected eyelid movement matches any of the identifiers within a table (e.g., table 316 of FIG. 3). In some embodiments, the control circuitry may analyze the sensor output and compare such output with stored identifiers (e.g., a predetermined number of blinks, blink pattern, amount of eyelid movement, eyelid level, etc.). In some embodiments, the stored identifiers may include eyelid movement in combination with a voice command or other input.

[0103] At 1212, the control circuitry determines whether the detected eyelid movement matches a stored eyelid movement identifier. In some embodiments, the control circuitry may calculate a match score and determine that there is a match between the detected eyelid movement and the stored eyelid movement identifier when the calculated match score exceeds a predetermined threshold.

[0104] In response to determining that the detected eyelid movement matches a stored eyelid movement identifier at 1214, the control circuitry causes an updated version of the XR environment to be presented for display based on an action corresponding to the matched eyelid movement. In the example of FIG. 3, the updated version of the XR environment 300 includes a book object 304 having a turned page (in the environment 300 illustrated in the lower portion of FIG. 3 as compared to the environment 300 illustrated in the upper portion of FIG. 3 where, e.g., the book object 304 is closed).

[0105] Figure 13 is a flowchart of a detailed illustrative process for receiving a voice command while an indicator is near an object within an XR environment, according to some embodiments of the present disclosure. For example, process 1300 may be commanded by one or more programs of computer-executable instructions that may be implemented on network device 800, and / or server 900, and / or devices 920, 925, 930, such as for distributing control of an extended reality environment among multiple devices. Process 1300 may be executed by the control circuitry of network device 800 (e.g., via control circuitry 804), and / or the control circuitry of server 900, and / or the control circuitry of devices 920, 925, 930. Additionally, one or more steps of process 1300 may be incorporated into, or combined with, one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1400 of FIG. 14, process 1500 of FIG. 15).

[0106] At 1302, an I / O circuitry (e.g., the I / O circuitry of the head-mounted display 402 of FIGS. 4A-4B) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such a user selection may be received via an input (e.g., a voice command or a touch screen).

[0107] At 1304, in response to receiving the user selection, a control circuitry (e.g., control circuitry 804 of FIG. 8 and / or the control circuitry of the head-mounted display 402 of FIG. 4A) may cause an XR environment including an object (e.g., object 408 of FIG. 4) to be displayed (e.g., via the head-mounted device 402 of FIG. 4). In some embodiments, the control circuitry may cause multiple objects (e.g., objects 408 and 410 of FIG. 4) to be generated.

[0108] In 1306, the control circuit network may detect the user's gaze. For example, a sensor (e.g., a camera) may be used to track the user's retinal movement, and such retinal movement of the user may be plotted on the display of the XR environment (e.g., the environment 400 in FIGS. 4A-4B).

[0109] In 1308, the control circuit network may determine whether the user's gaze has shifted to a second portion of the XR environment that includes an object (e.g., the object 410 in FIGS. 4A-4B). In some embodiments, the user's gaze may shift from a first portion of the display that does not include any objects. Alternatively, such a first portion of the display may include an object (e.g., the object 408 in FIGS. 4A-4B).

[0110] In 1310, in response to determining that the user's gaze has shifted to such a second portion of the XR environment that includes an object (e.g., the object 410 in FIGS. 4A-4B), the control circuit network may generate an indicator of the shift in the gaze (e.g., the indicator 406 in FIG. 4) for display. For example, such an indicator allows the user to confirm that the system is accurately tracking their gaze.

[0111] In 1312, in response to determining that the user's gaze has not shifted to such a second portion of the XR environment, the control circuit network may determine that the user's gaze has not shifted or has shifted to a different portion of the display, and may generate an indicator (e.g., the indicator 406 in FIG. 4) for display at the portion of the display where the user's gaze is directed.

[0112] At 1314, an I / O circuit network (e.g., the I / O circuit network of the head-mounted display 402 in FIGS. 4A-4B) may receive a voice command while an indicator (e.g., indicator 406 in FIG. 4) is near a target object (e.g., object 408 in FIG. 4A or object 410 in FIG. 4B). In some embodiments, a control circuit network (e.g., the control circuit network 804 in FIG. 8 and / or the control circuit network of the head-mounted display 402 in FIG. 4A) may determine whether a voice command is related to such a target object. In the example of FIG. 4A, the control circuit network extracts keywords from a voice command of "Play The Dark Knight", compares the extracted keywords with metadata associated with an object (e.g., object 408) to which the user's gaze is directed, and may determine whether the voice command is related to the target object.

[0113] At 1316, the control circuit network may execute an action corresponding to a voice command related to an object. For example, in the embodiment of FIG. 4B, an identifier for an object 410, i.e., a media content item "American Psycho", is near the indicator 406, and in response to receiving a voice command 405 ("Play American Psycho"), presentation of such a media asset to the user may be initiated.

[0114] FIG. 14 is a flowchart of a detailed illustrative process for varying the opacity of an indicator within an XR environment, according to some embodiments of the present disclosure. Note that process 1400 or any of its steps may be implemented on or provided by any of the devices shown in FIGS. 8-9. For example, process 1400 may be commanded by one or more programs of computer-executable instructions implemented on network device 800 (e.g., via control circuitry 804), and / or on control circuitry of server 900, and / or on control circuitry of devices 920, 925, 930, such as for distributing control of an extended reality environment among a plurality of devices. Additionally, one or more steps of process 1400 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1500 of FIG. 15).

[0115] At 1402, an I / O circuitry (e.g., the I / O circuitry of the head-mounted display 502 of FIG. 5) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such a user selection may be received via an input (e.g., a voice command or a touch screen).

[0116] At 1404, in response to receiving the user selection, a control circuitry (e.g., control circuitry 804 of FIG. 8 and / or control circuitry of the head-mounted display 502 of FIG. 5) may cause an XR environment, including an object (e.g., object 508 of FIG. 5), to be displayed (e.g., via the head-mounted device 502 of FIG. 5). In some embodiments, the control circuitry may cause a plurality of objects (e.g., objects 508 and 510 of FIG. 5) to be generated.

[0117] In 1406, the control circuit network may detect the user's gaze. For example, a sensor (e.g., a camera) may be used to track the user's retinal movement, and such retinal movement of the user may be plotted on the display of the XR environment (e.g., environment 500 in FIG. 5). The control circuit network may determine whether the user's gaze is directed towards a part of the XR environment that includes an object (e.g., object 508 in FIG. 5).

[0118] In 1410, in response to determining that the user's gaze is not directed towards a part of the XR environment that includes an object (e.g., object 508 in FIG. 5), the control circuit network may determine whether the user's gaze is directed towards a different part of the XR environment that includes a different object (e.g., object 510 in FIG. 5).

[0119] In 1410, in response to determining that the user's gaze is directed towards a part of the XR environment that includes an object (e.g., object 508 in FIG. 5), the control circuit network may generate an opacity-based indicator (e.g., indicator 506 in FIG. 6) for display within the XR environment in the vicinity of such a part of the XR environment. On the other hand, in response to determining that the user's gaze is directed towards a different part of the XR environment that includes a different object (e.g., object 510 in FIG. 5), the control circuit network may generate an opacity-based indicator (e.g., indicator 506 in FIG. 6) for display within the XR environment in the vicinity of such a different part of the XR environment.

[0120] In 1412, the control circuit network may determine the boundary of the object of interest (e.g., object 508 in FIG. 5). For example, the control circuit network may perform edge detection techniques and / or pixel comparison to determine the boundary of the object.

[0121] At 1414, the control circuitry varies the opacity of at least one opacity-based indicator (e.g., indicator 506 of FIG. 5) based on the identified boundaries of the object. In some embodiments, the opacity of indicators that overlap the boundaries of the object of interest is varied. For example, the opacity of an indicator that overlaps the boundaries of the object of interest may be decreased (FIG. 6) to avoid obscuring the object of interest, or increased (FIG. 5) to emphasize that the user's gaze is directed at the object of interest.

[0122] FIG. 15 is a flowchart of a detailed illustrative process for presenting additional information related to an item within an XR environment, according to some embodiments of the present disclosure. Note that process 1500 or any of its steps may be implemented on or provided by any of the devices shown in FIGS. 8-9. For example, process 1500 may be implemented by computer-executable instructions of one or more programs that may be implemented on network device 800 (e.g., via control circuitry 804), and / or control circuitry of server 900, and / or control circuitry of devices 920, 925, 930, such as for distributing control of an extended reality environment among multiple devices. Additionally, one or more steps of process 1500 may be incorporated into or combined with one or more steps of any other process or embodiment (e.g., process 1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14).

[0123] In 1502, an I / O circuit network (e.g., the I / O circuit network of the head-mounted display 702 in FIG. 7) may receive a user selection to enter an XR environment (e.g., a VR, AR, or MR environment). Such a user selection may be received via an input (e.g., a voice command or a touch screen).

[0124] In 1504, a control circuit network (e.g., the control circuit network 804 in FIG. 8 and / or the control circuit network of the head-mounted display 702 in FIG. 7) may, in response to receiving the user selection, generate an XR environment including an object (e.g., object 708 in FIG. 7) for display (e.g., via the head-mounted device 702 in FIG. 7). In some embodiments, the control circuit network may generate a plurality of objects (e.g., objects 702, 704, 708 in FIG. 7).

[0125] In 1506, a control circuit network (e.g., the control circuit network 804 in FIG. 8 and / or the control circuit network of the head-mounted display 702 in FIG. 7) may detect the user's eye movement (e.g., using a sensor).

[0126] In 1508, the control circuit network may determine whether an object has been within the user's field of view for at least a predetermined period. In some embodiments, the control circuit network may generate a data structure regarding the field of view that includes an object identifier associated with a virtual object within the field of view, and such a data structure may include coordinates representing the position of the field of view within the VR environment. The control circuit network may determine the current field of view and identify objects within the user's field of view based on the data structure and / or image captured by the head-mounted display 702. In one embodiment, the control circuit network may additionally detect whether the user's gaze is directed at an object within the field of view.

[0127] At 1510, in response to determining that an object has been within the field of view for at least a predetermined period (e.g., 5 seconds), the control circuitry causes one or more items associated with the object (e.g., item 714 in FIG. 7) to be generated for display within the XR environment. In some embodiments, when determining whether an object has been within the field of view for at least a predetermined period, the control circuitry may determine whether the object has remained within the user's field of view for a continuous (or discontinuous) period during the XR session.

[0128] The processes discussed above are intended to be illustrative, not limiting. Those skilled in the art will understand that the steps of the processes discussed herein may be omitted, modified, combined, and / or rearranged, and that any additional steps may be implemented without departing from the scope of the invention. More generally, the above disclosure is meant to be illustrative, not limiting. Only the following claims set the bounds as to what the invention encompasses. Further, note that the features and limitations described in any one embodiment may be applied to any other embodiment herein, and that a flowchart or example relating to one embodiment may be performed in a different order, combined with any other embodiment in a suitable manner, or performed in parallel. Additionally, the systems and methods described herein may be implemented in real time. Also note that the systems and / or methods described above may be applied to or used in accordance with other systems and / or methods. This specification discloses embodiments that include, but are not limited to, the following. 1. A method for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment comprising an object; Detecting, by using a first sensor, that a gaze has shifted from a first part of the extended reality environment to a second part of the extended reality environment, wherein an object is excluded from the first part of the extended reality environment and is included within the second part of the extended reality environment; In response to detecting the gaze shift, generating an indicator of the shift in the gaze for display within the extended reality environment; Detecting, by using a second sensor, an audio command while the indicator is near the object; In response to detecting the audio command, performing an action corresponding to the audio command; A method comprising the above steps. 2. The method according to item 1, wherein a two-way media guide is provided on a display, and the action is an instruction related to a media asset accessible via the two-way media guide. 3. The gaze is detected based on eye retinal movement, and the method further comprises: Converting the retinal movement into movement of an indicator on a display, according to the method described in item 1 or 2. 4. Determining whether a rate of the retinal movement exceeds a predetermined value; Normalizing the retinal movement when converting the retinal movement into movement of an indicator on a display in response to determining that the rate of the retinal movement exceeds the predetermined value, according to the method described in item 3. The method according to item 3, further comprising the above steps. 5. The method according to any one of items 2-4, wherein the audio command includes identification of a media asset and a command for performing an action. 6. The method according to any one of items 2-5, wherein the audio command includes an instruction for presenting a new media asset on a display. 7. The method according to any one of items 2-6, wherein the object is associated with an entity, and the audio command comprises an instruction for reading out content related to the entity. The method according to any one of items 1-7, wherein the step of detecting a voice command while the indicator is in the vicinity of the object includes determining that the gaze is directed at the object for at least a predetermined threshold period. The method according to any one of items 1-8, wherein the indicator being in the vicinity of the object includes the indicator overlapping the object. The method according to any one of items 1-9, wherein the display is presented via an extended reality head-mounted device. 11. A system for extended reality environment interaction, a display, generating an extended reality environment with an object for display, detecting, by using a first sensor, that the gaze is shifted from a first part of the extended reality environment to a second part of the extended reality environment, wherein the object is excluded from the first part of the extended reality environment and is included within the second part of the extended reality environment, in response to detecting the gaze shift, generating an indicator of the shift in the gaze for display within the extended reality environment, detecting, by using a second sensor, a voice command while the indicator is in the vicinity of the object, in response to detecting the voice command, performing an action corresponding to the voice command, a control circuitry configured as such, A system comprising. The system according to item 11, wherein a two-way media guide is provided on the display and the action is an instruction related to a media asset accessible via the two-way media guide. 13. The control circuitry further detects the gaze based on eye retinal movement, converts the retinal movement into movement of an indicator on the display. The system according to item 11 or 12, configured as such. 14. The control circuit network further determines whether the rate of retinal movement exceeds a predetermined value, and normalizes the retinal movement when converting the retinal movement into movement of an indicator on a display in response to determining that the rate of retinal movement exceeds a predetermined value. The system according to item 13, configured as such. 15. The voice command includes identification of a media asset and a command for executing an action. The system according to any one of items 12 - 14. 16. The voice command includes an instruction for presenting a new media asset on a display. The system according to any one of items 12 - 15. 17. The object is associated with an entity, and the voice command includes an instruction for reading out content related to the entity. The system according to any one of items 12 - 16. 18. The control circuit network is further configured to determine that the gaze is directed at the object for at least a predetermined threshold period when detecting a voice command while the indicator is in the vicinity of the object. The system according to any one of items 11 - 17. 19. The indicator being in the vicinity of the object includes the indicator overlapping the object. The system according to any one of items 11 - 18. 20. The display is presented via an extended reality head - mounted device. The system according to any one of items 11 - 19. 21. A method for extended reality environment interaction, comprising: generating an extended reality environment including a first object for display; receiving an input from one or more sensors; identifying a first object within the field of view based on the received input; detecting eyelid movement based on the received input; responding to detecting eyelid movement, regenerating a first object for display using a modified level of detail; A method comprising. 22. The extended reality environment comprises a plurality of objects, the plurality of objects including a first object and a second object, The identifying step further includes identifying a second object within the field of view, The step of regenerating the first object for display using a modified level of detail is performed in response to determining that the detected eyelid movement is associated with the first object. The method according to item 21. 23. The method according to item 22, further comprising, in response to determining that the detected eyelid movement is associated with the second object, regenerating the second object for display using a modified level of detail. 24. The method according to item 22 or 23, wherein the first object is in one of the foreground or background within the field of view in the extended reality environment, and the second object is in the other of the foreground or background within the field of view in the extended reality environment. 25. The method according to any one of items 21-24, wherein the step of regenerating the first object for display using a modified level of detail includes presenting the object at a higher resolution. 26. The method according to any one of items 21-25, further comprising performing one or more actions on the identified object based on one or more detected eyelid movements. 27. Further comprising calculating an individual virtual distance of a plurality of objects with respect to the user, The step of identifying a first object within the field of view includes determining that the first object is at a virtual distance closest to the user of the individual virtual distances. The method according to any one of items 22-26. 28. The method according to any one of items 21-27, wherein the step of detecting eyelid movement includes the step of determining the amount of eyelid movement. 29. The method according to any one of items 21-28, wherein the step of detecting eyelid movement includes the step of determining one or more eyelid levels. 30. A step of detecting that a user is navigating from a first position to a new position within an extended reality environment while a first object remains within the user's field of view, and a step of generating an updated version of the first object for display based on the user's viewpoint at the new position, The method according to any one of items 21-29, further comprising: 31. A system for extended reality environment interaction, configured to generate an extended reality environment for display, including a first object, a control circuitry network; an input / output (I / O) circuitry network configured to receive inputs from one or more sensors, and comprising: The control circuitry network is further configured to identify a first object within the field of view based on the received inputs, detect eyelid movement based on the received inputs, and in response to detecting eyelid movement, reproduce the first object for display using a modified level of detail, A system configured as such. 32. The extended reality environment comprises a plurality of objects, the plurality of objects including a first object and a second object, The control circuitry network is further configured to identify a second object within the field of view, In response to determining that the detected eyelid movement is associated with the first object, the control circuitry is further configured to perform a step of regenerating the first object for display using the modified level of detail. The system according to item 31. 33. The control circuitry is further The system according to item 32, configured to regenerate the second object for display using the modified level of detail in response to determining that the detected eyelid movement is associated with the second object. 34. The system according to item 31 or 33, wherein the first object is one of a foreground or a background within the field of view in the extended reality environment, and the second object is the other of the foreground or the background within the field of view in the extended reality environment. 35. The system according to any one of items 31 - 34, wherein the control circuitry is further configured to present the object at a higher resolution when regenerating the first object for display using the modified level of detail. 36. The control circuitry is further The system according to any one of items 31 - 35, configured to perform one or more actions on the identified object based on one or more detected eyelid movements. 37. The control circuitry is further calculate an individual virtual distance of a plurality of objects with respect to the user, and determine that the first object is at a virtual distance closest to the user of the individual virtual distances when identifying the first object within the field of view. The system according to any one of items 32 - 36, configured as such. 38. The system according to any one of items 31 - 37, wherein the control circuitry is further configured to determine the amount of movement of the eyelid when detecting the eyelid movement. 39. The control circuitry further determines one or more eyelid levels when detecting eyelid movement, the system according to any one of items 31-38 configured to do so. 40. The control circuitry further while a first object remains within the field of view, detects that the user is navigating from a first position to a new position within the extended reality environment, and based on the user's viewpoint at the new position, generates an updated version of the first object for display, the system according to any one of items 31-38 configured to do so. 41. A method for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment including an object; storing in memory a table of eyelid movement identifiers and corresponding actions executable on an object within the extended reality environment; detecting eyelid movement by using a sensor; matching the detected eyelid movement to one of the stored eyelid movement identifiers; in response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, generating an updated version of the extended reality environment for display based on the action corresponding to the matched eyelid movement; the method including the above steps. 42. The method according to item 41, further comprising selecting an object from a plurality of objects within the extended reality environment by detecting that the user's gaze is directed at the object. 43. The method according to item 42, further comprising generating for display a subset of eyelid movement identifiers executable on the object at which the user's gaze is directed. 44. The actions of the plurality of actions correspond to manipulating the object, the method according to any one of items 41-43. 45. An action is a method according to any one of items 41-44, corresponding to modifying the appearance of an object. 46. A method according to any one of items 41-45, associated with a user profile, that defines the relationship between an eyelid movement identifier executable on an object within an extended reality environment and a corresponding action. 47. While the user's gaze remains on an object, detecting that the user is navigating from a first position to a new position within the extended reality environment, and generating, for display, an updated version of the first object based on the user's perspective at the new position, wherein the updated version of the object has a modified appearance. A method according to items 45-46, further comprising. 48. An action executable on an object varies based on the type of the object. A method according to any one of items 41-47. 49. The step of detecting eyelid movement A method according to any one of items 41-48, including determining whether the eyelids remain closed for a predetermined period. 50. The step of matching the detected eyelid movement to one of the memorized eyelid movement identifiers is performed in response to determining that the eyelids remain closed for a predetermined period. A method according to item 49. 51. A system for extended reality environment interaction, configured to generate, for display, an extended reality environment including an object, a control network, configured to store a table of eyelid movement identifiers and corresponding actions executable on objects within the extended reality environment, a memory, comprising, the control network further By using a sensor, eyelid movement is detected, the detected eyelid movement is matched to one of the stored eyelid movement identifiers, in response to matching the detected eyelid movement to one of the stored eyelid movement identifiers, a updated version of the extended reality environment is caused to be displayed for viewing based on the action corresponding to the matched eyelid movement, A system configured as described above. 52. The control circuitry further, configured to select an object from a plurality of objects within the extended reality environment by detecting that the user's gaze is directed at the object, the system according to item 51. 53. The control circuitry further, configured to cause a subset of the eyelid movement identifiers that are applicable to the object at which the user's gaze is directed to be displayed for viewing, the system according to item 51 or 52. 54. The actions of the plurality of actions correspond to manipulating an object, the system according to any one of items 51-53. 55. The action corresponds to modifying the appearance of an object, the system according to any one of items 51-54. 56. The user is associated with a user profile that defines the relationship between the eyelid movement identifiers that are applicable to the objects within the extended reality environment and the corresponding actions, the system according to any one of items 51-55. 57. The control circuitry further, while the user's gaze remains on an object, detects that the user is navigating from a first position to a new position within the extended reality environment, based on the user's perspective at the new position, causes an updated version of the first object to be displayed for viewing, the updated version of the first object having a modified appearance, A system configured as described above, according to item 55 or 56. 58. The actions that can be performed on an object vary based on the type of the object, and the system described in any of items 51 - 57. 59. The control circuitry is further configured to, when detecting eyelid movement, determine whether the eyelid remains closed for a predetermined period, and the system described in any of items 51 - 58. 60. The control circuitry is further configured to perform a step of matching the detected eyelid movement to one of the stored eyelid movement identifiers in response to determining that the eyelid remains closed for a predetermined period, and the system described in item 59. 61. A method for extended reality environment interaction, comprising: generating an extended reality environment for display, the extended reality environment comprising an object; detecting, by using a sensor, that a gaze is directed to a first portion of the extended reality environment, wherein the object is included within the first portion of the extended reality environment; generating, for display within the extended reality environment, a plurality of opacity - based indicators in the vicinity of the first portion of the extended reality environment; identifying the boundary of the object; varying the opacity of at least one of the plurality of opacity - based indicators based on the identified boundary of the object; The method includes the above steps. 62. The method further includes determining whether at least one of the plurality of opacity - based indicators overlaps with the boundary of the object, and the step of varying the opacity of at least one of the plurality of opacity - based indicators based on the boundary of the object includes varying the individual opacity of at least one of the plurality of opacity - based indicators that overlap with the boundary, and the method described in item 61. 63. The method according to item 61 or 62, wherein the plurality of opacity-based indicators are arrows directed towards the object. 64. Detecting, by using a sensor, whether the gaze has shifted to a second part of the extended reality environment; In response to determining that the gaze has shifted to the second part, overlaying a plurality of opacity-based indicators in the vicinity of the second part of the display; The method according to any one of items 61-63, further comprising: 65. The method according to any one of items 61-64, wherein the individual opacities are varied based on the distance from the object. 66. The method according to item 65, wherein the individual opacity of the indicator increases as the distance between the individual indicator and the object decreases. 67. The method according to item 65 or 66, wherein the individual opacity of the indicator increases as the distance between the individual indicator and the object increases. 68. The method according to any one of items 61-67, wherein a bidirectional media guide is provided on the display, and actions related to media assets accessible via the bidirectional media guide are received at least in part based on the detected gaze. 69. The method according to any one of items 61-68, wherein the display is presented via an extended reality head-mounted device. 70. The method according to any one of items 61-69, wherein the display is presented without the use of an extended reality head-mounted device. 71. A system for extended reality environment interaction, comprising: A display; Generating an extended reality environment comprising an object for display; Detecting, by using a sensor, that the gaze is directed to a first part of the extended reality environment, wherein the object is included within the first part of the extended reality environment; In the vicinity of a first portion of the extended reality environment, generate a plurality of opacity-based indicators for display within the extended reality environment, Identify the boundaries of the object, Based on the identified boundaries of the object, vary the opacity of at least one of the plurality of opacity-based indicators, A control circuitry configured to, A system comprising. 72. The control circuitry further, Determine whether at least one of the plurality of opacity-based indicators overlaps the boundary of the object, Based on the boundary of the object, when varying the opacity of at least one of the plurality of opacity-based indicators, vary the individual opacity of at least one of the plurality of opacity-based indicators that overlap the boundary, The system according to item 71, configured to. 73. The plurality of opacity-based indicators are arrows directed towards the object, the system according to item 71 or 72. 74. The control circuitry further, By using a sensor, detect whether the gaze has shifted to a second portion of the extended reality environment, In response to determining that the gaze has shifted to the second portion, overlay the plurality of opacity-based indicators in the vicinity of the second portion of the display, The system according to any one of items 71-73, configured to. 75. The individual opacity is varied based on the distance from the object, the system according to any one of items 71-74. 76. The individual opacity of the indicator increases as the distance between the individual indicator and the object decreases, the system according to item 75. 77. The system according to item 75 or 76, wherein the individual opacity of the indicator increases as the distance between the individual indicator and the object increases. 78. The system according to any one of items 71 - 77, wherein a bidirectional media guide is provided in the display, and actions related to media assets accessible via the bidirectional media guide are received at least in part based on detected gaze. 79. The system according to any one of items 71 - 78, wherein the display is presented via an extended reality head - mounted device. 80. The system according to any one of items 71 - 79, wherein the display is presented without the use of an extended reality head - mounted device. 81. A method for extended reality environment interaction, comprising: generating an extended reality environment including an object for display; detecting eye movements; determining, based on the detection, whether the object is within the field of view for at least a predetermined period; in response to determining that the object is within the field of view for at least a predetermined period, generating one or more items related to the object for display within the extended reality environment; and the method includes the above steps. 82. The method according to item 81, wherein the one or more items related to the object comprise text information, images, videos, or any combination thereof. 83. Determining that at least a second predetermined period has elapsed since the start of the display of one or more items without the object being present within the field of view for a first predetermined period; and in response to determining that at least a second predetermined period has elapsed since the start of the display of one or more items without the object being present within the field of view, stopping the display of the one or more items. The method according to item 81 or 82, further comprising 84. The method according to any one of items 81-83, wherein the step of detecting eye movement includes a step of monitoring eyelid movement. 85. The step of generating one or more items related to an object for display in an extended reality environment is performed in response to determining that the object has been within the field of view and fixated for at least a predetermined period. The method according to any one of items 81-84. 86. The step of determining whether an object has been within the field of view for a predetermined period based on detection includes the step of determining that the field of view has been continuously over the object for a predetermined period during an extended reality session. The method according to any one of items 81-85. 87. The step of determining whether an object has been within the field of view for a predetermined period based on detection includes the step of determining that the field of view has been discontinuously over the object for a predetermined period during an extended reality session. The method according to any one of items 81-86. 88. Determining that a new object has been within the field of view for at least a predetermined time; In response to determining that a new object has been within the field of view for at least a predetermined period, generating one or more items related to the new object for display in an extended reality environment, while continuing to generate one or more items related to the object for display in the extended reality environment; The method according to any one of items 81-87, further comprising 89. The method according to any one of items 81-88, further comprising a step of enabling modification of the amount of time regarding a predetermined period. 90. The extended reality environment is presented via an extended reality head-mounted device. The method according to any one of items 81-89. 91. A system for extended reality environment interaction, comprising: a display, and generating an extended reality environment including an object for display, detecting eye movement, determining, based on the detection, whether the object has been within the field of view for at least a predetermined period, in response to determining that the object has been within the field of view for at least a predetermined period, generating one or more items related to the object for display within the extended reality environment, a control circuitry configured to; A system comprising. 92. The system of item 91, wherein the one or more items related to the object comprise text information, an image, a video, or any combination thereof. 93. The control circuitry further: determining that at least a second predetermined period has elapsed since the start of the display of the one or more items without the object being present within the field of view for a first predetermined period, in response to determining that at least a second predetermined period has elapsed since the start of the display of the one or more items without the object being present within the field of view, stopping the display of the one or more items, The system of item 91 or 92, configured to. 94. The system of any of items 91-93, wherein the control circuitry is further configured to monitor eyelid movement when detecting eye movement. 95. The system of any of items 91-94, wherein the control circuitry is further configured to generate one or more items related to the object for display within the extended reality environment in response to determining that the object has been within the field of view and fixation for at least a predetermined period. 96. The control circuit network is further configured to determine whether the object is within the field of view for a predetermined period, such that the field of view is continuously over the object for a predetermined period during an extended reality session, for the system according to any one of items 91-95. 97. The control circuit network is further configured to determine whether the object is within the field of view for a predetermined period, such that the field of view is discontinuously over the object for a predetermined period during an extended reality session, for the system according to any one of items 91-96. 98. The control circuit network is further configured to determine that a new object is within the field of view for at least a predetermined time, and in response to determining that a new object is within the field of view for at least a predetermined period, generate one or more items related to the new object for display within the extended reality environment, while continuing to generate one or more items related to the object for display within the extended reality environment, for the system according to any one of items 91-97. 99. The control circuit network is further configured to enable modification of the amount of time related to a predetermined period, for the system according to any one of items 91-98. 100. The extended reality environment is presented via an extended reality head-mounted device, for the system according to any one of items 91-99.

Claims

1. A method for extended reality environment interaction, comprising: generating an extended reality environment including objects for display; detecting, by using a first sensor, that a gaze has shifted from a first portion of the extended reality environment to a second portion of the extended reality environment, wherein the object is excluded from the first portion of the extended reality environment and is included within the second portion of the extended reality environment; in response to detecting the shift of the gaze, generating an indicator of the shift in the gaze for display within the extended reality environment; detecting, by using a second sensor, a voice command while the indicator is in the vicinity of the object, wherein the voice command includes information indicating that the gaze is directed to a portion of the extended reality environment different from the second portion of the extended reality environment associated with the indicator of the shift in the gaze; and in response to detecting the voice command, performing an action corresponding to the voice command. A method comprising the above steps.

2. The method according to claim 1, wherein a two-way media guide is provided on the display, and the action is an instruction related to a media asset accessible via the two-way media guide.

3. A method for extended reality environment interaction, comprising: generating an extended reality environment including a first object having an initial level of detail for display; receiving input from one or more sensors; identifying the first object within the field of view based on the received input; detecting a first instance of eyelid movement associated with the first object based on the received input; and in response to detecting the first instance of eyelid movement associated with the first object, regenerating the first object having a modified level of detail different from the initial level of detail for display. Further, while the first object having the modified level of detail is being regenerated for display based on the further received input, detecting a second instance of the eyelid movement associated with the first object; In response to detecting the second instance of the eyelid movement associated with the first object, returning to generating the first object having the initial level of detail for display; A method comprising. **Claim 4** A method for extended reality environment interaction, comprising: Generating an extended reality environment including an object for display; Storing in memory a table of a plurality of eyelid movement identifiers and a plurality of corresponding actions executable on the object within the extended reality environment; In a particular portion of the extended reality environment and based on the table stored in the memory, (1) a particular eyelid movement corresponding to a particular eyelid movement identifier among the plurality of eyelid movement identifiers, and (2) a particular action to be performed on the object if the particular eyelid movement corresponding to the particular eyelid movement identifier is detected, and (3) the object on which the particular action is to be performed if the particular eyelid movement corresponding to the particular eyelid movement identifier is detected Generating an explicit description for display; Detecting eyelid movement by using a sensor; Matching the detected eyelid movement to the particular eyelid movement identifier among the plurality of stored eyelid movement identifiers; In response to matching the detected eyelid movement to the particular eyelid movement identifier among the plurality of stored eyelid movement identifiers, generating an updated version of the extended reality environment for display by performing the particular action corresponding to the particular eyelid movement on the object A method comprising. **Claim 5** A method for extended reality environment interaction, comprising: Generating an extended reality environment including an object for display; Detecting, by using a sensor, that a gaze is directed at a first portion of the extended reality environment, wherein an object is included within the first portion of the extended reality environment, Generating, within the extended reality environment for display, a plurality of opacity-based indicators in the vicinity of the first portion of the extended reality environment, Identifying a boundary of the object, Varying an opacity of at least one of the plurality of opacity-based indicators based on the identified boundary of the object A method comprising. **Claim 6** A method for extended reality environment interaction, comprising: Generating an extended reality environment including an object for display, Detecting eye movement of a user, Based on the detecting, determining whether the object is within the user's field of view for at least a first predetermined period, In response to determining that the object is within the user's field of view for at least the first predetermined period, generating, for display within the extended reality environment, one or more items associated with the object, Determining that at least a second predetermined period has elapsed since starting to display the one or more items associated with the object while the object is not within the user's field of view, In response to determining that at least the second predetermined period has elapsed since starting to display the one or more items associated with the object while the object is not within the user's field of view, stopping the display of the one or more items associated with the object A method comprising. **Claim 7** A computer-implemented method, comprising: Generating an extended reality environment including an object for display during an extended reality session, During the extended reality session, determining whether the object is within the field of view for a first period, wherein the first period is determined using a timer and the first period is less than a threshold period, During the extended reality session, determining that the object is no longer within the field of view; Based on determining that the object is no longer within the field of view, pausing the timer; During the extended reality session, determining that the object is again within the field of view; Based on determining that the object is again within the field of view, resuming the use of the timer; Determining that the object was within the field of view for a second period after resuming the use of the timer; In response to determining that the combination of the first period and the second period during the extended reality session is at least the threshold period, generating one or more items associated with the object for display within the extended reality environment A computer-implemented method comprising.

8. A computer-implemented method for extended reality environment interaction, comprising: Determining whether the user's gaze is directed at a first object within the extended reality environment for at least a specific period; In response to determining that the user's gaze is directed at the first object for at least the specific period, generating one or more items associated with the first object for display within the extended reality environment; Determining that the user's gaze has shifted from the first object to a second object within the extended reality environment; While continuing to generate one or more items associated with the first object for display within the extended reality environment in response to determining that the user's gaze remains directed at the second object for at least the specific period without shifting back to the first object, generating one or more items associated with the second object for display within the extended reality environment A computer-implemented method comprising.

9. A system for extended reality environment interaction, the system comprising a control circuitry and a non-transitory computer-readable medium having instructions encoded thereon, the instructions, when executed by the control circuitry, causing the control circuitry to perform the method according to any one of claims 1 to 8. **Claim 10** A non-transitory computer-readable medium having instructions encoded thereon, the instructions, when executed by a control circuitry, causing the control circuitry to perform the method according to any one of claims 1 to 8.

Images