Improving interaction accuracy of gaze-enabled AR objects while moving

Abstract

Methods and systems for providing augmented reality coverage associated with real-world objects include detecting a gaze target of a user viewing the real-world environment using a pair of AR glasses by tracking the user's gaze. Location parameters that affect the gaze of the user are tracked, and one or more attributes of the gaze target are selectively corrected to allow the user to maintain their gaze on the gaze target. An AR trigger element associated with the gaze target is triggered in response to the gaze by the user. The AR trigger element provides additional information related to the gaze target selected by the user.

Description

Technical Field

[0001] The present disclosure relates to detecting gaze-enabled augmented reality world objects, and more particularly to improving the accuracy of detecting gaze-enabled augmented reality world objects while in motion. Background Art

[0002] When a user wears an augmented reality (AR) device (such as smart glasses), some objects in the real world will be augmented with additional information that can only be viewed in AR. To avoid cognitive overload, the additional information can be hidden until the user interacts with the AR object in some way. One way for a user to interact with an AR object is by looking at the AR object, and the AR device is equipped with gaze tracking capabilities to detect the user's gaze interaction. The AR object may be equipped with an AR trigger element associated with the AR display. In response to detecting a user gaze on the AR object, the AR trigger element may be triggered to render the relevant AR display.

[0003] Often, simply looking at the real-world object is not enough. For example, if a quick glance triggers an AR display, this may cause visual confusion and sensory overload. Further, if both the person and the real-world object are stationary, it is easy to identify an appropriate real-world object as the target of the user's gaze from among multiple real-world objects within the real-world environment. If one or both of the person and the real-world object are in motion, it may be difficult or impossible for the gaze to be fixed on the target object (e.g., the real-world object) for a sufficient amount of time to trigger the AR display.

[0004] The present invention is produced in this context. Summary of the invention

[0005] Implementations of the present disclosure relate to systems and methods for detecting user interaction with an object by tracking the user's gaze and triggering an augmented reality (AR) overlay associated with the object. Triggering of the AR overlay causes information associated with the object to be rendered for viewing through a pair of AR glasses. Gaze tracking requires that the gaze remain fixed on a specific point (e.g., a real-world object or gaze target) for a specific period of time in order to trigger the AR overlay.

[0006] However, many positional parameters may affect the difficulty of maintaining the user's gaze on the gaze target long enough to trigger the AR overlay. Some of the factors that may affect the user's gaze from being able to focus on the gaze target correspond to various positional parameters related to the user and the gaze target, including the user's movement when the gaze target is stationary, the movement of the gaze target when the user is stationary, the movement of both the gaze target and the user, the user's movement speed, direction, the movement speed, direction of the gaze target, changes in the aforementioned positional parameters, etc.

[0007] In order to reliably identify the gaze target, the relevant position parameters that affect the user's gaze are identified and the size of the gaze target is adjusted proportionally. The adjustment of the gaze target allows the user's gaze to better focus on the gaze target. In various implementations, the adjustment of the gaze target can be performed in two or three dimensions. Therefore, the adjustment of the gaze target is based on the gaze target being stationary when the user moves. In addition or alternatively, when it is determined that the gaze target moves relative to the user, the gaze target is adjusted to adapt to the movement of the gaze target. Further, when it is determined that the gaze target and the user are both moving, and the gaze target moves in a different direction from the user, the gaze target is then adjusted to take into account the direction of movement of the user and the gaze target. By solving the position parameters that affect the user's gaze, the user can maintain their gaze on the gaze target long enough to trigger the AR overlay. Further, by considering these position parameters, the AR overlay is maintained long enough to allow the user to process the information included in the AR overlay in his mind.

[0008] A method for providing an augmented reality (AR) overlay is disclosed. The method includes detecting a gaze target of a user viewing a real-world environment through a pair of AR glasses. The gaze target is a real-world object in the real-world environment, and the gaze target is identified by tracking the gaze of the user. The gaze target is associated with an AR trigger element. One or more position parameters that affect the gaze of the user are tracked based on the relative position of the user relative to the gaze target. One or more properties of the gaze target are selectively corrected in proportion to the one or more position parameters to allow the user to maintain their gaze on the real-world object representing the gaze target. When the gaze of the user directed to the gaze target exceeds a predefined time threshold, the AR trigger element associated with the gaze target is activated. The activation of the AR trigger element causes the AR overlay to be rendered in association with an image of the real-world object at a display portion of the AR glasses when the user views the real-world environment. The AR overlay provides information related to the real-world object representing the gaze target.

[0009] In another implementation, a method for providing an augmented reality overlay is disclosed. The method includes tracking a user's gaze to identify a gaze target that has aroused the user's interest. The gaze target identifies a real-world object in a real-world environment viewed by the user through a pair of AR glasses. The gaze target is associated with an AR trigger element. Movement of the user relative to the gaze target is detected. The movement of the user is analyzed to identify one or more position parameters of the user. One or more attributes of the gaze target are selectively corrected in proportion to the position parameters to allow the user to maintain the gaze on the real-world object representing the gaze target. When the gaze of the user on the gaze target exceeds a predefined time threshold, the AR trigger element associated with the gaze target is activated. The activation causes the AR overlay to be rendered in association with an image of the real-world object at a display portion of the pair of AR glasses when the user views the real-world environment. The AR overlay provides information related to the real-world object.

[0010] Other aspects of the present disclosure will become apparent from the following detailed description read in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present disclosure may be best understood by reference to the following description taken in conjunction with the accompanying drawings.

[0012] Figure 1 A simplified block diagram of a system for providing augmented reality (AR) overlays for real-world objects viewed through a pair of AR glasses according to one implementation of the present disclosure is provided.

[0013] Figure 2 Various modules of an AR object detection engine for rendering an AR overlay of a real-world object are illustrated according to one implementation of the present disclosure.

[0014] Figure 3A A simple example of correcting the size of a fixation target in response to a gait of a user moving relative to the fixation target according to one implementation of the present disclosure is illustrated.

[0015] Figure 3B A simple example of correcting the size of a fixation target in response to a user's motion relative to the fixation target is illustrated according to one implementation of the present disclosure.

[0016] Figure 3C A simple example of correcting the size of a fixation target in response to the gait and motion of a user moving relative to the fixation target is illustrated in accordance with one implementation of the present disclosure.

[0017] Figure 4AAn example of a gaze target extracted from a real-world environment with an AR overlay is illustrated according to one implementation of the present disclosure.

[0018] Figure 4B An example of an extracted gaze target with an AR overlay that remains in view when the view of the real-world environment changes is illustrated in accordance with one implementation of the present disclosure.

[0019] Figure 5A An example of a gaze target extracted from a real-world environment with an AR overlay in the presence of an occluding object approaching is illustrated according to one implementation of the present disclosure.

[0020] Figure 5B An example of a gaze target extracted from a real-world environment with an AR overlay in a situation where an occluding object occludes a real-world object according to one implementation of the present disclosure is illustrated.

[0021] Figure 6 A simplified operational flow of a method for verifying a digital twin of a real-world object owned by a user according to one implementation of the present disclosure is illustrated.

[0022] Figure 7 Components of an example computing device that may be used to perform aspects of various implementations of the present disclosure are illustrated. DETAILED DESCRIPTION

[0023] In the following description, a number of specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that the present disclosure may be practiced without some or all of these specific details. In other cases, well-known process steps are not described in detail in order to avoid making the present disclosure unclear.

[0024] As humans increasingly live online and in virtual worlds, people are increasingly interested in devices that allow users to have a satisfying interactive experience. Some devices, such as head-mounted displays (HMDs), provide users with a truly immersive experience by completely blocking the real world environment, allowing users to interact in the virtual world without interference. Alternatively, users can use augmented reality (AR) devices to interact with the virtual world. AR devices allow users to interact with virtual objects while providing users with the ability to see through and experience the real world environment around them. Therefore, when a user wears an AR device (such as smart glasses, etc.), the user is able to view real-world objects and additional information that can only be viewed in AR, where the additional information is related to one or more of the real-world objects.

[0025] To avoid cognitive overload, the additional information is typically hidden and activated only when the user interacts with the relevant real-world object. The real-world object may be equipped with an AR trigger element, such as a placard with a QR code, and when the user scans (i.e., activates) the QR code, additional information related to the real-world object is rendered. To avoid visual clutter when the user views the real-world scene through an AR device (e.g., a pair of AR glasses), the AR trigger element is activated to render the additional information after determining that the user's interest in the real-world object lasts for a defined period of time rather than instantaneously. The user's attention can be determined by tracking user interactions with the real-world object (e.g., user gaze) and ensuring that the user interaction at the real-world object lasts for at least a certain period of time.

[0026] For example, a gaze interaction with a real-world object requires that the gaze remain fixed on the real-world object and continue for a specific period of time to trigger an AR overlay. When both the real-world object and the user are in a stagnant state (i.e., stationary), it is easy to track the gaze interaction. However, if either or both of the real-world object and the user are in motion, many position parameters (such as type of motion, speed of motion, direction of motion, etc.) may make it difficult or impossible for the gaze to remain fixed on the real-world object and continue for a specific period of time to trigger the AR display. In order to alleviate the problems caused by the movement of the user and / or the real-world object, the position parameters that affect the user's gaze on the real-world object (i.e., the gaze target) are identified, and one or more attributes of the gaze target are adjusted in proportion to the one or more position parameters, so that the user can maintain their gaze on the gaze target for a certain time without changing their movement.

[0027] The speed and / or direction of the user's movement when the user is in a moving vehicle (such as riding a car or riding a bicycle) can make it difficult for the user to maintain their gaze on the real-world object for a long time, so that the real-world object cannot be regarded as the gaze target. The faster the movement, the more difficult it is to detect that the user's gaze is fixed on the gaze target for a long enough time to accurately establish the user's interest in the gaze target in order to trigger the corresponding AR overlay. In addition to the user's movement speed and direction, other properties related to the movement need to be considered in order to identify and adjust one or more position parameters of the gaze target. For example, when the user walks, their gaze will naturally jump slightly based on their gait. Slower gaits have minimal jumps and allow users to maintain their gaze on the gaze target for a longer time, resulting in more accurate gaze detection, while faster gaits (i.e., brisk walking, jogging, running, sprinting) have larger jumps and will result in less accurate gaze detection. Therefore, in addition to speed and direction, gait-related properties must also be considered.

[0028] In order to accurately identify a fixation target on which the user's gaze is maintained for a defined period of time, an AR object detection engine executed on a computing system is used to track the user's motion, identify a fixation target based on the user's gaze, and adjust one or more dimensions of the identified fixation target in proportion to one or more attributes of the user's motion. For example, the user's gait (i.e., the user's motion) may be tracked, and the fixation target may be adjusted in proportion to the magnitude of the motion caused by the user's gait. Instead of or in addition to the user's motion, the fixation target itself may be in motion, and the AR object detection engine tracks the motion of the fixation target relative to the user and adjusts one or more attributes of the fixation target to account for the motion of the fixation target.

[0029] Adjustment of one or more properties enables the user to maintain their gaze on the gaze target long enough to trigger an AR overlay associated with the gaze target. The AR overlay can be rendered long enough for the user to view the information. In some cases, an image of the gaze target is also captured and rendered in association with the AR overlay information, and such data can be rendered even after the user passes by the gaze target.

[0030] With the present disclosure generally understood, specific implementations will be described with reference to the various figures.

[0031] Figure 1 Implementations of a system 10 for identifying a gaze target and providing an overlay of information associated with the gaze target on a display portion of a pair of smart glasses 102 are illustrated. In some implementations, a pair of smart glasses 102 is configured as an augmented reality (AR) device, wherein a user is able to see through and view a real-world environment near the user and augmented reality (AR) content associated with real-world objects. A view of the real-world environment is seen via a real-world viewing portion 102a, and AR content is viewed through an AR display portion 102b of the smart glasses 102. Because the smart glasses 102 are configured for viewing AR content, the smart glasses 102 are also referred to herein as AR glasses 102.

[0032] like Figure 1 As shown, a user 100 is shown as physically present in a real-world space 108 and viewing a real-world environment through a pair of AR glasses 102. The real-world environment includes a plurality of real-world objects and the user 100 is shown as stationary. Figure 1In the illustrated implementation, a pair of AR glasses 102 is wirelessly connected to a cloud computing system 106 via a network 104. In other implementations, a pair of AR glasses 102 may be connected to the cloud computing system 106 via a computing device (not shown) located in a real-world space 108 in which the user operates. The connection between the pair of AR glasses 102 and the computing device may be via a wired or wireless connection. The user 100 may interact with an interactive application or provide input related to a real-world object via a user interface provided at the AR glasses 102 or via a controller (not shown) communicatively connected to the AR glasses 102 and / or the cloud computing system 106. For example, the input provided by the user may be transmitted to the cloud computing system 106 via the network 104 and used to affect the state of the interactive application. Alternatively, the input related to the real-world object provided by the user may be processed by the cloud computing system 106 to identify additional information associated with the real-world object and return it to the pair of AR glasses 102 for rendering. Depending on how the pair of AR glasses 102 are configured, the AR object detection engine 122 may be executed on a processor of the pair of AR glasses 102, or on a computing device coupled to the pair of AR glasses 102, or on a server of the cloud computing system 106. The AR object detection engine 122 is used to process user input to confirm the selection of a real-world object and identify additional information related to the real-world object.

[0033] In some implementations, the user may move and the real-world objects in the real-world environment may be stationary. In alternative implementations, one or more of the real-world objects in the real-world environment may move, while the user may be stationary. In yet other implementations, both the user and one or more of the real-world objects may move. In all such implementations, the view of the real-world environment changes continuously as different real-world objects enter the field of view of the user 100. Some of the real-world objects in the real-world environment may be associated with AR trigger elements, which may be used to initiate an augmented display (i.e., an AR display). An example of an AR trigger element associated with a real-world object may be a placard or label with a quick response (QR) code. The AR object detection engine 122 is configured to track real-world objects equipped with AR trigger elements, tracking the gaze of the user 100 as the user moves in the real-world environment, so as to identify real-world objects that attract the attention of the user 100 (i.e., gaze targets with associated trigger elements). The user's attention on the real-world object may be determined by tracking the user's gaze and determining that the user's gaze on the real-world object persists for at least a defined period of time to establish the real-world object as the user's gaze target. However, due to the movement of the user and / or the real-world object, it may be difficult or impossible for the user's gaze to remain fixed on the real-world object for a period of time long enough to establish the user's interest in the real-world object as the gaze target and trigger an AR display associated with the real-world object. Various positional parameters associated with the user and / or the gaze target (e.g., type of motion, speed of motion, direction of travel, etc.) due to the movement of the user and / or the real-world object may affect whether the user's gaze remains fixed on a particular point.

[0034] The AR object detection engine 122 considers positional parameters associated with the movement of the user and / or the real-world object and proportionally adjusts one or more dimensions of the real-world object on which the user's gaze is detected to determine whether the user's gaze continues on the real-world object for a defined period of time (e.g., 5 seconds, 3 seconds, etc.) in order to establish the real-world object as the user's gaze target. Once the gaze target is determined, the size of the gaze target is adjusted according to the positional parameters that affect the user's gaze in order to allow the user to maintain their gaze on the gaze target long enough to instantiate the AR display associated with the gaze target. Figures 2 to 5B The details of tracking user gaze and adjusting the size of the gaze target are described in detail. Once the user's gaze target in the real world environment is established, the AR trigger element associated with the gaze target is activated. The activated AR trigger element provides information associated with the gaze target for the user to use.

[0035] Figure 2Various modules of the AR object detection engine 122 are illustrated. In one implementation, the AR object detection engine is used to establish a user's gaze target in the real world environment and render an AR display of the gaze target on a pair of AR glasses for the user to use. The AR object detection engine 122 includes multiple modules. The AR object detection engine 122 can be a hardware component or a software component. In some implementations, some of the modules of the AR object detection engine 122 can be implemented as hardware components of an application-specific integrated circuit (ASIC). Broadly speaking, the AR object detection engine 122 is used to identify positional parameters that affect the user's gaze in the real world environment, attributes associated with the user or the gaze target, proportionally adjust the size of the gaze target to assist the user in maintaining their gaze on the gaze target, and activate AR trigger elements associated with the gaze target. Once the gaze target is identified, the AR object detection engine 122 can also predict when the second object will occlude the gaze target based on an analysis of the movement of the second object in the real world environment, identify attributes related to the movement of the second object, and determine the duration of AR coverage that must be maintained due to the second object occluding the gaze target.

[0036] The AR object detection engine 122 first identifies the user wearing a pair of AR glasses 102 by retrieving user data (121a) and verifying it with the captured user identification information of the user 100 wearing a pair of AR glasses 102. The capture of identification information and verification of the user can be performed using any known standard or proprietary process or any new process available at the AR object detection engine 122. Upon successful verification, the AR object detection engine 122 starts (engages) the user attribute recognition engine 123 to track the user to determine whether the user is stationary or moving around in the real world environment. The user is first tracked by mapping the real world environment near the user. The mapping can be two-dimensional or three-dimensional. Using the mapping of the real world environment, attributes including the position of the user and real world objects in the real world environment are determined. The attribute recognition engine 123 uses the position of the user relative to the real world object to determine whether the user is moving and the real world object is stationary, or whether the real world object is moving and the user is stationary, or whether the real world object and the user are moving relative to each other.

[0037] Once the user is successfully authenticated and the attributes of the user and the real-world object are determined, in one implementation, a gaze detection engine 121b is initiated to determine the user's gaze. The gaze detection engine 121b initiates an inward-facing camera disposed on a pair of AR glasses 102 and / or other image capture devices coupled to the pair of AR glasses 102 to capture an image of the user's eyes and uses the data captured by the image capture device / camera to determine the user's gaze direction. The gaze direction is then used to identify real-world objects in the real-world environment in the direction of the user's gaze. When the user's gaze on the identified real-world object lasts for at least a defined period of time, the identified real-world object is established as the user's gaze target. Once the user's gaze target is established, the attributes of the user and the real-world object are analyzed to identify position parameters (including motion characteristics) of the user and the real-world object. The position parameters of the user and the real-world object are used to determine the user's movement and / or the movement of one or more real-world objects, so that adjustments can be made to the gaze target to keep the user's gaze fixed on the gaze target.

[0038] The user's movement is tracked using a mapping of the real-world environment and various data captured by multiple sensors and devices. For example, the mapping and images captured by sensors (e.g., externally facing cameras and other image capture devices) can be analyzed to determine whether the user is moving around or stationary. Similarly, data captured by an inertial measurement unit sensor (IMU, including gyroscopes, magnetometers, accelerometers, GPS, etc.) provided in a pair of AR glasses 102 can be used to determine whether the user 100 wearing a pair of AR glasses 102 is moving or stationary. When the user attribute recognition engine 123 detects that the user is moving in the real-world environment, the data captured from the various sensors / devices is used to determine various position parameters (related to gait and movement) that may affect the user's gaze due to the user's movement in the real-world environment.

[0039] As previously noted, the user's gait while walking causes their gaze to jump. Slower gaits have fewer jumps and therefore can result in more accurate gazes, while faster gaits resulting from brisk walking, jogging, running, sprinting have larger jumps and therefore can result in less accurate gazes. When a user walks and sets their gaze on a gaze target (i.e., a real-world object that supports a triggering element) within a real-world environment, the user must slow down their gait in order to continue to maintain their gaze on the gaze target. In order to avoid the user from getting into an unsafe situation when continuing to maintain their gaze at the gaze target while moving (e.g., due to slowing down their gait when the user is in the middle of the street), the user attribute recognition engine 123 tracks the user's gait and identifies characteristics of the gait (123a). Using data captured from various sensors / devices associated with a pair of AR glasses 102, some of the characteristics of the gait identified from the user attributes include the intensity of the gait (e.g., slow or fast), the amplitude of the gait (i.e., the level of jumping), the length of the stride, etc.

[0040] Similar to walking, when the user is in motion (i.e., riding), depending on the speed of the motion, it may be difficult for the user to continue to fix their gaze on the gaze target long enough to trigger the associated AR overlay. The user may be driving or riding a vehicle (e.g., a car, a bicycle, a train, etc.) or some other type of moving surface, such as a moving sidewalk, an escalator, etc. As with gait, the faster the user's motion, the more difficult it is for the user to continue to keep the user's gaze on the gaze target. In order to enable the user to fix their gaze on the gaze target, the characteristics of the user's motion are identified from the user's attributes. The characteristics of the motion are examined and used to adjust the gaze target to allow the user to continue their gaze on the gaze target. Motion is a vector and has at least two related characteristics-speed and direction. Data captured by the sensor and / or device is analyzed to determine the speed and direction of the user's motion. In some implementations, the characteristics of the user's motion in the real world environment are considered independently of the gait (123b) to correct the gaze target. In other implementations, the characteristics of the user's motion are considered in conjunction with the attributes of the gait to correct the gaze target (123c). Data from the analysis of user movement is provided as input to the gaze target correction engine 125 for further processing.

[0041] Instead of or in addition to tracking the user's movement and / or gait, the AR object detection engine 122 is also configured to track real-world objects within the user's real-world environment to determine a stationary selected real-world object among the real-world objects and a moving selected other real-world object among the real-world objects. In some implementations, only the real-world objects associated with the AR trigger element are tracked. In other implementations, all real-world objects in the real-world environment are tracked. When the real-world object moves around in the real-world environment, the real-world environment and therefore the view of the real-world environment change in relation to the movement of the real-world object. The gaze target attribute recognition engine 124 is activated by the AR object detection engine 122 to identify the real-world object in motion based on the attributes of the real-world object captured using multiple devices and sensors. When the real-world object is the user's gaze target, the position parameters of the real-world object (i.e., the gaze target) and other motion characteristics that affect the user's gaze are evaluated. In one implementation, the evaluation includes determining the speed of movement (124a), the direction of movement (124b), and other characteristics (124c), such as the type of gaze target, the size of the gaze target, the physical characteristics, etc. These characteristics and position parameters are provided to the gaze target correction engine 125 for further processing.

[0042] The gaze target correction engine 125 analyzes data related to the user's movement and / or the movement of the gaze target to determine the amount of correction required for the gaze target in order to allow the user to maintain their gaze on the gaze target. In one implementation, the size of the gaze target is adjusted 125a in proportion to the user's gait. For example, the size of the gaze target may be adjusted in proportion to the amplitude of the user's gait. As previously noted, as the jump in the gait increases due to a change in the user's gait (i.e., when the user switches from walking to jogging to running), the size and proportion of the gaze target is changed to match the amplitude of the user's gait.

[0043] Figure 3A Some examples of using the amplitude of a user's gait to proportionally adjust the size of a fixation target in one implementation are illustrated. Figure 3A The amplitude of the user's gait is shown when the user is walking, jogging, and running. As can be seen, the amplitude of the user's gait when jogging is significantly increased compared to the amplitude of the user's gait when walking. Similarly, the amplitude of the user's gait when running is significantly greater than the amplitude when the user is jogging or walking. The size and scale of the fixation target are changed to match the corresponding amplitude determined from the user's gait when walking, jogging, or running. The increase in the size and scale of the fixation target makes it easier for the user to maintain their fixation on the fixation target without affecting their gait. Figure 3AThe tracking and adjustment of the gaze target in two dimensions (2D) is illustrated. The above implementation can be extended to accommodate tracking and adjustment in three dimensions (3D). The gaze target correction engine 125 is configured to track not only the up and down movement of the user, but also the forward and backward and left and right movement. Therefore, Figure 3A The 2D circular fixation target shown may instead be represented by a 3D sphere, and Figure 3A The distortion shown as being proportional to the user's gait in (x, y) space (ie, 2D space) can also be represented by a distortion proportional to the user's gait in (x, y, z) space (ie, 3D space).

[0044] Re-reference Figure 2 , similar to gait, the size and scale of the gaze target can be dynamically adjusted to accommodate motion characteristics identified from the movement of the user and / or the gaze target in the real world environment. Just like gait, speed and direction (i.e., motion characteristics) can be tracked in 2D or 3D, and the size and scale of the gaze target can be adjusted accordingly in 2D or 3D (125b).

[0045] Figure 3B One such example is illustrated, where the motion characteristics of a user are tracked and the corresponding size and scale of a gaze target is distorted or altered. Figure 3B The examples in show the motion characteristics of a user riding a bicycle, riding in a car, and riding in a train. Figure 3B In the illustration of, the size of the rectangle corresponds to the speed of the user when riding a bicycle, riding in a car and riding in a train. As shown in the figure, the speed when the user rides a bicycle is less than the speed when the user drives a car, and the speed when driving a car is shown as less than the speed when the user rides a train. As with gait, the size and proportion of the fixation target are changed to adapt to the speed and direction of the user's movement. Tracking movement in 2D, and the fixation target is shown as changing in 2D, but can be extended to tracking movement in 3D, and the fixation target is also shown as changing in 3D. In some implementations, the direction of the user's movement determines the direction in which the fixation target is changed. Instead of or in addition to tracking the movement of the user and dynamically changing the size and size of the fixation target, the movement of the fixation target can also be tracked and the size and size of the fixation target can be changed proportionally to match the motion characteristics of the fixation target.

[0046] In some implementations, based on the analysis of the properties of the gaze target and the user, it can be determined that the user and the gaze target are both in motion. Further, this can be determined from the motion characteristics of the gaze target and the user moving in the same direction. In such implementations, the user's motion speed is compared with the motion speed of the gaze target. If the gaze target moves with the user at the same speed, the size and size (i.e., property) of the gaze target is not changed. This is because the user will be able to keep their gaze fixed on the gaze target during the motion of the user and the gaze target. On the other hand, if the user's direction of motion is opposite to the direction of motion of the gaze target and / or the user's speed of motion is different from the speed of motion of the gaze target, the size and size of the gaze target are changed proportionally to adapt to the speed and / or direction difference of the user and the target. For example, if the direction of motion of the gaze target is opposite to the direction of motion of the user, but the speed is the same, the gaze target will appear to go "faster", and the direction and speed of the gaze target need to be taken into consideration when correcting the size and size of the gaze target.

[0047] In some implementations, the movement of the user and / or the gaze target and the user's gait may all be considered when correcting the size and dimensions of the gaze target (125c). In these implementations, changes to the gaze target take into account the user, the speed and direction of movement of the gaze target, and the user's gait attributes.

[0048] Figure 3C An example of such an implementation is illustrated, where a combination of the user's motion characteristics (i.e., speed and direction) and gait are considered when proportionally adjusting the size and dimensions of the fixation target. The user's movements are shown as being associated with walking, jogging, and running. The size and dimensions of the fixation target are varied by simultaneously considering all three inputs (i.e., speed, direction, and gait), and the adjustments increase as the user moves from walking to jogging to running. Although Figure 3C The examples shown are shown in 2D, but the examples can also be extended to 3D.

[0049] Once the gaze target is positively identified based on the user's gaze, the detailed information of the gaze target is provided to the trigger activation engine 126. The gaze target is associated with an AR trigger element, which renders an AR overlay when triggered. The trigger activation engine 126 tracks the amount of time the user's gaze is on the gaze target, and activates the AR trigger element associated with the gaze target when it is determined that the amount of time on the gaze target has lasted for at least a predefined period of time. The activated AR trigger element provides additional information related to the gaze target for rendering as an AR overlay on the AR display portion 102a of a pair of AR glasses 102. In some implementations, when the user's gaze on the gaze target is detected, the AR overlay is rendered together with a snapshot of the gaze target captured by the image capture device. The AR overlay is rendered in association with the snapshot.

[0050] In some implementations, the AR overlay will be visible for a long enough time for the user to mentally process the information included in the AR overlay. In some implementations, the AR overlay is rendered as long as the user's gaze remains on the gaze target, and when the gaze is interrupted, the AR overlay is removed. In some implementations, the length of time that the AR overlay is rendered as an overlay may not be sufficient for the user to process the information included in the AR display. This may be due to the motion of the gaze target and / or the user. For example, if the user is driving a car, the user may see a billboard on the side of the street with a gaze target that supports an AR overlay. The user's gaze on the motion-adjusted gaze target triggers the AR overlay. However, if the car passes the billboard before the user finishes reading the information included in the AR overlay, the user's gaze will be interrupted as the car passes the gaze target (i.e., the billboard) and the AR overlay will disappear with the gaze target. In order to allow the user to have enough time to process the information in the AR overlay even after the user's gaze on the gaze target is interrupted, in some implementations, the AR object detection engine 122 provides a "sticky" function (126a).

[0051] The user may select the sticky function 126a to indicate that the user wants the AR overlay to remain in place even after the user's gaze on the gaze target is interrupted. The sticky function (126a) may be provided on the user interface for user selection. In some implementations, the user interface may be a gaze-based interface that can be operated by a user blinking or winking. In other implementations, the user interface may be operated by voice commands or gestures or touch or user input. When the user selects to stick the AR overlay, the AR overlay will remain in view for a certain period of time after the gaze target is no longer visible. The time period may be specified by the user or may be a pre-set period. In some implementations, in response to detecting that the user initiated the sticky function, a "snapshot" of the gaze target is captured, and the captured snapshot of the gaze target is rendered in association with the stuck AR overlay to provide appropriate content for the AR overlay. In one implementation, the user may be looking at a jacket in a store window while riding by. When it is detected that the user's gaze on the jacket has lasted for at least a predefined period of time, the jacket is considered to be the user's gaze target. When passing by the store window, the user may activate the sticky function. In response to the user activating the sticky functionality, the trigger activation engine maintains the AR overlay in the user's view for a defined period of time after the gaze target has passed, and upon expiration of the defined period, removes the AR overlay from the display portion of the pair of AR glasses 102. Additionally, the trigger activation engine 126 signals an image capture device associated with the pair of AR glasses 102 to take a photo of the AR trigger-enabled jacket and include the snapshot of the jacket with the AR overlay. The defined period of time for maintaining the AR overlay in the user's field of view may be defined based on the amount of information included in the AR overlay, wherein an AR overlay with less information is maintained for a shorter period of time, and an AR overlay with more information is maintained for a longer period of time.

[0052] Figure 4A and Figure 4B An example implementation of the "sticky" function in operation in one implementation is shown. A user can see a leather jacket in a store window with an AR overlay of the price. Upon detecting a user gaze on the leather jacket for more than a predefined period of time, the AR overlay is activated. Figure 4A One such example is illustrated, where a leather jacket displayed in a store window is shown with a price AR overlay being activated. Once the gaze target (i.e., the leather jacket in the store window) is identified, when the pinning function is activated due to the user passing by the store window, a snapshot of the leather jacket in the store window is captured, along with the price AR overlay. For example, the captured image of the leather jacket with the price AR overlay is maintained in the user's field of view for a predefined period of time. Figure 4BA snapshot of a leather jacket with a price AR overlay is shown that remains in the user's field of view even after the user has passed a store window and the view of the real-world environment viewed by the user has changed to a landscape including trees.

[0053] Re-reference Figure 2 In some implementations, the user may select a "delayed view" option (126b) from the user interface to replace the pinned functionality. The delayed view option may be useful to prevent the pinned AR overlay from obscuring a portion of the display through which the user is viewing the real world environment for a longer period of time. When the delayed view option (126b) is selected, a snapshot of the gaze target captured along with the associated AR overlay is stored in a local database or a remote database or cache for later retrieval. The user may later recall the stored view of the gaze target with the AR overlay when it is safe for the user to view the gaze target with the AR overlay.

[0054] In some implementations, the user may select a “rewind” option (126c) from the user interface to allow the user to process past a gaze target with an associated AR overlay. For example, a rewind option may be provided to allow the user to “rewind” to a previously triggered target. For example, a user driving a car may pass by a billboard and engage a gaze-triggered AR overlay associated with a gaze target. The user may forget to activate the pinned function or the delayed view function before the gaze target is no longer in view. In order to provide the user with the option to view these gaze targets, the trigger activation engine 126 may automatically store a history of a certain number of the last gaze-triggered AR overlays for later recall. In some implementations, the gaze-triggered AR overlays are for gaze targets that the user previously expressed interest in. In other implementations, the gaze-triggered AR overlays stored in the history may be for real-world objects that the user passed by. In such implementations, the user's gaze may not have been on any single real-world object that passed by, because multiple real-world objects with associated AR overlays may all be focused on a single portion of the real-world scene that the user is passing by. In such scenarios, images of real-world objects are captured and provided to the user to see if the user is interested in any of the real-world objects with associated AR overlays. For example, the user may be driving and passing by multiple real-world objects with associated AR overlays. In order to allow the user to sort through multiple real-world objects, the user may select "rewind" to view previously seen AR overlays, even if the gaze target is not in the field of view. In some implementations, along with the AR overlay, a snapshot of the corresponding gaze target may also be stored so that when the user rewinds and views a certain AR overlay, an image of the gaze target is also presented to provide content to the AR overlay. The identified AR overlay is forwarded to the display portion of a pair of AR glasses for use by the user. In some implementations, the real-world object captured in the image may be a real-world object associated with an AR trigger element. In some specific implementations, when the rewind option is selected, the image of the real-world object captured and provided to the user is for an object in which the user has not previously shown any interest.

[0055] In some implementations, a situation may arise in which a gaze target triggered via the user's gaze is occluded. This may occur due to the presence of some kind of occlusion in which a second object appears or passes between the user and the gaze target. In some implementations, the occlusion may occur on a static second object, such as a tree, a billboard, a parked car, etc. In an alternative implementation, the occlusion may occur on a moving second object, such as another car, a cart, a pedestrian, etc. To address these obstacles, a "grace period" is built into the viewing of the AR overlay, which is activated in response to the user's gaze being occluded. The grace period allows the AR overlay to continue to exist for an additional amount of time after the user's gaze on the gaze target is interrupted (i.e., the gaze target moves out of the user's field of view due to the occlusion). The AR object detection engine 122 can use one or more externally facing cameras associated with a pair of AR glasses 102 to capture images of real-world objects, and use object recognition based on machine learning (ML) algorithms to build situational awareness of the state of different real-world objects in the real-world environment and the occlusion state. Context awareness is the identification of stationary and moving real-world objects in order to determine the cause of occlusion of the gaze target.

[0056] Using this information, the AR object detection engine 122 can identify which of the real-world objects (stationary or moving) will appear between the user and the gaze target during the gaze-based interaction. The stationary object can block the gaze target from the user's field of view based on the movement of the gaze target and / or the user. Similarly, the moving object can block the gaze target from the user's field of view based on the movement of the moving object, the user, and / or the gaze target. The AR object detection engine 122 can initiate an ML algorithm to create an artificial intelligence (AI) module 120 and train the AI ​​module by tracking various position parameters that affect the user's gaze (i.e., position parameters related to the movement of the user, the gaze target, and one or more real-world objects in stationary or moving real-world objects) and other characteristics of the user, the gaze target, and the real-world objects, the other characteristics being obtained by tracking and analyzing the user's motion and gait data and the motion data of the gaze target and / or the real-world object.

[0057] The ML algorithm 140 uses motion and gait data to identify from real-world objects a second object that is approaching a gaze target and will appear between the user and the gaze target during a gaze-based interaction, and predictively determines a grace period for viewing an AR overlay associated with the gaze target during a period when the second object should occlude the gaze target. For example, the ML algorithm 140 uses properties related to the motion of the second object to determine when the second object approaching the gaze target is likely to occlude the gaze target, the duration of such occlusion, and the like. Based on the analyzed data, the ML algorithm 140 activates a grace period, where the grace period is identified as matching the duration that the gaze target will be occluded by the second object. Activating the grace period causes the AR display associated with the gaze target to be maintained for use by the user.

[0058] For example, the AR overlay is triggered in response to detecting that the user selects a gaze target. During the activation of the AR overlay, the ML algorithm 140 embedded in the object detection algorithm analyzes data related to various real-world objects of the real-world environment in which the user exists, and recognizes that the bus is approaching and will eventually appear between the gaze target and the user. Responsively, the ML algorithm 140 starts the occluding object attribute recognition engine 127 to determine / calculate the size of the bus (i.e., OO size 127c), direction of travel, speed, etc. Using this data, the ML algorithm 140 determines the duration of the user's gaze being blocked by the bus, and activates a grace period matching the duration. Grace period activation allows the AR overlay to be maintained for gaze targets that support gaze. In some implementations, the AR overlay is superimposed on the bus during the time when the bus blocks the gaze target. In some implementations, the superposition is performed in a manner that the AR overlay moves relative to the movement of the bus. Once the bus passes (i.e., no longer blocks the gaze target) and the user has reestablished gaze contact with the gaze target, the AR overlay is removed from the rendering on the bus. In some implementations, the AR overlay may continue to be rendered next to or relative to the gaze target after the bus no longer obscures the gaze target. This may occur when the time period for rendering the AR overlay has not expired, or when the amount of information included in the AR overlay is large and the grace period is insufficient for the user to use the information included in the AR overlay. In some implementations, the time period for rendering the AR overlay may be user-specific. In some implementations, a snapshot of the gaze target is also rendered on the bus in association with the AR overlay.

[0059] FIG. 5A to FIG. 5B An example is illustrated in some implementations where an AR overlay is rendered on an occluding object. Figure 5AThe user's gaze target is shown to be a jacket in a store window, and an AR price overlay is rendered in association with the gaze target in response to detecting that the leather jacket is the target of the user's gaze. In addition, the ML algorithm 140 detects that a bus (i.e., an occluding object) is approaching and may occlude the gaze target by appearing between the user and the leather jacket in the store window. In response to detecting the approaching bus, the occlusion duration is calculated by identifying at least the size, shape, speed, and direction of the bus. The occlusion duration is used to define a grace period during which the gaze target and associated AR overlay need to be maintained for the user to view. Therefore, as the bus passes between the user and the gaze target, a snapshot of the leather jacket and the AR price overlay are rendered on the side of the bus. Figure 5B As an example, once the bus moves away and the gaze target is visible, the AR overlay is rendered in association with the gaze target, such as Figure 5A shown.

[0060] Various implementations define ways to adjust the size of real-world objects to allow users to maintain their gaze without the user having to change their gait or movement. Changing the size of the real-world object allows the real-world object to be accurately identified as a gaze target, so that the size of the gaze target can be corrected according to the motion characteristics of the user and / or the gaze target. Correction for position parameters such as gait and movement allows the user to maintain their gaze on the target long enough to trigger the AR overlay. Various options, such as a top-up function, a delayed viewing function, a rewind function, occlusion considerations, etc., allow the user to use the information included in the AR overlay during or when it is convenient and safe for the user, making the options a flexible and useful application. After reading the various implementations of the present disclosure, other advantages may occur to those skilled in the art.

[0061] Figure 6The process operation of a method for providing an augmented reality (AR) overlay associated with a real-world object in some implementations is illustrated. The method begins when a gaze target of a user viewing a real-world environment through a pair of AR glasses is detected, as shown in operation 610. The gaze target is a real-world object that the user is viewing among the real-world objects included in the real-world environment and is identified by tracking the user's gaze. When the user's gaze is directed to the real-world object for at least a predefined period of time, the real-world object is established as the user's gaze target. The real-world object that is the user's gaze target is associated with an AR trigger element. Once the user's gaze target is established, one or more position parameters of the user are tracked, as shown in operation 620. The user's gaze may be affected by position parameters and other motion characteristics, such as the user's gait or movement and / or the movement of the gaze target relative to the user. Various position parameters and other motion characteristics are used to determine the gait amplitude, speed, and direction associated with the movement of the user and / or the gaze target.

[0062] One or more attributes of the gaze target are selectively corrected using position parameters and other characteristics, as shown in operation 630, so as to allow the user to continue to maintain their gaze on the real-world object representing the gaze target. One or more attributes are corrected to be proportional to one or more identified position parameters, characteristics. For example, the speed and direction of the user's movement relative to the gaze target are used to proportionally adjust the size of the gaze target so that the gaze target can be easily identified and verified, where the verification can be in the form of the user continuing to maintain their gaze on the real-world object. The user's gaze on the gaze target is used to trigger an AR overlay associated with the real-world object, as shown in operation 640. The triggered AR overlay is rendered on a display portion of a pair of AR glasses worn by the user. The AR overlay provides additional information related to the real-world object representing the gaze target.

[0063] Figure 7Components of an example computing device 700 that can be used to perform aspects of various embodiments of the present disclosure are illustrated. This block diagram illustrates a computing device 700 that can be combined with or can be a personal computer, video game console, personal digital assistant, head-mounted display (HMD), wearable computing device, laptop or desktop computing device, server, or any other digital device suitable for practicing embodiments of the present disclosure. For example, in various implementations discussed herein, the device 700 represents a first device as well as a second device. The device 700 includes a central processing unit (CPU) 702 for running software applications and optionally an operating system. Further, the CPU 702 may include a machine learning algorithm 140 in which an object recognition algorithm (not shown) is embedded to identify real-world objects using physical attributes captured for real-world objects. The CPU 702 may be composed of one or more homogeneous or heterogeneous processing cores. For example, the CPU 702 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments may be implemented using one or more CPUs having a microprocessor architecture that is particularly well suited for highly parallel and computationally intensive applications such as processing operations to interpret queries, identify context-related resources, and immediately implement and render the context-related resources in a video game. Device 700 may be local to a player playing a game segment (e.g., a game console), or remote from the player (e.g., a backend server processor), or one of many servers in a gaming cloud system that uses virtualization to remotely stream game play to client devices.

[0064] Memory 704 stores applications and data for use by CPU 702. Storage 706 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other optical storage devices, as well as signal transmission and storage media. User input device 708 transmits user input from one or more users to device 700, examples of which may include keyboards, mice, joysticks, touch pads, touch screens, static or video recorders / cameras, tracking devices for identifying gestures, and / or microphones. Network interface 714 allows device 700 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communications through local area networks and wide area networks (such as the Internet). Audio processor 712 is suitable for generating analog or digital audio output from instructions and / or data provided by CPU 702, memory 704 and / or storage 706. The components of device 700 , including CPU 702 , memory 704 , data storage 706 , user input device 708 , network interface 714 , and audio processor 712 , are connected via one or more data buses 722 .

[0065] The graphics subsystem 720 is further connected to the data bus 722 and the components of the device 700. The graphics subsystem 720 includes a graphics processing unit (GPU) 716 and a graphics memory 718. The graphics memory 718 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory 718 may be integrated with the GPU 716 in the same device, connected to the GPU 716 as a separate device, and / or implemented in the memory 704. The pixel data may be provided directly from the CPU 702 to the graphics memory 718. Alternatively, the CPU 702 provides the GPU 716 with data and / or instructions defining the desired output image, and the GPU 716 generates pixel data for one or more output images according to the data and / or instructions. The data and / or instructions defining the desired output image may be stored in the memory 704 and / or the graphics memory 718. In one embodiment, the GPU 716 includes a 3D rendering capability to generate pixel data for an output image from instructions and data defining the geometry, lighting, shadows, textures, motion, and / or camera parameters of the scene. GPU 716 may further include one or more programmable execution units capable of executing shader programs.

[0066] Graphics subsystem 720 periodically outputs pixel data for an image to be displayed on display device 710 from graphics memory 718. Display device 710 may be any device capable of displaying visual information in response to a signal from device 700, including CRT, LCD, plasma, and OLED displays. In addition to display device 710, the pixel data may also be projected onto a projection surface. For example, device 700 may provide an analog or digital signal to display device 710.

[0067] It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) typically use cloud computing. Cloud computing is a computing method in which dynamically scalable and usually virtualized resources are provided as services over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing services typically provide common applications (such as video games) accessed from a web browser online, while software and data are stored on servers in the cloud. Based on the way the Internet is depicted in computer network diagrams, the term cloud is used as a metaphor for the Internet and is an abstraction of the complex infrastructure that it hides.

[0068] In some embodiments, a game server can be used to perform the operation of a duration information platform for video game players. Most video games played over the Internet are operated by connection to a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, video games can be performed by a distributed game engine. In these embodiments, a distributed game engine can be performed on multiple processing entities (PEs) so that each PE performs a functional fragment of a given game engine on which a video game runs. The game engine simply regards each processing entity as a computing node. The game engine typically performs a series of operations with diverse functions to perform video game applications and additional services for user experience. For example, a game engine implements game logic, performs game calculations, physical effects, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communications, game play / playback functions, help functions, etc. Although a game engine can sometimes be performed on an operating system virtualized by a hypervisor of a specific server, in other embodiments, the game engine itself is distributed between multiple processing entities, each of which can reside on different server units in a data center.

[0069] According to this embodiment, the corresponding processing entity used to perform the operation can be a server unit, a virtual machine, or a container, depending on the needs of each game engine fragment. For example, if a game engine fragment is responsible for camera transformations, a virtual machine associated with a graphics processing unit (GPU) can be provided to this particular game engine fragment because it will perform a large number of relatively simple mathematical operations (e.g., matrix transformations). Other game engine fragments that require fewer but more complex operations can be provided with processing entities associated with one or more higher-powered central processing units (CPUs).

[0070] By distributing the game engine, the game engine is provided with elastic computing properties that are not constrained by the capabilities of the physical server units. Instead, more or fewer computing nodes are provided to the game engine as needed to meet the needs of the video game. From the perspective of the video game and the video game player, a game engine distributed across multiple computing nodes is no different than a non-distributed game engine executed on a single processing entity, because the game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output components to the end user.

[0071] The user accesses the remote service using a client device, which includes at least a CPU, a display, and an I / O. The client device can be a PC, a mobile phone, a notebook computer, a PDA, etc. In one embodiment, the network executed on the game server recognizes the type of device used by the client and adjusts the communication method used. In other cases, the client device uses a standard communication method such as HTML to access the application on the game server through the Internet.

[0072] It should be understood that a given video game or gaming application may be developed for a specific platform and a specific associated controller device. However, when such a game is made available via a gaming cloud system as presented herein, a user may access the video game with different controller devices. For example, a game may have been developed for a gaming console and its associated controller, while a user may access a cloud-based version of the game from a personal computer using a keyboard and mouse. In such scenarios, an input parameter configuration may define a mapping of inputs generated from a user's available controller devices (i.e., a keyboard and mouse in this case) to acceptable inputs for execution of the video game.

[0073] In another example, a user may access the cloud gaming system via a tablet computing device, a touch screen smart phone, or other touch screen driven device. In this case, the client device and the controller device are integrally formed in the same device, wherein input is provided by means of a detected touch screen input / gesture. For such devices, the input parameter configuration may define a specific touch screen input corresponding to the game input of the video game. For example, during the running of a video game, a button, a directional pad, or other type of input element may be displayed or overlaid to indicate a position on the touch screen where the user can touch to generate a game input. Gestures (such as swipes or specific touch movements in a particular direction) may also be detected as game inputs. In one embodiment, guidance may be provided to the user, indicating, for example, how to provide input for gameplay via a touch screen before starting to play a video game, so as to adapt the user to the operation of the controls on the touch screen.

[0074] In some embodiments, the client device is used as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device can process these inputs in turn, and then transmit the input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller itself can be a networked device with the ability to directly transmit input to the cloud game server via a network without first transmitting such input through a client device. For example, the controller can be connected to a local networking device (such as the above-mentioned router) to send data to and receive data from the cloud game server. Therefore, although the client device may still be required to receive video output from a cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send input directly to the game cloud server over the network, thereby bypassing the client device.

[0075] In one embodiment, the networked controller and client device may be configured to send certain types of input directly from the controller to the cloud gaming server, and to send other types of input via the client device. For example, inputs whose detection does not rely on any additional hardware or processing other than the controller itself can be sent directly from the controller to the cloud gaming server via the network, thereby bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or require processing by the client device may be sent by the client device to the cloud gaming server. These may include video or audio captured from the gaming environment, which may be processed by the client device before being sent to the cloud gaming server. In addition, inputs from the motion detection hardware of the controller may be processed by the client device in conjunction with the captured video to detect the position and motion of the controller, which is then transmitted to the cloud gaming server by the client device. It should be understood that the controller device according to various embodiments may also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0076] In one embodiment, various technical examples can be implemented using a virtual environment via a head mounted display (HMD). The HMD may also be referred to as a virtual reality (VR) head mounted device. As used herein, the term "virtual reality" (VR) generally refers to the interaction of a user with a virtual space / environment, which involves viewing the virtual space in a real-time response to the movement of the HMD (controlled by the user) through the HMD (or VR head mounted device) to provide the user with a feeling in a virtual space or metaverse. For example, a user can see a three-dimensional (3D) view of the virtual space when facing a given direction, and when the user turns to one side and thus also turns the HMD, the view of the side in the virtual space is then rendered on the HMD. The HMD can be worn in a manner similar to glasses, goggles or a helmet, and is configured to display video games or other metaverse content to the user. The HMD can provide a very immersive experience to the user by means of its display mechanism being close to the user's eyes. Therefore, the HMD can provide a display area that occupies most of the user's or even the entire field of view to each eye of the user, and can also provide viewing with three-dimensional depth and perspective.

[0077] In one embodiment, the HMD may include a gaze tracking camera configured to capture images of the user's eyes as the user interacts with the VR scene. The gaze information captured by the gaze tracking camera may include information related to the user's gaze direction and specific virtual objects and content items in the VR scene that the user is paying attention to or interested in interacting with. Thus, based on the user's gaze direction, the system can detect specific virtual objects and content items that may be of potential attention to the user (where the user is interested in interacting with and engaging with them), such as game characters, game objects, game props, etc.

[0078] In some embodiments, the HMD may include an outward-facing camera configured to capture images of the user's real-world space, such as the user's body movements and any real-world objects that may be located in the real-world space. In some embodiments, the images captured by the outward-facing camera may be analyzed to determine the position / orientation of the real-world object relative to the HMD. Using the HMD, the known position / orientation of the real-world object, and the inertial sensor data from the inertial motion unit (IMU) sensor, the user's posture and movement may be continuously monitored and tracked during the user interaction with the VR scene. For example, when interacting with a scene in a game, a user may make various gestures, such as pointing and walking toward a specific content item in the scene. In one embodiment, the system may track and process gestures to generate predictions of interactions with specific content items in the game scene. In some embodiments, machine learning may be used to facilitate or assist the predictions.

[0079] During the use of the HMD, various single-hand and two-hand controllers can be used. In some implementations, the controller itself can be tracked by tracking the lights included in the controller or tracking the shapes, sensors and inertial data associated with the controller. Using these different types of controllers or even just using gestures made and captured by one or more cameras, the virtual reality environment or metaverse rendered on the HMD can be docked, controlled, manipulated, interacted with and participated in the virtual reality environment or metaverse. In some cases, the HMD can be wirelessly connected to the cloud computing and gaming system through a network. In one embodiment, the cloud computing and gaming system maintain and execute the video game played by the user. In some embodiments, the cloud computing and gaming system are configured to receive input from the HMD and interface objects through the network. The cloud computing and gaming system are configured to process input to affect the game state of the video game being executed. Output (such as video data, audio data and tactile feedback data) from the video game being executed is transmitted to the HMD and the interface object. In other implementations, the HMD can communicate wirelessly with the cloud computing and gaming system through alternative mechanisms or channels such as cellular networks.

[0080] In addition, although the implementations in the present disclosure may be described with reference to a head-mounted display, it should be understood that in other implementations, a non-head-mounted display may be substituted, including but not limited to a portable device screen (e.g., a tablet, a smart phone, a laptop computer, etc.) or any other type of display that can be configured to render video and / or provide a display of an interactive scene or virtual environment according to the present implementation. It should be understood that the various embodiments defined herein may be combined or assembled into a specific implementation using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that can define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0081] As noted, embodiments of the present disclosure for communicating between computing devices may be practiced using a variety of computer device configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, head-mounted displays, wearable computing devices, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked over wired or wireless networks.

[0082] In some embodiments, wireless technology can be used to facilitate communication. Such technology may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which the service area covered by the provider is divided into small geographic areas called cells. Analog signals representing sound and images are digitized in the phone, converted by an analog-to-digital converter, and transmitted as a bit stream. Through the channels allocated by the transceiver from the frequency pool reused in other cells, all 5G wireless devices in the cell communicate with the local antenna array and low-power automatic transceiver (transmitter and receiver) in the cell through radio waves. The local antenna is connected to the telephone network and the Internet via a high-bandwidth optical fiber or wireless backhaul connection. As in other cell networks, a mobile device that crosses from one cell to another automatically goes to the new cell. It should be understood that the 5G network is only an exemplary type of communication network, and the embodiments of the present disclosure can utilize an earlier generation of wireless or wired communications, as well as a newer generation of wired or wireless technology after 5G.

[0083] In view of the above embodiments, it should be understood that the present disclosure may employ various computer-implemented operations involving data stored in a computer system. These operations are those operations that require physical manipulation of physical quantities. Any operation described herein that forms a part of the present disclosure is a useful machine operation. The present disclosure also relates to a device or apparatus for performing these operations. The apparatus may be specially constructed for the desired purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used together with a computer program written according to the teachings of this article, or it may be more convenient to construct a more specialized apparatus to perform the desired operation.

[0084] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed in a system that allows the processing operations to occur at various time intervals related to the processing, so long as the processing of telemetry and game state data used to generate the modified game state is performed in the desired manner.

[0085] One or more embodiments may also be manufactured as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium may include a computer readable tangible medium distributed on a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0086] Although the foregoing embodiments have been described in some detail for the purpose of clarity of understanding, it is apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments should be considered as illustrative rather than restrictive, and the embodiments are not limited to the details given herein, but may be modified within the scope and equivalent scope of the appended claims.

[0087] It should be understood that the various features disclosed herein may be used to combine or assemble the various embodiments defined herein into a specific implementation. Therefore, the examples provided are only some possible examples and are not limited to various implementations that can define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

Claims

1. A method for providing an augmented reality (AR) overlay, comprising: Detecting a gaze target of a user viewing a real-world environment through a pair of AR glasses, the gaze target being a real-world object of the real-world environment identified by tracking the gaze of the user, the gaze target being associated with an AR trigger element; tracking one or more position parameters of the user, the one or more position parameters being related to a change in position of the user relative to the gaze target; selectively correcting one or more properties of the gaze target in proportion to the one or more position parameters to allow the user to maintain the gaze on the real-world object representing the gaze target; as well as When the user's gaze towards the gaze target exceeds a predefined time threshold, the AR trigger element associated with the gaze target is activated, the activation causing the AR overlay to be rendered in association with an image of the real-world object at a display portion of the pair of AR glasses when the user views the real-world environment, the AR overlay providing information related to the real-world object.

2. The method of claim 1, wherein tracking the one or more position parameters comprises: A gait of the user is tracked as the user moves relative to the gaze target, the gait of the user being used to dynamically adjust one or more dimensions of the gaze target in proportion to a magnitude of the gait of the user.

3. The method of claim 1 , wherein tracking the one or more position parameters comprises: Tracking the user's movement relative to the gaze target, the movement including speed and direction, the movement of the user being used to dynamically adjust one or more dimensions of the gaze target in proportion to the speed, the direction, or the speed and direction of the user's movement.

4. The method of claim 1, wherein tracking the one or more position parameters comprises: The user's gait and movement relative to the gaze target are tracked, and the user's movement and gait are used to dynamically adjust one or more dimensions of the gaze target in proportion to the user's gait, speed and direction of movement relative to the gaze target.

5. The method of claim 1 , wherein tracking the one or more position parameters comprises: Tracking the movement of the gaze target relative to the user, the movement of the gaze target including a target speed and a target direction, the movement of the gaze target being used to dynamically adjust one or more dimensions of the gaze target in proportion to the target speed, the target direction, or the target speed and target direction identified from the movement of the gaze target.

6. The method of claim 1, wherein selectively correcting the one or more attributes comprises: One or more sizes and proportions of the gaze target are dynamically adjusted based on the one or more position parameters, and the dynamic adjustment is completed without affecting the one or more position parameters to allow the user to maintain the gaze on the gaze target.

7. The method of claim 1, wherein selectively correcting the one or more attributes comprises: The two-dimensional property or the three-dimensional property of the gaze target is selectively adjusted to account for the one or more positional parameters affecting the gaze of the user.

8. The method of claim 7, wherein the gaze of the user is affected by the movement of the gaze target, the movement of the gaze target including a target speed and a target direction, and Wherein selectively correcting the one or more properties of the fixation target further comprises: One or more properties of the gaze target are selectively adjusted to account for the target speed and the target direction associated with the movement of the gaze target.

9. The method of claim 1, wherein activating the AR trigger element comprises: activating a sticky functionality to cause the AR overlay to be rendered in a portion of the display portion of the pair of AR glasses for a period of time after the gaze target is beyond the gaze of the user, the sticky functionality being activated using a gaze-based interface, or by a voice command, or a gesture or touch-activated interface, or as specified in a user profile setting, and The time period is specified by the user or preset.

10. The method of claim 9, wherein activating the sticky function further comprises, capturing a snapshot of the gaze target associated with the activated AR trigger element; and As the user views changes to the real-world environment, the snapshot of the gaze target is rendered on the display portion of the pair of AR glasses alongside the AR overlay, the AR overlay providing information related to the gaze target included in the snapshot.

11. The method of claim 1 , wherein activating the AR trigger element comprises activating a delayed viewing function from a user interface, wherein activating the delayed viewing function comprises, capturing a snapshot of the gaze target associated with the activated AR trigger element; and The snapshot is stored together with the associated AR overlay for later recall by the user.

12. The method of claim 1, wherein activating the AR trigger element comprises: automatically storing a history of the last predefined number of AR overlays, each AR overlay being associated with a corresponding gaze target identified by tracking the gaze of the user and being associated with a corresponding AR trigger element; and A rewind option is provided on a user interface for a user to select a particular AR overlay stored in the history, the user selection causing the AR overlay to be rendered on the display portion of the pair of AR glasses for use by the user.

13. The method of claim 1, wherein activating the AR trigger element further comprises: predicting that a second object occludes the gaze target, the prediction being determined by tracking positions of real-world objects in the real-world environment relative to the gaze target, the occlusion preventing the user from clearly seeing the gaze target; and The AR overlay associated with the gaze target is presented for a duration in which the gaze target is predicted to be occluded by the second object.

14. The method of claim 13, wherein the prediction comprises, collecting attributes of each of a plurality of real-world objects that are part of the real-world environment, the attributes of the plurality of real-world objects being used to identify selected ones of the real-world objects that are stationary and selected other ones of the real-world objects that are moving; and Analyzing properties of each of the selected other real-world objects that are moving among the real-world objects to identify the second object moving toward the gaze target, and predicting a time range when the second object will occlude the gaze target, the predicted time range being used to render the AR overlay associated with the gaze target in the display portion of the pair of AR glasses for a duration of the predicted time range.

15. The method of claim 14, wherein the length of the predicted time range varies based on the motion of the second object and the motion of the fixation target, and The attributes that affect the length of the predicted time range include at least the size, shape, moving direction and speed of the second object and the gaze target.

16. The method of claim 14, wherein the AR overlay and the image of the real-world object representing the gaze target are superimposed on the second object for the duration of a predicted time range, and when the predicted time range expires, the image of the real-world object representing the gaze target and the AR overlay are removed from the second object.

17. The method of claim 1, wherein the predefined time threshold is dynamically adjusted downward each time the gaze of the user is directed toward the gaze target.

18. A method for providing an augmented reality (AR) overlay, comprising: tracking a user's gaze to identify a gaze target that has captured the user's interest, the gaze target identifying a real-world object within a real-world environment viewed by the user through a pair of AR glasses, the gaze target being associated with an AR trigger element; detecting movement of the user relative to the gaze target, and analyzing the movement to identify one or more position parameters of the user; selectively correcting one or more properties of the gaze target in proportion to the one or more position parameters to allow the user to maintain the gaze on the real-world object representing the gaze target; as well as When the gaze of the user on the gaze target exceeds a predefined time threshold, the AR trigger element associated with the gaze target is activated, the activation causing the AR overlay to be rendered next to the image of the real-world object at the display portion of the pair of AR glasses when the user views the real-world environment, the AR overlay providing information related to the real-world object.

19. The method of claim 18, wherein selectively correcting further comprises, detecting movement of the gaze target relative to the movement of the user, analyzing the movement of the gaze target to identify additional position parameters of the user; and An additional correction is performed on the one or more properties in proportion to the movement of the fixation target.

20. The method of claim 18, wherein activating the AR trigger element to render the AR overlay further comprises, collecting attributes of each of a plurality of real-world objects in the real-world environment, the attributes being used to identify a stationary selected real-world object of the plurality of real-world objects and a moving selected other real-world object of the plurality of real-world objects; and Analyzing properties of each of the selected other real-world objects that are moving among the real-world objects to identify a second object that is moving to occlude the gaze target, and predicting a time range when the second object will occlude the gaze target, the predicted time range being used to render the AR overlay associated with the gaze target in the display portion of the pair of AR glasses for a duration of the predicted time range.

Images