Projector-assisted Augmented Reality: Adjusting the size, shape, and presentation of AR based on real-world space

Abstract

A method and system for normalizing a view of a real-world space and a virtual environment of an interactive application viewed through a pair of augmented reality glasses includes receiving content of an interactive application for rendering on a display screen of the pair of augmented reality glasses. The content is streaming content provided in response to input provided by a user and includes a virtual object. The virtual object is dynamically scaled to match a scale of the real-world object to a scale of the virtual object. The dynamically scaled content of the interactive application is projected as an overlay on top of the real-world object in the real-world space viewed through the pair of augmented reality glasses. Once the content is dynamically scaled, a normalized view between the real-world space and the virtual environment is enabled.

Description

[Technical Field]

[0001] The present disclosure relates to normalizing views of content presented to a user while interacting within an interactive application. [Background technology]

[0002] Playing video games and sharing media content from gameplay has become mainstream. The growing popularity of multiplayer video games, social media applications, and other interactive applications has enabled users to simultaneously share and view media content. Users may be co-located or remotely located. Gameplay and viewing of media content occurs from within the confines of the users' own spaces, without the users having to travel to a common venue.

[0003] A user may use a wearable computing device, such as a head-mounted display (HMD), glasses, or the like, to interact with a video game and view gameplay content generated from gameplay of the video game rendered on a display screen associated with the wearable computing device. To make interacting with and viewing the content easier and more enjoyable, each user may set up their wearable device to clearly view the gameplay content of the video game. This may include adjusting the optics of the wearable computing device according to the user's visual characteristics or preferences, or adjusting the image of the gameplay content to allow the user to discern details in the image of the gameplay content rendered on the display screen. The adjustment of the optics or amount of image is made according to the amount of detail the user wants to discern.

[0004] However, some users may never have used the option to adjust the imagery or optics to view the imagery of gameplay content. This may be due to a lack of availability of the option on their respective wearable computing devices or a lack of user knowledge. Even if users have the option and / or knowledge, the level of adjustment to the optics or imagery may not be the same among all users. Particularly in video game environments where users may be playing against each other, such setup mismatches may give some users an unfair advantage over other users during head-to-head competition.

[0005] It is against this background that embodiments of the present disclosure have been made. Summary of the Invention

[0006]

[0003] Embodiments of the present disclosure relate to systems and methods for dynamically scaling objects viewed through a pair of wearable computing devices, which may be a pair of smart glasses or a head-mounted display. The wearable computing devices (or simply "wearable devices" hereinafter) may be augmented reality (AR) devices used to view a real-world space in the vicinity of a user wearing the wearable devices, and to view content from an interactive application, such as a video game application, on a display screen associated with the AR devices. The AR device may be communicatively connected to a local console or server computing device at a cloud computing site, which is used to run interactive applications and provide interactive application content for rendering on an associated display screen. Alternatively, the AR device itself may run interactive applications and provide content for viewing along with real-world content in a real-world space. The interactive application may be a multiplayer video game, or may be a single-player video game or a multi-user interactive application. The content of the interactive application may be streamed to the AR device (i.e., client device) in response to input provided by a user(s). The content of the interactive application is provided as an overlay for rendering on top of real-world content viewed by the user. Dynamic scaling of the content of the interactive application or the real-world content from the real-world space may be performed before rendering the content of the interactive application as an overlay. Dynamic scaling of the content of the real-world space or the content of the interactive application is performed to normalize the view between the content of the real-world space and the content of the interactive application. The content of the interactive application is considered to be virtual content (i.e., AR content).

[0007] In a video game environment where users may be playing against each other, normalizing the AR content helps normalize head-to-head gameplay between users (i.e., players). Normalization prevents a first user from gaining an unfair advantage over a second user due to the particular size or characteristics of the AR content in the game or real-world space when viewed through the AR glasses as opposed to the second user. In some embodiments, the AR content may be scaled to match the scale of real-world objects in the real-world space. Alternatively, real-world objects in the real-world space may be dynamically scaled when viewed through the AR glasses. In addition to scaling the size of objects (real-world or virtual objects), various embodiments may also be extended to scale sounds emanating from or received by the AR content, so that the sounds are normalized within the context of the AR content (i.e., gameplay content) being viewed by a user(s).

[0008] In some embodiments where the interactive application is a video game, the video game may be played between two players, and dynamic scaling of the content allows the real-world spaces of the two players to be normalized. For example, if a first real-world space (e.g., a first living room) near a first player is shared with a second player, and a second real-world space (e.g., a second living room) near the second player is shared with the first player, dynamic scaling allows the space and dimensions of the first and second real-world spaces of the first and second players that are shared with each other to be normalized and normalized to each other. In this example, dynamic scaling provides a level playing field for each player, ensuring that a video game (or any other interactive application) played between two players does not give one player an unfair advantage over the other. Additionally, dynamic scaling to match the scale of real-world objects to the scale of virtual objects, or vice versa, assists in accurately blending the virtual and real worlds and avoids problems associated with a first user gaining an unfair advantage over a second user based on the first user's real-world setting or positioning within the real-world scene.

[0009] In one embodiment, a method is disclosed. The method includes receiving content of an interactive application for rendering on a display screen of a pair of augmented reality (AR) glasses (or AR glasses pair) worn by a user. The pair of AR glasses is used to view a real-world space proximate the user. The content is provided by the interactive application in response to input provided by the user and one or more other users during interaction with the interactive application. The content includes virtual objects. The virtual objects of the interactive application content are dynamically scaled to match the scale of real-world objects in the real-world space viewed through the pair of AR glasses. The scaled content of the interactive application is presented as an overlay on top of the real-world objects in the real-world space viewed through the pair of AR glasses. The scaling allows the user a normalized view of the interactive content when viewed along the real-world objects.

[0010] In one embodiment, a scanned image of a real-world space near the user is received and analyzed to identify a layout of real-world objects in the real-world space and determine attributes of the real-world objects. The attributes of the real-world objects are used to determine the scale of the real-world objects in the real-world space viewed through the pair of AR glasses. Images of virtual objects included in the content of the interactive application are projected as overlays on top of the real-world objects. The images of the virtual objects are dynamically adjusted to match the scale of the virtual objects to the scale of the real-world objects in the real-world space viewed through the pair of AR glasses.

[0011] In one embodiment, the scanned image is received from a scanner for scanning an image of a real-world space in response to a first signal, and the image of the virtual object is dynamically adjusted using a projector in response to a second signal. The first signal and the second signal are triggered by a machine learning algorithm in response to receiving content of an interactive application for rendering on a display screen of a pair of AR glasses. The machine learning algorithm analyzes the scanned image received in response to the first signal and generates a second signal to project the dynamically adjusted image onto the real-world object.

[0012] In one embodiment, the attributes of each real-world object determined from analyzing the scanned image include at least its location in real-world space when viewed through the pair of AR glasses, its location relative to other real-world objects, its location relative to a user interacting with it in real-world space, the size of the real-world object, and the rendering distance of the real-world object.

[0013] In one embodiment, presenting the scaled content further includes filtering one or more virtual objects from the content before rendering the scaled interactive content, where the filtering enables the scaled interactive content to be viewed along real-world space through the pair of AR glasses.

[0014] In one embodiment, presenting the scaled content includes providing a visual indicator that indicates the presence of one or more virtual objects included in the overlay.

[0015] In one embodiment, the dynamic scaling further includes scaling sounds emanating from one or more virtual objects included in the interactive content, the dynamic scaling of the sounds being performed based on the context of the interactive content.

[0016] In one embodiment, the interactive application is a video game played between two remotely located players and the content is gameplay content.

[0017] In one embodiment, the dynamic scaling includes scaling gameplay content within the context of real-world objects in a vicinity of a first player before overlaying the gameplay content over the real-world objects viewed through a first pair of AR glasses for the first player, and scaling gameplay content within the context of real-world objects in a vicinity of a second player before overlaying the gameplay content over the real-world objects viewed through a second pair of AR glasses for the second player.

[0018] In one embodiment, the dynamic scaling includes scaling gameplay content within the context of real-world objects in a vicinity of a first player before overlaying the gameplay content over the real-world objects viewed through a first pair of AR glasses for the first player, and scaling gameplay content within the context of real-world objects in a vicinity of a second player before overlaying the gameplay content over the real-world objects viewed through a second pair of AR glasses for the second player.

[0019] In one embodiment, a real-world object from a first real-world space near a first player is detected as selected for inclusion as the first virtual object by the first player, and a second real-world object from a second real-world space near a second player is detected as selected for inclusion as the second virtual object by the second player, the first virtual object is dynamically scaled to match the scale of the first real-world object viewed by the first player through the first pair of AR glasses, and the second virtual object is dynamically scaled to match the scale of the second real-world object viewed by the second player.

[0020] In one embodiment, the scaled first virtual object is presented as an overlay on a first real world space viewed through a first pair of AR glasses for a first player, and the second virtual object is presented as an overlay on a second real world space viewed through a second pair of AR glasses for a second player, the first virtual object and the second virtual object are rendered along with the content of the interactive application, and the dynamic scaling normalizes the view of the selected first virtual object relative to the first real world space and the selected second virtual object relative to the second real world space.

[0021] In one embodiment, a method includes detecting a selection of a first real-world object from a first real-world space proximate a first player for sharing with a second player or detecting a selection of a second real-world object from a second real-world space proximate a second player for sharing with the first player. The first real-world object is rendered as a first virtual object when rendered as an overlay on the second real-world space, and the second real-world object is rendered as a second virtual object when rendered as an overlay on the first real-world space. Dynamic scaling of the first virtual object is performed to match the scale of the real-world object in the second real-world space viewed by the second player, and dynamic scaling of the second virtual object is performed to match the scale of the real-world object in the first real-world space viewed by the first player. The selection of the first virtual object for sharing is performed by the first player, and the selection of the second virtual object for sharing is performed by the second player.

[0022] In one embodiment, the dynamic scaling is performed in response to feedback provided by a user, by a server running an interactive application, by the interactive application, or by a machine learning algorithm.

[0023] In one embodiment, a method is disclosed. The method includes receiving content of an interactive application for rendering on a display screen of a pair of augmented reality (AR) glasses worn by a user. The content is provided by the interactive application in response to input provided by the user. Dynamically scaling real-world objects in a real-world space proximate the user viewed through the pair of AR glasses worn by the user to match the scale of the real-world objects to the scale of virtual objects included in the content of the interactive application. The virtual objects of the content of the interactive application are presented as overlays on top of the scaled real-world objects in the real-world space viewed through the pair of AR glasses. The dynamic scaling provides the user with a normalized view of the content relative to the real-world objects.

[0024] In one embodiment, dynamic scaling of real-world objects is performed by adjusting the optics of a pair of AR glasses through which a user views the real-world space.

[0025] In one embodiment, the optical system is adjusted according to the visual characteristics of a user viewing the real-world space through a pair of AR glasses.

[0026] In one embodiment, the virtual objects presented in the overlay are selectively filtered to remove one or more virtual objects included in the content before presenting the virtual objects on top of the real-world objects viewed through the pair of AR glasses.

[0027] In one embodiment, feedback is generated to the user requesting adjustments to the user's position in real-world space relative to a virtual object in content provided as an overlay on the real-world space viewed through a pair of AR glasses.

[0028] In one embodiment, the interactive application runs on a server at the cloud site, and the feedback is generated as a signal from the server running the interactive application or as a suggestion from a machine learning algorithm running on the interactive application or the server.

[0029] In one embodiment, the interactive application is a multiplayer video game played between a first player and a second player, and the content of the interactive application includes gameplay content generated in response to input from the first player and the second player.

[0030] In one embodiment, the dynamic scaling of the real-world objects includes scaling a first real-world space viewed through a first pair of AR glasses by a first player to match the scale of a second real-world space viewed through a second pair of AR glasses by a second player, where the dynamic scaling normalizes the view of the first real-world space relative to the second real-world space viewed by the first player and the second player, and the scaling of the first real-world space and the second real-world space matches the scaling of virtual objects from content of the interactive application presented as an overlay on the first real-world space and the second real-world space, respectively.

[0031] In one embodiment, presenting the virtual object includes receiving a selection of a first real-world object in the first real-world space from the first player or a selection of a second real-world object in the second real-world space from the second player for inclusion in the first and second real-world spaces viewed by the first and second players, and rendering the first real-world object as the first virtual object or the second real-world object as the second virtual object as an overlay on the first and second real-world spaces viewed by the first and second players, scaling the first virtual object or the second virtual object to match the scaling of the first and second real-world spaces, and then rendering along with the virtual objects of the gameplay content rendered on the pairs of AR glasses of the first and second players.

[0032] In one embodiment, one or more virtual objects included in the gameplay content are filtered before being included as an overlay on top of real-world objects viewed by the first and second players, and the amount of filtering is defined to prevent clutter from appearing on the display screens of the first and second pairs of AR glasses worn by the first and second players.

[0033] In one embodiment, a method is disclosed. The method includes rendering gameplay content of a video game being played between a first player and a second player on display screens of a first pair of augmented reality (AR) glasses and a second pair of AR glasses worn by the first player and the second player. The gameplay content is provided by the video game in response to inputs provided by the first player and the second player. Dynamic scaling of real-world objects in a first real-world space near the first player viewed through the first pair of AR glasses of the first player is performed to match the scale of real-world objects in a second real-world space near the second player viewed through the second pair of AR glasses of the second player. The dynamic scaling is performed by normalizing a view of the first real-world space viewed by the first player to a second real-world space viewed by the second player. Virtual objects included in the video game content are presented as overlays on the scaled real-world objects in the first real-world space and the second real-world space viewed through the respective pairs of AR glasses of the first player and the second player.

[0034] In one embodiment, the dynamic scaling of the real-world object is performed by adjusting the optics of a first pair of AR glasses and a second pair of AR glasses through which the first player and the second player view the first and second real-world spaces.

[0035] In one embodiment, the optical system is adjusted according to the visual characteristics of a user viewing the real-world space through a pair of AR glasses.

[0036] In one embodiment, the dynamic scaling includes scaling gameplay content of the video game to match the scaling of real-world objects from the first and second real-world spaces as viewed by the first and second players, and the dynamic scaling of the gameplay content is performed before rendering it as an overlay on top of the real-world objects in the first and second real-world spaces.

[0037] In one embodiment, the virtual objects of the gameplay content are presented by dynamically filtering one or more of the virtual objects of the gameplay content before rendering them as an overlay, the amount of filtering being defined to prevent clutter on the display screens of the first and second pairs of AR glasses worn by the first and second players.

[0038] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.

[0039] The present disclosure may be best understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0040] [Figure 1A] 1 illustrates an exemplary cloud gaming system used to access a video game and provide gameplay content along with a real-world view in which the video game is viewed through a pair of AR glasses, according to an embodiment of the present disclosure. [Figure 1B] 1 illustrates a simplified block diagram of different sub-modules of a content scaling module used to normalize an image viewed through a pair of augmented reality (AR) glasses worn by a user, according to one embodiment of the present disclosure. [Figure 1C] 1 illustrates a simplified block diagram of a dynamic scaling engine used to dynamically scale a real-world space viewed through a pair of AR glasses worn by a user as part of normalizing a view between the real-world space and a virtual environment provided by a video game, according to one embodiment of the present disclosure. [Figure 2A]1 illustrates the display screen of a pair of augmented reality glasses worn by a user that renders an augmented reality image in which the size of real-world objects is rendered disproportionate to virtual objects from the content of an interactive application, according to one embodiment. [Figure 2B] 1 illustrates a display screen of a pair of augmented reality glasses worn by a user that renders an augmented reality image in which the size of real-world objects is scaled to match the scale of virtual objects from the content of an interactive application, according to one embodiment of the present disclosure. [Figure 3A] 1 illustrates the display screen of a pair of augmented reality glasses worn by a user that renders an augmented reality image in which the size of real-world objects is rendered disproportionate to virtual objects from the content of an interactive application, according to one embodiment. [Figure 3B] 1 illustrates a display screen of a pair of augmented reality glasses worn by a user that renders an augmented reality image in which the size of virtual objects from the content of an interactive application are dynamically scaled to match the scale of real-world objects, according to one embodiment of the present disclosure. [Figure 4A] 1 illustrates an image of a first real-world space near a first player as viewed through a first pair of AR glasses worn by the first player, and an image of a second real-world space near a second player as viewed through a second pair of AR glasses worn by the second player, in which the scale of real-world objects in the first real-world space does not match the scale of real-world objects in the second real-world space, according to one embodiment of the present disclosure. [Figure 4B]1 illustrates an image of a first real-world space near a first player viewed through a first pair of AR glasses and an image of a second real-world space near a second player viewed through a second pair of AR glasses, in which the view is normalized by adjusting the scale of a first set of real-world objects in the first real-world space to match the scale of a second set of real-world objects in the second real-world space, according to one embodiment of the present disclosure. [Figure 5A] 1 illustrates flow operations of a method for normalizing a view of a real-world object in a real-world space viewed through a pair of AR glasses to a virtual object included in content of an interactive application, according to one embodiment of the present disclosure. [Figure 5B] 1 illustrates flow operations of a method for normalizing a view of a real-world object in a real-world space viewed through a pair of AR glasses to a virtual object included in content of an interactive application, according to an alternative embodiment of the present disclosure. [Figure 5C] 1 illustrates flow operations of a method for normalizing views of a first set of real-world objects in a first real-world space to views of a second set of real-world objects in a second real-world space as viewed through a first pair of AR glasses and a second pair of AR glasses for a first player and a second player, and rendering virtual objects from content of an interactive application as an overlay, according to an example embodiment of the present disclosure. [Figure 6] 1 illustrates an exemplary system used to load interactive applications (e.g., video games) available to a cloud gaming site, according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a diagram of an embodiment of a client device. [Figure 8] FIG. 1B illustrates components of a head-mounted display (HMD), which is an example of the client device of FIG. 1A. DETAILED DESCRIPTION OF THE INVENTION

[0041] In the following detailed description, several specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without some or all of these specific details. In other instances, well-known process steps have not been described in detail in order not to obscure the present disclosure.

[0042] As more media content becomes available online, users can access a variety of media content and become fully immersed in viewing and interacting with it. These users can view media content while viewing a real-world space using a pair of augmented reality (AR) glasses. AR glasses allow media content to be presented alongside the real-world space, helping users to continue experiencing the real-world space while keeping track of the media content. The media content may be streaming media content generated from an interactive application with which the user is interacting or made available to the user by a media content provider or other user. Streaming media content may include content of live private events (e.g., birthdays, anniversaries, travel, etc.) or live public events (e.g., live gaming, live video game gameplay, live concerts, live debates, etc.) provided by a media content provider or user, or pre-recorded media content of past events (e.g., lectures, public or private events, video games, etc.) provided by a content provider or user, or pre-recorded media content (e.g., TV shows, movies, etc.) available for viewing and sharing. A user's interaction experience can be further enhanced by interacting with interactive applications such as video games, either individually or with two or more players playing together, or individually or in groups, and online access allows these users to participate in such interactions from the confines of their own space, which may be co-located or remotely located. User interaction may be in the form of input into the interactive application or commentary related to the media content the user is viewing or related to other users.

[0043] In the case of video games, users use their own computing devices to view and interact. These computing devices may be customized according to each user's preferences. Customization may occur at the viewing level, such as customized display screens (e.g., large and small screens, multiple screens, or a single screen), or at the spatial level (e.g., customized real estate space), in addition to the interactive input level (e.g., customized controllers, customized input buttons, customized controls, etc.). The display screen may be part of a wearable computing device, such as a head-mounted display (HMD) or smart glasses (e.g., AR glasses) used to view virtual or augmented reality (AR) content. The customized view and / or customized space in which a user interacts with an interactive application, such as a video game, may give the user an unfair advantage over other users. For example, in a gaming environment in which a first player may be pitted against a second player, if the first player has a display screen of a particular size or resolution for viewing media content (e.g., virtual or AR content) of the video game and / or real-world objects in the real-world space, the first player may have an unfair advantage over the second player who may not have an expanded display screen for viewing the same media content. The enhanced display screen may enable a first player to notice and appropriately interact with finer details in the media content or real-world space viewed through, for example, a pair of AR glasses, while a second player may not have that advantage. Similarly, a first player may move around a larger real-world space (e.g., a large game room) while interacting with a video game, while a second player may move around a smaller real-world space (e.g., a dorm room). Due to a mismatch in the size of the real-world spaces of the first and second players, the view of the real-world space viewed through the first player's pair of AR glasses may be larger than the view of the real-world space viewed through the second player's pair of AR glasses. As a result, the first player may be able to see more details in the real-world space than the second player, giving the first player an unfair advantage over the second player.

[0044] To eliminate an unfair advantage of a first player over a second player and normalize the real-world space and / or AR content viewed through a pair of AR glasses, e.g., a first player and a second player, the system is designed to normalize the view of real-world objects from the real-world space viewed through the pair of AR glasses of the first player and the second player, as well as virtual objects included in the media content of a video game (i.e., an interactive application). In some embodiments, the real-world space near two remotely located players is normalized. For example, if a first player's living room is shared with a second player's living room, the space and dimensions of each player's living room are normalized relative to one another, providing an equal advantage to both players in a video game played between two players in a shared real-world space using AR content. This type of normalization levels the playing field for both players by avoiding problems associated with one player gaining an unfair advantage over the other based on the real-world setting or positioning in the real-world setting. In addition to normalizing the views of the two players, the system may also normalize virtual objects included in the media content (of the video game) that need to be positioned over real-world objects in the real-world space viewed through the AR glasses pair. Normalization may be performed by scaling virtual objects to match the scale of real-world objects in real-world space or by scaling real-world objects to match the scale of virtual objects viewed through the pair of AR glasses. This type of normalization prevents one user from gaining an unfair advantage over another user by having a special size or characteristic defined for the game or virtual content in real-world space viewed through the pair of AR glasses or the like (e.g., an HMD). Instead of or in addition to normalizing the view of the virtual and / or real-world objects seen through the pair of AR glasses, the system may also scale sounds emanating from or received by the virtual content, so that sounds are normalized for interactive applications within the context of the game content or virtual objects viewed through the pair of AR glasses. The normalization may be based on suggestions from the content scaling module, the video game (i.e., the interactive application), or a server running a machine learning algorithm. These suggestions, when provided as feedback, may take into account the user's current position in the real-world space and may also provide suggestions for best positioning the user in the real-world space, for best positioning a character associated with the user in a video game, or for best positioning the user in the real-world space relative to the virtual content. Normalizing the views between the virtual content and the real-world space, and between the real-world space shared between two users, ensures that games played or activities performed by the two users in the shared real-world space are comparable.

[0045] Normalization helps ensure accurate blending of the virtual and real worlds, for example, for a user during gameplay of a video game and for multiple users sharing their real-world space with other users. Additionally, normalization of the virtual and real worlds helps provide a normalized interaction experience for two users playing a video game (i.e., an interactive application) from two different environments by providing a level playing field (i.e., an equivalent challenge) for the two users. Scanners and / or other sensors may be used to scan the real-world environment. Data provided by the scanners and other sensors may be analyzed to determine the layout of a real-world space in which a user is interacting with an interactive application (e.g., a video game) and to determine attributes of real-world objects within the real-world space. These attributes are used to determine the scale of the real-world objects viewed through the pair of AR glasses. Based on the attributes, a projector is used to either scale the virtual object to match the scale of the real-world object or scale the real-world object to match the scale of the virtual object, and then present the virtual object as an overlay on top of the real-world object viewed through the pair of AR glasses.

[0046] In some embodiments, a dynamic scaling engine of the content scaling module may be used to generate a signal to a projector to scale the virtual object and then render it as an overlay on top of the real-world object viewed through the pair of AR glasses. In some embodiments, a machine learning algorithm may be used to determine the layout and attributes of the identified real-world object in the real-world space. In one embodiment, in response to receiving the content, the dynamic scaling engine may send a first signal to a scanner, image capture device, and sensor available in or near the pair of AR glasses or a computing device communicatively connected to the pair of AR glasses to capture an image of the real-world space in which the user is present. Once the image is captured by the image capture device and sensor, it is forwarded to a machine learning algorithm, which analyzes the captured image to identify the layout and attributes of the real-world object. The layout and attributes of the real-world objects are then used to determine the scale of the real-world objects. The dynamic scaling engine may generate a second signal to adjust the scale of the virtual objects from the content based on the scale of the real-world objects to match the scale of the real-world objects determined from analyzing the images of the real-world space captured by the image capture device and sensor. In some embodiments, one or more real-world objects may be selected and introduced as virtual objects alongside the virtual objects of the content of the interactive application. The virtual objects introduced in this manner may be scaled prior to introduction to match the scale of the real-world objects and virtual objects viewed through the pair of AR glasses.

[0047] With a general understanding of the present disclosure, specific embodiments will now be described with reference to various drawings.

[0048] 1A illustrates a simplified block diagram of a system used in one embodiment to normalize a view between a virtual object viewed through a pair of wearable computing devices, such as a pair of AR glasses, and a real-world object in a real-world space. Note that various embodiments are described with reference to a pair of AR glasses as the wearable computing device used to view real-world objects in the real-world space and virtual objects of an interactive application. However, embodiments may be extended to include other wearable computing devices, such as a head-mounted display (HMD).

[0049] The system includes multiple client devices 100 operated by multiple users who access a cloud gaming site 104 over a network 102, such as the Internet. The users may be co-located or remotely located and may access the cloud gaming site 104 over the Internet from one or more geographic locations. The client devices 100 are communicatively connected to the cloud gaming site by wired or wireless means. The cloud gaming site 104 may include multiple servers distributed across multiple data centers. The multiple servers host one or more interactive applications that authenticate users, instantiate one or more interactive applications that provide the media content necessary for rendering on the users' client devices 100, and provide the resources necessary to normalize the view between real-world content and virtual content included in the media content when rendered on the display screen of the client device (e.g., HMD, AR glasses, etc.).

[0050] For example, server 104a of cloud gaming site 104 may be used to perform user authentication before providing client device 100 with access to an interactive application. Server 104b may include one or more consoles (e.g., game consoles, not shown), each configured to run one or more instances of one or more video games or interactive applications. Alternatively, server 104b may be configured to run one or more instances of one or more interactive applications. Server 104b may be a single server or may represent multiple servers, each server providing the resources necessary to run one or more instances of the interactive application. Alternatively, each of the multiple servers may provide specific resource(s) for running the interactive application, and one of the multiple servers may coordinate the resources for successful execution of the interactive application. One or more servers 104c of the cloud gaming site 104 may run an instance of a content scaling module that is used to normalize the view between real-world objects in real-world space and virtual objects included in the media content of the interactive application viewed through the user's pair of AR glasses. In one embodiment, the user's client device 100 may be a separate computing device used to provide input to the interactive application, and the media content may be provided to the client device 100 for viewing through the user's pair of AR glasses. In this embodiment, each pair of AR glasses may be communicatively connected via a wired or wireless connection to the user's respective client device 100, which is located locally in or near the user's real-world space and communicatively connected to the cloud gaming site 104 via a network 102 (such as the Internet). In alternative embodiments, the pair of AR glasses may themselves be client devices 100 used to provide input to interactive applications running on one or more servers of the cloud gaming site 104. In these and other embodiments, the pair of AR glasses is communicatively connected to a server on the cloud gaming site 104 via the Internet.

[0051] It should be noted that while different servers (104a-c) are shown performing different services, all or a combination of the services (authentication, running interactive applications, and running content scaling modules) may be performed by a single server. User authentication server 104a may interact with user data database 108 to access user accounts 106 contained therein to authenticate users of client devices 100 requesting access to video games running on server 104b. User accounts 106 may include biometric and other authentication data used to authenticate users. Upon successful authentication of the user, the user is provided with access to select a video game for gameplay or to view gameplay.

[0052] Upon successful user authentication of the user, and in response to the user's request to select a video game for gameplay, server 104b may access game titles 110 stored in game database 112 to authenticate the user's request. Game database 112 maintains a list of game titles 110 available to the cloud gaming site and a list of games to which each user is eligible to access, either through a paid membership / subscription or through access granted by the game developer or game host. Once the access request is authenticated by server 104b, the user is provided with access to an instance of the video game by executing the game code of the video game on server 104b. In this case, server 104b may be a standalone server that stores game code for multiple video games and executes instances of the video games (i.e., has resource functionality) in response to requests. Alternatively, server 104b may access and retrieve game code from a video game stored on another server during gameplay of the video game, instantiate the video game using resources of server 104b, and generate a video content stream (i.e., gameplay data). The generated gameplay data is compressed using compression technology of an encoder available to server 104b and transmitted as streaming frames of game data over network 200 to client device 100.

[0053] In one embodiment, the video game may be a multiplayer video game accessed for gameplay by multiple users, who may be distributed across multiple geographic locations. In this embodiment, the game code of the video game may be instantiated on multiple servers (e.g., cloud servers) 104b distributed across multiple data centers. In some embodiments, the data centers on which the video game may be instantiated may be identified at the geographic locations of one or more users, or at least near one or more users, to provide gameplay data with minimal latency. Once instantiated on the multiple cloud servers 104b, gameplay data generated for the video game may be synchronized among the multiple cloud servers 104b and transferred as media content to the users' client devices 100. In other embodiments, server 104b may be configured to manage one or more virtual machines capable of running instances of a video game and providing gameplay data streams, where streaming may occur in real time or with a delay. Other types of cloud servers may include blade servers. Input provided by users through their client devices 100 is used to update the game state of the video game and generate updated gameplay data. The updated gameplay data is streamed to users as frames of gameplay data.

[0054] A video game engine running on server 104b (e.g., a cloud server or a standalone server or a game console) is communicatively connected to the game logic of the video game and provides the framework for the video game. The game engine is a software layer that serves as the foundation for the video game and provides the basic infrastructure (i.e., framework) for developing the video game. The game engine abstracts the details of performing common related tasks (i.e., game engine tasks) required for all video games, while the game code of the video game provides the game logic that details how the video game should be played. The game code of a video game uses modules included in a game engine to control virtual objects, including virtual characters and virtual scenes within the video game, and to generate gameplay data. Some of the basic core modules of a game engine used by the game code of any video game include a physics engine (for collision detection, collision response, object movement based on trajectory, gravity, friction, etc.), a rendering engine (i.e., renderer) for 2D and 3D graphics, a sound engine, scripting, animation, artificial intelligence, threading, networking, streaming, memory management, scene graphics, etc. These basic core modules of a game engine can be reused by various video games to generate gameplay data based on the game state of the video game. The generated gameplay data includes animations provided by various virtual characters, where the animations are based on the context of the video game used to generate the gameplay data.

[0055] A user's client device 100, communicatively connected to the server 104b via the network 102, receives frames of gameplay data, decompresses the gameplay data, and renders it on a display screen associated with the client device. The display screen of each user's client device may be configured to view the real-world space while having the ability to interact with the gameplay data. Each client device (100) has a processor, memory, and communication capabilities and may be portable or non-portable to access the network 102 using wired, wireless, or 4G / 5G communications, etc. The client device 100 may run an operating system and include a network interface, or may be a thin client with a network interface for accessing the network (Internet) 102 and communicating with the server 104b, which provides the computing capabilities. The network 102 may be a 3G, 4G, or 5G network.

[0056] A client device 100 equipped with 5G communication capabilities can be connected to a 5G network. In one embodiment, the 5G network is a digital cellular network whose service area is divided into multiple "cells" (i.e., small geographic areas). Analog data generated by a mobile device is digitized and transmitted as radio waves to a local antenna within the cell using frequency channels that can be reused among geographically separated cells. The local antenna is connected to the Internet and telephone networks by high-bandwidth optical fiber or other similar wireless communications. Because the 5G network uses high-frequency radio waves for communication, it can transmit data at higher data rates, resulting in lower network latency.

[0057] In other embodiments, the media content may be provided by a media content provider hosted on a server separate from server 104b used to store the game code and run the video game. The media content may relate to a live event (e.g., a live game, a live concert, a live debate, a live video capture of an event, etc.) or a recorded event and may be streamed or otherwise provided to client device 100 for rendering. The media content may be generated by a media content provider and transmitted to the client device, or may be generated by another user and transmitted to the user's client device using resources available to the media content provider. Similar to gameplay data for a video game, a display screen associated with a user's client device that renders the media content provided by a media content provider may be configured to provide the user with an augmented reality experience by allowing the user to view the real-world space in their vicinity while rendering the media content along with a view of real-world objects in the real-world space.

[0058] The display screen may be part of a pair of AR glasses worn by the user, and the client device may be the pair of AR glasses or a computing device communicatively connected to each user's pair of AR glasses. The media content rendered along with the real-world objects may be dynamically scaled to match the scale of the real-world objects using a content scaling module executing on the server 104 c, or alternatively, the real-world objects viewed through the pair of AR glasses may be dynamically scaled to match the scale of virtual objects included in the media content before the media content is rendered along with the real-world objects in real-world space. The dynamic scaling provides a normalized view between the real-world objects and the virtual objects included in the media content.

[0059] 1B shows a simplified block diagram of various sub-modules included in a content scaling module 120, which, in one embodiment, is used to normalize the view between real-world objects and virtual objects in media content viewed through a pair of AR-enabled glasses. The content scaling module 120 is shown running on server 104c. However, the content scaling module may run on any server of a cloud gaming site, including server 104a used to authenticate users or server 104b running instances of a video game. Some of the sub-modules included in the content scaling module 120 include a real-world scene identification engine 121, an interactive application content reception engine 122, a dynamic scaling engine 123, a content frame generation engine 127, and an image encoder 128. The identified sub-modules are provided as examples only and should not be considered limiting. The content scaling module 120 may include fewer or additional sub-modules. In some embodiments, the content scaling module 120 may be executed on a computing device local to the AR glasses pair or by a processor available to the AR glasses pair.

[0060] The real-world scene identification engine 121 is configured to obtain details of real-world objects contained in a real-world space near the user viewed through the pair of AR glasses. The details may be obtained using an image capture device, scanner, or sensor available on the pair of AR glasses, on a computing device or controller communicatively connected to the pair of AR glasses, or distributed within the real-world space. The details of the real-world objects are processed to determine the scale of the real-world objects. The visual characteristics of a first user may differ from the visual characteristics of a second user. As a result, the scale of a real-world object viewed through the lenses of a first pair of AR glasses worn by a first user may be the actual scale of the real-world object, and the scale of the real-world object viewed through the lenses of a second pair of AR glasses worn by a second user may be adjusted according to the visual characteristics of the second user. Once processed, the information related to the scale of the real-world object is provided to the virtual object scaling engine 124. The virtual object scaling engine 124 uses this information to adjust the scaling of the virtual object provided to the pair of AR glasses for rendering. Details of the processing of real-world objects viewed through the pair of AR glasses are described in more detail with reference to FIG. 1C .

[0061] The interactive application content reception engine 122 of the content scaling module 120 receives content from an interactive application, such as a video game, generated by the interactive application execution engine. The content includes gameplay data that is updated in response to input provided by a user / player through their respective client device(s) 100. The input from the user / player is used to adjust the game state of the video game and update the content included in frames of gameplay data. Interactive application content reception engine 122 processes frames of gameplay data transmitted by the interactive application to determine the type of content to receive (e.g., streaming or non-streaming content, gameplay content or other interactive application, live or recorded content, etc.), the amount of content, the types of virtual objects contained therein, the positions and orientations of the virtual objects, the depth of the virtual objects, etc. The processed content of the interactive application (i.e., frames of gameplay data in the case of a video game) is provided to dynamic scaling engine 123 for further processing.

[0062] The dynamic scaling engine 123 receives processed input of real-world objects captured from the real-world space and virtual objects included in the content of the interactive application and dynamically scales the virtual objects of the content or the real-world objects in the real-world space before making them available for viewing through the pair of AR glasses. In one embodiment shown in FIG. 1B , the dynamic scaling engine 123 cooperates with the virtual object scaling engine 124 to dynamically scale the virtual objects of the content being streamed by the video game application so that the scale of the virtual objects matches the scale of the real-world objects. In this embodiment, the dynamic scaling engine 123 uses the processed data obtained from the real-world scene identification engine 121 to determine the scale of real-world objects viewed through the pair of AR glasses, and uses the processed data of the real-world objects to determine the scale to which virtual objects in content being streamed by the video game (i.e., interactive application) need to be dynamically scaled before transferring this content to the client device for rendering. Details related to the scaling of the virtual objects are provided to the projection engine 137. The projection engine 137 uses the details to adjust the scaling of images of virtual objects in the content included in the frames and forwards the frames of content with the virtual objects adjusted for scaling to the image encoder 128. If and when new content is streamed from the video game, the projection engine 137 dynamically scales the virtual objects included in the new content according to the scaling information provided by the dynamic scaling engine 123. The frames of the scaled content are configured as an overlay and forwarded to the image encoder 128, where they are encoded using available encoding technology. The encoded frames of the scaled content are forwarded to the client device and rendered as an overlay on top of the real-world objects in the real-world space viewed through the pair of AR glasses. The scale of the virtual objects in the content rendered in the overlay matches the scale of the real-world objects viewed through the pair of AR glasses. 1B , the virtual object is scaled according to the scale of the real-world object viewed through the pair of AR glasses. Instead of dynamically scaling the virtual object of the video game content, in an alternative embodiment, the real-world object viewed through the pair of AR glasses may be adjusted to match the scale of the virtual object. In one embodiment, the virtual object in the overlay may be highlighted or provide other visual indicators to alert the user to the presence of the virtual object.

[0063] FIG. 1C illustrates an alternative embodiment in which the dynamic scaling engine 123 cooperates with the real-world object scaling engine 125 to dynamically adjust the scaling of real-world objects in real-world space viewed through the pair of AR glasses to match the scale of virtual objects included in the video game content. In this embodiment, the dynamic scaling engine 123 can obtain details about the scale of virtual objects included in the video game content and dynamically adjust the scaling of the real-world objects to match the scale of the virtual objects. In one embodiment, the real-world object scaling engine 125 cooperates with the optics adjustment engine 126 to adjust the optical properties of the lenses of the pair of AR glasses so that the real-world objects can be viewed through the pair of AR glasses at a scale that matches the scale of the virtual objects included in the video game content. In one embodiment, the optical properties of the lenses may be adjusted based on the user's visual properties. The user's visual property data 129 may be stored in the user data database 108 and may be made available to the optics adjustment engine 126 based on user credentials provided during user authentication.

[0064] 1C illustrates additional sub-modules included within the dynamic scaling engine 123 that can be used to scale real-world objects in real-world space viewed through a pair of AR glasses. The purpose of scaling the real-world objects is to normalize the view between the real-world objects viewed through the pair of AR glasses and the virtual objects included in the content of the video game (i.e., interactive application). The dynamic scaling engine 123 may utilize a machine learning algorithm 135 to determine the amount of scaling that needs to be performed on the real-world objects or the virtual objects included in the content of the interactive application, and then render the virtual objects as an overlay on top of the real-world space viewed through the pair of AR glasses. The virtual object scaling engine 124 is engaged when virtual objects included in an interactive application need to be dynamically scaled to match the scale of real-world objects in a real-world scene viewed through the pair of AR glasses. Similarly, the real-world object scaling engine 125 is engaged when real-world objects need to be dynamically scaled to match the scale of virtual objects in video game content before presenting the virtual objects on top of the scaled real-world objects. In one embodiment, the dynamic scaling of real-world objects may be performed using the optics adjustment engine 126.

[0065] In one embodiment, when a real-world object needs to be scaled, the dynamic scaling engine 123 sends a signal to the real-world object scaling engine 125 to initiate a scan of the real-world space in which the user is present and interacting with the interactive application. In response to the signal from the dynamic scaling engine 123, the real-world object scaling engine 125 activates one or more image capture devices and / or scanners (132), one or more sensors 133 including motion sensors, etc., to capture images and data related to the real-world space in the user's vicinity. The captured images are processed to identify data related to the real-world space. The data related to the real-world space is analyzed to identify the layout of the real-world objects in the real-world space and attributes of the real-world objects. One or more attributes of a real-world object identified from analysis of image data obtained from the image capture device (e.g., camera) and scanner 132 may include the geometry of the real-world space in the user's vicinity in which the real-world object is located when viewed through the pair of AR glasses, the boundaries of the real-world space, the position of the real-world object within the real-world space, the identity of the real-world object, the position, direction and orientation of the real-world object relative to other real-world objects and relative to the user, the size of the real-world object, the rendering distance of the real-world object, etc. Data provided by various sensors (including motion sensors) may be used to identify the amount of space available between the user and various real-world objects and between real-world objects, the depth of the real-world object viewed through the pair of AR glasses, etc. The amount of space between the real-world object and the user may vary based on the user's movement within real-world space as the user interacts with an interactive application, and sensor data captured by sensors 133 may be used to detect changes in the user's position and orientation within real-world space relative to the real-world object and calculate the space between the real-world object and the user. Attributes of the real-world object are used to determine the scale of the real-world object viewed through the pair of AR glasses. The attributes and other data related to the real-world object are provided as input to machine learning algorithm 135.

[0066] To assist in determining the scale of real-world objects, the dynamic scaling engine 123 cooperates with the virtual object scaling engine 124. The virtual object scaling engine 124 analyzes content from an interactive application to determine various attributes of virtual objects included in the content. In a video game application, the virtual objects are part of a virtual scene within the game environment, and various attributes of the virtual objects determined from analyzing the content include the type of virtual object, the number of virtual objects of each type, the identity of the virtual object, the position and orientation of the virtual object relative to the virtual character representing the user (if applicable) and other virtual characters (either representing other users or available in the game environment), the position and orientation of the virtual object relative to other virtual objects, the depth of the virtual object, etc. The attributes of the virtual objects are used to determine the scale of virtual objects included in the content's virtual scene. The attributes and other data related to the content are provided as input to a machine learning algorithm 135.

[0067] Layout generation engine 134 is configured to receive and analyze data from image capture devices and / or scanners 132 and sensors 133 to determine a layout of the real-world space. In one embodiment, layout generation engine 134 is a separate module within scan engine 131. In alternative embodiments, layout generation engine 134 may be part of scan engine 131. Data from scan engine 131 is provided as input to machine learning algorithms 135, which include data from the image capture devices, images captured by the scanner, data from the sensors, and layout details from layout generation engine 134.

[0068] The machine learning algorithm 135 receives attributes associated with virtual objects included in the content from the virtual object scaling engine 124, attributes associated with real-world objects from the scan engine 131, and layout data associated with the real-world space. The input is classified using a classifier 136, which may include one or more attributes of the real-world objects and / or one or more attributes of the virtual objects. The machine learning algorithm then uses the classifier 136 to build an artificial intelligence (AI) model. The AI ​​model includes a plurality of nodes and edges defined between consecutive pairs of nodes. Each node corresponds to one or more attributes of a real-world object and / or a virtual object, and the edges between any two consecutive nodes define the relationship between the attributes contained in the nodes. The machine learning algorithm 135 generates various outputs based on the content of the video game and various output goals in the real-world space defined for the user. For example, an output goal may be defined to ensure the user does not collide with real-world objects. Another output goal may be to ensure the user does not trip or fall. Another output goal may be to ensure the user does not approach or cross boundaries of the real-world space, and to safely guide the user away from boundaries around real-world objects if the user approaches a boundary. Yet another output goal may be to determine scaling factors for various visual characteristics of the user. As additional content is received and additional attributes of real-world and virtual objects are identified, the AI ​​model is continuously trained to fine-tune the output goals.

[0069] The dynamic scaling engine 123 cooperates with the optics adjustment engine 126 or the content frame generation engine 127 to adjust the scale of either the real-world content or the virtual content based on an output goal defined for or by the user. In one embodiment, the dynamic scaling engine 123 generates a signal to the optics adjustment engine 126 when a real-world object viewed through the pair of AR glasses needs to be dynamically scaled. In response to the signal from the dynamic scaling engine 123, the optics adjustment engine 126 may adjust the optics of the lenses of the pair of AR glasses such that the size of the real-world object viewed through the pair of AR glasses is scaled to match the scale of the virtual object in the interactive application.

[0070] The optical properties of the lenses of the AR glasses pair can be adjusted by the optical system adjustment engine 126 by generating an appropriate signal based on the type of lenses used. The AR glasses pair can be adjustable-focus eyeglasses in that the focal length of the lenses can be varied by utilizing electro-optical or optomechanical methods. In an embodiment using an electro-optically enabled AR glasses pair, the lenses use liquid crystal as a liquid medium, and the focal length of the lenses can be varied by changing the refractive index of the liquid. By applying an appropriate electrical potential, changes to the refractive properties of the liquid can be achieved. In another embodiment using an optomechanically enabled AR glasses pair, the lenses are comprised of a combination of flexible and rigid lenses, and a slider is provided to vary the focal length of the flexible lenses. In one embodiment, the slider may move using a signal provided by the optics adjustment engine 126. In another embodiment, the lens may exert hydraulic pressure against a diaphragm. In this embodiment, each lens is configured to enclose a reservoir of fluid silicone, and the focal length can be varied by adjusting the fluid level, which can be either by electrical pulses or other means. The electrical pulses or other means for controlling the fluid level can be initiated using signal(s) from the optics adjustment engine 126. The above-described methods or lens designs for adjusting the focal length are provided by way of example only and should not be considered limiting. Other lens designs may also be employed to adjust the optical properties of the lens to enable dynamic scaling of real-world objects. Alternatively, other methods for adjusting the scale of a real-world object viewed through a pair of AR glasses may be envisioned.

[0071] If adjusting the optical properties of the lenses, once the optical properties of the lenses for appropriately scaling the real-world space have been determined, the optical property settings are transferred to image encoder 128 for encoding. Along with the optical property settings, a rate of change of the optical property settings can also be provided to ensure that scaling of real-world objects viewed across the display screen of the pair of AR glasses does not have any disorienting effect on the user. Adjusting the optical property settings of the lenses is one way to dynamically scale images of real-world objects viewed through the lenses of the pair of AR glasses.

[0072] In an alternative embodiment, dynamic scaling of real-world objects is enabled by rendering the real-world objects as virtual objects. In this embodiment, an image of the real-world object is captured, rescaled, and presented as a virtual object. Thus, if a real-world object viewed through a pair of AR glasses needs to be dynamically scaled, dynamic scaling engine 123 generates a signal to real-world object scaling engine 125 to perform dynamic rescaling of the real-world object. The real-world object scaling engine 125 cooperates with the scan engine 131 to scan a real-world scene near the user, identify real-world objects contained therein, and capture images of the real-world objects. In response to the signal to rescale the real-world objects, the scan engine 131 generates a signal to apply an occlusion mask to the lenses of the pair of AR glasses to prevent incoming real-world light from reaching the eyes. The occlusion mask transitions the lenses from a transparent mode to a non-transparent mode. Simultaneously, the scan engine 131 activates one or more image capture devices / scanners 132 to capture images of the real-world objects. The one or more image capture devices / scanners 132 are disposed on the surface of the pair of AR glasses and / or within the real-world scene and are connected either directly to the pair of AR glasses or via a console / computing device using a wired or wireless connection.

[0073] In one embodiment, scan engine 131 analyzes the captured image to identify real-world objects in the real-world scene. Once the real-world objects are identified, scan engine 131 selectively rescales some of the real-world objects to match the scale of the virtual objects viewed through the AR glasses pair, leaving other real-world objects in the captured image at their original scale. In an alternative embodiment, scan engine 131 may rescale all real-world objects in the captured image to match the scale of the virtual objects viewed through the glasses pair.

[0074] The scan engine 131 then forwards the rescaled real-world objects to the layout generation engine 134. As described above, the layout generation engine 134 determines the layout of the real-world objects in the real-world space / scene and provides the layout information and the rescaled real-world objects to the machine learning algorithm 135. Based on the goals defined for the real-world and virtual objects, the machine learning algorithm may use the images and layout information of the real-world and virtual objects to identify the content that needs to be included in each frame of content. The content and layout details from the machine learning algorithm are provided to the content frame generation engine 127, which generates the frames of content.

[0075] The content frame generation engine 127 receives input of scaled virtual objects of the content from the virtual object scaling engine 124 or unscaled virtual objects of the content from the interactive application content receiving engine 122. Additionally, the content frame generation engine 127 also receives as input images of scaled real-world objects, if available. These inputs are used to generate frames of the content. The frames of the content may include images of dynamically scaled virtual objects or the content's original scaled virtual objects (i.e., the unscaled content), if available, and / or images of dynamically scaled real-world objects. The frames of the content are provided to the image encoder 128.

[0076] Image encoder 128 encodes frames of content (scaled or unscaled) and data related to lens optical property settings. The encoded content and optical property settings (if available) are transferred to a user's client device for rendering, which may be a pair of AR glasses or a computing device communicatively connected to a pair of AR glasses.

[0077] The client device receives the encoded data and decodes the data using a decoder available on the client device. The decoded frames of content are provided to the pair of AR glasses and rendered as overlays on top of real-world objects viewed through the pair of AR glasses. In addition to rendering the decoded frames of content, lens optical properties (if available) are also applied to the lenses of the pair of AR glasses so that real-world objects viewed through the adjusted lenses are scaled to match the scale of the virtual objects rendered as overlays. As described above, the optical properties applied to the lenses take into account the user's visual characteristics. In some embodiments, one or more virtual objects included in the video game content are filtered before frames of the content are generated and transmitted to the client device, where the content is decoded and rendered. The filtering may be performed to minimize the amount of virtual objects rendered as overlays so as not to overclutter the content (real-world objects and virtual objects from real-world space) viewed through the pair of AR glasses, thereby preventing the user from becoming confused or overwhelmed.

[0078] In some embodiments, in addition to scaling the virtual content (i.e., virtual objects in the content generated by the interactive application), sounds originating from or received by the virtual content can also be scaled to normalize sounds occurring in the real world and the virtual world. The sound scaling may be based on the context of the interactive application (e.g., a video game) and the context of the real-world object being viewed. In one embodiment, the content scaling module can provide feedback to the user so that the user can place themselves within the context of the real-world space or virtual content so that, when viewed through a pair of AR glasses, they can be properly positioned relative to the virtual content that is added as an overlay on top of the real-world space. The feedback can be provided by a server running the content scaling module 120, by the interactive application (e.g., a video game), or by a machine learning algorithm 135.

[0079] In an embodiment in which a video game is being played between two players who share their respective real-world spaces with each other, the real-world spaces of the two players may be normalized. For example, if a first player shares the first player's living room (i.e., a large space) with a second player who is playing the video game, and the second player shares the second player's dorm room (i.e., a small space) with the first player, the spaces and dimensions of the first and second players may be normalized to equally advantage each player during gameplay of the video game played within the shared real-world space. This type of normalization ensures that each player has a level playing field and prevents one player with a large amount of space from gaining an unfair advantage over another player with a limited amount of space. That is, it prevents a first player from gaining an unfair advantage over a second player based on differences in real-world settings or positioning within different real-world settings. Such normalization allows two players from two different environments to have a normalized experience and an equivalent playing space. Similar benefits can be achieved by adjusting the optical properties of the two players to allow the two players to have a normalized view of the video game content and real-world space.

[0080] In alternative embodiments, virtual objects included in the gameplay content may be scaled to match the scale of real-world objects in a vicinity of the user before overlaying the gameplay content over the real-world space viewed through the pair of AR glasses. In the above example of a video game played between two remotely located players, virtual objects in the gameplay content may be dynamically scaled to match the scale of a first set of real-world objects included in a first real-world space in a vicinity of the first player before overlaying the gameplay content over the first set of real-world objects viewed through the first player's first pair of AR glasses. Similarly, virtual objects included in the gameplay content may be dynamically scaled to match the scale of the second set of real-world objects included in a second real-world space near the second player before overlaying the gameplay content on top of the second set of real-world objects viewed through the second pair of AR glasses for the second player. The dynamic scaling may be performed in response to input from the first player or the second player, or in response to a signal from a content scaling module or a machine learning algorithm performed by the content scaling module or by a server executing the content scaling module. In some embodiments, in a video game application providing gameplay content, real-world objects from a vicinity of a first player and / or a second player may be selected for inclusion as virtual objects that can be rendered alongside virtual objects in the gameplay content of the video game. The real-world objects for inclusion may be selected by the first player or the second player, or by both the first player and the second player. In an embodiment in which real-world objects are selected by only the first player or the second player, an image of a virtual object representing the selected real-world object may be included in the view of only the player who selected the real-world object (i.e., only the first player or the second player who made the selection). In other embodiments, real-world objects selected for inclusion as virtual objects by either the first player or the second player may be shared with the other player, in which case virtual objects representing the real-world objects selected for inclusion by the first player or the second player are presented alongside the virtual objects of the video game content in an overlay viewable through both the first and second pairs of AR glasses of the first and second players, respectively. The images of the real-world objects are dynamically scaled to match the scale of the virtual objects in the gameplay content being rendered in the first player's and / or second player's AR glasses pair, or in an overlay on top of the real-world objects viewed through the AR glasses pair. In some embodiments, a first real-world object is selected by a first player from a first real-world space proximate the first player for inclusion as a first virtual object. Concurrently, a second real-world object is selected by a second player from a second real-world space proximate the second player for inclusion as a second virtual object, and the first real-world object and the second real-world object are selected for sharing with each other. In these embodiments, both the first and second real-world objects are projected as the first and second virtual objects, respectively, and the projections are rendered and scaled to match the scale of the virtual objects of the gameplay content included in the overlay. Dynamic scaling includes scaling the dimensions of the first and second virtual objects before being included in the overlay.

[0081] In one embodiment, the content scaling module 120 may be used to add one or more real-world objects from the user's real-world space as virtual objects viewed through the pair of AR glasses. The virtual objects added from the real-world space are added to the virtual objects of the content that are overlaid on top of the real-world objects viewed by the user through the pair of AR glasses. For example, a dog, cat, or child in the user's real-world space may be added as a virtual character alongside the virtual objects of the content in a video game. Alternatively, from the above example of two players sharing a real-world space, one or more real-world objects in the first real-world space of the first player and one or more real-world objects in the second real-world space of the second player may be included with the virtual objects of the content being rendered on top of the real-world objects of the first and second players. This type of normalization of adding content from each other's real-world spaces helps to create an accurate blend between the virtual and real-world spaces of the two players. Those skilled in the art may associate other advantages with the various illustrations and descriptions provided herein.

[0082] 2A illustrates, in one embodiment, a rendering of a real-world space viewed through a pair of AR glasses onto which virtual objects from a virtual game content are overlaid, where neither the view of the real-world objects in the real-world space nor the virtual objects in the content are scaled for normalization. The real-world space shows multiple real-world characters in the background (e.g., under a tree) with virtual characters from the video game content rendered as an overlay in the foreground. However, the scale of the virtual characters does not match the scale of the real-world characters in the background.

[0083] 2B shows a rendering of the real-world space of FIG. 2A in which, in one embodiment, the virtual character has been scaled to normalize the view of the real-world space containing the virtual object. In this embodiment, the virtual character is projected onto the real-world space as an overlay at a scale that matches the scale of the real-world character.

[0084] FIG. 3A shows, in one embodiment, a rendering of real-world space as viewed through a user's pair of AR glasses, with virtual objects from the video game content projected on top of the real-world space as an overlay, with neither the view of the real-world objects nor the virtual objects from the content normalized. FIG. 3B shows, in one embodiment, a rendering of a view of real-world space scaled to match the scale of the virtual objects from the content. In one embodiment, the view of real-world space may be adjusted by adjusting the optical properties of the lenses to provide a normalized view of the real-world objects relative to the virtual objects from the content. In this embodiment, the virtual objects from the content are projected unchanged. However, dynamic scaling of the view of real-world space ensures that the user is presented with a normalized view of real-world space and the video game content projected on top of the real-world space.

[0085] FIG. 4A illustrates, in one embodiment, a view of a first real-world space near a first player as seen through a first pair of AR glasses for the first player and a view of a second real-world space near a second player as seen through a second pair of AR glasses for the second player. The first player is playing a video game with a second player. In the embodiment illustrated in FIG. 4A , the first real-world space and the second real-world space are not normalized to each other. As shown, the first player may be playing from a large living room space while the second player may be playing from a small dorm room space, thereby giving the first player a spatial advantage over the second player. For example, to name a few examples, the first real-world space may include a display screen with a width “d1” on which the content of the video game is rendered, the first player is separated from the display screen by a space length “L1,” a large sofa is separated from a smaller sofa by a length “L2,” the large sofa having a width “d2,” and the small sofa being of size “S2.” These dimensions can be determined by analyzing images captured from the first real-world space by scanners, image capture devices, sensors, etc., on the first AR glass pair, on a computing device communicatively connected to the first AR glass pair, or distributed within the first real-world space. Similarly, the second real-world space includes a display screen that is separated from the second player by a distance "L3", and the size of the display screen of the second player is represented by "d3". As can be seen, the space available to the first player within the first real-world space is larger than the space available to the second player within the second real-world space, so the first player has an undue advantage over the second player.

[0086] Figure 4B shows a normalized view of the first real-world space and the second real-world space as viewed through the first AR glass pair and the second AR glass pair, respectively. Normalization of the view is achieved by dynamically scaling the space and dimensions of the first set of real-world objects distributed within the first real-world space and viewed through the first AR glass pair of the first player to match the scaling of the view of the second set of real-world objects in the second real-world space of the second player viewed through the second AR glass pair. For example, as part of the normalization, the space between the first player and the display screen within the first real-world space viewed through the first AR glass pair can be adjusted to L1', which can be approximately equal to the separation distance L3 between the second player and the display screen within the second real-world space. Similarly, when viewed through the first AR glass pair, the length between the small sofa and the large sofa, represented as L2', can be dynamically adjusted such that L2' < L2. The size of the small sofa viewed through the first AR glass pair of the first player can be dynamically adjusted such that the image of the small sofa is S2' (S2' < S2). Additionally, the size of the display screen in the first real-world space viewed through the first pair of AR glasses may be dynamically adjusted to d1′, which may be approximately equal to d3, the size of the display screen used by the second player in the second real-world space. This dynamic scaling is achieved by adjusting the dimensions of real-world objects and the spacing between real-world objects viewed through each of the first and second pairs of AR glasses, normalizing the views of the first and second real-world spaces to allow each player to have a level playing field, thereby preventing one player from gaining an unfair advantage over the other player.

[0087] It should be noted that the terms “engine” or “module” used to describe various sub-modules of content scaling module 120 refer, in one embodiment, to portions of a program or code configured to perform a specific function. The program or code may use deep learning algorithms to query and / or process specific pieces of data (e.g., input from a user, images of a real-world space, images of media content, etc.). In the case of dynamic scaling engine 123, a program may be used to analyze various data (of a virtual environment or real-world space) to determine various attributes of real-world and virtual objects, use the various attributes to determine the scale at which various objects (virtual or real-world objects) are rendered or viewed, dynamically scale virtual objects, or adjust the optical properties of the lenses of a pair of AR glasses to normalize views of the real world and virtual environment. Content scaling module 120 may be implemented as software, hardware, or firmware.

[0088] 5A illustrates an operational flow of a method used in one embodiment to normalize a view of a real-world space and a virtual environment viewed through a pair of AR glasses. The method begins at operation 510, where content of an interactive application is received and rendered on a display screen of a pair of AR glasses worn by a user to view a real-world space near the user. The interactive application may be a video game, and the content may be gameplay content generated in response to input provided by the user during gameplay. The content may be streaming content that changes based on changes in the game state of the game, where the game state is determined from the user's input.

[0089] In response to receiving AR content (i.e., virtual content from gameplay), as shown in operation 520, a scale of a real-world object viewed through the pair of AR glasses is determined, and the scale of the virtual object is dynamically adjusted to match the scale of the real-world object. The scale of the real-world object may be determined by capturing images of the real-world object in the real-world space using image capture devices, such as scanners and cameras, and sensors available to the pair of AR glasses, a computing device communicatively connected locally to the pair of AR glasses, or distributed within the real-world space. The captured images may be analyzed to identify the layout of the real-world space, various real-world objects distributed within the real-world space, and various attributes of the real-world objects, including the type of real-world object and the number of each type, an estimate of the size, depth, position, and orientation of the real-world objects relative to each other and to the user, etc. The various attributes of the real-world object may be used to determine the scale of the real-world object viewed through the pair of AR glasses. Based on the scale of the real-world object determined from the analysis, the virtual object may be dynamically scaled such that the scale of the virtual object matches the scale of the real-world object.

[0090] The scaled virtual object is then provided as an overlay for rendering on top of the real-world object being viewed through the pair of AR glasses, as shown in operation 530. Dynamic scaling of the virtual object allows for providing a normalized view of the real-world object and the virtual object when viewed through the pair of AR glasses.

[0091] FIG. 5B illustrates an operational flow of a method used in another embodiment to normalize a view of a real-world space and a virtual environment near a user viewed through a pair of AR glasses. The method begins at operation 540, where content of an interactive application is received and rendered on a pair of AR glasses. As described with reference to FIG. 5A, the interactive application may be a video game application that generates streaming gameplay content in response to input from a user. The gameplay content represents a virtual environment and includes virtual objects, including virtual characters and virtual scenes. The pair of AR glasses is used to view the real-world space near the user while interacting with the gameplay content of the video game.

[0092] In response to receiving gameplay content of the video game, the content scaling module may determine a scale of the gameplay content and a scale of the real-world object being provided for rendering. Based on this determination, if the scale of the real-world object does not match the scale of the virtual object, the content scaling module may dynamically scale the real-world object viewed through the pair of AR glasses, as shown in operation 550. Dynamic scaling is matching the scale of the real-world object to the scale of the virtual object included in the gameplay content of the video game. In one embodiment, the dynamic scaling of the real-world object is achieved by adjusting the optical properties of the lenses of the pair of AR glasses so that the scale of the real-world object viewed through the pair of AR glasses matches the scale of the gameplay content. As described with reference to FIG. 1C , various methods of adjusting the optical properties of the lenses can be envisioned depending on the type of lenses used.

[0093] The virtual objects of the gameplay content are then projected as an overlay onto the scaled real-world objects in real-world space, as shown in operation 560. The scaling of the real-world objects ensures normalization of the views of the real-world objects and the virtual objects of the gameplay content. This type of normalization allows a user to have a level playing field when playing a video game with another user.

[0094] 5C illustrates an operational flow for a method for normalizing a view of a real-world space shared between two players and including virtual objects from an interactive application in accordance with yet another embodiment of the present invention. The method begins at operation 570, in which gameplay content for a video game being played between a first player and a second player is received. The first player plays and interacts with the video game and receives the gameplay content for rendering on a display screen associated with a first pair of AR glasses. Similarly, the second player plays and interacts with the video game and receives the gameplay content for rendering on a display screen associated with a second pair of AR glasses. The first player and the second player view real-world objects in their vicinity using the first pair of AR glasses and the second pair of AR glasses, respectively.

[0095] In operation 580, a first set of real-world objects in a first real-world space near the first player are dynamically scaled to match the scale of a second set of real-world objects in a second real-world space near the second player viewed through the first and second pair of AR glasses. The dynamic scaling of the real-world objects in the first and second real-world spaces may be performed by adjusting optical properties of lenses of the first and / or second pair of AR glasses. The dynamic scaling enables a normalized view of the first real-world space viewed through the first pair of AR glasses for the first player and the second real-world space viewed through the second pair of AR glasses for the second player.

[0096] In operation 590, virtual objects included in gameplay content of a video game played between a first player and a second player are projected as overlays onto a first real-world space viewed by the first player through a first pair of AR glasses and a second real-world space viewed by the second player through a second pair of AR glasses. Dynamic scaling of the real-world spaces viewed by the first and second players allows for a normalized view of the real-world space shared between the two players, and when gameplay content is rendered onto each real-world space, further normalizes the view between the real-world space and the virtual environment of the video game. This method of providing a normalized view of the real-world space and the virtual environment allows users to enjoy the video game on an equal footing and prevents one user from gaining an unfair advantage over another user based on their respective real-world settings or their respective positioning within the real-world setting.

[0097] Other advantages will become apparent to those skilled in the art upon reading the various embodiments provided herein.

[0098] 6 shows an exemplary system used to load game files for games available through a cloud gaming site. The system includes multiple client devices 600 (100 in FIG. 1A) communicatively connected to a cloud gaming site 604 (104 in FIG. 1A) via a network 602 (102 in FIG. 1A), which may include a LAN, wired, wireless, cellular (e.g., 4G, 5G, etc.), or any other type of data network, including the Internet. When a request to access the cloud gaming site 604 is received from a client device 600, the cloud gaming site 604 accesses user account information 606 (106 in FIG. 1A) stored in a user data database 608 (108 in FIG. 1A) to identify the user associated with the client device from which the request is initiated. In some embodiments, the cloud gaming site may also verify the identified user to determine all games the user is authorized to view / play. Following user account identification / verification, the cloud gaming site accesses game title data store 610 (110 in FIG. 1A) to identify game titles available at the game cloud site for the user account that initiated the request. Game title data store 610 then interacts with game database 612 (112 in FIG. 1A) to retrieve the game titles of all games available to the cloud gaming site. When a new game is introduced, the game database 612 is updated with the game code, and game title information for the newly introduced game is provided to the game title data store 610. The client device from which the request is initiated may or may not be registered with the cloud gaming site when the request is initiated. If the user of the client device initiating the request is not a registered user, the cloud gaming site may identify the user as a new user and select an appropriate game title for the new user (e.g., a default set of game titles). As shown in FIG. 6, the identified game title is returned to the client device for presentation on display screen 600-a.

[0099] A user interaction with one of the game titles rendered on the client device is detected, and a signal is sent to the cloud gaming site. The signal includes information about the game title on which the user interaction was detected and the user interaction registered with the game title. In response to the signal received from the client device, the cloud gaming site proactively identifies a data center from among multiple data centers (614-a through 614-d) where the game is hosted and sends a signal to the identified data center 614 to load the game associated with the game title on which the user interaction was detected. In some embodiments, multiple data centers may host the game. In such embodiments, the cloud gaming site may determine the geographic location of the client device originating the request, identify a data center geographically close to the client device, and send a signal to the data center to preload the game. The user's geographic location may be determined using a global positioning system (GPS) mechanism in the client device, the client's IP address, or the client's ping information, to name a few. Of course, the above-described methods for detecting a user's geographic location are exemplary, and other types of mechanisms or tools may be used to determine a user's geographic location. Identifying a data center close to the client device can minimize latency when a user interacts with a game. In some embodiments, the identified data center may not have the bandwidth / capacity / resources necessary to host the game or may be overutilized. In these embodiments, the cloud gaming site may identify a second data center that is geographically closer to the client device. Loading a game includes loading game code and running an instance of the game.

[0100] In response to receiving a signal from the cloud gaming site, the identified data center can select a server 616 from among multiple servers at the data center 614 and instantiate the game on the server. The server is selected based on available hardware / software processing power and game requirements. The server can include multiple game consoles 618, and the server can determine which of the multiple game consoles to use to load the game. The game consoles can resemble standalone game consoles or can be rack-mounted or blade servers. A blade server can, in turn, include multiple server blades, each blade having the circuitry necessary to instantiate a single dedicated interactive application, such as a video game. Of course, the aforementioned game consoles are exemplary and should not be considered limiting. Other types of game consoles, including game stations and the like, and other forms of blade servers, can also be used to host the identified game.

[0101] Once the game console is identified, the general game-related code for the game is loaded onto the game console, and a signal identifying the game console instantiating the game is sent back over the network to the client device via the cloud gaming site, making the loaded game available to the user.

[0102] 7 is a diagram of an embodiment of a client device 700 (100 of FIG. 1A). The client device 700 includes several components, such as a network interface card (NIC) 702, a processor 704, a memory device 706, an audio memory device 708, one or more speakers 710, an audio-video frame synchronizer 712, a video decoder system 714, an audio-video frame separator 715, and a display device 718. The components of the client device 700 are coupled together via a bus 716.

[0103] An example of the audio video frame separator 715 is a processor or FPGA or PLD or controller or microprocessor or microcontroller or CPU. An example of the audio video frame synchronizer 712 is a processor or FPGA or PLD or controller or microprocessor or microcontroller or CPU.

[0104] The NIC 702 applies a communication protocol to the stream containing the encoded image frames and audio frames to parse the stream, obtain the encoded image frames and audio frames, and transmit the encoded image frames and audio frames to the audio-video frame separator 715 via bus 716.

[0105] The audio video frame separator 715 separates the encoded image frames from the audio frames, for example, by identifying the difference between the extension of the encoded image frames and the extension of the audio frames to distinguish the encoded image frames from the audio frames.

[0106] The audio video frame separator 715 sends the audio frames to the audio memory device 708 via bus 716 and sends the encoded image frames to the video decoder system 714 via bus 716. The audio frames are stored in the audio memory device 708.

[0107] The video decoder system 714 applies a decoding protocol to output decoded image frames based on the encoded image frames and sends the decoded image frames to an audio video frame synchronizer 712. The audio video frame synchronizer 712 accesses audio frames from the audio memory device 708 and synchronizes the audio frames with the decoded image frames. For example, the audio video frame synchronizer 712 matches the timing at which each decoded image frame is displayed on the display device 718 with the timing at which sound based on the corresponding one of the audio frames should be output by one or more speakers 710.

[0108] The audio video frame synchronizer 712 transmits the decoded image frames via a bus 716 to a display device 718, which displays the decoded images on a display screen of the display device 718. Concurrent with the display of the decoded images, one or more speakers 710 output synchronized sound based on the image frames.

[0109] The processor 704 controls one or more of the components of the client device 700. For example, the processor 704 executes an operating system that enables communication between components via the bus 716.

[0110] Referring to FIG. 8, a diagram illustrating components of an HMD 801 is shown. The HMD 801 includes a processor 800 for executing program instructions. A memory device 802 (memory device 706 in FIG. 7) is provided for storage purposes. Examples of the memory device 802 include volatile memory, non-volatile memory, or a combination thereof. A display device 804 is included to provide a visual interface viewed by a user (FIG. 1A), such as displaying image frames generated from saved data. A battery 806 is provided to power the HMD 801. A motion detection circuit 808 includes any of various types of motion sensing hardware, such as a magnetometer 810, an accelerometer 812, and a gyroscope 814.

[0111] An accelerometer is a device for measuring acceleration and reaction forces caused by gravity. Single-axis and multi-axis models are available to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 812 are used to provide the direction of gravity, which provides an absolute reference for two angles, such as world-space pitch and world-space roll.

[0112] The magnetometer measures the strength and direction of the magnetic field near the HMD 801. In some embodiments, three magnetometers 810 are used within the HMD 801 to ensure an absolute reference for the world-space yaw angle. In various embodiments, the magnetometers are designed to span the Earth's magnetic field, which is ±80 microtesla. The magnetometers are affected by metal and provide yaw measurements that are monotonic with the actual yaw. In some embodiments, the magnetic field is distorted by metal in real-world environments, which causes distortion in the yaw measurement. In various embodiments, this distortion is calibrated using information from other sensors, such as the gyroscope 814 and camera 816. In one embodiment, the accelerometer 812 is used in conjunction with the magnetometer 810 to obtain the tilt and azimuth angles of the HMD 801.

[0113] A gyroscope is a device for measuring or maintaining orientation based on the principles of angular momentum. In one embodiment, instead of gyroscope 814, three gyroscopes provide information about movement across each axis (X, Y, and Z) based on inertial sensing. Gyroscopes are useful for detecting high-speed rotations. However, in some embodiments, gyroscopes drift over time in the absence of an absolute reference. This causes the gyroscope to be periodically reset, which can be done using other available information, such as determining position / orientation based on visual tracking of objects, accelerometers, magnetometers, etc.

[0114] Camera 816 is provided to capture images and image streams of the real-world environment surrounding user A, e.g., a room, a cabin, a natural environment, etc. In various embodiments, multiple cameras are included in HMD 801, including a rear-facing camera that faces away from user A, e.g., when user A is looking at a display or the like of HMD 801, and a front-facing camera that faces towards user A, e.g., when user A is looking at a display or the like of HMD 801. Additionally, in some embodiments, depth camera 818 is included in HMD 801 to sense depth information of objects in the real-world environment.

[0115] The HMD 801 includes a speaker 820 for providing audio output and, in some embodiments, a microphone 822 for capturing audio from the real-world environment, including sounds from the surrounding environment, speech made by user A, etc. The HMD 801 includes a haptic feedback circuit 824, such as a vibration device, for providing haptic feedback to user A. In one embodiment, the haptic feedback circuit 824 may generate movement and / or vibration of the HMD 801 to provide haptic feedback to user A.

[0116] An LED 826 is included as a visual indicator of the status of the HMD 801. For example, the LED may indicate battery level, power on, etc. A card reader 828 is included to allow the HMD 801 to read information from and write information to a memory card. A USB interface 830 is included as an example of an interface to allow connection of peripheral devices or to other devices, such as other portable devices, computers, etc. In various embodiments of the HMD 801, any of a variety of types of interfaces may be included to allow greater connectivity of the HMD 801.

[0117] Wi-Fi™ circuitry 832 is included to enable connection to the Internet via wireless networking technology. The HMD 801 also includes Bluetooth™ circuitry 834 to enable wireless connection to other devices. In some embodiments, a communications link 836 is also included for connection to other devices. In one embodiment, communications link 836 utilizes infrared transmission for wireless communication. In other embodiments, communications link 836 utilizes any of a variety of wireless or wired transmission protocols for communication with other devices.

[0118] Input buttons / sensors 838 are included to provide an input interface for User A (FIG. 1). Any of a variety of types of input interfaces may be included, such as buttons, touchpads, joysticks, trackballs, etc. In various embodiments, ultrasonic communication circuitry 840 is included in HMD 801 to facilitate communication with other devices via ultrasonic technology.

[0119] Biosensor 842 is included to enable detection of physiological data from user A or B. In one embodiment, biosensor 842 includes one or more dry electrodes to detect bioelectrical signals of user A or B through the user A's or B's skin.

[0120] The aforementioned components of the HMD 801 are described as merely exemplary components that may be included within the HMD 801. In various embodiments, the HMD 801 may or may not include some of the various components described above.

[0121] In some embodiments, communication between the server system and the client device may be facilitated using wireless technology. Such technology may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which a service area covered by a provider is divided into small geographic areas known as cells. Analog signals representing sound and video are digitized by the phone, converted by an analog-to-digital converter, and transmitted as a bitstream. All 5G wireless devices within a cell communicate over electromagnetic waves with a local antenna array and low-power automatic transceivers (transmitters and receivers) within the cell via frequency channels assigned by the transceiver from a frequency pool reused by other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cell networks, mobile devices moving from one cell to another are automatically transferred to the new cell. Of course, 5G networks are merely an example type of communications network, and embodiments of the present disclosure may use previous generations of wireless or wired communications, as well as later generations of wired or wireless technologies that come after 5G.

[0122] It should be noted that in various embodiments, one or more features of some of the embodiments described herein may be combined with one or more features of one or more of the remaining embodiments described herein.

[0123] The embodiments described in this disclosure may be implemented with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. In practice, the embodiments described in this disclosure are practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wired or wireless network.

[0124] With the foregoing embodiments in mind, it should be understood that, in practice, the embodiments described herein employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments described herein are useful machine operations. Some embodiments described herein also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, in one embodiment, various general-purpose machines are used with computer programs written in accordance with the teachings herein. Alternatively, it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0125] In implementation, some embodiments described in this disclosure are embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that stores data which can subsequently be read by a computer system. Examples of computer-readable media include hard drives, network-attached storage (NAS), ROM, RAM, compact disc ROM (CD-ROM), CD-recordable (CD-R), CD-rewritable (CD-RW), magnetic tape, optical data storage devices, non-optical data storage devices, etc. As an example, the computer-readable medium includes computer-readable tangible media distributed over network-coupled computer systems so that the computer-readable code is stored and executed in a distributed fashion.

[0126] Additionally, although some of the above embodiments are described with respect to a gaming environment, in some embodiments other environments are used instead of a game, such as, for example, a video conferencing environment.

[0127] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be adjusted to occur at slightly different times, or may be distributed within a system that allows processing operations to occur at various intervals relative to processing, so long as the processing of the overlay operation is performed in the desired manner.

[0128] Although the foregoing embodiments set forth in this disclosure have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Thus, the present embodiments are to be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein but may be modified within the scope of the appended claims and their equivalents.

Claims

1. receiving, at a content scaling module capable of communicating with a pair of augmented reality (AR) glasses worn by a user, content of an interactive application for rendering on a display screen of the pair of AR glasses, the pair of AR glasses being used to view a real-world space near the user, the content being provided by the interactive application in response to input provided by the user and one or more other users during interaction with the interactive application, the content including a virtual object; the content scaling module dynamically scaling the virtual objects of the content of the interactive application to match the scale of real-world objects in the real-world space viewed through the pair of AR glasses; presenting the dynamically scaled virtual objects of the content of the interactive application as overlays on top of the real-world objects in the real-world space viewed through the pair of AR glasses, the scaled interactive content providing the user with a view of the interactive content that normalizes the space and dimensions of the real-world spaces of the user and the other user relative to each other when viewed along the real-world objects; method.

2. In the dynamic scaling, receiving a scanned image of the real-world space in the vicinity of the user; analyzing the scanned image to identify a layout of the real-world objects in the real-world space and to determine attributes of the real-world objects, the attributes of the real-world objects being used to determine a scale of the real-world objects in the real-world space as viewed through the pair of AR glasses; generating an image of the virtual object included in the content of the interactive application for projection as an overlay on the real-world object, the image of the virtual object being dynamically adjusted to match the scale of the virtual object to the scale of the real-world object in the real-world space viewed through the pair of AR glasses before projecting it as the overlay on the real-world object; The method of claim 1.

3. the scanned image is received from a scanner in response to a first signal; the image of the virtual object is dynamically adjusted using a projector in response to a second signal; the first signal and the second signal are triggered by a machine learning algorithm in response to receiving content of the interactive application for rendering, the machine learning algorithm analyzing the scanned image received in response to the first signal and generating the second signal to project the dynamically adjusted image onto the real-world object. The method of claim 2.

4. the attributes of the real-world object determined from analyzing the scanned image include at least a rendering distance of the real-world object when viewed through the pair of AR glasses, a size of the real-world object, an orientation relative to the user interacting in the real-world space, an orientation relative to another real-world object, and a position within the real-world space; The method of claim 2.

5. Presenting the dynamically scaled content further comprises filtering one or more virtual objects from the content before rendering the scaled interactive content; the filtering enables viewing of the scaled interactive content along the real-world space through the pair of AR glasses. The method of claim 1.

6. providing a visual indicator for the virtual object to indicate the presence of the virtual object included in the overlay in the dynamically scaled presentation of the content; The method of claim 1.

7. The dynamic scaling further comprises scaling sounds emanating from one or more virtual objects included in the content of the interactive application, the dynamic scaling of the sounds being performed based on the context of the content of the interactive application. The method of claim 1.

8. the interactive application is a video game played between a first player and a second player, the first player and the second player being remotely located, and the content is gameplay content; The method of claim 1.

9. In the dynamic scaling, scaling the gameplay content according to the context of a first set of real-world objects in a first real-world space of the first player, and then overlaying the gameplay content on top of the first set of real-world objects viewed through a first pair of AR glasses of the first player; scaling the game play content according to the context of a second set of real-world objects in a second real-world space of the second player, and then overlaying the game play content over the second set of real-world objects viewed through a second pair of AR glasses of the second player. The method of claim 8.

10. Detecting a first selection of a first real-world object from a first real-world space near the first player for inclusion as a first virtual object and a second selection of a second real-world object from a second real-world space near the second player for inclusion as a second virtual object; the dynamic scaling includes scaling dimensions of the first virtual object and the second virtual object to match the scale of the virtual objects of the content of the interactive application presented in the overlay; the first selection is initiated by the first player and the second selection is initiated by the second player; The method of claim 8.

11. Dynamically scaled content presentation the first virtual object is included in the overlay rendered on the first real world space viewed through a first pair of AR glasses of the first player, and the second virtual object is included in the overlay rendered on the second real world space viewed through a second pair of AR glasses of the second player; the first virtual object and the second virtual object are rendered in accordance with the content of the interactive application; the dynamic scaling normalizes dimensions of the selected first virtual object with respect to the first real world space and normalizes dimensions of the selected second virtual object with respect to the second real world space; The method of claim 10.

12. and detecting a first selection of first real-world objects from a first real-world space proximate the first player for sharing with a second player; the first real-world object is rendered as a first virtual object when rendered within the overlay of the content of the interactive application on the second real-world space; the dynamic scaling scaling a dimension of the first virtual object to match the scale of the real-world object in the second real-world space as viewed by the second player; the first real-world object is selected for sharing by the first player; The method of claim 8.

13. The dynamic scaling includes: a server running the interactive application; or in response to a signal from the interactive application or a machine learning algorithm. The method of claim 1.

14. receiving, at a content scaling module capable of communicating with a pair of augmented reality (AR) glasses worn by a user, content of an interactive application for rendering on a display screen of the pair of AR glasses, the content being provided by the interactive application in response to input provided by the user and one or more other users; the content scaling module dynamically scaling a real-world object in a real-world space viewed through the pair of AR glasses worn by the user, the dynamic scaling being performed to match a scale of the real-world object to a scale of a virtual object included in the content of the interactive application; presenting the virtual objects of the content of the interactive application as overlays on top of the scaled real-world objects in the real-world space viewed through the pair of AR glasses; the dynamic scaling normalizes the real-world space and dimensions of the user and the other users' neighborhoods relative to one another to provide the user with a view of the content relative to the real-world objects; method.

15. the dynamic scaling of the real-world object is performed by adjusting the optics of the pair of AR glasses through which the user views the real-world space.

15. The method of claim 14.

16. The optical system is adjusted according to the visual characteristics of the user viewing the real world space through the pair of AR glasses.

16. The method of claim 15.

17. Presenting the virtual objects includes selectively filtering one or more of the virtual objects of the content before presenting the virtual objects over real-world objects viewed through the pair of AR glasses.

15. The method of claim 14.

18. Further, generating feedback to the user to adjust the user's position in the real-world space relative to a virtual object of the content provided as the overlay on the real-world space viewed through the pair of AR glasses.

15. The method of claim 14.

19. the interactive application runs on a server at a cloud site, and the feedback is generated as a signal from the server running the interactive application or as a suggestion from a machine learning algorithm running on the interactive application or on the server; 20. The method of claim 18.

20. the interactive application is a multiplayer video game played between a first player and a second player, and the content of the interactive application includes gameplay content generated in response to game inputs from the first player and the second player; 15. The method of claim 14.

21. dynamic scaling of the real-world object includes scaling a first real-world space viewed through a first pair of AR glasses of the first player to match the scale of a second real-world space viewed through a second pair of AR glasses of the second player; the dynamic scaling is performed by adjusting dimensions of a first set of real-world objects and spaces between the first set of real-world objects, and the dynamic scaling normalizes the space and dimensions of the first real-world space relative to the second real-world space viewed by the first player and the second player; the scaling of the first real world space and the second real world space matches the scaling of the virtual objects from the content of the interactive application presented as an overlay on the first real world space and the second real world space, respectively.

21. The method of claim 20.

22. The presentation of the virtual object further comprises: receiving from the first player a first selection of a first real-world object in the first real-world space or a second selection of a second real-world object in the second real-world space from the second player for inclusion in the overlay rendered on the first real-world space and the second real-world space viewed by the first player and the second player; the first real-world object is rendered as a first virtual object or the second real-world object is rendered as a second virtual object in the overlay rendered on the first real-world space and the second real-world space viewed by the first player and the second player, and the first virtual object or the second virtual object is scaled to match the scale of the first real-world space and the second real-world space before being included in the overlay for rendering along with the virtual objects of the game play content viewed through the pair of AR glasses of the first player and the second player.

21. The method of claim 20.

23. one or more virtual objects included in the gameplay content are filtered before being included as the overlay on the real-world objects viewed by the first player and the second player, the amount of filtering being defined to prevent clutter from occurring on the display screens of the first pair of AR glasses and the second pair of AR glasses worn by the first player and the second player; 21. The method of claim 20.

24. receiving gameplay content of a video game played between a first player and a second player at a content scaling module capable of communicating with a first pair of augmented reality (AR) glasses and a second pair of AR glasses worn by the first player and the second player for rendering on display screens of the first pair of AR glasses and a second pair of AR glasses, the gameplay content being provided by the video game in response to inputs provided by the first player and the second player; the content scaling module dynamically scaling real-world objects in a first real-world space near the first player as viewed through the first pair of AR glasses for the first player to match the scale of real-world objects in a second real-world space near the second player as viewed through the second pair of AR glasses for the second player, the dynamic scaling normalizing the space and dimensions of the first real-world space with respect to the second real-world space viewed by the first player and the second player; presenting virtual objects included in the gameplay content of the video game as overlays on top of the real-world objects included in the dynamically scaled first and second real-world spaces for viewing through the first and second pairs of AR glasses of the first and second players; method.

25. the dynamic scaling of the real-world object is performed by adjusting optics of the first pair of AR glasses and the second pair of AR glasses used by the first player and the second player to view the first real world space and the second real world space.

25. The method of claim 24.

26. the optical system of the first pair of AR glasses is adjusted according to the visual characteristics of the first player, and the optical system of the second pair of AR glasses is adjusted according to the visual characteristics of the second player; 26. The method of claim 25.

27. the dynamic scaling further comprises scaling the gameplay content of the video game to match scaling of the real-world objects from the first real world space and the second real world space as viewed by the first player and the second player, the dynamic scaling of the gameplay content being performed prior to rendering as the overlay on the real-world objects in the first real world space and the second real world space, respectively.

25. The method of claim 24.

Images