Projector-assisted augmented reality, adjusting the size, shape and presentation of AR based on real-world space

Dynamic scaling of content in wearable devices adjusts virtual and real-world objects to match scales, addressing unequal viewing issues in multiplayer games and ensuring a fair interactive experience.

JP2024543547AActive Publication Date: 2024-11-21SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024531055
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-24
Filing Date
2022-11-09
Publication Date
2024-11-21
Estimated Expiration
2042-11-09

AI Technical Summary

Technical Problem

In multiplayer video games and interactive applications, users often face unequal viewing experiences due to mismatched adjustments of wearable computing devices, leading to unfair advantages in competitive gameplay, as users may not have the option or knowledge to adjust optics or image settings to suit their preferences, affecting the fairness of head-to-head competition.

Method used

Dynamic scaling of interactive application content and real-world content is performed to normalize the view between real-world and virtual objects, ensuring that the size and characteristics of virtual content match the scale of real-world objects, thereby providing an equal playing field for all users.

Benefits of technology

This solution ensures that all users have a fair and balanced interactive experience by accurately blending virtual and real-world environments, preventing one user from gaining an unfair advantage over others, and enhancing the overall gaming experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543547000001_ABST
    Figure 2024543547000001_ABST
Patent Text Reader

Abstract

A method and system for normalizing a view of a real-world space and a virtual environment of an interactive application viewed through a pair of augmented reality glasses includes receiving content of an interactive application for rendering on a display screen of the pair of augmented reality glasses. The content is streaming content provided in response to input provided by a user and includes a virtual object. The virtual object is dynamically scaled to match a scale of the real-world object to a scale of the virtual object. The dynamically scaled content of the interactive application is projected as an overlay on top of the real-world object in the real-world space viewed through the pair of augmented reality glasses. Once the content is dynamically scaled, a normalized view between the real-world space and the virtual environment is enabled.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to normalizing views of content presented to a user while interacting within an interactive application. [Background technology]

[0002] Playing video games and sharing media content from gameplay has become mainstream. The growing popularity of multiplayer video games, social media applications, and other interactive applications has enabled users to simultaneously share and view media content. Users may be co-located or remotely located. Gameplay and viewing of media content occurs from within the confines of the users' own spaces, without the users having to travel to a common venue.

[0003] A user may use a wearable computing device, such as a head mounted display (HMD), glasses, etc., to interact with a video game and view gameplay content generated from gameplay of the video game rendered on a display screen associated with the wearable computing device. To make interacting with and viewing the content easier and more enjoyable, each user may have set up their wearable device to enable clear viewing of the gameplay content of the video game. This may include adjusting the optics of the wearable computing device according to the user's visual characteristics or preferences, or adjusting the image of the gameplay content to allow the user to discern details in the image of the gameplay content rendered on the display screen. The adjustment of the optics or amount of image is made according to the amount of detail the user wants to discern.

[0004] However, some users may never have used the option to adjust the imagery or optics to view imagery of gameplay content. This may be due to a lack of availability of the option on their respective wearable computing devices or a lack of knowledge by the users. Even if users have the option and / or knowledge, the level of adjustment to the optics or imagery may not be the same among all users. Particularly in a video game environment where users may be playing against one another, such a mismatch in setup may give some users an unfair advantage over other users during head-to-head competition.

[0005] It is against this background that embodiments of the present disclosure have been made. Summary of the Invention

[0006] The embodiments of the present disclosure relate to a system and method for dynamically scaling an object viewed through a pair of wearable computing devices, which may be a pair of smart glasses or a head mounted display. The wearable computing devices (or simply referred to hereinafter as "wearable devices") may be augmented reality (AR) devices that are used to view a real world space in the vicinity of a user wearing the wearable devices, and also view content from an interactive application, such as a video game application, on a display screen associated with the AR devices. The AR device may be communicatively connected to a local console, or a server computing device at a cloud computing site, which is used to run interactive applications and provide interactive application content for rendering on an associated display screen. Alternatively, the AR device itself may run interactive applications and provide content for viewing along with real-world content in a real-world space. The interactive application may be a multiplayer video game, or may be a single-player video game or a multi-user interactive application. The content of the interactive application may be streamed to the AR device (i.e., the client device) in response to input provided by the user(s). The content of the interactive application is provided as an overlay for rendering on top of the real-world content viewed by the user. The interactive application content or the real-world content from the real-world space may be dynamically scaled before rendering the interactive application content as an overlay. The dynamic scaling of the real-world space content or the interactive application content is performed to normalize the view between the real-world space content and the interactive application content. The interactive application content is considered to be virtual content (i.e., AR content).

[0007] In a video game environment where users may be playing against one another, normalization of the AR content helps normalize head-to-head gameplay between users (i.e., players). Normalization prevents a first user from gaining an unfair advantage over a second user due to the special size or characteristics of the AR content in the game or real-world space when viewed through the AR glasses as opposed to the second user. In some embodiments, the AR content may be scaled to match the scale of the real-world object in the real-world space. Alternatively, the real-world object in the real-world space may be dynamically scaled when viewed through the AR glasses. In addition to scaling the size of the object (real-world or virtual object), various embodiments may be extended to also scale the sound emanating from or received by the AR content, so that the sound is normalized within the context of the AR content (i.e., gameplay content) that the user(s) are viewing.

[0008] In some embodiments where the interactive application is a video game, the video game may be played between two players, and dynamic scaling of the content allows the real world spaces of the two players to be normalized. For example, if a first real world space (e.g., a first living room) near a first player is shared with a second player, and a second real world space (e.g., a second living room) near the second player is shared with the first player, dynamic scaling allows the space and dimensions of the first and second real world spaces shared with each other to be normalized and normalized with each other. In this example, dynamic scaling provides an even playing field for each player, ensuring that a video game (or any other interactive application) played between two players does not give one player an unfair advantage over the other. Additionally, dynamic scaling to match the scale of real-world objects to the scale of virtual objects, or vice versa, helps accurately blend the virtual and real worlds, avoiding problems associated with a first user having an unfair advantage over a second user based on the first user's real-world setting or positioning within a real-world scene.

[0009] In one embodiment, a method is disclosed. The method includes receiving content of an interactive application for rendering on a display screen of an augmented reality (AR) pair of glasses (or AR glasses pair) worn by a user. The AR glasses pair is used to view a real-world space proximate to the user. The content is provided by the interactive application in response to input provided by the user and one or more other users during interaction with the interactive application. The content includes virtual objects. The virtual objects of the interactive application content are dynamically scaled to match a scale of the real-world objects in the real-world space viewed through the AR glasses pair. The scaled content of the interactive application is presented as an overlay on top of the real-world objects of the real-world space viewed through the AR glasses pair. The scaling allows the user a normalized view of the interactive content when viewed along the real-world objects.

[0010] In one embodiment, a scanned image of a real-world space near the user is received and analyzed to identify a layout of real-world objects in the real-world space and determine attributes of the real-world objects. The attributes of the real-world objects are used to determine a scale of the real-world objects in the real-world space viewed through the pair of AR glasses. Images of virtual objects included in the content of the interactive application are projected as an overlay on top of the real-world objects. The images of the virtual objects are dynamically adjusted to match the scale of the virtual objects to the scale of the real-world objects in the real-world space viewed through the pair of AR glasses.

[0011] In one embodiment, the scanned image is received from a scanner for scanning an image of a real-world space in response to a first signal, and the image of the virtual object is dynamically adjusted using a projector in response to a second signal. The first signal and the second signal are triggered by a machine learning algorithm in response to receiving content of an interactive application for rendering on a display screen of the AR glasses pair. The machine learning algorithm analyzes the scanned image received in response to the first signal and generates a second signal to project the dynamically adjusted image onto the real-world object.

[0012] In one embodiment, the attributes of each real-world object determined from analyzing the scanned images include at least its location in real-world space when viewed through the pair of AR glasses, its location relative to other real-world objects, its location relative to a user interacting with it in real-world space, the size of the real-world object, and the rendering distance of the real-world object.

[0013] In one embodiment, presenting the scaled content further includes filtering one or more virtual objects from the content before rendering the scaled interactive content, where the filtering enables the scaled interactive content to be viewed along the real-world space through the pair of AR glasses.

[0014] In one embodiment, presenting the scaled content includes providing a visual indicator that indicates the presence of one or more virtual objects included in the overlay.

[0015] In one embodiment, the dynamic scaling further includes scaling sounds emanating from one or more virtual objects included in the interactive content. The dynamic scaling of the sounds is performed based on the context of the interactive content.

[0016] In one embodiment, the interactive application is a video game played between two remotely located players and the content is gameplay content.

[0017] In one embodiment, the dynamic scaling includes scaling gameplay content within the context of real-world objects in a vicinity of a first player before overlaying the gameplay content over the real-world objects viewed through a first pair of AR glasses for the first player, and scaling gameplay content within the context of real-world objects in a vicinity of a second player before overlaying the gameplay content over the real-world objects viewed through a second pair of AR glasses for the second player.

[0018] In one embodiment, the dynamic scaling includes scaling gameplay content within the context of real-world objects in a vicinity of a first player before overlaying the gameplay content over the real-world objects viewed through a first pair of AR glasses for the first player, and scaling gameplay content within the context of real-world objects in a vicinity of a second player before overlaying the gameplay content over the real-world objects viewed through a second pair of AR glasses for the second player.

[0019] In one embodiment, a real-world object from a first real world space in the vicinity of a first player is detected as selected for inclusion as the first virtual object by the first player, and a second real-world object from a second real world space in the vicinity of a second player is detected as selected for inclusion as the second virtual object by the second player, the first virtual object is dynamically scaled to match a scale of the first real world object viewed by the first player through the first pair of AR glasses, and the second virtual object is dynamically scaled to match a scale of the second real world object viewed by the second player.

[0020] In one embodiment, the scaled first virtual object is presented as an overlay on a first real world space viewed through a first pair of AR glasses for a first player, and the second virtual object is presented as an overlay on a second real world space viewed through a second pair of AR glasses for a second player, the first virtual object and the second virtual object are rendered along with the content of the interactive application, and the dynamic scaling normalizes the view of the selected first virtual object relative to the first real world space and the selected second virtual object relative to the second real world space.

[0021] In one embodiment, the method includes detecting a selection of a first real-world object from a first real-world space in a vicinity of a first player for sharing with a second player or detecting a selection of a second real-world object from a second real-world space in a vicinity of a second player for sharing with the first player. The first real-world object is rendered as a first virtual object when rendered as an overlay on the second real-world space, and the second real-world object is rendered as a second virtual object when rendered as an overlay on the first real-world space. Dynamic scaling of the first virtual object is performed to match a scale of the real-world object in the second real-world space viewed by the second player, and dynamic scaling of the second virtual object is performed to match a scale of the real-world object in the first real-world space viewed by the first player. The selection of the first virtual object for sharing is performed by the first player, and the selection of the second virtual object for sharing is performed by the second player.

[0022] In one embodiment, the dynamic scaling is performed in response to feedback provided by a user, by a server running an interactive application, by the interactive application, or by a machine learning algorithm.

[0023] In one embodiment, a method is disclosed. The method includes receiving content of an interactive application for rendering on a display screen of a pair of augmented reality (AR) glasses worn by a user. The content is provided by the interactive application in response to input provided by the user. Dynamically scaling real-world objects of a real-world space proximate the user viewed through the pair of AR glasses worn by the user to match the scale of the real-world objects to the scale of virtual objects included in the content of the interactive application. Presenting the virtual objects of the content of the interactive application as an overlay on top of the scaled real-world objects of the real-world space viewed through the pair of AR glasses. The dynamic scaling provides the user with a normalized view of the content related to the real-world objects.

[0024] In one embodiment, dynamic scaling of a real-world object is performed by adjusting the optics of a pair of AR glasses through which a user views the real-world space.

[0025] In one embodiment, the optical system is adjusted according to the visual characteristics of a user viewing the real-world space through a pair of AR glasses.

[0026] In one embodiment, the virtual objects presented in the overlay are selectively filtered to remove one or more of the virtual objects included in the content before presenting the virtual objects over the real-world objects viewed through the pair of AR glasses.

[0027] In one embodiment, feedback is generated to the user requesting adjustments to the user's position in real-world space relative to a virtual object in content provided as an overlay on the real-world space viewed through a pair of AR glasses.

[0028] In one embodiment, the interactive application runs on a server at the cloud site, and the feedback is generated as a signal from the server running the interactive application or as a suggestion from a machine learning algorithm running on the interactive application or on the server.

[0029] In one embodiment, the interactive application is a multiplayer video game played between a first player and a second player, and content of the interactive application includes gameplay content generated in response to input from the first player and the second player.

[0030] In one embodiment, the dynamic scaling of the real-world objects includes scaling a first real world space viewed through a first pair of AR glasses for a first player to match a scale of a second real world space viewed through a second pair of AR glasses for a second player, where the dynamic scaling normalizes the view of the first real world space to the second real world space viewed by the first player and the second player, and the scaling of the first real world space and the second real world space matches the scaling of virtual objects from content of the interactive application presented as an overlay on the respective first real world space and second real world space.

[0031] In one embodiment, presenting the virtual object includes receiving a selection of a first real world object in the first real world space from the first player or a selection of a second real world object in the second real world space from the second player for inclusion in the first and second real world spaces viewed by the first and second players, and rendering the first real world object as the first virtual object or the second real world object as the second virtual object as an overlay on the first and second real world spaces viewed by the first and second players, scaling the first virtual object or the second virtual object to match the scaling of the first and second real world spaces before rendering along with the virtual objects of the gameplay content rendered on the AR glasses pair of the first and second players.

[0032] In one embodiment, one or more virtual objects included in the game play content are filtered before being included as an overlay on top of real-world objects viewed by the first and second players, with the amount of filtering defined to prevent clutter on the display screens of the first and second pairs of AR glasses worn by the first and second players.

[0033] In one embodiment, a method is disclosed. The method includes rendering gameplay content of a video game played between a first player and a second player on a display screen of a first pair of augmented reality (AR) glasses and a second pair of AR glasses worn by the first player and the second player. The gameplay content is provided by the video game in response to inputs provided by the first player and the second player. Dynamically scaling real-world objects in a first real-world space in the vicinity of the first player viewed through the first pair of AR glasses of the first player to match the scale of real-world objects in a second real-world space in the vicinity of the second player viewed through the second pair of AR glasses of the second player. The dynamic scaling is normalizing a view of the first real-world space viewed by the first player to the second real-world space viewed by the second player. Virtual objects included in the content of the video game are presented as an overlay on the scaled real-world objects in the first real-world space and the second real-world space viewed through the respective pairs of AR glasses of the first player and the second player.

[0034] In one embodiment, dynamic scaling of the real-world object is performed by adjusting the optics of a first pair of AR glasses and a second pair of AR glasses through which the first and second players view the first and second real-world spaces.

[0035] In one embodiment, the optical system is adjusted according to the visual characteristics of a user viewing the real-world space through a pair of AR glasses.

[0036] In one embodiment, the dynamic scaling includes scaling gameplay content of the video game to match the scaling of real-world objects from the first and second real world spaces viewed by the first and second players, and the dynamic scaling of the gameplay content is performed prior to rendering it as an overlay on top of the real-world objects in the first and second real world spaces.

[0037] In one embodiment, the virtual objects of the gameplay content are presented by dynamically filtering one or more of the virtual objects of the gameplay content before rendering them as an overlay, with an amount of filtering defined to prevent clutter on the display screens of the first and second pairs of AR glasses worn by the first and second players.

[0038] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.

[0039] The disclosure may be best understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0040] [Figure 1A] 1 illustrates an exemplary cloud gaming system used to access a video game and provide gameplay content along with a real-world view in which the video game is viewed through a pair of AR glasses, in accordance with an embodiment of the present disclosure. [Figure 1B] FIG. 1 illustrates a simplified block diagram of different sub-modules of a content scaling module used to normalize an image viewed through a pair of augmented reality (AR) glasses worn by a user, according to one embodiment of the present disclosure. [Figure 1C] FIG. 1 illustrates a simplified block diagram of a dynamic scaling engine, according to one embodiment of the present disclosure, that is used to dynamically scale a real-world space viewed through a pair of AR glasses worn by a user as part of normalizing a view between the real-world space and a virtual environment provided by a video game. [Figure 2A]1 illustrates a display screen of a pair of augmented reality glasses worn by a user that renders an augmented reality image in which the size of real-world objects is rendered disproportionately to virtual objects from the content of an interactive application, according to one embodiment. [Figure 2B] 1 illustrates a display screen of a pair of augmented reality glasses worn by a user that renders an augmented reality image in which the size of real-world objects is scaled to match the scale of virtual objects from the content of an interactive application, according to one embodiment of the present disclosure. [Figure 3A] 1 illustrates a display screen of a pair of augmented reality glasses worn by a user that renders an augmented reality image in which the size of real-world objects is rendered disproportionately to virtual objects from the content of an interactive application, according to one embodiment. [Figure 3B] 1 illustrates a display screen of a pair of augmented reality glasses worn by a user that renders an augmented reality image in which the size of virtual objects from the content of an interactive application are dynamically scaled to match the scale of real-world objects, according to one embodiment of the present disclosure. [Figure 4A] 1 illustrates an image of a first real world space near a first player as viewed through a first pair of AR glasses worn by the first player, and an image of a second real world space near a second player as viewed through a second pair of AR glasses worn by a second player, in which the scale of real-world objects in the first real world space does not match the scale of real-world objects in the second real world space, according to one embodiment of the present disclosure. [Figure 4B]1 illustrates an image of a first real world space near a first player viewed through a first pair of AR glasses and an image of a second real world space near a second player viewed through a second pair of AR glasses, where the view is normalized by adjusting the scale of a first set of real world objects in the first real world space to match the scale of a second set of real world objects in the second real world space, according to one embodiment of the present disclosure. [Figure 5A] 1 illustrates a flow operation of a method for normalizing a view of a real-world object in a real-world space viewed through a pair of AR glasses to a virtual object included in content of an interactive application according to one embodiment of the present disclosure. [Figure 5B] 1 illustrates flow operations of a method for normalizing a view of a real-world object in a real-world space viewed through a pair of AR glasses to a virtual object included in content of an interactive application according to an alternative embodiment of the present disclosure. [Figure 5C] 1 illustrates flow operations of a method for normalizing views of a first set of real-world objects in a first real world space to views of a second set of real-world objects in a second real world space as viewed through a first pair of AR glasses and a second pair of AR glasses for a first player and a second player, and rendering virtual objects from content of an interactive application as an overlay, according to an example embodiment of the present disclosure. [Figure 6] 1 illustrates an exemplary system used to load interactive applications (e.g., video games) available to a cloud gaming site, according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a diagram of an embodiment of a client device. [Figure 8] FIG. 1B illustrates components of a head-mounted display (HMD), an example of a client device of FIG. 1A. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] In the following detailed description, several specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without some or all of these specific details. In other instances, well-known process steps have not been described in detail in order not to obscure the present disclosure.

[0042] As more media content becomes available online, users have the ability to access a variety of media content and become fully immersed in viewing and interacting with the media content. These users can view the media content even while viewing the real-world space using a pair of augmented reality (AR) glasses. The AR glasses allow the media content to be presented alongside the real-world space, helping the user to continue experiencing the real-world space while keeping track of the media content. The media content may be streaming media content generated from an interactive application with which the user is interacting or made available to the user by a media content provider or other user. Streaming media content may include content of live private events (e.g., birthdays, anniversaries, travel, etc.) or live public events (e.g., live games, live gameplay of video games, live concerts, live debates, etc.) provided by a media content provider or user, or pre-recorded media content of past events (e.g., lectures, public or private events, video games, etc.) provided by a content provider or user, or pre-recorded media content (e.g., TV shows, movies, etc.) available for viewing and sharing. The user's interaction experience can be further enhanced by interacting with interactive applications such as video games, either individually or with two or more players playing together, or individually or in groups, and online access allows these users to participate in such interactions from the confines of their own space, which may be co-located or remotely located. User interaction may be in the form of input into the interactive application or commentary related to the media content the user is viewing or related to other users.

[0043] In the case of video games, users use their own computing devices to view and interact. These computing devices may be customized according to the preferences of each user. Customization may occur at the viewing level, such as customized display screens (e.g., large and small screens, multiple screens or single screens, etc.), or at the spatial level (e.g., customized real estate space, etc.), in addition to the interactive input level (e.g., customized controllers, customized input buttons, customized controls, etc.). The display screen may be part of a wearable computing device, such as a head mounted display (HMD) or smart glasses (e.g., AR glasses) used to view virtual or augmented reality (AR) content. The customized view and / or customized space in which a user interacts with an interactive application, such as a video game, may give the user an unfair advantage over other users. For example, in a gaming environment in which a first player may be pitted against a second player, if the first player has a display screen of a particular size or resolution for viewing the media content (e.g., virtual or AR content) of the video game and / or real-world objects in the real-world space, the first player may have an unfair advantage over the second player who may not have an expanded display screen for viewing the same media content. The enhanced display screen allows the first player to notice and appropriately interact with finer details in the media content or real-world space viewed through the pair of AR glasses, for example, while the second player may not have that advantage. Similarly, the first player may move around a larger real-world space (e.g., a large game room) while interacting with the video game, while the second player may move around a smaller real-world space (e.g., a dorm room). Due to a mismatch in the size of the real-world spaces of the first and second players, the view of the real-world space viewed through the pair of AR glasses of the first player may be larger than the view of the real-world space viewed through the pair of AR glasses of the second player. As a result, the first player may be able to see more details of the real-world space than the second player, and the first player may have an unfair advantage over the second player.

[0044] To eliminate an unfair advantage of a first player over a second player and to normalize the real-world space and / or AR content viewed through a pair of AR glasses, e.g., a first player and a second player, the system is designed to normalize the view of real-world objects from the real-world space viewed through the pair of AR glasses of the first player and the second player as well as virtual objects included in the media content of a video game (i.e., an interactive application). In some embodiments, the real-world space in the vicinity of two remotely located players is normalized. For example, if a first player's living room is shared with a second player's living room, the space and dimensions of each player's living room are normalized relative to one another, so that a video game played between two players in a shared real-world space using AR content is equally advantageous to both players. This type of normalization levels the playing field for both players by avoiding problems associated with one player having an unfair advantage over the other player based on the real-world setting or positioning in the real-world setting. In addition to normalizing the views of the two players, the system may also normalize virtual objects included in the media content (of the video game) that need to be placed over real-world objects in the real-world space viewed through the AR glasses pair. Normalization may be performed by scaling the virtual object to match the scale of the real-world object in the real-world space, or by scaling the real-world object to match the scale of the virtual object viewed through the AR glasses pair. This type of normalization avoids one user having an unfair advantage over another user by having a special size or characteristic defined for the game or virtual content in the real-world space viewed through the AR glasses pair, etc. (e.g., HMD). Instead of or in addition to normalizing the view of the virtual and / or real-world objects seen through the AR glasses pair, the system may also scale the sounds emanating from or received by the virtual content, so that the sounds are normalized for interactive applications within the context of the game content or virtual object viewed through the AR glasses pair. The normalization may be based on suggestions from the content scaling module, the video game (i.e., the interactive application), or a server running a machine learning algorithm. These suggestions, when provided as feedback, may take into account the user's current position in the real-world space and may provide suggestions to best position the user in the real-world space, to best position a character associated with the user in a video game, or to best position the user in the real-world space relative to the virtual content. Normalizing the views between the virtual content and the real-world space, and between the real-world space shared between the two users, makes the games played or activities performed by the two users in the shared real-world space comparable.

[0045] Normalization helps ensure that the blending of the virtual and real worlds is accurate, for example, for a user during gameplay of a video game, and for multiple users who share their real-world space with other users. Additionally, normalization of the virtual and real worlds helps provide a normalized interaction experience for two users playing a video game (i.e., an interactive application) from two different environments by providing an equal playing field (i.e., equal challenge) for the two users. Scanners and / or other sensors may be used to scan the real-world environment. The data provided by the scanners and other sensors may be analyzed to determine the layout of a real-world space in which a user is interacting with an interactive application (e.g., a video game) and to determine attributes of real-world objects within the real-world space. These attributes are used to determine the scale of the real-world object viewed through the AR glasses pair. Based on the attributes, a projector is used to either scale the virtual object to match the scale of the real-world object or scale the real-world object to match the scale of the virtual object before presenting the virtual object as an overlay on top of the real-world object viewed through the AR glasses pair.

[0046] In some embodiments, a dynamic scaling engine of the content scaling module may be used to generate a signal to the projector to scale the virtual object and then render it as an overlay on top of the real-world object viewed through the AR glass pair. In some embodiments, a machine learning algorithm may be used to determine the layout and attributes of the real-world object identified in the real-world space. In one embodiment, in response to receiving the content, the dynamic scaling engine may send a first signal to a scanner, image capture device, and sensor available in or near the AR glass pair or a computing device communicatively connected to the AR glass pair to capture an image of the real-world space in which the user is present. Once the image is captured by the image capture device and sensor, it is forwarded to a machine learning algorithm that analyzes the captured image to identify the layout and attributes of the real-world object. The layout and attributes of the real-world objects are then used to determine a scale of the real-world objects. The dynamic scaling engine may generate a second signal to adjust the scale of the virtual objects from the content based on the scale of the real-world objects to match the scale of the real-world objects determined from analyzing the images of the real-world space captured by the image capture device and sensor. In some embodiments, one or more real-world objects may be selected and introduced as virtual objects along with the virtual objects of the content of the interactive application. The virtual objects thus introduced may be scaled to match the scale of the real-world objects and virtual objects viewed through the pair of AR glasses prior to introduction.

[0047] With a general understanding of the present disclosure, specific embodiments will now be described with reference to various drawings.

[0048] 1A illustrates a simplified block diagram of a system used in one embodiment to normalize a view between a virtual object viewed through a pair of wearable computing devices, such as a pair of AR glasses, and a real-world object in a real-world space. Note that various embodiments are described by referring to the pair of AR glasses as the wearable computing device used to view real-world objects in the real-world space and virtual objects of an interactive application. However, the embodiments may be extended to include other wearable computing devices, such as a head-mounted display (HMD).

[0049] The system includes multiple client devices 100 operated by multiple users accessing a cloud gaming site 104 via a network 102, such as the Internet. The users may be co-located or remotely located and may access the cloud gaming site 104 via the Internet from one or more geographic locations. The client devices 100 are communicatively connected to the cloud gaming site by wired or wireless means. The cloud gaming site 104 may include multiple servers distributed across multiple data centers. The multiple servers host one or more interactive applications that authenticate users, instantiate one or more interactive applications that provide the media content required to render on the user's client device 100, and provide the resources required to normalize the view between the real world content and the virtual content included in the media content when rendered on the display screen of the client device (e.g., HMD, AR glasses, etc.).

[0050] For example, server 104a of cloud gaming site 104 may be used to perform user authentication before providing client device 100 with access to an interactive application. Server 104b may include one or more consoles (e.g., game consoles, not shown), each of which may be configured to run one or more instances of one or more video games or interactive applications. Alternatively, server 104b may be configured to run one or more instances of one or more interactive applications. Server 104b may be a single server or may represent multiple servers, each of which may provide the resources necessary to run one or more instances of the interactive application. Alternatively, each of the multiple servers may provide specific resource(s) for running the interactive application, and one of the multiple servers may coordinate the resources for successful execution of the interactive application. One or more servers 104c of the cloud gaming site 104 may run an instance of a content scaling module that is used to normalize the view between real-world objects in a real-world space and virtual objects included in the media content of an interactive application viewed through the user's pair of AR glasses. In one embodiment, the user's client device 100 may be a separate computing device used to provide input to the interactive application, and the media content may be provided to the client device 100 for viewing through the user's pair of AR glasses. In this embodiment, each pair of AR glasses may be communicatively connected via a wired or wireless connection to a user's respective client device 100, which is located locally in or near the user's real-world space and communicatively connected to the cloud gaming site 104 via a network 102 (such as the Internet). In alternative embodiments, the pair of AR glasses may themselves be a client device 100 used to provide input to an interactive application running on one or more servers of the cloud gaming site 104. In these and other embodiments, the pair of AR glasses is communicatively connected to a server on the cloud gaming site 104 via the Internet.

[0051] It should be noted that although different servers (104a-c) are shown performing different services, all or a combination of the services (authentication, running interactive applications, and running a content scaling module) may be performed by a single server. User authentication server 104a may interact with user data database 108 to access user accounts 106 contained therein to authenticate users of client devices 100 requesting access to video games running on server 104b. User accounts 106 may include biometric data and other authentication data used to authenticate users. Upon successful authentication of the user, the user is provided with access to select a video game for gameplay or to view gameplay.

[0052] Upon successful user authentication of the user, and in response to the user's request to select a video game for gameplay, server 104b may access game titles 110 stored in game database 112 to authenticate the user's request. Game database 112 maintains a list of game titles 110 available to the cloud gaming site, and a list of games that each user is eligible to access, either through a paid membership / subscription, or through access granted by the game developer or game host. Once the access request is authenticated by server 104b, the user is provided with access to an instance of the video game by executing the game code of the video game on server 104b. In this case, server 104b may be a standalone server, capable of storing game code for multiple video games and executing instances of the video games (i.e., having resource functionality) in response to requests. Alternatively, server 104b may access and obtain game code from a video game stored on another server during gameplay of the video game, instantiate the video game using resources of server 104b, and generate a video content stream (i.e., gameplay data). The generated gameplay data is compressed using compression techniques of an encoder available to server 104b and transmitted as streaming frames of game data over network 200 to client device 100.

[0053] In one embodiment, the video game may be a multiplayer video game accessed for gameplay by multiple users, and the multiple users may be distributed across multiple geographic locations. In this embodiment, the game code of the video game may be instantiated on multiple servers (e.g., cloud servers) 104b that are distributed across multiple data centers. In some embodiments, the data centers where the video game may be instantiated may be identified at the geographic location of one or more users, or at least near one or more users, and may provide gameplay data with minimal latency. Once instantiated on the multiple cloud servers 104b, the gameplay data generated for the video game may be synchronized between the multiple cloud servers 104b and transferred as media content to the user's client device 100. In other embodiments, the server 104b may be configured to manage one or more virtual machines capable of running instances of a video game and providing a gameplay data stream, where the streaming may occur in real time or with a delay. Other types of cloud servers may include blade servers. Input provided by users through their client devices 100 is used to update the game state of the video game and generate updated gameplay data. The updated gameplay data is streamed to the user as frames of gameplay data.

[0054] A video game engine executing on server 104b (e.g., a cloud server or a standalone server or a game console) is communicatively connected to the game logic of the video game and provides a framework for the video game. The game engine is a software layer that serves as the foundation for the video game and provides the basic infrastructure (i.e., framework) for developing the video game. The game engine abstracts the details of performing common related tasks (i.e., game engine tasks) required for every video game, while the game code of the video game provides the game logic that details how the video game should be played. The game code of the video game uses modules contained in the game engine to control virtual objects including virtual characters and virtual scenes in the video game and generate gameplay data. Some of the basic core modules of the game engine used by the game code of any video game include a physics engine (for object movement based on collision detection, collision response, trajectory, gravity, friction, etc.), a rendering engine (i.e., renderer) for 2D and 3D graphics, a sound engine, scripting, animation, artificial intelligence, threading, networking, streaming, memory management, scene graphics, etc. These basic core modules of the game engine can be reused by various video games and generate gameplay data based on the game state of the video game. The generated gameplay data includes animations provided by various virtual characters, where the animations are based on the context of the video game used to generate the gameplay data.

[0055] The user's client device 100, communicatively connected to the server 104b via the network 102, receives the frames of gameplay data, decompresses the gameplay data, and renders it on a display screen associated with the client device. The display screen of each user's client device may be configured to view the real-world space while having the ability to interact with the gameplay data. Each client device (100) has a processor, memory, and communication capabilities, and may be portable or non-portable, to access the network 102, such as using wired, wireless, or 4G / 5G communications. The client device 100 may run an operating system and include a network interface, or may be a thin client with a network interface to access the network (Internet) 102 and communicate with the server 104b, which provides the computing capabilities. The network 102 may be a 3G, 4G, or 5G network.

[0056] The client device 100 with 5G communication capabilities can be connected to a 5G network. In one embodiment, the 5G network is a digital cellular network with a coverage area divided into multiple "cells" (i.e., small geographic areas). Analog data generated by the mobile device is digitized and transmitted as radio waves to a local antenna in the cell using frequency channels that can be reused in geographically separate cells. The local antenna is connected to the Internet and telephone networks by high-bandwidth optical fiber or other similar wireless communication. Because the 5G network uses high-frequency radio waves for communication, data can be transmitted at a higher data rate, resulting in lower network latency.

[0057] In other embodiments, the media content may be provided by a media content provider hosted on a server separate from server 104b used to store the game code and run the video game. The media content may relate to a live event (e.g., a live game, a live concert, a live debate, a live video capture of an event, etc.) or a recorded event and may be streamed or otherwise provided to client device 100 for rendering. The media content may be generated by a media content provider and transmitted to the client device, or may be generated by another user and transmitted to the user's client device using resources available to the media content provider. Similar to gameplay data for a video game, a display screen associated with a user's client device that renders the media content provided by a media content provider may be configured to provide an augmented reality experience to the user by allowing the user to view a real-world space in the user's vicinity while rendering the media content along with a view of real-world objects in the real-world space.

[0058] The display screen may be part of a pair of AR glasses worn by the user, and the client device may be a computing device communicatively connected to the pair of AR glasses or to each user's pair of AR glasses. The media content rendered along with the real-world object may be dynamically scaled to match the scale of the real-world object using a content scaling module executing on the server 104c, or alternatively, the real-world object viewed through the pair of AR glasses may be dynamically scaled to match the scale of the virtual object included in the media content before the media content is rendered along with the real-world object in the real-world space. The dynamic scaling provides a normalized view between the real-world object and the virtual object included in the media content.

[0059] 1B shows a simplified block diagram of various sub-modules included in a content scaling module 120 that, in one embodiment, is used to normalize the view between real-world objects and virtual objects of media content viewed through a pair of AR-enabled glasses. The content scaling module 120 is shown running on a server 104c. However, the content scaling module may run on any server of the cloud gaming site, including the server 104a used to authenticate users, or the server 104b running an instance of a video game. Some of the sub-modules included in the content scaling module 120 include a real-world scene identification engine 121, an interactive application content reception engine 122, a dynamic scaling engine 123, a content frame generation engine 127, and an image encoder 128. The identified sub-modules are provided by way of example only and should not be considered limiting. The content scaling module 120 may include fewer or additional sub-modules. In some embodiments, the content scaling module 120 may be executed on a computing device local to the AR glasses pair or by a processor available to the AR glasses pair.

[0060] The real-world scene identification engine 121 is configured to obtain details of real-world objects contained within a real-world space in the vicinity of the user viewed through the pair of AR glasses. The details may be obtained using an image capture device, scanner, sensor available on the pair of AR glasses, on a computing device or controller communicatively connected to the pair of AR glasses or distributed in the real-world space. The details of the real-world objects are processed to determine the scale of the real-world objects. The visual characteristics of the first user may be different from the visual characteristics of the second user. As a result, the scale of the real-world object viewed through the lenses of a first pair of AR glasses worn by the first user may be the actual scale of the real-world object, and the scale of the real-world object viewed through the lenses of a second pair of AR glasses worn by the second user may be adjusted according to the visual characteristics of the second user. Once processed, the information related to the scale of the real-world object is provided to the virtual object scaling engine 124. The virtual object scaling engine 124 uses the information to adjust the scaling of the virtual object provided to the pair of AR glasses for rendering. Details of the processing of the real-world object viewed through the pair of AR glasses are described in more detail with reference to FIG. 1C.

[0061] The interactive application content receiving engine 122 of the content scaling module 120 receives content from an interactive application, such as a video game, generated by the interactive application execution engine. The content includes gameplay data that is updated in response to input provided by a user / player through the respective client device(s) 100. The input from the user / player is used to adjust the game state of the video game and update the content included in the frames of gameplay data. The frames of gameplay data transmitted by the interactive application are processed by an interactive application content reception engine 122 to determine the type of content to receive (e.g., streaming or non-streaming content, gameplay content or other interactive application, live or recorded content, etc.), the amount of content, the types of virtual objects contained therein, the positions and orientations of the virtual objects, the depth of the virtual objects, etc. The processed content of the interactive application (i.e., the frames of gameplay data in the case of a video game) is provided to a dynamic scaling engine 123 for further processing.

[0062] The dynamic scaling engine 123 receives processed input of real-world objects captured from the real-world space and virtual objects included in the content of the interactive application and dynamically scales the virtual objects of the content or the real-world objects in the real-world space before making them available for viewing through the AR glasses pair. In one embodiment shown in FIG. 1B, the dynamic scaling engine 123 cooperates with the virtual object scaling engine 124 to dynamically scale the virtual objects of the content being streamed by the video game application such that the scale of the virtual objects matches the scale of the real-world objects. In this embodiment, the dynamic scaling engine 123 uses the processed data obtained from the real-world scene identification engine 121 to determine the scale of real-world objects viewed through the AR glasses pair, and uses the processed data of the real-world objects to determine the scale at which virtual objects in content being streamed by the video game (i.e., interactive application) need to be dynamically scaled before transferring this content to the client device for rendering. Details related to the scaling of the virtual objects are provided to the projection engine 137. The projection engine 137 uses the details to adjust the scaling of images of virtual objects of the content included in the frame and forwards the frame of the content with the virtual objects adjusted for scaling to the image encoder 128. If and when new content is streamed from the video game, the projection engine 137 dynamically scales the virtual objects included in the new content according to the scaling information provided by the dynamic scaling engine 123. The frames of the scaled content are configured as an overlay and forwarded to the image encoder 128 where they are encoded using available encoding techniques. The encoded frames of the scaled content are forwarded to the client device and rendered as an overlay on top of the real-world objects in the real-world space viewed through the AR glasses pair. The scale of the virtual objects of the content rendered in the overlay matches the scale of the real-world objects viewed through the AR glasses pair. In the embodiment shown in FIG. 1B, the virtual object is scaled according to the scale of the real-world object viewed through the AR glasses pair. Instead of dynamically scaling the virtual object of the video game content, in an alternative embodiment, the real-world object viewed through the AR glasses pair may be adjusted to match the scale of the virtual object. In one embodiment, the virtual object in the overlay may be highlighted or provided with other visual indicators to inform the user of the presence of the virtual object.

[0063] FIG. 1C illustrates an alternative embodiment in which the dynamic scaling engine 123 cooperates with the real-world object scaling engine 125 to dynamically adjust the scaling of real-world objects in real-world space viewed through the AR glasses pair to match the scale of virtual objects included in the video game content. In this embodiment, the dynamic scaling engine 123 can obtain details of the scale of virtual objects included in the video game content and dynamically adjust the scaling of the real-world objects to match the scale of the virtual objects. In one embodiment, the real-world object scaling engine 125 cooperates with the optics adjustment engine 126 to adjust optical properties of the lenses of the AR glasses pair so that the real-world objects can be viewed through the AR glasses pair at a scale that matches the scale of the virtual objects included in the video game content. In one embodiment, the optical properties of the lenses may be adjusted based on the visual properties of the user. The user's visual property data 129 may be stored in the user data database 108 and may be made available to the optics adjustment engine 126 based on user credentials provided during user authentication.

[0064] 1C illustrates additional sub-modules included within the dynamic scaling engine 123 that can be used to scale real-world objects in the real-world space viewed through the AR glasses pair. The purpose of scaling the real-world objects is to normalize the view between the real-world objects viewed through the AR glasses pair and the virtual objects included in the content of the video game (i.e., interactive application). The dynamic scaling engine 123 may utilize machine learning algorithms 135 to determine the amount of scaling that needs to be done to the real-world objects or virtual objects included in the content of the interactive application, and then render the virtual objects as an overlay on the real-world space viewed through the AR glasses pair. The virtual object scaling engine 124 is engaged when virtual objects included in an interactive application need to be dynamically scaled to match the scale of real-world objects in a real-world scene viewed through the AR glasses pair. Similarly, the real-world object scaling engine 125 is engaged when real-world objects need to be dynamically scaled to match the scale of virtual objects in the video game content before presenting the virtual objects on top of the scaled real-world objects. In one embodiment, the dynamic scaling of real-world objects may be performed using the optics adjustment engine 126.

[0065] In one embodiment, when a real-world object needs to be scaled, the dynamic scaling engine 123 sends a signal to the real-world object scaling engine 125 to initiate a scan of the real-world space in which the user is present and interacting with the interactive application. In response to the signal from the dynamic scaling engine 123, the real-world object scaling engine 125 activates one or more image capture devices and / or scanners (132), one or more sensors 133 including motion sensors, etc., to capture images and data related to the real-world space in the vicinity of the user. The captured images are processed to identify data related to the real-world space. The data related to the real-world space is analyzed to identify a layout of the real-world objects in the real-world space and attributes of the real-world objects. One or more attributes of a real-world object identified from analysis of image data obtained from the image capture device (e.g., camera) and scanner 132 may include the geometry of the real-world space in the vicinity of the user in which the real-world object is located when viewed through the AR glasses pair, the boundaries of the real-world space, the position of the real-world object within the real-world space, the identity of the real-world object, the position, direction and orientation of the real-world object relative to other real-world objects and relative to the user, the size of the real-world object, the rendering distance of the real-world object, etc. Data provided by various sensors (including motion sensors) may be used to identify the amount of space available between the user and various real-world objects and between the real-world objects, the depth of the real-world object viewed through the AR glasses pair, etc. The amount of space between the real-world object and the user may also vary based on the user's movement in the real-world space as the user interacts with the interactive application, and the sensor data captured by the sensor 133 may be used to detect changes in the user's position and orientation in the real-world space relative to the real-world object and calculate the space between the real-world object and the user. Attributes of the real-world object are used to determine the scale of the real-world object viewed through the AR glasses pair. The attributes and other data related to the real-world object are provided as input to the machine learning algorithm 135.

[0066] To assist in determining the scale of the real-world objects, the dynamic scaling engine 123 cooperates with a virtual object scaling engine 124. The virtual object scaling engine 124 analyzes content from an interactive application to determine various attributes of virtual objects included in the content. In a video game application, the virtual objects are part of a virtual scene in the game environment, and various attributes of the virtual objects determined from the analysis of the content include the type of virtual object, the number of virtual objects of each type, the identity of the virtual object, the position and orientation of the virtual object relative to the virtual character representing the user (if applicable) and other virtual characters (either representing other users or available in the game environment), the position and orientation of the virtual object relative to other virtual objects, the depth of the virtual object, etc. The attributes of the virtual object are used to determine the scale of the virtual object included in the virtual scene of the content. The attributes and other data related to the content are provided as input to a machine learning algorithm 135.

[0067] The layout generation engine 134 is configured to receive and analyze data from the image capture devices and / or scanners 132 and sensors 133 to determine a layout of the real-world space. In one embodiment, the layout generation engine 134 is a separate module within the scan engine 131. In alternative embodiments, the layout generation engine 134 may be part of the scan engine 131. Data from the scan engine 131 is provided as input to the machine learning algorithms 135, which includes the image capture devices, images captured by the scanner, data from the sensors, and layout details from the layout generation engine 134.

[0068] The machine learning algorithm 135 receives attributes associated with virtual objects included in the content from the virtual object scaling engine 124, attributes associated with real-world objects from the scan engine 131, and layout data associated with the real-world space. The input is classified using a classifier 136, which may include one or more attributes of the real-world objects and / or one or more attributes of the virtual objects. The machine learning algorithm then uses the classifier 136 to build an artificial intelligence (AI) model. The AI ​​model includes a number of nodes and edges defined between successive pairs of nodes. Each node corresponds to one or more attributes of real-world and / or virtual objects, and the edges between any two successive nodes define the relationship between the attributes contained within the nodes. The machine learning algorithm 135 generates various outputs based on the content of the video game and various output goals in the real-world space defined for the user. For example, an output goal may be defined to ensure that the user does not collide with real-world objects. Another output goal may be to ensure that the user does not trip or fall. Another output goal may be to ensure that the user does not approach or cross boundaries of the real-world space, and to guide the user safely around real-world objects away from boundaries if the user approaches a boundary. Yet another output goal may be to determine scaling factors for various visual characteristics of the user. As additional content is received and additional attributes of real-world and virtual objects are identified, the AI ​​model is continually trained to fine-tune the output goals.

[0069] The dynamic scaling engine 123 cooperates with the optics adjustment engine 126 or the content frame generation engine 127 to adjust the scale of either the real-world content or the virtual content based on an output goal defined for or by the user. In one embodiment, the dynamic scaling engine 123 generates a signal to the optics adjustment engine 126 when a real-world object viewed through the AR glass pair needs to be dynamically scaled. In response to the signal from the dynamic scaling engine 123, the optics adjustment engine 126 may adjust the optics of the lenses of the AR glass pair such that the size of the real-world object viewed through the AR glass pair is scaled to match the scale of the virtual object of the interactive application.

[0070] The optical properties of the lenses of the AR glasses pair can be adjusted by the optical system adjustment engine 126 by generating an appropriate signal based on the type of lenses used. The AR glasses pair can be adjustable-focus glasses in that the focal length of the lenses can be varied by utilizing electro-optical or opto-mechanical methods. In an embodiment using an electro-optical enabled AR glasses pair, the lenses can use liquid crystal as a liquid medium, and the focal length of the lenses can be varied by changing the refraction of the liquid. By applying an appropriate electric potential, changes to the refractive properties of the liquid can be achieved. In another embodiment using an opto-mechanical enabled AR glasses pair, the lenses are made of a combination of flexible and rigid lenses, and a slider is provided to vary the focal length of the flexible lenses. In one embodiment, the slider may move using a signal provided by the optics adjustment engine 126. In another embodiment, the lens may perform hydraulic pressure against a diaphragm. In this embodiment, each lens is configured to enclose a reservoir of fluid silicone, and the focal length may be varied by adjusting the fluid level, which may be by either an electrical pulse or other means. The electrical pulse or other means for controlling the fluid level may be initiated using a signal(s) from the optics adjustment engine 126. The above-mentioned methods or lens designs for adjusting the focal length are provided merely as examples and should not be considered limiting. Other lens designs may also be employed to adjust the optical properties of the lens to enable dynamic scaling of real-world objects. Alternatively, other methods of adjusting the scale of a real-world object viewed through a pair of AR glasses may also be envisioned.

[0071] If adjusting the optical properties of the lenses, once the optical properties of the lenses for appropriately scaling the real-world space are determined, the optical property settings are transferred to the image encoder 128 for encoding. Along with the optical property settings, a rate of change of the optical property settings can also be provided to ensure that the scaling of the real-world object viewed through the entire display screen of the AR glasses pair does not have any disorienting effect on the user. Adjusting the optical property settings of the lenses is one way to dynamically scale the image of the real-world object viewed through the lenses of the AR glasses pair.

[0072] In an alternative embodiment, dynamic scaling of the real-world object is enabled by rendering the real-world object as a virtual object. In this embodiment, an image of the real-world object is captured, rescaled, and presented as a virtual object. Thus, when a real-world object viewed through the AR glasses pair needs to be dynamically scaled, the dynamic scaling engine 123 generates a signal to the real-world object scaling engine 125 to perform dynamic rescaling of the real-world object. The real-world object scaling engine 125 cooperates with the scan engine 131 to scan a real-world scene in the vicinity of the user, identify real-world objects contained therein, and capture images of the real-world objects. In response to the signal to rescale the real-world objects, the scan engine 131 generates a signal to apply an occlusion mask to the lenses of the AR glasses pair to prevent incoming real-world light from reaching the eyes. The occlusion mask transitions the lenses from a transparent mode to a non-transparent mode. At the same time, the scan engine 131 activates one or more image capture devices / scanners 132 to capture images of the real-world objects. The one or more image capture devices / scanners 132 are disposed on the surface of the AR glasses pair and / or in the real-world scene and are connected either directly to the AR glasses pair or through a console / computing device using a wired or wireless connection.

[0073] In one embodiment, the scan engine 131 analyzes the captured image to identify real-world objects in the real-world scene. Once a real-world object is identified, the scan engine 131 selectively rescales some of the real-world objects to match the scale of the virtual object viewed through the AR glasses pair, leaving other real-world objects in the captured image at their original scale. In an alternative embodiment, the scan engine 131 may rescale all real-world objects in the captured image to match the scale of the virtual object viewed through the glasses pair.

[0074] The scan engine 131 then forwards the rescaled real-world objects to the layout generation engine 134. As described above, the layout generation engine 134 determines the layout of the real-world objects in the real-world space / scene and provides the layout information and the rescaled real-world objects to the machine learning algorithm 135. Based on the goals defined for the real-world and virtual objects, the machine learning algorithm may use the images and layout information of the real-world and virtual objects to identify the content that needs to be included in each frame of content. The content and layout details from the machine learning algorithm are provided to the content frame generation engine 127, which generates the frames of content.

[0075] The content frame generation engine 127 receives inputs of scaled virtual objects of the content from the virtual object scaling engine 124 or unscaled virtual objects of the content from the interactive application content receiving engine 122. Additionally, the content frame generation engine 127 also receives as inputs images of scaled real-world objects, if available. These inputs are used to generate a frame of the content. The frame of the content may include images of dynamically scaled virtual objects or original scaled virtual objects of the content (i.e., unscaled content), if available, and / or images of dynamically scaled real-world objects. The frame of the content is provided to an image encoder 128.

[0076] The image encoder 128 encodes frames of content (scaled or unscaled) and data related to the optical property settings of the lenses. The encoded content and optical property settings (if available) are transferred to the user's client device for rendering, which may be a pair of AR glasses or a computing device communicatively connected to a pair of AR glasses.

[0077] The client device receives the encoded data and decodes the data using a decoder available to the client device. The decoded frames of content are provided to the pair of AR glasses and rendered as an overlay on top of real-world objects viewed through the pair of AR glasses. In addition to rendering the decoded frames of content, lens optical properties (if available) are also applied to the lenses of the pair of AR glasses such that real-world objects viewed through the adjusted lenses are scaled to match the scale of the virtual objects rendered as an overlay. As mentioned above, the optical properties applied to the lenses take into account the visual properties of the user. In some embodiments, one or more of the virtual objects included in the content of the video game are filtered before the frame of the content is generated and transferred to the client device, where the content is decoded and rendered. The filtering can be performed to minimize the amount of virtual objects rendered as an overlay, so that the content viewed through the AR glasses pair (real-world objects and virtual objects from the real-world space) is not too cluttered, so that the user is not confused or overwhelmed.

[0078] In some embodiments, in addition to scaling the virtual content (i.e., virtual objects of the content generated by the interactive application), sounds originating from or received by the virtual content can also be scaled to normalize the sounds occurring in the real and virtual worlds. The sound scaling can be based on the context of the interactive application (e.g., a video game) and the context of the real-world object being viewed. In one embodiment, the content scaling module can provide feedback to the user so that the user can place themselves in the real-world space or the context of the virtual content such that when viewed through the AR glasses pair, they can be properly positioned with respect to the virtual content that is added as an overlay on top of the real-world space. The feedback can be provided by a server running the content scaling module 120, or by the interactive application (e.g., a video game), or by the machine learning algorithm 135.

[0079] In an embodiment in which a video game is being played between two players, and the two players share their respective real-world spaces with each other, the real-world spaces of the two players may be normalized. For example, if a first player shares the first player's living room (i.e., a large space) with a second player who is playing a video game, and the second player shares the first player's dorm room (i.e., a small space), the spaces and dimensions of the first and second players may be normalized to be equally advantageous to each player during gameplay of the video game played within the shared real-world space. This type of normalization ensures that each player has an equal playing field and avoids giving one player with a large space an unfair advantage over another player with a limited space, i.e., avoiding an unfair advantage of a first player over a second player based on differences in real-world settings or positioning within different real-world settings. Such normalization allows two players from two different environments to have a normalized experience and comparable game space. Similar benefits can be achieved by adjusting the optical properties of the two players to allow the two players to have a normalized view of the video game content and real-world space.

[0080] In alternative embodiments, virtual objects included in the gameplay content may be dynamically scaled to match the scale of real-world objects in a vicinity of the user before overlaying the gameplay content on the real-world space viewed through the pair of AR glasses. In the above example of a video game played between two remotely located players, virtual objects in the gameplay content may be dynamically scaled to match the scale of a first set of real-world objects included in a first real-world space in a vicinity of the first player before overlaying the gameplay content on the first set of real-world objects viewed through the first pair of AR glasses of the first player. Similarly, virtual objects included in the gameplay content may be dynamically scaled to match the scale of the second set of real-world objects included in a second real-world space proximate the second player before overlaying the gameplay content on top of the second set of real-world objects viewed through the second pair of AR glasses for the second player. The dynamic scaling may be performed in response to input from the first player or the second player, or in response to a signal from a content scaling module or from a machine learning algorithm performed by the content scaling module or by a server executing the content scaling module. In some embodiments, in a video game application providing gameplay content, real-world objects from a vicinity of a first player and / or a second player may be selected for inclusion as virtual objects that can be rendered along with virtual objects of the gameplay content of the video game. The real-world objects for inclusion may be selected by the first player or the second player, or by both the first player and the second player. In an embodiment in which real-world objects are selected by only the first player or the second player, an image of a virtual object representing the selected real-world object may be included in the view of only the player who selected the real-world object (i.e., only the first player or the second player who made the selection). In other embodiments, real-world objects selected for inclusion as virtual objects by either the first player or the second player may be shared with the other player, in which case the virtual objects representing the real-world objects selected for inclusion by the first player or the second player are presented along with the virtual objects of the video game content in an overlay viewable through both the first and second pairs of AR glasses of the first and second players, respectively. The images of the real-world objects are dynamically scaled to match the scale of the virtual objects in the gameplay content being rendered in the first player's and / or second player's AR glasses pair, or in an overlay on top of the real-world objects viewed through the AR glasses pair. In some embodiments, a first real world object is selected by a first player from a first real world space proximate the first player for inclusion as a first virtual object. Concurrently, a second real world object is selected by a second player from a second real world space proximate the second player for inclusion as a second virtual object, and the first real world object and the second real world object are selected for sharing with each other. In these such embodiments, both the first and second real world objects are projected as the first and second virtual objects, respectively, and the projections are rendered and scaled to match the scale of the virtual objects of the gameplay content included in the overlay. The dynamic scaling includes scaling dimensions of the first virtual object and the second virtual object before being included in the overlay.

[0081] In one embodiment, the content scaling module 120 may be used to add one or more real-world objects from the user's real-world space as virtual objects viewed through the AR glasses pair. The virtual objects added from the real-world space are added to the virtual objects of the content that are overlaid on top of the real-world objects viewed by the user through the AR glasses pair. For example, a dog, cat, or child in the user's real-world space may be added as a virtual character along with the virtual objects of the content of a video game. Alternatively, from the above example of two players sharing a real-world space, one or more real-world objects in the first real-world space of the first player and one or more real-world objects in the second real-world space of the second player may be included with the virtual objects of the content being rendered on top of the real-world objects of the first and second players. This type of normalization of adding content from each other's real-world spaces helps to provide an accurate blend between the virtual and real-world spaces of the two players. Those skilled in the art may derive other advantages from the various figures and descriptions provided herein.

[0082] 2A illustrates, in one embodiment, a rendering of a real-world space viewed through a pair of AR glasses onto which virtual objects from the content of a virtual game are overlaid, where neither the view of the real-world objects in the real-world space nor the virtual objects of the content are scaled for normalization. The real-world space shows multiple real-world characters in the background (e.g., under a tree) with virtual characters from the video game content rendered as an overlay in the foreground. However, the scale of the virtual characters does not match the scale of the real-world characters in the background.

[0083] Figure 2B shows a rendering of the real-world space of Figure 2A where, in one embodiment, the virtual character has been scaled to normalize the view of the real-world space containing the virtual object. In this embodiment, the virtual character is projected onto the real-world space as an overlay at a scale that matches the scale of the real-world character.

[0084] FIG. 3A illustrates, in one embodiment, a rendering of the real-world space as viewed through a user's pair of AR glasses, with the video game content's virtual objects projected onto the real-world space as an overlay, with neither the view of the real-world objects nor the content's virtual objects being normalized. FIG. 3B illustrates, in one embodiment, a rendering of the view of the real-world space scaled to match the scale of the content's virtual objects. In one embodiment, the view of the real-world space may be adjusted by adjusting the optical properties of the lenses to provide a normalized view of the real-world objects relative to the content's virtual objects. In this embodiment, the content's virtual objects are projected unchanged. However, dynamic scaling of the view of the real-world space ensures that the user is presented with a normalized view of the real-world space and the video game content projected onto the real-world space.

[0085] FIG. 4A illustrates, in one embodiment, a view of a first real world space near a first player as viewed through a first pair of AR glasses for the first player, and a view of a second real world space near a second player as viewed through a second pair of AR glasses for the second player. The first player is playing a video game with a second player. In the embodiment illustrated in FIG. 4A, the first real world space and the second real world space are not normalized to each other. As illustrated, the first player may play from a large living room space while the second player may play from a small dorm room space, thereby allowing the first player to have a spatial advantage over the second player. For example, to name a few examples, the first real world space includes a display screen of width "d1" on which the content of the video game is rendered, the first player is separated from the display screen by a space of length "L1", a large sofa is separated from a small sofa by a length "L2", the large sofa has a width "d2", and the small sofa is of size "S2". These dimensions can be determined by analyzing images captured from the first real-world space by scanners, image capture devices, sensors, etc., on the first AR glass pair, on a computing device communicatively connected to the first AR glass pair, or distributed within the first real-world space. Similarly, the second real-world space includes a display screen that is separated from the second player by a distance "L3", and the size of the display screen of the second player is represented by "d3". As can be seen, the space available to the first player within the first real-world space is larger than the space available to the second player within the second real-world space, so the first player has an undue advantage over the second player.

[0086] Figure 4B shows a normalized view of the first real-world space and the second real-world space as viewed through the first AR glass pair and the second AR glass pair, respectively. The normalization of the view is achieved by dynamically scaling the space and dimensions of the first set of real-world objects distributed within the first real-world space and viewed through the first AR glass pair of the first player to match the scaling of the view of the second set of real-world objects in the second real-world space of the second player viewed through the second AR glass pair. For example, as part of the normalization, the space between the first player and the display screen within the first real-world space viewed through the first AR glass pair can be adjusted to L1', which can be approximately equal to the separation distance L3 between the second player and the display screen within the second real-world space. Similarly, when viewed through the first AR glass pair, the length between the small sofa and the large sofa, represented as L2', can be dynamically adjusted such that L2' < L2. The size of the small sofa viewed through the first AR glass pair of the first player can be dynamically adjusted such that the image of the small sofa is S2' (S2' < S2). Additionally, the size of the display screen in the first real world space viewed through the first pair of AR glasses may be dynamically adjusted to d1′, which may be approximately equal to d3, the size of the display screen used by the second player in the second real world space. This dynamic scaling is done by adjusting the dimensions of real-world objects and the space between real-world objects viewed through each of the first and second pairs of AR glasses, normalizing the views of the first and second real world spaces to allow each player to have a level playing field, thereby avoiding an unfair advantage for one player over the other player.

[0087] It should be noted that the terms "engine" or "module" used to describe the various sub-modules of the content scaling module 120 refer, in one embodiment, to portions of a program or code configured to perform a particular function. The program or code may use deep learning algorithms to query and / or process certain pieces of data (e.g., input from a user, images of real-world space, images of media content, etc.). In the case of the dynamic scaling engine 123, a program may be used to analyze various data (virtual environment or real-world space) to determine various attributes of real-world and virtual objects, use the various attributes to determine the scale at which various objects (virtual or real-world objects) are rendered or viewed, and perform dynamic scaling of virtual objects or adjust the optical properties of the lenses of an AR glasses pair to normalize the view of the real-world and virtual environments. The content scaling module 120 may be implemented as software, hardware, or firmware.

[0088] 5A illustrates an operational flow of a method used in one embodiment to normalize a view of a real-world space and a virtual environment viewed through a pair of AR glasses. The method begins at operation 510 with receiving and rendering content of an interactive application on a display screen of a pair of AR glasses worn by a user to view a real-world space in the user's vicinity. The interactive application may be a video game, and the content may be gameplay content generated in response to input provided by the user during gameplay. The content may be streaming content that changes based on changes in a game state of the game, the game state being determined from the user's input.

[0089] In response to receiving the AR content (i.e., virtual content from the game play), as shown in operation 520, a scale of a real-world object viewed through the pair of AR glasses is determined, and a scale of a virtual object is dynamically adjusted to match the scale of the real-world object. The scale of the real-world object may be determined by capturing an image of the real-world object in the real-world space using the pair of AR glasses, image capture devices such as scanners, cameras, and sensors available to the computing device communicatively connected locally to the pair of AR glasses or distributed within the real-world space. The captured images may be analyzed to identify the layout of the real-world space, various real-world objects distributed in the real-world space, and various attributes of the real-world objects, including the type of real-world object and the number of each type, an estimate of the size, depth, position, orientation of the real-world objects relative to each other and to the user, and the like. The various attributes of the real-world object may be used to determine the scale of the real-world object viewed through the pair of AR glasses. Based on the scale of the real-world object determined from the analysis, the virtual object may be dynamically scaled such that the scale of the virtual object matches the scale of the real-world object.

[0090] The scaled virtual object is then provided as an overlay for rendering on top of the real-world object being viewed through the AR glass pair, as shown in operation 530. Dynamic scaling of the virtual object makes it possible to provide a normalized view of the real-world object and the virtual object when viewed through the AR glass pair.

[0091] FIG. 5B illustrates an operational flow of a method used in another embodiment to normalize a view of a real-world space and a virtual environment near a user viewed through a pair of AR glasses. The method begins with operation 540, where content of an interactive application is received and rendered on a pair of AR glasses. As described with reference to FIG. 5A, the interactive application may be a video game application that generates streaming gameplay content in response to input from a user. The gameplay content represents a virtual environment and includes virtual objects including virtual characters and virtual scenes. The pair of AR glasses is used to view a real-world space near the user while interacting with the gameplay content of the video game.

[0092] The content scaling module may determine a scale of the gameplay content being provided for rendering and a scale of the real-world object in response to receiving the gameplay content of the video game. Based on this determination, if the scale of the real-world object does not match the scale of the virtual object, the content scaling module may dynamically scale the real-world object viewed through the AR glasses pair, as shown in operation 550. Dynamic scaling is matching the scale of the real-world object to the scale of the virtual object included in the gameplay content of the video game. The dynamic scaling of the real-world object, in one embodiment, is by adjusting optical properties of the lenses of the AR glasses pair such that the scale of the real-world object viewed through the AR glasses pair matches the scale of the gameplay content. As described with reference to FIG. 1C, various ways of adjusting the optical properties of the lenses can be envisioned depending on the type of lenses used.

[0093] The virtual objects of the gameplay content are then projected as an overlay onto the scaled real-world objects in real-world space, as shown in operation 560. The scaling of the real-world objects ensures normalization of the views of the real-world objects and the virtual objects of the gameplay content. This type of normalization allows a user to have an even playing field when playing a video game with another user.

[0094] 5C illustrates an operational flow of a method for normalizing a view of a real-world space shared between two players and including a virtual object from an interactive application in yet another embodiment of the present invention. The method begins at operation 570, where gameplay content of a video game played between a first player and a second player is received. The first player plays and interacts with the video game and receives the gameplay content for rendering on a display screen associated with a first pair of AR glasses. Similarly, the second player plays and interacts with the video game and receives the gameplay content for rendering on a display screen associated with a second pair of AR glasses. The first player and the second player view real-world objects in the vicinity of the first player and the second player using the first pair of AR glasses and the second pair of AR glasses, respectively.

[0095] In operation 580, a first set of real-world objects in a first real world space proximate the first player are dynamically scaled to match the scale of a second set of real-world objects in a second real world space proximate the second player viewed through the first and second pair of AR glasses. The dynamic scaling of the real-world objects in the first and second real world spaces may be performed by adjusting optical properties of lenses of the first and / or second pair of AR glasses. The dynamic scaling allows for a normalized view of the first real world space viewed through the first pair of AR glasses for the first player and the second real world space viewed through the second pair of AR glasses for the second player.

[0096] In operation 590, virtual objects included in gameplay content of a video game played between the first player and the second player are projected as an overlay onto a first real-world space viewed by the first player through a first pair of AR glasses and a second real-world space viewed by the second player through a second pair of AR glasses. Dynamic scaling of the real-world spaces viewed by the first and second players allows for a normalized view of the real-world space shared between the two players, and further normalizes the view between the real-world space and the virtual environment of the video game when the gameplay content is rendered onto the respective real-world spaces. This method of providing a normalized view of the real-world space and the virtual environment allows users to enjoy the video game on an equal footing and avoids an unfair advantage that one user may have over the other user based on their respective real-world settings or their respective positioning within the real-world settings.

[0097] Other advantages will become apparent to those skilled in the art upon reading the various embodiments provided herein.

[0098] 6 illustrates an exemplary system used to load game files for games available through a cloud gaming site. The system includes a number of client devices 600 (100 in FIG. 1A) communicatively connected to a cloud gaming site 604 (104 in FIG. 1A) via a network 602 (102 in FIG. 1A), which may include a LAN, wired, wireless, cellular (e.g., 4G, 5G, etc.), or any other type of data network, including the Internet. When a request to access the cloud gaming site 604 is received from a client device 600, the cloud gaming site 604 accesses user account information 606 (106 in FIG. 1A) stored in a user data database 608 (108 in FIG. 1A) to identify the user associated with the client device from which the request is initiated. In some embodiments, the cloud gaming site may also verify the identified user to determine all games the user is authorized to view / play. Following user account identification / verification, the cloud gaming site accesses a game title data store 610 (110 in FIG. 1A) to identify game titles available at the game cloud site for the user account that initiated the request. The game title data store 610 then interacts with a game database 612 (112 in FIG. 1A) to obtain the game titles of all games available to the cloud gaming site. When a new game is introduced, the game database 612 is updated with the game code and game title information of the newly introduced game is provided to the game title data store 610. The client device from which the request is initiated may or may not be registered with the cloud gaming site when the request is initiated. If the user of the client device initiating the request is not a registered user, the cloud gaming site may identify the user as a new user and select an appropriate game title for the new user (e.g., a default set of game titles). As shown in FIG. 6, the identified game title is returned to the client device for presentation on the display screen 600-a.

[0099] A user interaction with one of the game titles rendered on the client device is detected and a signal is sent to the cloud gaming site. The signal includes game title information for which the user interaction was detected and the user interaction registered with the game title. In response to the signal received from the client device, the cloud gaming site proactively identifies a data center among multiple data centers (614-a to 614-d) where the game is hosted and sends a signal to the identified data center 614 to load the game associated with the game title for which the user interaction was detected. In some embodiments, multiple data centers may host the game. In such embodiments, the cloud gaming site may determine the geographic location of the client device originating the request, identify a data center geographically close to the client device, and send a signal to the data center to preload the game. The user's geographic location may be determined using a Global Positioning System (GPS) mechanism in the client device, the client's IP address, or the client's ping information, to name a few. Of course, the above-mentioned methods of detecting the geographic location of the user are exemplary, and other types of mechanisms or tools may be used to identify the geographic location of the user. Identifying a data center close to the client device can minimize latency when the user interacts with the game. In some embodiments, the identified data center may not have the necessary bandwidth / capacity / resources to host the game or may be overused. In these embodiments, the cloud gaming site may identify a second data center that is geographically closer to the client device. Loading the game includes loading the game code and running an instance of the game.

[0100] In response to receiving a signal from the cloud gaming site, the identified data center can select a server 616 from among multiple servers at the data center 614 and instantiate the game on the server. The server is selected based on available hardware / software processing power and game requirements. The server can include multiple game consoles 618, and the server can determine which of the multiple game consoles to use to load the game. The game consoles can resemble independent game consoles or can be rack-mounted or blade servers. The blade servers can in turn include multiple server blades, each blade having the circuitry necessary to instantiate a single dedicated interactive application, such as a video game. Of course, the aforementioned game consoles are exemplary and should not be considered limiting. Other types of game consoles, including game stations and the like, and other forms of blade servers may also be used to host the identified game.

[0101] Once the game console is identified, the general game-related code for the game is loaded onto the game console and a signal identifying the game console instantiating the game is sent back over the network to the client device via the cloud gaming site, such that the loaded game is made available to the user.

[0102] 7 is a diagram of an embodiment of a client device 700 (100 of FIG. 1A). The client device 700 includes several components, such as a network interface card (NIC) 702, a processor 704, a memory device 706, an audio memory device 708, one or more speakers 710, an audio video frame synchronizer 712, a video decoder system 714, an audio video frame separator 715, and a display device 718. The components of the client device 700 are coupled together via a bus 716.

[0103] An example of the audio video frame separator 715 is a processor or FPGA or PLD or controller or microprocessor or microcontroller or CPU. An example of the audio video frame synchronizer 712 is a processor or FPGA or PLD or controller or microprocessor or microcontroller or CPU.

[0104] The NIC 702 applies a communication protocol to the stream containing the encoded image frames and the encoded audio frames to parse the stream to obtain the encoded image frames and audio frames and transmits the encoded image frames and audio frames to an audio video frame separator 715 via bus 716.

[0105] The audio video frame separator 715 separates the encoded image frames from the audio frames, for example, the audio video frame separator 715 identifies the difference between the extension of the encoded image frames and the extension of the audio frames to distinguish the encoded image frames from the audio frames.

[0106] The audio video frame separator 715 sends the audio frames to the audio memory device 708 via bus 716 and sends the encoded image frames to the video decoder system 714 via bus 716. The audio frames are stored in the audio memory device 708.

[0107] The video decoder system 714 applies a decoding protocol to output decoded image frames based on the encoded image frames and sends the decoded image frames to an audio video frame synchronizer 712. The audio video frame synchronizer 712 accesses audio frames from the audio memory device 708 and synchronizes the audio frames with the decoded image frames. For example, the audio video frame synchronizer 712 matches the timing at which each of the decoded image frames is displayed on the display device 718 with the timing at which sound based on corresponding ones of the audio frames should be output by one or more speakers 710.

[0108] The audio video frame synchronizer 712 transmits the decoded image frames over a bus 716 to a display device 718, which displays the decoded images on a display screen of the display device 718. Concurrent with the display of the decoded images, one or more speakers 710 output sound based on the image frames in a synchronized manner.

[0109] The processor 704 controls one or more of the components of the client device 700. For example, the processor 704 executes an operating system that enables communication between components via the bus 716.

[0110] Referring to FIG. 8, a diagram showing components of the HMD 801 is shown. The HMD 801 includes a processor 800 for executing program instructions. A memory device 802 (memory device 706 in FIG. 7) is provided for storage purposes. Examples of the memory device 802 include volatile memory, non-volatile memory, or a combination thereof. A display device 804 is included to provide a visual interface for the user (FIG. 1A) to view, such as displaying image frames generated from the saved data. A battery 806 is provided as a power source for the HMD 801. A motion detection circuit 808 includes any of various types of motion sensing hardware, such as a magnetometer 810, an accelerometer 812, and a gyroscope 814.

[0111] Accelerometers are devices for measuring acceleration and reaction forces caused by gravity. Single-axis and multi-axis models are available to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 812 are used to provide the direction of gravity, which provides an absolute reference for two angles, such as world-space pitch and world-space roll.

[0112] The magnetometer measures the strength and direction of the magnetic field near the HMD 801. In some embodiments, three magnetometers 810 are used in the HMD 801 to ensure an absolute reference for the world space yaw angle. In various embodiments, the magnetometers are designed to span the Earth's magnetic field, which is ±80 microtesla. The magnetometers are affected by metal and provide a yaw measurement that is monotonic with the actual yaw. In some embodiments, the magnetic field is distorted due to metal in the real world environment, which causes distortion in the yaw measurement. In various embodiments, this distortion is calibrated out using information from other sensors, such as, for example, the gyroscope 814, the camera 816, etc. In one embodiment, the accelerometer 812 is used in conjunction with the magnetometer 810 to obtain the tilt and azimuth angles of the HMD 801.

[0113] A gyroscope is a device for measuring or maintaining orientation based on the principles of angular momentum. In one embodiment, instead of gyroscope 814, three gyroscopes provide information about movement across each axis (X, Y, and Z) based on inertial sensing. Gyroscopes are useful for detecting high speed rotations. However, in some embodiments, gyroscopes will drift over time in the absence of an absolute reference. This causes the gyroscope to be periodically reset, which can be performed using other available information, such as determining position / orientation based on visual tracking of objects, accelerometers, magnetometers, etc.

[0114] A camera 816 is provided to capture images and image streams of a real-world environment surrounding user A, e.g., a room, a cabin, a natural environment, etc. In various embodiments, multiple cameras are included in the HMD 801, including a rear-facing camera that faces away from user A, e.g., when user A is looking at a display of the HMD 801, and a front-facing camera that faces towards user A, e.g., when user A is looking at a display of the HMD 801, etc. Additionally, in some embodiments, a depth camera 818 is included in the HMD 801 to sense depth information of objects in the real-world environment.

[0115] The HMD 801 includes a speaker 820 for providing audio output and, in some embodiments, a microphone 822 for capturing audio from the real world environment, including sounds from the surrounding environment, speech made by user A, etc. The HMD 801 includes a haptic feedback circuit 824, such as a vibration device, for providing haptic feedback to user A. In one embodiment, the haptic feedback circuit 824 may generate movement and / or vibration of the HMD 801 to provide haptic feedback to user A.

[0116] An LED 826 is included as a visual indicator of the status of the HMD 801. For example, an LED may indicate battery level, power on, etc. A card reader 828 is included to allow the HMD 801 to read and write information from and to a memory card. A USB interface 830 is included as an example of an interface to allow connection of peripheral devices, or connection to other devices, such as other portable devices, computers, etc. In various embodiments of the HMD 801, any of a variety of types of interfaces may be included to allow greater connectivity of the HMD 801.

[0117] Wi-Fi™ circuitry 832 is included to enable connection to the Internet via wireless network technology. The HMD 801 also includes Bluetooth™ circuitry 834 to enable wireless connection to other devices. In some embodiments, a communications link 836 is also included for connection to other devices. In one embodiment, the communications link 836 utilizes infrared transmission for wireless communication. In other embodiments, the communications link 836 utilizes any of a variety of wireless or wired transmission protocols for communication with other devices.

[0118] Input buttons / sensors 838 are included to provide an input interface for User A (FIG. 1). Any of a variety of types of input interfaces may be included, such as buttons, touchpads, joysticks, trackballs, etc. In various embodiments, ultrasonic communication circuitry 840 is included in the HMD 801 to facilitate communication with other devices via ultrasonic technology.

[0119] Biosensor 842 is included to enable detection of physiological data from user A or B. In one embodiment, biosensor 842 includes one or more dry electrodes to detect bioelectrical signals of user A or B through the skin of user A or B.

[0120] The aforementioned components of the HMD 801 have been described merely as example components that may be included within the HMD 801. In various embodiments, the HMD 801 may or may not include some of the various components described above.

[0121] In some embodiments, communication between the server system and the client device may be facilitated using wireless technology. Such technology may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. 5G networks are digital cellular networks in which the service area covered by a provider is divided into smaller geographical areas called cells. Analog signals representing sound and video are digitized in the telephone, converted by an analog-to-digital converter, and transmitted as a bit stream. All 5G wireless devices within a cell communicate with local antenna arrays and low-power automatic transceivers (transmitters and receivers) within the cell via electromagnetic waves over frequency channels assigned by the transceivers from a frequency pool reused in other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cell networks, mobile devices moving from one cell to another are automatically moved to the new cell. Of course, 5G networks are merely an example type of communication network, and embodiments of the present disclosure may use previous generations of wireless or wired communications, as well as later generations of wired or wireless technologies that come after 5G.

[0122] It should be noted that in various embodiments, one or more features of some of the embodiments described herein may be combined with one or more features of one or more of the remaining embodiments described herein.

[0123] The embodiments described in this disclosure may be implemented in a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. In implementation, the embodiments described in this disclosure are practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wired or wireless network.

[0124] With the above embodiments in mind, it should be understood that in practice, the embodiments described in this disclosure employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments described in this disclosure are useful machine operations. Some embodiments described in this disclosure also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, in one embodiment, various general-purpose machines are used with computer programs written in accordance with the teachings herein. Alternatively, it may be more convenient to construct more specialized apparatus to perform the required operations.

[0125] In implementation, some embodiments described in this disclosure are embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that stores data which is subsequently read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), ROM, RAM, compact disc ROM (CD-ROM), CD-recordable (CD-R), CD-rewritable (CD-RW), magnetic tape, optical data storage devices, non-optical data storage devices, and the like. As an example, the computer readable medium includes computer readable tangible media distributed over network-coupled computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0126] Additionally, although some of the above embodiments are described with respect to a gaming environment, in some embodiments other environments are used in place of a game, such as, for example, a video conferencing environment.

[0127] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted to occur at slightly different times, or may be distributed within a system that allows processing operations to occur at various intervals related to processing, so long as the processing of the overlay operation is performed in the desired manner.

[0128] Although the foregoing embodiments described in this disclosure have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. receiving content of an interactive application for rendering on a display screen of a pair of augmented reality (AR) glasses worn by a user, the pair of AR glasses being used to view a real-world space in the user's vicinity, the content being provided by the interactive application in response to input provided by the user and one or more other users during interaction with the interactive application, the content including virtual objects; dynamically scaling the virtual objects of the content of the interactive application to match a scale of a real-world object in the real-world space viewed through the pair of AR glasses; presenting the virtual objects of the content of the interactive application dynamically scaled as an overlay on top of the real-world objects in the real-world space viewed through the pair of AR glasses, the scaled interactive content providing the user with a normalized view of the interactive content when viewed along the real-world objects. method.

2. In the dynamic scaling, receiving a scanned image of the real world space in the vicinity of the user; analyzing the scanned image to identify a layout of the real-world objects in the real-world space and to determine attributes of the real-world objects, the attributes of the real-world objects being used to determine a scale of the real-world objects in the real-world space as viewed through the pair of AR glasses; generating an image of the virtual object included in the content of the interactive application for projecting as an overlay onto the real-world object, the image of the virtual object being dynamically adjusted to match the scale of the virtual object to the scale of the real-world object in the real-world space viewed through the pair of AR glasses prior to projecting as the overlay onto the real-world object; The method of claim 1.

3. the scanned image is received from a scanner in response to a first signal; the image of the virtual object is dynamically adjusted using a projector in response to a second signal; the first signal and the second signal are triggered by a machine learning algorithm in response to receiving content of the interactive application for rendering, the machine learning algorithm analyzing the scanned image received in response to the first signal and generating the second signal to project the dynamically adjusted image onto the real-world object. The method of claim 2.

4. the attributes of the real-world object determined from analyzing the scanned images include at least a rendering distance of the real-world object when viewed through the pair of AR glasses, a size of the real-world object, an orientation relative to the user interacting in the real-world space, an orientation relative to another real-world object, and a position within the real-world space; The method of claim 2.

5. Presenting the dynamically scaled content further includes filtering one or more virtual objects from the content before rendering the scaled interactive content; the filtering enables viewing of the scaled interactive content along the real-world space through the pair of AR glasses. The method of claim 1.

6. providing a visual indicator for the virtual object to indicate the presence of the virtual object included in the overlay in the dynamically scaled presentation of the content; The method of claim 1.

7. The dynamic scaling further comprises scaling sounds emanating from one or more virtual objects included in the content of the interactive application, the dynamic scaling of the sounds being performed based on a context of the content of the interactive application. The method of claim 1.

8. the interactive application is a video game played between a first player and a second player located remotely, and the content is game play content; The method of claim 1.

9. In the dynamic scaling, scaling the game play content according to the context of a first set of real-world objects in a first real-world space of the first player, and then overlaying the game play content over the first set of real-world objects viewed through a first pair of AR glasses of the first player; scaling the game play content according to the context of a second set of real-world objects in a second real-world space of the second player, and then overlaying the game play content over the second set of real-world objects viewed through a second pair of AR glasses of the second player. The method according to claim 8.

10. Detecting a first selection of a first real world object from a first real world space proximate the first player for inclusion as a first virtual object and a second selection of a second real world object from a second real world space proximate the second player for inclusion as a second virtual object; and dynamically scaling the first virtual object and the second virtual object to match the scale of the virtual objects of the content of the interactive application presented in the overlay. the first selection is initiated by the first player and the second selection is initiated by the second player; The method according to claim 8.

11. In dynamically scaled content presentation, the first virtual object is included in the overlay rendered on the first real world space viewed through a first pair of AR glasses of the first player, and the second virtual object is included in the overlay rendered on the second real world space viewed through a second pair of AR glasses of the second player; the first virtual object and the second virtual object are rendered in accordance with the content of the interactive application; the dynamic scaling normalizes a view of the selected first virtual object to the first real world space and normalizes a view of the selected second virtual object to the second real world space. The method of claim 10.

12. and detecting a first selection of first real world objects from a first real world space proximate the first player for sharing with a second player; the first real-world object is rendered as a first virtual object when rendered within the overlay of the content of the interactive application on the second real-world space; the dynamic scaling includes scaling a dimension of the first virtual object to match the scale of the real-world object in the second real-world space as viewed by the second player; the first real-world object is selected for sharing by the first player; The method according to claim 8.

13. The dynamic scaling is performed in response to a signal from a server executing the interactive application, or from the interactive application, or from a machine learning algorithm. The method of claim 1.

14. receiving content of an interactive application for rendering on a display screen of a pair of augmented reality (AR) glasses worn by a user, said content being provided by the interactive application in response to input provided by the user; performing dynamic scaling of real-world objects in a real-world space viewed through the pair of AR glasses worn by the user, the dynamic scaling being performed to match a scale of the real-world objects to a scale of virtual objects included in the content of the interactive application; presenting the virtual objects of the content of the interactive application as an overlay on top of the scaled real-world objects in the real-world space viewed through the pair of AR glasses; the dynamic scaling providing the user with a normalized view of the content in relation to the real-world objects. method.

15. the dynamic scaling of the real-world object is performed by adjusting the optics of the pair of AR glasses through which the user views the real-world space. The method of claim 14.

16. The optical system is adjusted according to the visual characteristics of the user viewing the real world space through the pair of AR glasses. The method of claim 15.

17. Presenting the virtual objects includes selectively filtering one or more of the virtual objects of the content before presenting the virtual objects over a real-world object viewed through the pair of AR glasses. The method of claim 14.

18. Further, generating feedback to the user to adjust the user's position in the real world space relative to a virtual object of the content provided as the overlay on the real world space viewed through the pair of AR glasses. The method of claim 14.

19. the interactive application is executed on a server at a cloud site, and the feedback is generated as a signal from the server executing the interactive application or as a suggestion from a machine learning algorithm executing on the interactive application or on the server.

20. The method of claim 18.

20. the interactive application is a multiplayer video game played between a first player and a second player, and the content of the interactive application includes gameplay content generated in response to game inputs from the first player and the second player. The method of claim 14.

21. Dynamically scaling the real-world object includes scaling a first real world space viewed through a first pair of AR glasses of the first player to match a scale of a second real world space viewed through a second pair of AR glasses of the second player; the dynamic scaling is performed by adjusting dimensions of and spaces between the first set of real world objects, the dynamic scaling normalizing views of the first real world space relative to the second real world space as viewed by the first player and the second player; the scaling of the first real world space and the second real world space matches the scaling of the virtual objects from the content of the interactive application presented as an overlay on the first real world space and the second real world space, respectively.

21. The method of claim 20.

22. The presentation of the virtual object further comprises: receiving from the first player a first selection of a first real world object in the first real world space or a second selection of a second real world object in the second real world space from the second player for inclusion in the overlay rendered on the first and second real world spaces viewed by the first and second players; rendering the first real world object as a first virtual object or the second real world object as a second virtual object in the overlay rendered on the first real world space and the second real world space viewed by the first player and the second player, the first virtual object or the second virtual object being scaled to match the scale of the first real world space and the second real world space prior to inclusion in the overlay for rendering along with the virtual objects of the game play content viewed through the pair of AR glasses of the first player and the second player; 21. The method of claim 20.

23. one or more virtual objects included in the game play content are filtered prior to inclusion as the overlay on top of the real-world objects viewed by the first player and the second player, an amount of filtering defined to avoid creating clutter on a display screen of the first pair of AR glasses and the second pair of AR glasses worn by the first player and the second player.

21. The method of claim 20.

24. receiving gameplay content of a video game being played between a first player and a second player for rendering on display screens of a first pair of augmented reality (AR) glasses and a second pair of AR glasses worn by the first player and the second player, the gameplay content being provided by the video game in response to inputs provided by the first player and the second player; dynamically scaling real-world objects in a first real world space proximate the first player as viewed through the first pair of AR glasses for the first player to match scales of real-world objects in a second real world space proximate the second player as viewed through the second pair of AR glasses for the second player, the dynamic scaling normalizing a view of the first real world space to the second real world space viewed by the first player and the second player; presenting virtual objects included in the gameplay content of the video game as overlays on the real-world objects included in the dynamically scaled first and second real-world spaces for viewing through the first and second pairs of AR glasses of the first and second players; method.

25. the dynamic scaling of the real-world object is performed by adjusting optics of the first pair of AR glasses and the second pair of AR glasses used by the first player and the second player to view the first real world space and the second real world space.

25. The method of claim 24.

26. The optical system of the first pair of AR glasses is adjusted according to visual characteristics of the first player, and the optical system of the second pair of AR glasses is adjusted according to visual characteristics of the second player.

26. The method of claim 25.

27. The dynamic scaling further comprises scaling the gameplay content of the video game to match scaling of the real-world objects from the first real world space and the second real world space as viewed by the first player and the second player, the dynamic scaling of the gameplay content being performed prior to rendering as the overlay on top of the real-world objects in the first real world space and the second real world space, respectively.

25. The method of claim 24.

Citation Information

Patent Citations

  • Information processing device, authoring method, and program

    JP2012168798A

  • Automatic placement of virtual objects in 3D space

    JP2019532382A

  • Controlling a virtual object with a real controller device

    US20130328762A1

  • Rendering virtual objects in 3D environments

    US20180218538A1

  • Real-world anchor in a virtual-reality environment

    US20200111256A1