AR systems used to provide augmented reality (AR) in video games

Abstract

An augmented reality (AR) system allows AR to be provided in video games. The disclosed AR system allows AR content to be layered on top of the built-in features of a video game to provide a unique "in-game" AR experience for gamers. A remote computing system provides a central data warehouse for AR content and related data, which can be accessed by selected client machines to render enhanced frames with AR content. The AR content can be spatially relevant AR content rendered at appropriate locations within the game world. The AR content can be event-specific, so that the AR content is added in response to game-related events. The disclosed AR system allows multi-player aspects to be added to otherwise single-player games and / or AR content to be shared in real time to provide enhanced features such as spectating, mixing of game worlds, and / or teleportation through AR objects.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a PCT application claiming priority to U.S. patent application serial number 16 / 143,192, filed on September 26, 2018, entitled “AUGMENTED REALITY (AR) SYSTEM FOR PROVIDING AR IN VIDEO GAMES,” which is hereby incorporated by reference in its entirety. Background Art

[0003] Traditionally, augmented reality (AR) technology involves augmenting a real-world environment with computer-generated content displayed on a see-through display. In this way, a user of such an AR device is able to see the computer-generated content in the context of real objects residing in their real-world environment. To date, AR technology has been primarily limited to augmenting the real world. The disclosures made herein are presented with respect to these and other considerations. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical components or features.

[0005] Figure 1 A block diagram illustrating example components of a client machine having an augmented reality (AR) component configured to render an augmented frame during execution of a video game, the augmented frame including video game content and augmented reality (AR) content.

[0006] Figure 2 is a diagram illustrating an example system including components of a remote computing system for creating AR content and maintaining the AR content in a spatial database so that the AR content can be selectively provided to client machines for use in a video game.

[0007] Figure 3 is a diagram illustrating an example system by which a client machine may subscribe to an AR channel that determines the AR content received by the client machine.

[0008] Figure 4 is a flow chart of an example process for: providing a content creation interface for authors to create AR content; storing the created AR content in a spatial database; and sending selected AR content to a requesting client machine based on one or more filtering criteria.

[0009] Figure 5 is a flow chart of an example process for receiving AR content from a remote computing system and rendering frames, including augmented frames that include AR content, during execution of a video game.

[0010] Figure 6 is a flow chart of an example process for augmenting a frame with spatially relevant AR content during execution of a video game.

[0011] Figure 7 is a flow chart of an example process for augmenting a frame with dynamic and / or interactive AR content during execution of a video game.

[0012] Figure 8 is a diagram showing example techniques for adding multiplayer aspects to a single-player video game using the disclosed AR system.

[0013] Fig. 9 is a flow chart of an example process for adding a multiplayer aspect to a single-player game using an AR system by exchanging data between client machines over a computer network.

[0014] Fig.10 is a diagram showing example techniques for sharing aspects of a game world between client machines using the disclosed AR system.

[0015] Fig.11 is a flow chart of an example process for sharing aspects of a game world between client machines using an AR system.

[0016] Fig.12 is a flow chart of an example process for exchanging events between a video game and a separately executed AR component on a client machine. DETAILED DESCRIPTION

[0017] Among other things, this article describes techniques, devices, and systems for providing augmented reality (AR) in video games. As mentioned, AR is traditionally considered to be a technology that can be used to enhance a user's experience of the real world (i.e., the user's physical environment). The AR system disclosed herein enhances not the user's experience of the real world, but the user's experience of the game world of the video game. The disclosed AR system allows AR content to be layered on top of the built-in features of the video game. The AR system in this game is universal in the sense that it allows authors to provide gamers with a unique "in-game" AR experience by creating AR content for any video game or multiple video games. In doing so, the disclosed AR system reduces the burden on game developers to provide the same type of enhanced features to their own games. If left to their own devices, game developers may end up custom building their own AR system, which may result in the AR system being game-specific and incompatible with other games published by other game developers. In contrast, the AR system in the disclosed game is compatible with multiple different video games.

[0018] In the disclosed game, the AR system may include, among other things, a remote computing system that acts as a central data repository for AR content and related data. In some embodiments, the remote computing system maintains the AR content in a spatial database that associates the AR content with various data (e.g., a game identifier (ID), spatial data related to the game world coordinates of a video game, event data specifying game-related events, etc.). Additionally or alternatively, the AR content may be an associated AR channel that acts as a filtering criterion for the AR content.

[0019] The remote computing system may further provide an interface for authors to create new AR content, which is thereafter maintained by the remote computing system and made accessible to selected audiences of gamers who wish to enhance their in-game AR experience in their video games. This content creation interface may support the creation of different types of AR content, including but not limited to information messages, two-dimensional (2D) objects, three-dimensional (3D) objects, screenshots with 2D and / or 3D pixel data, video clips, and the like. AR content may be "static" and therefore rendered at a fixed location within the game world. AR content may be "dynamic" and therefore may be moved or animated within the game world. In some embodiments, AR content may even be interactive by using plugins that allow authors to create executable programs that respond to real-time video game data as input to the program. In this way, players of video games may not only experience AR content that has been added to the game world, but may also interact with AR content in some cases, which is very similar to playing a secondary video game within the game world of a core video game.

[0020] In order to render AR content within the game world, the client machine may obtain access to the AR content, and may appropriately identify the relevant AR content and render the relevant AR content within the game world when executing the video game on the client machine. The client machine may access the AR content from any suitable storage location (e.g., from a remote computing system via a computer network; from a local memory after downloading the AR content from the remote computing system). In an example process, the client machine may execute a video game that is configured to output video game content in a series of frames. During game execution, the client machine may enhance any given frame in a series of frames with AR content by: (i) obtaining video game data about the current state of the video game from the video game; (ii) identifying AR content based at least in part on the video game data; (iii) generating an enhanced frame containing the video game content and the identified AR content; and (iv) rendering the enhanced frame on a display associated with the client machine. It is noteworthy that the AR content is not generated by the video game executed on the client machine, but is retrieved from a separate resource that maintains the AR content retrieved when rendering the enhanced frame by layering the AR content on the "top" of the video game content. Although this document often states that AR content is layered “on top of” video game content, this should not be taken literally, as AR content may be merged with video game content in any suitable manner such that certain video game content (e.g., semi-transparent graphics) is rendered “on top of” the AR content.

[0021] In some embodiments, video game data—which is obtained from the video game being executed and used to identify relevant AR content for the enhanced frame—can be spatial data related to game world coordinates within the game world of the video game. For example, AR content can be identified based on its association with coordinates in the game world that are somehow related to the current position of the character controlled by the player. In this way, spatially relevant AR content can be rendered at a certain location within the game world of the video game in the enhanced frame along with the video game content. In some embodiments, the video game data can also be event data related to game-related events. For example, AR content can be identified and rendered within the game world based on the occurrence of game-related events (e.g., shooting from a gun, a game character entering / exiting a vehicle, etc.).

[0022] The disclosed AR system also allows for the use of various multi-player aspects to enhance single-player video games. This can be achieved without completely modifying the code of a single-player game in order to make it a multi-player game. To implement such multi-player aspects, client machines can exchange data over a computer network. For example, a first client machine executing a video game can be connected to a remote computing system so that the first client machine can receive data sent by a second client machine that is also executing the video game via the remote computing system. The data sent by the video game executed on the second client machine can be spatial data that specifies the current position of a character controlled by a second player within a second instance of a game world rendered on the second client machine. Upon receiving this spatial data over the network, the first client machine can identify and retrieve AR content (e.g., an AR avatar of a character controlled by the second player), and the first client machine can render the retrieved AR content within a first instance of a game world being rendered on the first client machine, the AR content being rendered at a location within the game world corresponding to the received spatial data. For example, this technology may allow “speed running” to be added to an otherwise single-player game, whereby a first player using a first client machine sees an AR avatar of a second player's game character superimposed over the video game content of the first player's video game.

[0023] In some embodiments, the disclosed AR system can build (or reconstruct) a model of a portion of a game world from a 3D screenshot (i.e., an image with depth data). In this case, the AR content can be a 3D screenshot, and a 3D model of a portion of the game world captured in the 3D screenshot can be built to allow a first player to look around and / or move around a "slice" of the game world captured by a second player. This 3D slice of the game world can be rendered in an enhanced frame 122 on the first player's client machine while playing the video game.

[0024] In some embodiments, the disclosed in-game AR system may allow real-time sharing of AR content between client machines of different players over a computer network. For example, AR content may be rendered in a first video game as a viewport or even a portal into the game world of another player's video game. This technology may use 3D screenshots to build a 3D model of a portion of the game world presented in a particular 3D screenshot. This allows players to interact with each other through AR content rendered in each video game as a window into the virtual game world of another video game.

[0025] The techniques and systems described herein may allow one or more devices to save resources in terms of processing resources, memory resources, and / or networking resources. For example, sending data instead of actual content (e.g., images and / or video files) over a computer network reduces network bandwidth consumption, at least compared to live game streaming techniques used today that send content streams over computer networks with high bandwidth consumption. As another example, selectively downloading AR content to a client machine may reduce network bandwidth consumption and / or memory consumption generated by and / or on a local client machine that is configured to retrieve and render AR content during video game execution. Other examples are described throughout this disclosure.

[0026] Figure 1 A block diagram showing example components of a client machine 100 having an augmented reality (AR) component 102 configured to render an augmented frame during execution of a video game 104, the augmented frame containing both video game content and AR content. Figure 1 The client machine 100 shown in FIG. 1 may represent a computing device that may be used by a user 106 to execute programs and other software on the computing device. Figure 1 In the context of a user 106 of the illustrated client machine 100 using the client machine 100 for the specific purpose of playing a video game 104 that is executed on the client machine 100 or that is executed on a remote computing system and is playable on the client machine 100 as a streamed video game 104, the user 106 is generally referred to herein as a “player.” Thus, the terms “user 106,” “player 106,” and / or “gamer 106” may be used interchangeably herein to refer to a user of a client machine 100 where one of the many uses of the client machine is to play a video game.

[0027] The client machine 100 may be implemented as any suitable type of computing device configured to process graphics and render graphics on an associated display, including but not limited to a PC, a desktop computer, a laptop computer, a mobile phone (e.g., a smart phone), a tablet computer, a portable digital assistant (PDA), a wearable computer (e.g., a virtual reality (VR) headset, an augmented reality (AR) headset, smart glasses, etc.), an in-vehicle (e.g., in-car) computer, a television (smart TV), a set-top box (STB), a game console, and / or any similar computing device.

[0028] In the illustrated embodiment, the client machine 100 includes, among other components, one or more processors 108, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs), one or more displays 110, memory 112 (or non-transitory computer-readable media 112), and one or more communication interfaces 114. Figure 1 The example client machine 100 of implies that the client machine 100 includes an embedded display 110, but the client machine 100 may actually omit the display, but may instead be coupled to a peripheral display. Thus, display 110 is intended to mean an associated display 110, whether embedded in the client machine 100 or connected thereto (via a wired or wireless protocol).

[0029] The memory 112 (or non-transitory computer readable medium 112) may include volatile and non-volatile memory, removable or non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Such memory includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic box, tape, magnetic disk storage device or other magnetic storage device, RAID storage system or any other medium that can be used to store the desired information and can be accessed by the computing device. The computer readable medium 112 can be implemented as a computer readable storage medium ("CRSM"), which can be any available physical medium that can be accessed by one or more processors 108 to execute the instructions stored on the memory 112. In a basic embodiment, the CRSM can include random access memory ("RAM") and flash memory. In other embodiments, the CRSM can include but is not limited to read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM") or any other tangible medium that can be used to store the desired information and can be accessed by one or more processors 108.

[0030] As will be described in greater detail below, the client machine 100 may communicate with a remote computing system over a computer network via one or more communication interfaces 114. As such, the one or more communication interfaces 114 may employ any suitable communication protocol for communicating over a wired infrastructure (e.g., coaxial cable, fiber optic cable, etc.), a wireless infrastructure (e.g., radio frequency (RF), cellular, satellite, etc.), and / or other connection technologies.

[0031] In some embodiments, Figure 2A remote computing system such as the remote computing system 200 shown in FIG. 1 acts as or can access a platform for distributing (e.g., downloading) programs (and content) to client machines such as the client machine 100. Thus, the client machine 100 Figure 1 1 is shown as having a video game client 116 installed in the memory 112. The video game client 116 may represent an executable client application configured to launch and execute a program such as a video game (or video game program). In other words, the video game client 116 may contain game software that can be used to play a video game on the client machine 100. With the video game client 116 installed, the client machine 100 may then have the ability to receive (e.g., download, stream, etc.) a video game from a remote system over a computer network and execute the video game via the video game client 116. Any type of content distribution model may be used for this purpose, such as a direct purchase model in which video games may be purchased individually to be downloaded and executed on the client machine 100, a subscription-based model, a content distribution model in which video games are rented or leased for a certain period of time, etc. Thus, the client machine 100 may contain one or more video games, such as the video game 104, within the video game library 118. These video games may be retrieved and executed by loading the video game client 116. In an example, a user may choose to play one of a plurality of video games they have purchased and downloaded to a video game library 118 by loading a video game client 116 and selecting a video game 104 to begin execution of the video game 104. The video game client 116 may allow the user 106 to log into the video game service using credentials (e.g., user account, password, etc.).

[0032] like Figure 2 A remote computing system such as the remote computing system 200 of the video game 104 may further serve as or access a platform for distributing (e.g., streaming, downloading, etc.) augmented reality (AR) content 120 to client machines such as the client machine 100. Thus, the client machine 100 is shown as having the AR content 120 stored in the local memory 112, such that the AR content 120 is accessible from the local memory 112. Generally speaking, this AR content 120 may be received (e.g., downloaded, streamed, etc.) from the remote system 200 over a computer network and may be used in the process of rendering an enhanced frame containing the AR content 120 added to the video game content output by the video game 104 itself. In this way, the AR content 120 may be maintained remotely (e.g., at the remote computing system 200) and may be accessed over a computer network. Figure 1An example enhanced frame 122 is shown that may be rendered as one of a plurality of frames during execution of the video game 104. As used herein, a "frame" means an image frame that is one of a number of image frames in a series of image frames used to render a live video game on a display. Thus, the enhanced frame 122 is a composite frame that includes both the video game content 124 and the AR content 120. Notably, the AR content 120 represents content that is not generated by the video game 104 itself. Thus, the AR content 120 represents supplemental computer-generated graphics that are added to the video game content after the fact. Thus, Figure 1 The video game content 124 in represents the video game content of one of a series of frames output by the video game 104 itself when executed on the client machine 100. In this sense, the "game world" of the video game 104 can be defined by a coordinate system, and the portion of the game world rendered in each frame in the series of frames can depend on various factors, including the current position of the player-controlled character 126 within the game world. The coordinate system of the game world can define coordinates corresponding to positions within the game world. Figure 1 An example of a first-person shooter video game 104 is shown that allows a player 106 to control the movement of a game character 126 within a game world. For example, the player 106 may provide user input to the client machine 100 (e.g., via a game controller, a touch screen, etc.) to move the player-controlled character 126 from one location to another, where each location is designated according to specific coordinates that indicate where the player-controlled character 126 is located within the game world at any given moment.

[0033] As will be described in more detail below, AR content 120 accessible to a client machine 100 may be stored in association with spatial data that may specify particular coordinates of a game world of a particular video game 104. In this manner, each time the video game 104 renders a portion of the game world that includes coordinates associated with particular AR content 120, the AR content 120 may be identified based on its association with those coordinates, and an augmented frame 122 may be generated using the identified AR content 120 that includes the AR content 120 rendered at a location within the game world that corresponds to those coordinates.

[0034] To illustrate how client machine 100 may operate to provide in-game AR, consider a frame in a series of frames to be rendered on one or more displays 110 associated with client machine 100. To render a given frame, AR component 102 executing on client machine 100 may: obtain video game data 128 about the current state of video game 104 from video game 104; identify AR content 120 (e.g., based at least in part on video game data 128); Figure 1As shown by arrow 130 in the figure, AR content 120 is accessed locally or remotely); an enhanced frame 122 is generated, which includes video game content 124 output by the video game 104 and AR content 120 identified based on video game data 128; and the enhanced frame 122 is rendered on one or more displays 110.

[0035] The AR component 102 can be executed separately from the execution of the video game 104, so that if the video game 104 crashes, the AR component 102 will not crash, and vice versa. In this sense, the AR component 102 is decoupled from any specific video game 104 executed on the client machine 100, which provides the ability to have an AR system containing an AR component 102 that is compatible with multiple video games and can be transferred between multiple video games, so that AR content 120 can be added to any video game 104 to enhance the user experience. For example, the AR component 102 can be run as a separate process from the video game 104 (e.g., a .exe separate from the .exe of the video game 104), and the AR component 102 and the video game 104 can communicate back and forth. The AR process can potentially communicate with multiple video games and / or multiple non-game applications at once. This AR process can also contain plug-ins or be configured to load plug-ins. These plug-ins can be executed within a secure sandbox (or container). This decoupling of the AR component 102 from the video game 104 provides stability; the video game 104 will not crash the AR process, and vice versa. Security is another benefit because third-party plug-in code used to render the AR content 102 will not run in the same process as the video game 104 because it is sandboxed and kept separate, thereby mitigating any possibility of cheating with the AR system. In some embodiments, a video game in the form of an “AR viewer” described in more detail below can allow the user 106 to view the AR content 120 outside the context of the video game content 124. For example, the “AR viewer” can access the AR content 120 and render it on a blank background or a 3D model representation of the game world.

[0036] exist Figure 1In an example, the AR content 120 identified based on the video game data 128 includes a first AR content 120(1) and a second AR content 120(2). For example, the first AR content 120(1) is a screenshot (e.g., a 2D or 3D image), and the second AR content 120(2) is, for example, an information message. The first AR content 120(1) can be associated with a first coordinate within the game world, and the second AR content 120(2) can be associated with a second coordinate within the game world. In this case, the video game data 128 obtained from the video game 104 can include spatial data specifying a current position of a player-controlled character 126 within the game world and may include other spatial data, such as a current orientation of a virtual camera associated with the player-controlled character 126. The camera orientation data can indicate a field of view from the perspective of the player-controlled character 126, and thus, when coupled with the current position of the player-controlled character 126, a set of coordinates corresponding to a portion of the game world that is within the field of view of the player-controlled character 126 can be determined. In this way, AR content 120 to be rendered in an augmented frame 122 may be identified based at least in part on spatial data specifying the current position of a player-controlled character 126 and possibly based on additional spatial data such as camera heading data, indices, etc. These aspects of the spatial data are described in more detail below.

[0037] Thus, the AR component 102 may determine that the screenshot (first AR content 120(1)) is to be rendered at a first coordinate corresponding to a first location within the game world, and that the information message (second AR content 120(2)) is to be rendered at a second coordinate corresponding to a second location within the game world. In this manner, the AR content 120 may be "spatially dependent" AR content 120 in the sense that it is associated with specific coordinates within the game world. Thus, the player 106 may navigate the player-controlled character 126 around the AR content 120, which in some cases remains fixed at a location within the game world.

[0038] As mentioned, AR content 120 may additionally or alternatively be event-related AR content 120 in the sense that it is associated with a particular event when the particular event occurs in the video game 104. In this case, first AR content 120(1) may be associated with a first game-related event, and second AR content 120(2) may be associated with the first game-related event or the second game-related event, and video game data 128 obtained from the video game 104 may include event data indicating the occurrence of one or more game-related events.

[0039] In some embodiments, the AR component 102 may receive video game data 128 from the video game 104 as part of a function call made by the video game 104. In this case, a game developer of the video game 104 may implement an application programming interface (API) in the video game code to provide a rendering hook that makes such function calls to the AR component 102 during each frame cycle during game execution to pass the video game data 128 to the AR component 102. For example, a code library written by a service provider of a video game platform may be provided to game developers for integration into their video games 104, which allows AR-related processes to be provided to run within the game process and is responsible for communicating with an external AR component 102 running in a separate process. The AR component 102 may be responsible for rendering the augmented frame 122 based on the video game data 128, and for requesting the AR content 120 to be rendered in the augmented frame 122. In some embodiments, the timing of the function call during the frame cycle is such that the function call is made after the opaque graphics are rendered in the video game content 124, but before the semi-transparent graphics are rendered in the video game content 124, so that the AR content 120 can be rendered between the two types of graphics. In some embodiments, the depth data from the depth buffer is used to appropriately merge the video game content 124 and the AR content 120. The function call can provide the video game data 128 (e.g., spatial data such as game world coordinates, matrix transformations of the camera orientation of the player-controlled character 126, event data, etc.) to the AR component 102 so that the AR component 102 can retrieve relevant AR content 120 based on the video game data 128.

[0040] In some embodiments, AR content 120 may be automatically “injected” into a video game by AR component 102 as an overlay on an existing frame of video game content 124, independent of coordination with the game developer to implement any additional AR-related code into their video game. As will be described in more detail below, this automatic injection technique may be implemented using simultaneous localization and mapping (SLAM) technology. In short, the SLAM process may be accomplished by Figure 2The remote computing system 200 shown is executed offline and can be used to incrementally rebuild the game world geometry (e.g., a 3D model of the game world) from many images. This back-end process can be completed by a service provider of the video game platform, by a game developer, and / or by crowdsourcing game world images from a player client machine 100. In this way, SLAM can be used to automate the recognition of the game world geometry depicted in a screenshot of the video game content 124, and the 3D model of the game world generated by the SLAM process can be used by the client machine 100 to enhance the video game content 124 with AR content 120 in a manner that presents AR content 120 in the context of the game world geometry. This can also allow AR content 120 to be added to the video game content 124 of an old catalog game whose code is no longer updated by the game developer. In some embodiments, the video game 106 can be configured to explicitly request the AR content 120 itself and render the AR content 120 itself without relying on the AR component 102.

[0041] In some embodiments, the AR content 120 is superimposed on the video game content 124 during the rendering of the enhanced frame 122. For example, the video game 104 may output pixel data (e.g., color values, depth values, etc.) corresponding to graphics to be rendered on one or more displays 110 for the enhanced frame 122. The pixel data output by the video game 104 may indicate, for example, that opaque graphics are to be rendered first (e.g., at a larger depth value that is farther from the current position of the player-controlled character 126), and that semi-transparent graphics (e.g., particles, clouds, dust, etc.) are to be rendered after the opaque graphics (e.g., at a smaller depth value that is closer to the current position of the player-controlled character 126). Thus, in some cases, the AR content 120 may be rendered between the opaque graphics and the semi-transparent graphics, such as by rendering the AR content 120 at a depth value between the respective depth values ​​of the opaque graphics and the semi-transparent graphics.

[0042] In some embodiments, the AR content 120 may be presented in a subtle manner within the enhanced frame 122, such as with an icon that does not take up much space in the game world, and when the user 106 focuses on the AR content 120 (e.g., by hovering a pointer over the icon, moving near the icon, etc.), a pop-up window may be presented asking the user 106 if he / she wants to see more. If the user 106 indicates that he / she wants to see more through selection of a button, a full version of the AR content 120 may be presented (e.g., by expanding the icon into a screenshot, an information message, an object, or any other form of AR content 120). In some embodiments, unsubscribed AR content 120 may be presented in this manner. AR channels are discussed in more detail below (e.g., see Figure 3). In short, AR channels act as a filtering mechanism such that a user 106 can subscribe to one or more AR channels to see AR content related to those subscribed AR channels. However, in addition to receiving subscribed AR content from subscribed AR channels, the client machine 100 can also receive AR content 120 to which the user 106 has not yet subscribed, which is presented within the enhanced frame 122 in a subtle manner for visually distinguishing the unsubscribed AR content 120 from the subscribed AR content 120. This unsubscribed AR content 120 can be transmitted to the client machine 100 based on the current game world coordinates of the portion of the game world of the video game 104 to be rendered. Thus, the unsubscribed AR content 120 can be provided to the user 106 for subscription based on the location within the game world that the user 106 is currently experiencing.

[0043] Figure 22 is a diagram showing an example system 202, which includes components of a remote computing system 200 for creating and maintaining AR content 120 and related data in a spatial database 204 so that the AR content 120 is selectively provided to a client machine for use in a video game. In the illustrated embodiment, the remote computing system 200 includes one or more processors 206, one or more communication interfaces 208, and a memory 210 (or a non-transitory computer-readable medium 210) in addition to other components. The memory 210 (or non-transitory computer-readable medium 210) may include volatile and non-volatile memory, removable or non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, tape, magnetic disk storage device or other magnetic storage device, RAID storage system, or any other medium that can be used to store desired information and can be accessed by a computing device. The computer-readable medium 210 may be implemented as a computer-readable storage medium ("CRSM"), which may be any available physical medium that can be accessed by one or more processors 206 to execute instructions stored on the memory 210. In a basic embodiment, the CRSM may include random access memory ("RAM") and flash memory. In other embodiments, the CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that can be used to store desired information and can be accessed by one or more processors 206. The augmented reality (AR) module 212 may represent instructions stored in the memory 210 that, when executed by one or more processors 206, cause the remote computing system 200 to perform the techniques and operations described herein. The memory 210 is also shown as maintaining a video game catalog 214, which may store a catalog of video games such as the video game 104 for distribution to client machines such as the client machine 100 described herein.

[0044] The one or more communication interfaces 208 may employ any suitable communication protocol to communicate via a wired infrastructure (e.g., coaxial cable, fiber optic cable, etc.), a wireless infrastructure (e.g., radio frequency (RF), cellular, satellite, etc.), and / or other connection technologies. Figure 2Authors such as authors 216(1) and 216(2) shown in the figure can access the remote computing system 200 via a computer network 218 using corresponding user computing devices 220(1) and 220(2). The computer network 218 can represent and / or include, but is not limited to, the Internet, other types of data and / or voice networks, wired infrastructure (e.g., coaxial cable, fiber optic cable, etc.), wireless infrastructure (e.g., radio frequency (RF), cellular, satellite, etc.), and / or other connection technologies. In some cases, the remote computing system 200 can be part of a network-accessible computing platform that is maintained and accessible via the computer network 218. Network-accessible computing platforms such as such network-accessible computing platforms can be referred to using terms such as "on-demand computing", "software as a service (SaaS)", "platform computing", "network-accessible platform", "cloud service", "data center", etc. In general, the remote computing system 200 is configured to act as a central data repository for AR content 120 and related data.

[0045] The remote computing system 200 may be further configured to provide an interface (e.g., an application programming interface (API)) for the user computing device 120 to create new AR content 120. As such, the remote computing system 200 may receive instructions 222 for creating AR content 120 from the user computing device 120 via the content creation interface (e.g., API). Figure 2 A first author 216(1) is depicted using a first user computing device 220(1) to provide first instructions 222(1) for creating new AR content 120, and a second author 216(2) is depicted using a second user computing device 220(2) to provide second instructions 222(2) for creating new AR content 120. It should be appreciated that the remote computing system 220 may support a community of such authors 216 who wish to create new AR content 120, such that the new AR content is maintained by the remote computing system 200 for access by client machines while playing a video game.

[0046] In addition to the API that allows the author 216 to create AR content 120 outside of the execution of the video game 104, new AR content 120 can also be created on the client machine 100 via plug-in logic during the execution of the video game 104. For example, the AR component 102 executed on the client machine 100 can provide video game data 128 to one or more plug-ins that create new AR content 120 (e.g., sticky notes, screenshots, etc.) during game play. The AR content 120 created by this plug-in can be transient in the sense that it exists during the lifetime of the game session of the current player 106 and does not persist after the session ends. Alternatively, the AR content 120 created by the plug-in can be uploaded to the spatial database 204 so that it can be accessed in subsequent game sessions. In some embodiments, the AR content 120 created by the plug-in is shared in real time with other players 106 who are playing the same video game 104 or different video games. In this case, the remote computing system 200 acts as a server that relays the AR content 120 between client machines 100 during the game session. The author 216 may also use the content creation API to specify access permissions associated with new AR content and / or content previously created by the author 216. The content creation API may also allow AR content to be added to pre-existing screenshots or video clips associated with one or more video games 104.

[0047] Figure 2 A spatial database 204 is shown for storing AR content 120 created by an author 216. The spatial database 204 can store AR content 120 and various types of data (including but not limited to Figure 2 ) is associated with the type of data shown in . For example, Figure 2 It is shown how the spatial database 204 may contain multiple records of AR content 120 (1), 120 (2), ..., 120 (N). Each record of AR content 120 may be associated with a game identifier (ID) 224 that uniquely identifies a video game 104 within the video game catalog 214, for example. In this way, each record of AR content 120 may be tied to a specific video game. In some embodiments, individual records of AR content 120 may be associated with multiple game IDs 224 for multiple different video games, or the spatial database 204 may maintain separate records to associate the same AR content 120 with multiple different game IDs 224 for different video games. In some embodiments, the game ID 224 may allow aspects of a video game to be specified at any suitable level of granularity, such as a level of a video game (if the video game has multiple levels). For example, a record of AR content 120 may be associated with level 3 of a specific video game, but not with other levels.

[0048] Each record of the AR content 120 may also be associated with an AR channel ID 230 that uniquely identifies the AR channel. Figure 3 ) describes AR channels in more detail. In short, the AR channel can act as a filtering criterion to filter out irrelevant AR content 120 and send relevant AR content 120 to the client machine of the user 106 based on the AR channel subscription of the user 106.

[0049] Individual records of AR content 120 may also be associated with game world coordinates 226. Game world coordinates 226 may be considered to be spatial data 227 that specify particular coordinates within a game world of a particular video game, the game world being defined by a coordinate system. In this way, whenever game world coordinates 226 associated with a record of AR content 120 are to be rendered in a frame in order to present a portion of the game world to a player 106 of a video game 104, the AR content 120 associated with those game world coordinates 226 may be identified and used to generate an enhanced frame 122 containing the AR content 120. In an illustrative example, an author 216(1) may create AR content 120, such as an information message, for a given level of a video game to be presented in a doorway whenever the doorway is rendered in a frame in a series of frames. It should be appreciated that multiple records of AR content 120 may be associated with the same game world coordinates 226 (e.g., a doorway on a given level of a video game), and that some or all of the AR content 120 associated with those game world coordinates 226 may be presentable to a given player 106 of the video game, depending on the access permissions that the player 106 has to access the AR content 120. For example, multiple information messages may be associated with a doorway on level 3 of a particular video game, and some or all of those information messages may be visible as AR content 120 to a given player 106 when the doorway is in the field of view of a player-controlled character 126.

[0050] Referring again to the game ID 224, the game ID 224 may also be used to distinguish between multiple instances of game world coordinates 226 within the game world of the video game 104. In other words, the game ID 224 may make the game world coordinates 226 unique in situations where the game world coordinates are ambiguous. Consider an example in which the author 216(2) wants to attach a hologram as AR content 120 to a car provided as video game content 124. The AR component 102 executing on the client machine 100 may need to know which car, among potentially many cars of the same make and model, to attach the AR content 120 (e.g., a hologram) to. For mobile objects such as cars that can move around in the game world, the game world coordinates 226 associated with such mobile objects may be expressed relative to the mobile object, rather than relative to a portion of the game world environment outside of the mobile object. In this case, the game world coordinates 226 associated with the mobile object may not be sufficient to fully distinguish the portion of the game world to which the AR content 120 (e.g., a hologram) is to be attached, and the game ID 224 may therefore be used to fully distinguish between multiple instances of the game world coordinates 226.

[0051] As another example of how the game ID 224 may be used, consider a virtual hotel that appears in multiple different locations in the game world of the video game 104. Although the video game 104 may express the game world coordinates 226 of each hotel relative to the hotel itself (e.g., as if the hotel itself were a small game world), each instance of the hotel may be uniquely identified by a different game ID 224. In general, it should be understood that the game world coordinates 226 may not be truly similar to real world coordinates due to various aspects of the video game that are not shared by the real world (e.g., portals connecting different locations, pre-built rooms that are stitched together in a different order each time the video game is loaded (each session), etc.). For these and other reasons, the game ID 224 can help distinguish multiple instances of the same game world coordinates 226.

[0052] In an illustrative example, the recording of AR content 120 may correspond to a screenshot of a portion of the game world captured by a character controlled by a player (e.g., Figure 1). When the screenshot is captured, it may have been associated with the game world coordinates 226 and the game ID 224, as well as the camera orientation when the screenshot was captured. This data may be uploaded along with the associated AR content 120 to create a new record in the spatial database 204. Thus, during game play, when the video game 104 provides video game data 128 in the form of spatial data specifying the current game world coordinates 226 and the current game ID 224 associated with a player-controlled character 126, the screenshot associated with the spatial data may be rendered as AR content 120 on top of the video game content 124 for the frame, thereby allowing the first player 106 to see the same game world snapshot as that seen by the second player 106. In some embodiments, the actual screenshots are not displayed unless and until the current player's camera orientation matches the camera orientation associated with the screenshots, and in addition, when the camera orientations do not match, these screenshots may be presented as "floating" images in the game world, much like Figure 1 Example first AR content 120(1) is shown. In this way, if multiple screenshots are captured in the same location of the game world and uploaded to the remote computing system 200 as AR content 120, a given player 106 can see a cluster of floating images that can be viewed whenever the player 106 aligns his / her player-controlled character 126 with the camera heading associated with those screenshots.

[0053] Individual records of AR content 120 may also be associated with states 228. To illustrate how states 228 may be used, consider a video game that presents a game world that dynamically changes between different states over time (e.g., when a particular event occurs that changes what happens in the game). In an illustrative example, the game world may be presented in a first state before a boss is defeated and in a second state after the boss is defeated. In this sense, individual records of AR content 120 may be associated with these different game world states by means of states 228. That is, a first AR content 120 associated with a particular game world coordinate 226 may be presented in a first state of the game world by virtue of its association with the first state 228, and when the state of the game world changes, the first AR content 120 may be removed, and a second AR content associated with the same game world coordinate 226 may be presented in a second state of the game world by virtue of its association with the second state 228, and so on. In the illustrative example, when the player 106 starts a boss battle, the player 106 may see first AR content 120 in the form of an information message wishing the player "good luck" in battling the boss, and then when the player 106 defeats the boss, the player 106 may see second AR content 120 in the form of an information message congratulating the player on defeating the boss.

[0054] As yet another example of how the state 228 may be used, consider a video game that can be played in different modes (e.g., a solo mode where each player plays on their own, a partner mode where players play in pairs, a squad mode where players play in larger groups, etc.). These modes can be played independently, but within the same game world of the video game. Thus, the recording of AR content 120 can be associated with a state 228 corresponding to a particular game mode. In general, the state 228 can be any state used to filter the context (not necessarily the spatial context) in which the AR content 120 is to be rendered in the augmented frame 122.

[0055] Individual records of the AR content 120 may also be associated with pixel data 232. The pixel data 232 may be specifically associated with the AR content 120 in the form of a screenshot captured by the player 106 of the video game. For example, the pixel data 232 may include a 2D array of per-pixel values ​​(e.g., color values) to reconstruct a 2D screenshot of the game world. In some embodiments, the pixel data 232 includes per-pixel depth values ​​that provide a sense of depth to the scene. The AR component 102 of the client machine 100 may use the pixel data 232 containing 3D information about the scene to build a 3D model of the game world.

[0056] exist Figure 2The example data types shown as included in the records of AR content 120 in the spatial database 204 are merely examples, and there may be other types of data associated with specific records of AR content 120. For example, access permissions may be associated with individual records of AR content 120 to indicate specific users or user groups to whom the AR content 120 has visibility when playing a video game. For example, the AR content 120 may be associated with a tag that specifies that the AR content 120 is visible to the public, friends of the author 216 who created the AR content 120, or other specified users or user groups. In some instances, the AR content 120 may be associated with user interests, spoken languages ​​(e.g., English, Japanese, Spanish, etc.), geographic locations, times of day, etc. These types of data may act as filtering criteria to allow the AR content 120 to be sent to the requesting client machine 100 when one or more criteria are met. These types of data may additionally or alternatively serve as rendering criteria to determine whether to render the AR content 120 (e.g., render the AR content 120 if the current time corresponds to a particular time of day (e.g., within a particular time range), if the user 106 is currently located at a particular geographic location (e.g., within a particular geographic region / area), if the user 106 speaks a particular language, if the user 106 is interested in particular topics (e.g., as indicated in a user profile (where user interests are specified)), etc.).

[0057] Authors 216 can create various different types of AR content 120 through content creation interfaces (e.g., application programming interfaces (APIs)) provided by remote computing systems 200. Example types of AR content 120 include, but are not limited to: information messages (e.g., messages posted by gamers 106); virtual objects (e.g., shapes, avatars, shooting targets, etc.) - including 2D and / or 3D objects; screenshots captured by players while playing video games - including 2D and / or 3D screenshots; video clips; interactive objects (e.g., game characters or other virtual objects or graphics that move within the game world), etc.

[0058] To enable the creation of AR content 120 (e.g., static, dynamic, or otherwise interactive AR content 120), the remote computing system 200 may provide an API for the author 216 to write code (e.g., an executable program such as a plug-in, which may be implemented as a dynamic link library (DLL), a Javascript file, an .exe, etc.) in the record of the AR content 120 stored in the spatial database 204. In this case, instead of retrieving the already created AR content 120, the video game data 128 about the current state of the video game 104 may be provided as input to the executable program of the record of the AR content 120, and the executable program may generate and output the AR content 120 based on the program processing of the video game data 128. In this sense, in some embodiments, the "AR content" stored in the record of the spatial database 204 may include an "executable program" that is configured to generate the AR content based on the video game data 128 input to the executable program. In some embodiments, the AR component 102 executing on the client machine 100 may create a secure sandbox, load one or more executable programs or plug-ins (e.g., DLLs) corresponding to one or more AR channels subscribed by the user 106 of the client machine 100, and provide the plug-ins with video game data 128 so that the plug-ins run their corresponding logic and return AR content 120. For example, there may be a folder of DLLs, each representing a different plug-in. When the user 106 subscribes to one or more AR channels, the AR component 102 may load one or more corresponding DLLs within the secure sandbox, and then, for each frame to be rendered, the AR component 102 may provide the video game data 128 as input to the one or more corresponding DLLs that have been loaded, and may receive as output from the one or more DLLs the AR content 120 to be rendered in the frame as an enhanced frame 122. In an illustrative example, the author 216 may create a plug-in to allow animated objects (e.g., game characters) to be added to the game world of the video game 104 as an overlay of the AR content 120. Using such a plugin layer, the author 216 can create a secondary game that runs separately from the base (or core) video game 104 and is rendered as an overlay on the video game content of the video game 104. In this sense, the video game 104 does not need to know or care about the AR content 120 rendered on top of the video game content 124, but the interactive AR content 120 can still depend on the video game data 128 about the current state of the video game 104 so that the interactive AR content 120 can be presented in the appropriate context within the game world (e.g., at the appropriate location and in a sensible manner, at the appropriate time, given the geometry of the game world, etc.).In this way, the player 106 can interact with the AR content 120 by providing user input to control the player-controlled character 126. The AR plug-in can be executed locally on the client machine 100, or it can be executed remotely at the remote computing system 200. In the latter case, the AR content 120 can be received in real time by the client machine 100 over the network 218. The user 106 can select multiple executable programs (AR plug-ins) to download from the remote computing system 200, each plug-in generating AR content 120 for a specific purpose.

[0059] In some embodiments, a content creation interface (e.g., an API) provided by the remote computing system 200 may allow an author 216 to create an executable program (e.g., a plug-in) that is configured to receive as input video game data 128 related to the current scene (e.g., a 3D screenshot) of the game world being rendered on the screen of the client machine 100, and the executable program may output AR content. In this way, interactive AR content 120 may be presented in the context of the game world presented on the screen of the client machine 100. For example, an author 216 may use a plug-in to write an executable program that enables an AR game character to run around the game world of a video game, such as an enemy that the player 106 may try to shoot, capture, or otherwise interact with. The 3D screenshot data may allow such interactive content 120 to be added in a realistic manner, such as by having the character run around a wall in the game world instead of running through it. For video games 104 with similar player movement behaviors, similarly sized worlds, and / or similar game logic, the author 216 may create an executable program that is compatible with and works across multiple video games. In this way, an AR system including an AR module 212 can promote a culture of secondary game development, where authors 216 like to use plugin creation APIs to create secondary AR-based games that run "on top" of multiple different video games (especially video games with similar game worlds and player movement behaviors). In this sense, authors 216 using the AR system disclosed herein can actually be game developers engaged in developing secondary AR-based games. For example, a plugin can be used to create an AR game session that uses game states from multiple game instances to generate AR content 120, which can then be shared among multiple client machines 100. Users 106 of those client machines 100 may be able to participate in these AR game sessions without executing the same video game - some AR games may be designed to allow each user to be in a different game or in a video game in the form of an AR viewer. User interactions within an "AR game session" can be mediated by a network 218 (e.g., even if user interaction through the network 218 is not supported in one or more specific video games, the AR platform enables users to interact with each other).

[0060] Game developers can participate in this AR system by adding secondary features to their game code to support the use of executable programs that provide interactive AR content 120 as described herein. For example, a video game 104 can be developed by a game developer whereby the video game is configured to emit data to and receive data from an AR component 102 executing on a given client machine 100 for adding interactive AR content 120 as an additional layer on top of the video game content 124. For example, a game developer can add one or more lines of code to their video game 104 that, whenever a bullet is fired in the game, emits data that specifies a vector (or ray) within the game world that provides a directionality and possibly a magnitude (e.g., speed) associated with a bullet fired by a character 126 controlled by a player in the video game 104. The AR component 102 can receive this emitted data and provide it as input to an executable program for a particular record of AR content 120 associated with the event data corresponding to the bullet firing. The executable program can respond to this event by generating an AR-based shooting target that is positioned within the game world and that the player 106 can attempt to hit using the weapon (e.g., gun, knife, etc.) of the player-controlled character 126. The executable program can continue to receive data emitted by the video game 104 that informs the executable program about whether the AR-based shooting target has been hit, and the executable program can output an explosion target so that the player 106 can interact with the AR-based shooting target by shooting at the AR-based shooting target and causing it to explode. During the warm-up phase of the video game 104, this may be useful for the player 106 to practice their shooting rather than just running around aimlessly in the game world. In this sense, game developers can provide continuous support for such AR-based interactivity in their games by updating their video games 104 to emit and / or receive specific data used by the executable program written by the author 216 using the plugin layer.

[0061] In some embodiments, a content creation interface (e.g., API) provided by the remote computing system 200 may allow authors 216 to write executable programs (e.g., plug-ins) that provide data to the video game 104 as an additional layer of interactivity with the video game 104. For example, the API may be used to write programs (e.g., plug-ins) that send data to the video game 104 during its execution in order to control when a player-controlled character 126 gains or loses health, or to control when and / or where enemies appear in a particular game. Game developers may have to write game code that supports such two-way interaction with the AR component 102 executed on the client machine 100, and game developers may determine when to enable such functionality (e.g., by disabling interactive AR content 120 during online competitive play to prevent cheating, but allowing interactive AR content 120 during single-player mode to provide a more vivid and interesting AR experience on top of the basic video game 104). In some embodiments, different AR plug-ins (executable programs) written by different authors 216 may interact with each other by passing data back and forth at the management of the AR component 102 during video game execution. For example, the author 216 may create a “utility” plugin that provides “quality of life improvements” to the player-controlled character 126 (e.g., an AR plugin for visualizing useful information that is omitted from the built-in user interface of the video game 104).

[0062] In some embodiments, the remote computing system 200 may maintain a repository of 3D models of the game world 234. As mentioned, the SLAM process may be performed offline by the remote computing system 200 to reconstruct the game world geometry and store the game world geometry in the 3D model of the game world 234. The 3D model may be part of the game world, so that multiple 3D models may constitute the entire game world. SLAM is a technique for building a model (or map) based on recognized shapes, points, lines, etc. presented based on image data capturing at least a portion of the environment in an unknown terrain environment using, among other things, computational algorithms. For example, a 3D model may represent an aggregated area of ​​the game world reconstructed from many game frames (e.g., from a dense, time-continuous video stream (with or without depth buffer data) of color data) via SLAM. This reconstruction may be done as a separate backend process so that the 3D model 234 is available a priori before the AR plugin begins executing on the client machine 100. For a given video game, the 3D reconstruction process may be performed by the service provider of the video game platform, by the game developer of the video game 104, or by crowdsourcing video game screenshots from player client machines 100. User consent may be obtained prior to obtaining such data from the client machine 100 of the user 106. Furthermore, the user 106 may choose to participate in the process by voluntarily exploring the game world and uploading screenshot data to the remote computing system 200 for the purpose of building the 3D model 234.

[0063] The SLAM-created 3D model 234 may be retrieved by the client machine 100 for the purpose of identifying the game world from the image of the video game content 124 and determining the camera pose for adding the AR content 120 in the game. In some embodiments, the 3D model 234 may be obtained by the client machine 100 to automatically inject the AR content 120 into an unmodified video game and / or pre-existing screenshots and / or videos of the video game. This may enable a "streaming" use case where the video game 104 outputs a video stream without the AR content 120 and the AR system (e.g., the AR component 102 and / or the AR module 212) overlays the AR content 120 on top of the video stream (e.g., by obtaining sufficient information from the video game, such as the camera pose per frame).

[0064] Figure 3 is a diagram illustrating an example system 300 by which a client machine 100 can subscribe to an AR channel that determines the AR content 120 received by the client machine 100. In general, the AR content 120 may be received by the client machine 100. Figure 1The client machine 100, which is the same as the client machine 100 described above, may receive AR content 120 from the remote computing system 200 over the computer network 218 before and / or during execution of the video game 104. This may involve real-time streaming of the AR content 120 during execution of the video game 104, or when the AR content 120 is to be rendered as part of the enhanced frame 122, to reduce latency and / or network bandwidth consumption, the AR content 120 may be downloaded from the remote computing system 200 and may be accessed from the local memory of the client machine 100. In some embodiments, whenever the user 106 starts the video game 104 on the client machine 100, the video game client 116 sends a request containing an identifier of the video game 104 (e.g., the game ID 224) to the remote computing system 200 over the computer network 218, and the remote computing system 200 uses the game ID 224 to look up a record of the AR content 120 associated with the game ID 224 within the spatial database 204. Thus, of the available AR content 120 maintained in the spatial database 204, the remote computing system 200 may identify a subset of the available AR content 120 associated with the game ID 224 it received in the request from the client machine 100. This identified AR content 120 may be further filtered based on one or more filtering criteria, so that a filtered subset of AR content 120 is ultimately sent to the client machine 100, the filtered subset of AR content 120 being the AR content 120 associated with the game ID 224 that also satisfies the one or more filtering criteria. This filtering criteria may include one or more criteria that are satisfied if the user 106 sending the request has access rights to the AR content 120. For example, in the case where the record of the AR content 120 is visible only to a specific user group, the record of the AR content 120 is sent to the client machine 100 if the logged-in user account sending the request is included in the specific user group. Otherwise, if a user 106 not included in the particular user group sends a request for the AR content 120 , a record of the AR content 120 may not be sent to the client machine 100 , depending on why the user 106 may not have access rights to the AR content 120 .

[0065] Figure 3 Another example filtering criterion shown in is whether the AR content 120 is associated with a channel to which the user 106 of the client machine 100 has subscribed. Figure 3A user interface 302 is shown that a video game client 116 may display on a client machine 100. The user interface 302 may include a list of menu items 304 for the user to select for navigating to different aspects of the video game service provided by the remote computing system 200. For example, a "Store" menu item 304 may allow a user 106 of a client machine 100 to browse content such as video games within a video game catalog 214. A "Library" menu item 304 may allow a user 106 to browse a library of content, such as a video game library 118 accessible to the client machine 100 because the user 106 has acquired (e.g., purchased, rented, leased, etc.) a video game. A "News" menu item 304 may allow a user 106 to browse news articles published by a content publishing entity. A "Community" menu item 304 may allow a user 106 to interact with other users 106 of the video game service (e.g., friends and other users of the community). An "In-Game Augmented Reality" menu item 304 may allow a user 106 of a client machine 100 to access various AR features provided by the AR system disclosed herein. Figure 3 306, which specifies the types of AR content 120 that are sent to the client machine 100 over the computer network 218 and are therefore made visible to the user 106 of the client machine 100. In other words, the available AR content 120 maintained by the remote computing system 200 may be divided into AR channels 306, and the user 106 may subscribe to one or more of those AR channels 306, and possibly switch between channels in order to view different types of AR content 120 during execution of the video game 104. The AR channels 306 may also be considered “layers” of AR content 120 that become visible or remain hidden based on the user 106’s channel subscriptions.

[0066] Figure 3Four example AR channels 306(1)-(4) to which the user 106 may subscribe are shown, with a "Subscribe" button next to each AR channel 306 that the user 106 has not yet subscribed to. For channels that have already been subscribed to, an indication that the user 106 has subscribed to the AR channel 306 may be provided next to the AR channel 306, such as a "check box" next to the "Screenshots" channel 306(2), indicating that the user 106 has subscribed to the AR channel 306(2). For example, the user 106 may subscribe to the "Fan Fiction" channel 306(1) to make AR content 120 related to fan fiction visible during execution of the video game 104 (e.g., multiple information messages that tell a story and may give hints as to where to find the next information message (similar to geocaching) may be placed throughout the game world). The user 106 may subscribe to the "Screenshots" channel 306(2) to make AR content 120 including screenshots visible during execution of the video game 104. User 106 may subscribe to a “Friends” channel 306(3) to make AR content 120 associated with one or more of user 106’s friends visible during execution of video game 104. This type of AR channel 306(3) may be provided with a drop-down menu or similar selection mechanism to select the specific friends and / or groups of friends that user 106 wants to subscribe to. User 106 may subscribe to a “Pro Gamers” channel 306(4) to make AR content 120 associated with one or more professional gamers visible during execution of video game 104. Furthermore, this type of AR channel 306(4) may be provided with a drop-down menu or similar selection mechanism to select the specific professional gamer or group of gamers that user 106 wants to subscribe to. These are merely examples of types of AR channels 306 to which user 106 may subscribe.

[0067] In some embodiments, the user 106 may subscribe to an AR channel 306 that selects only AR content 120 that is currently popular (e.g., a volume of community activity with AR content 120 above a threshold, a volume of views or votes above a threshold, popular AR content 120 among the user's 106 friends, etc.) and makes it visible. In other words, one or more of the AR channels 306 may be configured to selectively make visible a subset of the AR content 120 of the AR channel 120 that the user 106 may be interested in based on a certain level of heuristics indicating that other users of the community are interested in the AR content 120. In some embodiments, the AR module 212 of the remote computing system 200 may be configured to provide the player 106 who is currently playing the video game with an option to subscribe to the AR channel 306 on-the-fly during gameplay. For example, a friend of the player 106 may currently be playing a secondary AR-based game of the type of treasure hunt game, and the player 106 using the client machine 100 may be presented with a pop-up option to subscribe to an AR channel 306 that provides AR content 120 for the same treasure hunt game that his / her friend is enjoying at the moment. After subscribing to the AR channel 306 for the treasure hunt game, the client machine 100 may download the AR content 120 that provides the treasure hunt game as a secondary game running on top of the core video game 104 executed on the client machine 100. As another example, the player 106 may be playing a video game 104 and the video game may be streamed to a user community. The player 106 using the client machine 100 may see the player's 106's avatar presented in the video game world and the number of viewing users on the avatar (e.g., 10k, representing 10,000 viewers) within the game world of the video game 104 that he / she is currently playing, thereby indicating that the AR content 120 is currently popular. A player 106 using a client machine 100 may click on an avatar or a subscribe button next to the avatar to bring up a 2D or 3D broadcast of the player's 106 gameplay.

[0068] Again specific reference Figure 2In response to the user 106 selecting a subscription button associated with a particular AR channel 306, the client machine 100 may send a subscription request to the remote computing system 200 via the computer network 218. Upon receiving the subscription request, the remote computing system 200 may identify AR content 120 that satisfies the filtering criteria (e.g., AR content 120 associated with the subscribed AR channel 306), and may send the AR content 120 to the client machine 100 for use in generating an enhanced frame 122 with the AR content 120. Thus, the client machine 100 receives the AR content 120 that satisfies one or more filtering criteria (e.g., AR content 120 associated with the subscribed AR channel 306), and the client machine 100 may store the AR content 120 in a local memory of the client machine 100 and associate it with relevant data (e.g., spatial data such as game world coordinates 226, indexes 228, camera orientation data, pixel data 232, event data, etc.). In this way, whenever the AR content 120 is to be rendered in the enhanced frame 122 , the AR content may be retrieved from the local memory, which may reduce latency and / or network bandwidth consumption.

[0069] In some embodiments, the one or more filtering criteria used to filter the AR content 120 that is ultimately sent to the client machine may further include filtering criteria intended to reduce local memory consumption and / or network bandwidth consumption. For example, consider a scenario where the amount of AR content 120 associated with a particular game is so large that it is difficult to download all relevant AR content to the local memory of the client machine 100. In such a scenario, the relevant AR content 120 may be filtered based on recency (e.g., creation date of the AR content 120), popularity (e.g., based on the number of views, the number of votes, etc.), the author of the AR content 120 (e.g., giving priority to AR content 120 created by friends relative to the general user), known user interests about the user 106 (e.g., user interests specified in the user profile, based on usage history, etc.), data volume (e.g., only downloading AR content 120 with less than a threshold data volume), etc. In this way, the most relevant AR content 120 may be selected, and less relevant AR content 120 may be filtered out, so that the client machine 120 receives only the most relevant AR content 120.

[0070] Figure 3Another AR feature depicted in FIG. 1 is an “AR viewer” 308 feature. This feature allows a user 106 to select an AR viewer 308 link to invoke an AR viewer, which is considered herein to be a type of video game 104 that can display AR content 120 outside the context of video game content 124. For example, the AR viewer can present the AR content 120 on a blank background or as an overlay on one or more SLAM-created 3D models of the game world. In any case, the use of the AR viewer 308 feature does not rely on loading the video game 104, and can therefore be executed on a client machine 100 (e.g., a phone) whose processing resources may not be suitable for running a PC game. In an illustrative example, a user 106 may want to see what his / her friend is currently looking at while playing a video game 104 enhanced with AR content 120. The user 106 can select the AR viewer 308 link on his / her phone. The AR viewer 308 link and a sub-link to a friend's current game session submit a query to the remote computing system 200 along with the game ID, the location of the friend's game character currently located in the game world, and the camera orientation. Based on this data, the remote computing system 200 can access the spatial database 204 and return the AR content 120 visible at the location given the camera orientation. This AR content 120 can be presented on the user's phone in a series of frames (e.g., in 3D), and the user 106 can look around and see his / her friend's character running around in the game world that the friend is currently experiencing. In this way, the user 106 can watch the AR content 120 that his / her friend is currently experiencing at the remote location.

[0071] The processes described herein are presented as a collection of blocks in a logical flow diagram, which represent a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, a block represents a computer-executable instruction that, when executed by one or more processors, performs the described operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any number of the described blocks may be combined in any order and / or in parallel to implement the processes.

[0072] Figure 4 4 is a flow chart of an example process 400 for providing an author with a content creation interface to create AR content; storing the created AR content in a spatial database; and sending selected AR content to a requesting client machine based on one or more filtering criteria. For discussion purposes, process 400 is described with reference to the previous figures.

[0073] At 402, the remote computing system 200 may provide an interface (eg, an API) for the user computing device 220 to create new AR content 120. The author 216 using the user computing device 220 may use this content creation interface to create any suitable type of AR content 120 as described herein.

[0074] As shown in sub-block 404, in some embodiments, providing a content creation interface at block 402 may allow the author 216 to create one or more executable programs (e.g., plug-ins) configured to generate interactive AR content 120 based on video game data 128 provided as input to the one or more executable programs. Such video game data 128 may include, but is not limited to, spatial data that specifies, among other things, the current location of a player-controlled character or some other object in the video game, event data that indicates the occurrence of a game-related event (e.g., a bullet being fired, a game character entering or exiting a vehicle, a porch, etc.).

[0075] At 406, the remote computing system 200 may receive instructions from the user computing device 220 through the interface for creating the new AR content 120. These instructions may include user input provided to an interface in the form of a graphical user interface that allows the author 216 to specify the type of AR content and other parameters related to the AR content, such as the game ID 224 of the video game, special data 227 (such as game world coordinates 226 within the game world of the video game at which the AR content 120 is to be presented), event data, and the like.

[0076] At 408, the remote computing system 200 may store the new AR content 120 in the spatial database 204. As shown in sub-block 410, storing the new AR content 120 in the spatial database 204 may involve associating the new AR content 120 with associated data within the spatial database 204. The associated data may include, but is not limited to, a game ID 224 of the video game, game world coordinates 226 within the game world of the video game, an index 228 for distinguishing the game world coordinates 226 when the game world coordinates 226 are otherwise ambiguous, camera orientation data (e.g., a camera orientation associated with a captured screenshot), pixel data 232 (e.g., pixel data associated with a captured screenshot), event data, and the like.

[0077] As shown in sub-block 412, storing the new AR content 120 in the spatial database 204 may involve associating the new AR content 120 with one or more AR channels 306 as described herein. In this way, the user 106 may subscribe to an AR channel 306 to receive specific AR content 120 that may be displayed within the game world of their video game.

[0078] At 414, the remote computing system 200 may receive, from the client machine 100 through the computer network 218, a subscription request from the user account to subscribe to one or more AR channels 306 as one or more subscribed AR channels 306. Figure 3 An example of an AR channel 306 is described.

[0079] At 416, the remote computing system 200 may receive video game data 128 associated with the video game 104 from the client machine 100 over the computer network 218. This video game 104 may be a video game executed by the client machine 100 (e.g., by launching a video game client 116 on the client machine 100 and executing the video game 104 via the video game client 116, causing the AR component 102 to send the video game data 128 to the remote computing system 200).

[0080] At 418, the remote computing system 200 may identify the game-specific AR content 120 among the available AR content 120 stored in the spatial database 204 based at least in part on the video game data 128. For example, the record of the AR content 120 in the spatial database 204 may be associated with the game ID 224 of the specific video game 104 executed on the client machine 100, the spatial data 227 associated with the portion of the game world to be rendered, etc., and the AR content 120 may be identified at block 418 because it is relevant to the specific video game 104 executed on the client machine 100 and to the current state of the video game 104 as indicated by the video game data 104.

[0081] At sub-block 420, the remote computing system 200 may filter the game-specific AR content 120 identified at block 418 based on one or more subscribed AR channels 306 of the user account in question. This may result in a subset of the game-specific AR content 120 associated with the subscribed AR channels 306. For example, the AR content 120 associated with the video game 104 in question may be related to multiple AR channels 306, so the AR channels 306 to which the user account in question has not subscribed may be used to filter out irrelevant AR content 120, and the remaining subset of game-specific AR content 120 may be a subset associated with one or more subscribed AR channels 306. At sub-block 422, the remote computing system 200 may further filter the game-specific AR content 120 based on one or more filtering criteria to obtain a subset of the game-specific AR content 120 to be ultimately sent to the client machine 100. If the popularity of the subset of game-specific AR content 120 is greater than a popularity threshold, the example filtering criteria may be satisfied at block 422. In this example, popularity may be determined based on at least one of a number of views or a number of upvotes associated with a subset of the game-specific AR content 120, and / or based on the user's previous activity or the user's social network relationships. As described herein, other filtering criteria are also contemplated, such as filtering game-specific AR content 120 based on access permissions associated with a user account, based on a creation date of the AR content 120 (filtering out AR content older than a threshold), based on user interests associated with a user account, based on time of day, based on a geographic location associated with a client machine 100 or a user account, and the like.

[0082] At 424, the remote computing system 200 may send the subset of the game-specific AR content 120 to the requesting client machine 100 over the computer network 218. In this manner, the client machine 100 is provided with access to the relevant AR content 120, which may be rendered “on top” of the video game content 124 in the enhanced frame 122 during execution of the video game. It should be appreciated that at least blocks 416-424 may occur in real-time during game execution to provide real-time AR content 120 to the client machine 100 over the network 218. In some embodiments, the process 400 is performed close to the time at which the video game begins execution, such that the client machine 100 receives filtered game-specific AR content 120, which may be stored locally for access during game execution.

[0083] In some embodiments, at block 418, the remote computing system 200 may further identify, among the available AR content stored in the spatial database 204, an additional subset of the game-specific AR content 120 associated with the unsubscribed AR channel 306 based at least in part on the video game data 128. In this case, at block 424, the remote computing system 200 may also send the additional subset of the game-specific AR content 120 to the client machine 100 over the computer network for presentation on the client machine 100 in a manner that visually distinguishes the additional subset of the game-specific AR content from the subset of the game-specific AR content. That is, the AR content 120 that the user 106 has not subscribed to may be presented in a subtle manner (e.g., by a relatively small icon that indicates that the AR content is available if the user is interested in clicking on the icon). Based on the video game data 128, this unsubscribed AR content may be spatially relevant.

[0084] Figure 5 is a flow chart of an example process 500 for receiving AR content from a remote computing system 200 and rendering frames, including an augmented frame 122 including AR content 120, during execution of a video game 104. For purposes of discussion, the process 500 is described with reference to the previous figures.

[0085] At 502, the client machine 100 may send a subscription request to subscribe to one or more AR channels 306 as one or more subscribed AR channels 306 to the remote computing system 200 via the computer network 218. Figure 3 An example of an AR channel 306 is described.

[0086] At 504, the client machine 100 may begin executing the video game 104. For example, the user 106 of the client machine 100 may launch the video game client 116 and may select a video game 104 from the video game library 118 to begin executing the video game 104.

[0087] At 506, the client machine 100 may send a request including the game ID 224 of the video game 104 to the remote computing system 200 over the computer network 218. This may be automated logic in response to the user 106 starting execution of the video game 104 on the client machine 100. The game ID 224 allows the remote computing system 200 to look up the game-specific AR content 120 associated with the game ID 224.

[0088] At 508, the client machine 100 may receive the subset of game-specific AR content 120 and associated data from the remote computing system 200 over the computer network 218. The subset of game-specific AR content 120 received at block 508 may have been selected by the remote computing system 200 based at least in part on the game ID 224 and the one or more subscribed AR channels 306. Thus, the subset of game-specific AR content 120 may be associated with both the video game 104 and the one or more subscribed AR channels 306.

[0089] At sub-block 510, the client machine 100 may store a subset of the game-specific AR content 120 in association with associated data in a local memory of the client machine 100. The associated data may include, but is not limited to, spatial data 227, pixel data 232, event data, etc., which may be used to determine when to render specific AR content 120 in an enhanced frame 122 during game execution.

[0090] At 512, the client machine 100 may render a series of frames during execution of the video game 104 via the video game client 116. For example, when the video game 104 is executed in a first process (or one or more threads), the video game 104 may output the video game content 124 in a series of frames. The operations that may be iteratively performed at block 512 to render individual frames in the series of frames are illustrated by sub-blocks 514-524.

[0091] At 514, the AR component 102—executed by the video game client 116 on the client machine 100 as a separate process (e.g., or one or more threads) from the video game 104—can obtain video game data 128 from the video game 104 regarding the current state of the video game 104. This video game data 128 can be spatial data 227 related to game world coordinates 226 within the game world of the video game 104. For example, the spatial data 227 can specify the current position of a player-controlled character within the game world of the video game 104, a set of game world coordinates 226 of a portion of the game world to be rendered in an upcoming frame, an index 228, the current heading of a virtual camera associated with the player-controlled character, etc. The video game data 128 can be event data specifying a game-related event associated with the video game 104. In some embodiments, obtaining the video game data 128 from the video game 104 at box 514 may include: the AR component 102 receiving the video game data 128 from the video game 104 as part of a function call made by the video game 104 to request AR content 120 from the AR component 102 to render the frame as an enhanced frame 122 when necessary.

[0092] At 516, the frame may be rendered as a normal frame without any AR content 120 (i.e., only with video game content 124). At 518, the AR component 102 may determine whether the AR content 120 is identifiable based on the video game data 128. For example, the AR component 102 may compare at least a portion of the video game data 128 it obtained at block 514 with associated data stored with accessible AR content 120 (e.g., a subset of the game-specific AR content 120 received by the client machine 100 at block 508). If the AR component 102 determines that there is no identifiable AR content 120 to display, the process 500 may follow the "no" route from block 518 to advance to the next frame at block 520. On the other hand, if the AR component 102 determines that there is identifiable AR content 120 to display, the AR component 102 identifies the AR content 120 based at least in part on the video game data 120 (e.g., by comparing the video game data 128 to associated data stored with the AR content 120), and the process 500 can follow the "yes" route from block 518 to block 522. The identification of the AR content 120 can include retrieving / receiving the AR content 120 from a source other than the video game 104 (e.g., from the local memory of the client machine 100, in real time from the remote computing system 200, etc.). In the case of real-time retrieval of the AR content 120 over the network 218, the reception of the AR content at block 508 can occur in conjunction with (e.g., in response to) the identification of the AR content 120 at block 518.

[0093] At 522, the client machine 100 may generate, via the AR component 102, an enhanced frame 122 that includes the video game content 124 output by the video game 104 and the AR content 120 identified and retrieved at block 516. As mentioned, the generation of the enhanced frame 122 may include overlaying the AR content 120 on the video game content 124. In some embodiments, the operations at block 522 include: obtaining depth data associated with the video game content 124 of a given frame; and merging the video game content 124 with the AR content 120 in the enhanced frame 122 based at least in part on the depth data. For example, the AR content 120 may be layered over some video game content 124 (e.g., opaque graphics) and layered under other video game content 124 (e.g., semi-transparent graphics).

[0094] At 524 , the client machine 100 may render the enhanced frame 122 on one or more displays 110 associated with the client machine 100 via the AR component 102 , and may then proceed to the next frame at box 520 to iterate boxes 514 - 524 , as indicated by the arrow from box 520 to box 514 .

[0095] The player 106 may effectively change AR channels 306 at any time during the process 500 by subscribing to one or more different AR channels 306, which causes the client machine 100 to send a new subscription request to the remote computing system 200 so that different or additional AR content 120 may be received from the remote computing system 200 based on the one or more newly subscribed AR channels 306. In this way, the player 106 may switch between AR channels 306 during game play to present different AR content 120 in the augmented frames 122 during execution of the video game.

[0096] Figure 6 1 is a flow chart of an example process 600 for enhancing a frame with spatially correlated AR content 120 during execution of a video game 104. Operations of process 600 may involve rendering a separate frame as an enhanced frame in a series of frames rendered during execution of the video game. For purposes of discussion, process 600 is described with reference to the previous figures.

[0097] At 602, the AR component 102—executed by the video game client 116 on the client machine 100—can obtain video game data 128 from the video game 104 in the form of spatial data 227 regarding the current state of the video game 104. For example, the spatial data 227 obtained at block 602 can relate to, but is not limited to: current game world coordinates, such as the current coordinates (i.e., current position) of a player-controlled character 126 within the game world of the video game 104, a set of game world coordinates 226 of a portion of the game world to be rendered in an upcoming frame; a game ID 224, a state 228, a current camera orientation of a virtual camera (e.g., a virtual camera associated with the player-controlled character 126); or any combination thereof.

[0098] At 604 , the AR component 102 may identify AR content 120 from available AR content (eg, a subset of the game-specific AR content 120 received from the remote computing system 200 ) based at least in part on the spatial data 227 .

[0099] At sub-block 606, AR content 120 may be identified based at least in part on the game world coordinates 226 in the received spatial data 227. For example, the AR component 102 may identify AR content 120 associated with game world coordinates 226 that are contained in a set of game world coordinates 226 in the spatial data 227 or that are within a threshold distance from a current position of a player-controlled character 126 and within the field of view of the game character 126 as determined from the current heading of the virtual camera.

[0100] At sub-block 610, the AR content 120 may be identified based at least in part on the game ID 224 specified in the spatial data 227. For example, the AR component 102 may identify the AR content 120 associated with the game ID that matches the game ID 224 in the spatial data 227. If the game world contains multiple instances of the game world coordinates 226 specified in the spatial data 227, this game ID may be used to distinguish between the multiple instances of the game world coordinates 226.

[0101] At 612, the client machine 100 may generate, via the AR component 102, an enhanced frame 122 that includes the AR content 120 (and possibly the video game content 124 output by the video game 104) identified (and retrieved) at block 604. As shown in sub-block 614, where the identified AR content 120 is a 3D screenshot (e.g., an image with depth data), the generation of the enhanced frame 122 at block 612 may include constructing a 3D model of a portion of the game world presented in the 3D screenshot based at least in part on the data contained in the 3D screenshot (e.g., a 2D array of pixel data plus depth data from a depth buffer). The resulting 3D model may be overlaid on the video game content 124 within the enhanced frame 122 at block 612. For example, the identified AR content 120 may be a 3D screenshot of the game world of the video game 104 currently executing on the client machine or a different game world of a different video game. In either case, the 3D screenshot may have been captured by another player 106, perhaps at an earlier time, capturing the entire context of the game world at the moment the 3D screenshot was captured. The constructed 3D model superimposed on the video game content 124 as AR content 120 can allow the player 106 of the currently executing video game 104 to navigate around the constructed 3D model (e.g., move around objects captured in the 3D screenshot) to see the "slice" of the game world captured by the other player 106 in a manner that simulates what the other player 106 saw when the 3D screenshot was captured. As shown in sub-box 616, in the case where the identified AR content 120 is a plurality of consecutive 3D screenshots, the generation of the enhanced frame 122 at box 612 can include starting playback of a 3D video based at least in part on the plurality of consecutive 3D screenshots.

[0102] At 618 , the client machine 100 may render the augmented frame 122 on one or more displays 110 associated with the client machine 100 via the AR component 102 , and may then proceed to the next frame as part of an iterative process of rendering a series of frames during execution of the video game 104 .

[0103] Figure 71 is a flow chart of an example process 700 for enhancing a frame with dynamic and / or interactive AR content 120 during execution of a video game 104. Operations of process 700 may involve rendering a separate frame as an enhanced frame in a series of frames rendered during execution of the video game. For purposes of discussion, process 700 is described with reference to the previous figures.

[0104] At 702, the AR component 102, executed by the video game client 116 on the client machine 100, can obtain video game data 128 about the current state of the video game 104 as described herein. As shown in sub-box 704, obtaining the video game data 128 can include obtaining a 3D model of the portion of the game world to be rendered in the upcoming frame. This 3D model can be retrieved from the remote computing system 200, which can pre-generate a 3D model of the game world of the video game based on a dense time-continuous video stream (e.g., containing color data with or without depth buffer data) and store the 3D model for on-demand access by the client machine 100. The SLAM process can be performed offline and can be used to incrementally reconstruct the game world geometry (e.g., a 3D model of the game world) from many images. This back-end process can be completed by a service provider of the video game platform, by a game developer, and / or by crowdsourcing game world images from player client machines 100.

[0105] At 706, the AR component 102 may identify the AR content 120 from the available AR content (e.g., a subset of the game-specific AR content 120 received from the remote computing system 200) based at least in part on the video game data 128. To identify the AR content 120, the AR component 102 may look up a record of the AR content 120 from the available AR content using the video game data 128, and may determine that the record of the AR content 120 provides an executable program configured to generate dynamic and / or interactive AR content 120. This executable program may have been created by using a plug-in as described herein.

[0106] At sub-block 708, the AR component 102 may provide the video game data 128 (e.g., the 3D model obtained at block 704) as input to an executable program configured to generate dynamic and / or interactive AR content 120 based at least in part on the video game data 128. In some embodiments, the AR component 102 may create a secure sandbox, load one or more executable programs or plug-ins (e.g., DLLs) corresponding to one or more AR channels subscribed to by the user 106 of the client machine 100, and provide the video game data 128 as input to the plug-ins so that the plug-ins run their corresponding logic and return AR content 120. For example, there may be a folder of DLLs, each representing a different plug-in. When the user 106 subscribes to one or more AR channels, the AR component 102 may load the one or more corresponding DLLs within the secure sandbox, and then run the process 700, where at sub-block 708, the AR component 102 may provide the video game data 128 as input to the one or more corresponding DLLs that have been loaded.

[0107] At sub-block 710, the AR component 102 may receive, as output, the identified AR content in the form of dynamic and / or interactive AR content 120 from the one or more executable programs. The AR component 102 may also receive, as output, from the one or more executable programs, positioning data for positioning the dynamic and / or interactive AR content 120 within the augmented frame 122, the positioning data being based at least in part on the 3D model of the portion of the game world to be rendered.

[0108] At 712, the client machine 100 may generate an augmented frame 122 via the AR component 102, the augmented frame containing the video game content 124 output by the video game 104 and the AR content 120 identified at block 706. By providing the 3D model as input to the executable program, dynamic (e.g., moving and / or animated) AR objects may be automatically injected into the game world in a manner that is sensitive to the geometry and / or topology of the game world. For example, a moving or animated AR game character may be moved within the game world by avoiding collisions with virtual objects (e.g., obstacles, walls, doors, etc.) in the game world and / or positioning the AR content against walls, on the floor, etc.

[0109] At 718 , the client machine 100 may render the augmented frame 122 on one or more displays 110 associated with the client machine 100 via the AR component 102 , and may then proceed to the next frame as part of an iterative process of rendering a series of frames during execution of the video game 104 .

[0110] Figure 8is a diagram showing an example technique for adding a multiplayer aspect to a single-player video game using the disclosed AR system. Figure 8 In the embodiment, the first client machine 100(1) and the second client machine 100(2) are each connected to a computer network 218 so as to communicate with a remote computing system 200( Figure 8 ) and exchange data with other client machines 100. Figure 8 In the example of FIG. 1 , a first player 106(1) is playing a video game 104 on a first client machine 100(1), and a second player 106(2) is also playing the same video game 104 on a second client machine 100(2). The video game 104 can be executed independently on each client machine 100 without any networking code, such that the video game 104 is not dependent on a network connection to execute. In this sense, in some examples, the video game 104 executed on each client machine can be a single-player video game that each player 106 can play on his or her own.

[0111] However, the network connection enables data 800 to be transferred between client machines 100 executing the same single-player video game 104 via the remote computing system 200, such as Figure 8As shown. This allows multiplayer aspects to be added to single-player video games without relying on the game developer to incorporate multiplayer aspects into their video game (which can be a costly endeavor). Instead, the code of the video game 104 can be configured to iteratively emit data 800 about the current state of the video game 104 (e.g., emit data 800 every frame, every two frames, every few frames, etc.). This data 800 emitted from the video game 104 executing on the second client machine 100(2) can be transmitted via the computer network 218 to the first client machine 100(1) that is executing the local AR component 102. The AR component 102 on the first client machine 100(1) can receive the data 800 sent from the second client machine 100(2) and can process the data 800 to generate an enhanced frame 122 containing AR content 120(1) that adds the multiplayer aspects to the first instance of the game world rendered on the first client machine 100(1). For example, a ghost image of the game character of the second player 106(2) may be rendered as AR content 120(1) on the screen of the first client machine 100(1) to enable multiplayer aspects, such as a "speed run," in which the first player 106(1) can see the position of the game character of the second player 106(2) relative to the game character of the first player 106(1). In this way, the first player 106(1) may compete with the second player 106(2) without having to implement multiplayer aspects into the code of the video game 104 itself. These multiplayer aspects may be added in real time by sending data 800 in real time over the network 218 as the video game 104 is played on each client machine 100. Additionally or alternatively, the game performance of the second player 106(2) may be saved and replayed at a later time during execution of the video game 104 on the first client machine 100(1) as an AR content stream. In this way, the AR content 120(1) retrieved and used to enhance the frame during game play on the first client machine 100(1) may be a live or replayed stream of the game play of the second player 106(2) in the same video game 104. In this way, the first player 106(1) may compete in real time, and / or the first player 106(1) may continually practice against a replay of the game play of the second player 106(2). In some embodiments, the frame containing the enhanced frame 122 rendered on the first client machine 100(1) may be saved and uploaded as a video clip to the remote computing system 200 so that others may replay the competitive experience as seen by the first player 106(1).

[0112] In another aspect, game world 802(2) may be stitched into game world 802(1) on first client machine 100(1) in a manner that aligns the coordinate systems of the two game worlds 802(1) and 802(2) to effectively blend the two game worlds 802(1) and 802(2) together. For example, if two player-controlled characters 826(1) and 826(2) are positioned close to each other (e.g., within a threshold distance) within the same game world of the same video game 104, first client machine 100(1) may simultaneously receive from second client machine 100(2) one or more 3D screenshots of second game world 802(2) and a set of coordinates of one or more portions of second game world 802(2) depicted in the one or more 3D screenshots, and AR component 102 of first client machine 100(1) may align the graphics in the received screenshots with the current set of coordinates in first game world 802(1) to blend the two game worlds together. For example, the first player 106(1) may be able to see objects that the second player 106(2) sees in the second game world 802(2) as AR content 120(2). The difference between the two game worlds 802(1) and 802(2) in the augmented frame 122 may be visually indicated by highlighting objects or graphics rendered as AR content 120 (e.g., using a different color, etc.). Figure 8 1, the difference between the two game worlds 802(1) and 802(2) is shown in the augmented frame 122 by the objects from the second game world 802(2) being shown in dashed lines, while the objects in the first game world 802(1) are shown in solid lines. The visual distinction may help the first player 106(1) distinguish between the video game content 124 and the AR content 120 when the AR content 120 is spliced ​​into the first game world 1002(1) in a manner that would otherwise make it difficult to discern what the AR content 120 is in the augmented frame 122.

[0113] To illustrate Figure 8 The operation of the technique shown in Fig. 9 , which is a flow chart of an example process 900 for adding multiplayer aspects to a single-player game using an AR system by exchanging data between client machines 100 via a computer network 218. Consider a scenario in which, during execution of a video game 104 on a first client machine 100(1), a first instance of a game world 802(1) is rendered on the first client machine, and during independent execution of the same video game 104 on a second client machine 100(2), a second instance of the same game world 802(2) is rendered on a second client machine 100(2). Figure 8As shown, during independent execution of the video game 104 on each client machine 100, the first player 106(1) may control the first player-controlled character 826(1), while the second player 106(2) may control the second player-controlled character 826(2).

[0114] At 902, the first client machine 100(1) may receive data 800 in the form of spatial data 227 from a second client machine 100(2) via a computer network 218, the spatial data specifying a current position of a second player-controlled character 826(1) within a game world rendered as a second instance of the game world 802(2) on the second client machine 100(2).

[0115] At 904, in order to render the enhanced frame 122 (e.g., as in Figure 8 ), the AR component 102 executing on the first client machine 100(1) can obtain video game data 128 from the video game 104 executing on the first client machine 100(1), wherein the video game data 128 is in the form of spatial data 227, wherein the spatial data specifies first game world coordinates 226 associated with a portion of the game world of the video game 104 to be rendered as a first instance of the game world 802(1) on the first client machine 100(1).

[0116] At 906, the AR component 102 of the first client machine 100(1) identifies AR content 120 for generating an enhanced frame 122. As shown in sub-block 908, the AR content 120 may be identified by determining that the current position of the second player-controlled character 826(2) is within the portion of the game world to be rendered on the screen of the first client machine 100(1) based at least in part on the first game world coordinates 226. As shown in sub-block 910, the AR content 120 may be retrieved as an AR avatar of the second player-controlled character 826(2). For example, the first client machine 100(1) may receive the AR content 120 in the form of an AR avatar from the remote computing system 200 (before or during execution of the video game 104). In this way, the first client machine 100(1) may access a record of the AR content including the AR avatar.

[0117] At 912, the first client machine 100(1) may generate, via the AR component 102, an augmented frame 122 that includes the video game content 124 output by the video game 104 executing on the first client machine 100(1) and the AR content 120 identified (and retrieved) at block 906. As shown in subblock 914, the generation of the augmented frame 122 may include presenting, as the AR content 120, an AR avatar of the second player-controlled character 826(2) on the video game content 124 at a location within the first instance of the game world 802(1) that corresponds to the current location of the second player-controlled character 826(2).

[0118] At 916, the first client machine 100(1) may render the enhanced frame 122 on one or more displays 110 associated with the first client machine 100(1) via the AR component 102, and may then proceed to the next frame as part of an iterative process of rendering a series of frames during execution of the video game 104. This instance of the enhanced frame 122 is shown in FIG. Figure 8 , where the AR content 120 in the augmented frame 122 is an AR avatar of a character 826(2) controlled by a second player at a location corresponding to the current location of the character in the game world. This allows a multi-player aspect to be added as an enhancement to a video game executed on the first client machine 100(1), which may be a single-player game.

[0119] Fig.10 is a diagram showing example techniques for sharing aspects of a game world between client machines using the disclosed AR system. Fig.10 In the embodiment, the first client machine 100(1) and the second client machine 100(2) are each connected to a computer network 218 so as to communicate with a remote computing system 200( Fig.10 ) and exchange data with other client machines 100. Fig.10 In the example of FIG. 1 , a first player 106(1) is playing a video game 104 on a first client machine 100(1), and a second player 106(2) is playing a video game 104 on a second client machine 100(2). The video games 104 executed on each client machine 100 may be the same video game or different video games, and the video games may be single-player or multi-player video games.

[0120] The network connection enables not only data to be transmitted between client machines 100 via the remote computing system 200, but also AR content 120 to be transmitted. For example, AR content 120 sent from the second client machine 100(2) to the first client machine 100(1) may represent a screenshot (e.g., a 2D or 3D screenshot) of a portion of the second game world 1002(2) rendered on the second client machine 100(2). This AR content 120 may be iteratively transmitted (e.g., every frame, every two frames, every few frames, etc.) during execution of the video game 104 on the second client machine 100(2). The AR component 102 on the first client machine 100(1) may receive the AR content 120 sent from the second client machine 100(2), and may use the AR content 120 to generate one or more enhanced frames 122 containing the received AR content 120. Thus, real-time transmission of AR content 120 in the form of screenshots over network 218 allows game worlds to be shared between client machines, such as sharing a second game world 1002(2) within a first game world 1002(1) rendered on a first client machine 100(1). For example, as a second player 106(2) controls a second player-controlled character 1026(2) to move around in the second game world 1002(2), a screenshot of the second game world 1002(2) as seen from the perspective of the second player-controlled character 1026(2) can be sent as AR content 120 over network 218 to the first client machine 100(1). As the first player 106(1) controls the first player-controlled character 1026(1) to move around in the first game world 1002(1), the received screenshots can be presented in an augmented frame 122 as AR content 120, such as by rendering a screenshot of a "portal" through a wall in the first game world 1002(1). This can enable different types of functionality related to sharing of game worlds between client machines.

[0121] For example, the AR content 120 may provide the first player 106(1) with a viewport into the second game world 1002(2). In this case, one or more screenshots of the second game world 1002(2) as seen from the perspective of the character 1026(2) controlled by the second player may be transmitted to the first client machine 100(1) for display within the first game world 1002(1) as the AR content 120. The series of screenshots may be transmitted over the network 218 to enable the gameplay of the second player 106(2) to be viewed in real time. For example, the first player 106(1) may be able to view the gameplay of the second player 106(2) through the AR content 120 presented as a viewport into the second game world 1002(2). The series of screenshots transmitted over the network 218 may be presented as a live AR video stream via the AR component 102 on the first client machine 100(1) so that the first player 106(1) can watch the character 1026(2) controlled by the second player move in the second game world 1002(2). To enable a 3D viewport into the second game world 1002(2), the first client machine 100(1) may receive AR content 120 in the form of one or more 3D screenshots, and the AR component 102 on the first client machine 100(1) may construct a 3D model of the game world 1002(2) depicted in the 3D screenshots and / or present the 3D screenshots positioned based on the camera orientation associated with the 3D screenshots. This may allow the first player 106(1) to look around and / or move around the reconstructed 3D model of the second game world 1002(2) to see even more details about the environment of the character 1026(2) controlled by the second player.

[0122] Another example of sharing game worlds between client machines 100 involves stitching or blending two game worlds 1002(1) and 1002(2) together in a manner that aligns the coordinate systems of the two game worlds 1002(1) and 1002(2). For example, if two player-controlled characters 1026(1) and 1026(2) are positioned close to each other (e.g., within a threshold distance) within the same game world of the same video game 104, a first client machine 100(1) may simultaneously receive from a second client machine 100(2) one or more 3D screenshots of the second game world 1002(2) and a set of coordinates of one or more portions of the second game world 1002(2) depicted in the one or more 3D screenshots, and the AR component 102 of the first client machine 100(1) may align the graphics in the received screenshots with the current set of coordinates in the first game world 1002(1) to blend the two game worlds together. Imagine that a character 1026(1) controlled by a first player is located at the same location within the game world as a character 1026(2) controlled by a second player, and that the first player 106(1) is able to see a 3D rendering of an enemy (AR content 120) that the second player 106(2) sees in the second game world 1002(2). The difference between the two game worlds 1002(1) and 1002(2) in the augmented frame 122 may be visually indicated by highlighting objects or graphics rendered as AR content 120 (e.g., using a different color, etc.). This may help the first player 106(1) distinguish between the video game content 124 and the AR content 120 when the AR content 120 is spliced ​​into the first game world 1002(1) in a manner that would otherwise make it difficult to discern what the AR content 120 is in the augmented frame 122.

[0123] For reference Fig.10 The operation of the described technique is now referred to Fig.11 , which is a flow diagram of an example process 1100 for sharing aspects of a game world between client machines using an AR system.

[0124] At 1102 , the AR component 102 —executed by the video game client 116 on the first client machine 100 ( 1 ) —may obtain video game data 128 as described herein from the video game 104 executing on the first client machine 100 ( 1 ).

[0125] At 1104, the AR component 102 of the first client machine 100(1) identifies the AR content 120 for generating the enhanced frame 122. As shown in sub-block 1106, the AR content 120 may be received from the second client machine 100(2) via the computer network 218 to identify the AR content 120. For example, the AR content 120 may be in the form of one or more screenshots of a portion of a game world 1002(2) rendered on the second client machine 100(2), which may be the same game world as the game world 1002(1) rendered on the first client machine 100(1) or a different game world. In some embodiments, the first client machine 100(1) may receive the AR content 120 from the second client machine 100(2) along with additional data 800, such as spatial data 227 relating to game world coordinates 226 of the second game world 1002(2) rendered on the second client machine 100(2). This may include coordinates corresponding to the current position of the second player-controlled character 1026(2) within the second game world 1002(2).

[0126] At 1108, the first client machine 100(1) may generate, via the AR component 102, an augmented frame 122 that includes the video game content 124 output by the video game 104 and the AR content 120 identified (and retrieved) at block 1104. In some embodiments, the video game data 128 obtained at block 1102 may allow the AR content 120 to be provided in the context of the game world 1002(1) in the augmented frame 122 to be rendered, such as by positioning the AR content 120 in the form of a screenshot on a wall in the game world 1002(1) or otherwise attaching the AR content 120 to an object in the game world 1002(1). As shown in sub-blocks 1110, 1112, and 1114, the generation of the augmented frame 122 may be implemented in a variety of ways.

[0127] At sub-block 1110, the AR component 102 of the first client machine 100(1) may construct a viewport into the game world 1002(2) rendered on the second client machine 100(2) based at least in part on the AR content 120 (e.g., one or more screenshots) received at block 1106. For example, the 3D screenshots may be used to reconstruct a 3D view of the game world 1002(2) of the second player 106(2) within a portal rendered as an AR object in the first game world 1002(1). The series of screenshots may be rendered much like a video stream of the AR content 120 to provide a live viewport that may be used to watch the gameplay of the second player 106(2). In some embodiments, the viewport constructed at sub-block 1110 may include other AR content 120 that the second player 106(2) of the second client machine 100(2) sees within the second game world 1002(2).

[0128] At sub-block 1112, the AR component 102 of the first client machine 100(1) may stitch together the game world 1002(2) rendered on the second client machine 100(2) with the game world 1002(1) rendered on the first client machine 100(1) based at least in part on the game world coordinates 226 received from the second client machine 100(2). For example, if both instances of the game worlds are within a threshold distance from a common location within the game worlds, the current game world coordinates may be aligned. In some embodiments, the AR content 120 is in the form of a 3D screenshot of the game world 1002(2), which may be used to obtain a camera pose associated with the 3D screenshot. The camera pose may be used to align and / or orient other AR content 120 (e.g., objects, game characters, etc.) within the game world 1002(1).

[0129] At 1116, the first client machine 100(1) may render the enhanced frame 122 on one or more displays 110 associated with the first client machine 100(1) via the AR component 102, and may then proceed to the next frame as part of an iterative process of rendering a series of frames during execution of the video game 104.

[0130] Fig.12 is a flow diagram of an example process 1200 for exchanging events between a video game and a separately executed AR component on a client machine.

[0131] At 1202 , the AR component 102 may be executed by the video game client 116 on the client machine 100 as a separate process from the video game 104 executing on the client machine 100 .

[0132] At 1204, the AR component 102 may receive one or more game-issued events from the video game 104. For example, a game developer may specify game events that the video game will issue during execution, and the author 216 of the AR content 120 may subscribe to those “game-issued events” to understand what is happening in the video game 104 for the purpose of generating the AR content 120 and returning the game-accepted events to the video game 104.

[0133] At 1206, the AR component 102 may identify the AR content 120. This identification may be based on providing one or more game-issued events as input to one or more executable programs (e.g., one or more plug-ins). In some embodiments, the identification may additionally or alternatively be based on video game data 128 (e.g., spatial data 227) obtained from the video game 104.

[0134] At 1208, the AR component 102 can send one or more game-accepted events to the video game 104 in response to receipt of the one or more game-issued events. Likewise, a game developer can specify the events that the video game 104 can accept. In some embodiments, one or more executable programs (e.g., one or more plug-ins) can output these game-accepted events based at least in part on the game-issued events provided as input.

[0135] At 1210 , the client machine 100 may generate, via the AR component 102 , an enhanced frame 122 that includes the video game content 124 output by the video game 104 and the AR content 120 identified at block 1206 .

[0136] At 1212 , the client machine 100 may render the augmented frame 122 on one or more displays 110 associated with the client machine 100 via the AR component 102 , and may then proceed to the next frame as part of an iterative process of rendering a series of frames during execution of the video game 104 .

[0137] Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.

Claims

1. A system for providing enhanced frames for a video game, the system comprising: one or more processors; as well as a memory storing computer-executable instructions that, when executed by the one or more processors, cause operations including: executing a video game that outputs video game content as a series of frames, the video game content representing a game world of the video game; identifying augmented reality content based at least in part on video game data regarding a current state of the video game; generating an enhanced frame including the augmented reality content rendered within the game world, wherein the enhanced frame is a composite frame including the video game content and the augmented reality content; as well as The enhanced frame is rendered for display.

2. The system of claim 1, wherein the identification of the augmented reality content comprises: providing the video game data to the executable program via an augmented reality component; as well as The augmented reality content generated by the executable program based at least in part on processing the video game data by the executable program is received by the augmented reality component.

3. The system of claim 1, wherein the video game data specifies game world coordinates associated with the portion of the game world to be rendered, and wherein the augmented reality content is identified based at least in part on the game world coordinates.

4. The system of claim 3, wherein the video game data further specifies a game identifier for the video game, and wherein the augmented reality content is identified based at least in part on the game identifier.

5. The system of claim 1, wherein the video game data specifies a current orientation of a virtual camera associated with a player-controlled character, and wherein the augmented reality content is identified based at least in part on the current orientation of the virtual camera.

6. The system of claim 1, wherein the augmented reality content is a screenshot of at least one of a portion of the game world or a portion of a different game world of a different video game.

7. The system of claim 1 , wherein the augmented reality content is a three-dimensional screenshot, and wherein the generating of the augmented frame comprises: constructing a three-dimensional model from the three-dimensional screenshot based at least in part on depth buffer data associated with the three-dimensional screenshot; as well as The three-dimensional model is superimposed on at least a portion of the video game content within the enhanced frame.

8. The system of claim 1, wherein the augmented reality content is a plurality of consecutive three-dimensional screenshots, and wherein the generating of the augmented frame comprises starting playback of a three-dimensional video based at least in part on the plurality of consecutive three-dimensional screenshots.

9. The system of claim 1, the operations further comprising executing an augmented reality component as a separate process from the video game.

10. A method for providing an enhanced frame for a video game, the method comprising: a video game executed by one or more processors, the video game outputting video game content; identifying, by the one or more processors, augmented reality content based at least in part on video game data regarding a current state of the video game; generating, by the one or more processors, an augmented frame comprising the augmented reality content rendered within a game world of the video game, wherein the augmented frame is a composite frame including the video game content and the augmented reality content; as well as The enhancement frame is rendered by the one or more processors for display.

11. The method of claim 10, wherein the identification of the augmented reality content comprises: providing the video game data to the executable program via an augmented reality component; as well as The augmented reality content generated by the executable program based at least in part on processing the video game data by the executable program is received by the augmented reality component.

12. The method of claim 10, further comprising executing an augmented reality component as a separate process from the video game.

13. The method of claim 10, wherein the video game data includes spatial data associated with a portion of the game world to be rendered.

14. The method of claim 13, wherein the spatial data specifies at least one of: game world coordinates associated with the portion of the game world to be rendered; or The current direction of the virtual camera associated with the player's character.

15. The method of claim 10, wherein the augmented frame is rendered for display on a first client machine, wherein the augmented reality content comprises a screenshot of a portion of the game world or a different game world rendered on a second client machine, and wherein the generating of the augmented frame comprises: Based at least in part on the screenshot, a viewport into the game world or the different game world rendered on the second client machine is constructed.

16. A method for providing an enhanced frame for a video game, the method comprising: receiving, by one or more processors, video game data associated with a video game that outputs video game content; identifying, by the one or more processors, augmented reality content based at least in part on the video game data; as well as The one or more processors send the augmented reality content to a client machine to generate an enhanced frame of the video game, the enhanced frame including the augmented reality content rendered within a game world of the video game, wherein the enhanced frame is a composite frame including the video game content and the augmented reality content.

17. The method of claim 16, wherein the identification of the augmented reality content comprises: providing the video game data to the executable program via an augmented reality component; as well as The augmented reality content generated by the executable program based at least in part on processing the video game data by the executable program is received by the augmented reality component.

18. The method of claim 16, wherein the video game data includes spatial data associated with a portion of the game world to be rendered.

19. The method of claim 16, wherein: The identifying of the augmented reality content includes identifying a subset of available augmented reality content associated with a subscribed augmented reality channel; as well as The sending of the augmented reality content to the client machine includes sending a subset of the available augmented reality content to the client machine.

20. The method of claim 19, wherein: The identifying of the augmented reality content includes identifying an additional subset of the available augmented reality content associated with an unsubscribed augmented reality channel; as well as The sending of the augmented reality content to the client machine includes sending the additional subset of the augmented reality content to the client machine for presentation on the client machine in a manner that visually distinguishes the additional subset of the augmented reality content from the subset of the available augmented reality content.

21. A client machine comprising: processor; as well as a memory storing computer-executable instructions that, when executed by the processor, cause the client machine to: executing a video game that outputs video game content as a series of frames; executing an augmented reality component as a separate process from the video game; and Render a frame from the series by: obtaining, by the augmented reality component, from the video game video game data regarding a current state of the video game; identifying, by the augmented reality component, augmented reality content based at least in part on the video game data; generating, by the augmented reality component, an augmented frame, the augmented frame being a composite frame including the video game content and the augmented reality content; as well as The enhanced frame is rendered on a display associated with the client machine.

22. The client machine of claim 21, wherein the computer executable instructions, when executed by the processor, further cause the client machine to: Before rendering the frame: sending a subscription request to subscribe to the augmented reality channel as a subscribed augmented reality channel to the remote computing system via the computer network; as well as loading one or more executable programs corresponding to the subscribed augmented reality channel and configured to generate the augmented reality content based at least in part on the video game data, The identification of the augmented reality content includes: providing the video game data as input to the one or more executable programs; as well as The augmented reality content is received as output from the one or more executable programs.

23. A client machine according to claim 21, wherein the video game data includes spatial data that specifies game world coordinates associated with the portion of the game world of the video game to be rendered, and wherein the augmented reality content is identified based at least in part on the game world coordinates.

24. The client machine of claim 23, wherein the spatial data further specifies a game identifier that can be used to distinguish between multiple instances of the game world coordinates, and wherein the augmented reality content is identified based at least in part on the game identifier.

25. The client machine of claim 23, wherein the spatial data further specifies a current orientation of a virtual camera associated with a player-controlled character, and wherein the augmented reality content is identified based at least in part on the current orientation of the virtual camera.

26. The client machine of claim 23, wherein the augmented reality content is a screenshot of at least one of a portion of the game world or a portion of a different game world of a different video game.

27. The client machine of claim 26, wherein the screenshot is a three-dimensional screenshot, and wherein the generating of the enhanced frame comprises: constructing a three-dimensional model from the three-dimensional screenshot based at least in part on depth buffer data associated with the three-dimensional screenshot; as well as The three-dimensional model is overlaid on the video game content within the enhanced frame.

28. The client machine of claim 21, wherein the augmented reality content is a plurality of consecutive three-dimensional screenshots, and wherein the generating of the augmented frame comprises starting playback of a three-dimensional video based at least in part on the plurality of consecutive three-dimensional screenshots.

29. The client machine of claim 21, wherein obtaining the video game data from the video game further comprises: The video game data is received by the augmented reality component from the video game as part of a function call made by the video game.

30. A method for providing an enhanced frame for a video game, the method comprising: executing the video game by a client machine; as well as A frame in a series of frames is rendered by the client machine by: determining spatial data associated with a portion of a game world to be rendered, the spatial data comprising game world coordinates associated with the portion of the game world to be rendered and comprising a camera orientation; identifying augmented reality content based at least in part on the spatial data; generating an augmented frame including the augmented reality content; as well as The enhanced frame is rendered on a display associated with the client machine.

31. The method of claim 30, wherein the determining of the spatial data comprises obtaining the spatial data from the video game by an augmented reality component executing on the client machine as a separate process from the video game, and wherein the identification of the augmented reality content comprises: providing the spatial data as input to an executable program configured to generate the augmented reality content based at least in part on the spatial data; as well as The augmented reality content is received as output from the executable program.

32. A method according to claim 31, wherein the spatial data comprises a three-dimensional model of the portion of the game world to be rendered, and wherein: said providing said spatial data as said input to said executable program comprises providing said three-dimensional model as said input to said executable program; and The output received from the executable program includes positioning data for positioning the augmented reality content within the augmented frame based on the three-dimensional model.

33. The method of claim 30, further comprising: executing an augmented reality component as a separate process from said video game; Receiving, by the augmented reality component, one or more game-generated events from the video game; as well as In response to the receipt of the events issued by the one or more games, the augmented reality component sends events accepted by one or more games to the video game.

34. The method of claim 30, wherein the client machine is a first client machine, the spatial data comprises first spatial data specifying the game world coordinates associated with the to-be-rendered portion of the game world to be rendered as a first instance of the game world on the first client machine, the method further comprising: receiving second spatial data from a remote computing system over a computer network, the second spatial data specifying a current position of a player-controlled character within the game world rendered on a second client machine as a second instance of the game world; The identification of the augmented reality content includes: determining, based at least in part on the game world coordinates, that the current position of the player-controlled character is within the portion of the game world to be rendered; as well as retrieving an augmented reality avatar of the character controlled by the player, Wherein the generating of the enhanced frame comprises presenting the augmented reality avatar of the player-controlled character in the enhanced frame at a location within the first instance of the game world corresponding to the current location of the player-controlled character.

35. The method of claim 30, wherein the client machine is a first client machine, wherein the identification of the augmented reality content comprises receiving the augmented reality content in the form of a three-dimensional screenshot of a portion of the game world rendered on a second client machine from a remote computing system over a computer network, and wherein the generation of the augmented frame comprises positioning the three-dimensional screenshot based at least in part on a second camera orientation associated with the three-dimensional screenshot.

36. The method of claim 30, wherein the client machine is a first client machine, wherein the identification of the augmented reality content comprises receiving the augmented reality content in the form of a screenshot of a portion of the game world or a different game world rendered on a second client machine from a remote computing system over a computer network, and wherein the generating of the augmented frame comprises: Based at least in part on the screenshot, a viewport into the game world or the different game world rendered on the second client machine is constructed.

37. The method of claim 36, wherein the viewport includes other augmented reality content seen by a user of the second client machine within the game world or the different game world rendered on the second client machine.

38. The method of claim 30, wherein the client machine is a first client machine, wherein the identification of the augmented reality content comprises receiving the augmented reality content in the form of a screenshot of a portion of the game world rendered on a second client machine from a remote computing system over a computer network, and wherein the generating of the augmented frame comprises: The game world rendered on the second client machine is stitched together with the game world rendered on the first client machine based at least in part on the game world coordinates contained in the spatial data and additional game world coordinates received from the second client machine.

39. A system for providing game-specific augmented reality content to a client machine, the system comprising: one or more processors; as well as a memory that stores available augmented reality content and associated data in a spatial database and that stores computer-executable instructions that, when executed by the one or more processors, cause the system to: receiving, from a client machine over a computer network, a subscription request from a user account to subscribe to an augmented reality channel as a subscribed augmented reality channel; receiving, from the client machine over the computer network, a request from the user account containing video game data associated with a video game; identifying game-specific augmented reality content among the available augmented reality content stored in the spatial database based at least in part on the video game data; filtering the game-specific augmented reality content based on the subscribed augmented reality channel to obtain a subset of the game-specific augmented reality content associated with the subscribed augmented reality channel; and The subset of the game-specific augmented reality content is sent to the client machine over the computer network.

40. The system of claim 39, wherein the computer executable instructions, when executed by the one or more processors, further cause the system to: Providing an application programming interface for a user computing device to create new augmented reality content; receiving, via the computer network and via the application programming interface, instructions from a user computing device for creating the new augmented reality content; and The new augmented reality content is stored in the spatial database.

41. A system according to claim 40, wherein providing the application programming interface includes providing the application programming interface to the user computing device to create an executable program, the executable program being configured to generate augmented reality content based at least in part on the video game data provided as input to the executable program.

42. The system of claim 39, wherein the associated data comprises: a game identifier for the video game; as well as Spatial data specifying game world coordinates within a game world of the video game.

43. A system according to claim 39, wherein the computer-executable instructions, when executed by the one or more processors, further cause the system to filter the game-specific augmented reality content to obtain the subset of the game-specific augmented reality content based on popularity of the subset of the game-specific augmented reality content being greater than a popularity threshold, wherein the popularity is determined based on at least one of: a number of views associated with the subset of the game-specific augmented reality content, a number of votes associated with the subset of the game-specific augmented reality content, a user's previous activity, or the user's social network relationships.

44. The system of claim 40, wherein providing the application programming interface comprises providing the application programming interface for an author of the new augmented reality content to specify access permissions associated with the new augmented reality content and / or augmented reality content previously created by the author.

45. The system of claim 40, wherein providing the application programming interface comprises providing the user computing device with the application programming interface to add augmented reality content to pre-existing screenshots or video clips associated with one or more video games.

46. ​​The system of claim 39, wherein the computer executable instructions, when executed by the one or more processors, further cause the system to: identifying, based at least in part on the video game data, an additional subset of the game-specific augmented reality content associated with an unsubscribed augmented reality channel among the available augmented reality content stored in the spatial database; and The additional subset of the game-specific augmented reality content is sent to the client machine via the computer network for presentation on the client machine in a manner that visually distinguishes the additional subset of the game-specific augmented reality content from the subset of the game-specific augmented reality content.

Images