Feature Similarity Scoring of Physical Environments for Augmented Reality Gameplay

Abstract

A method and system for processing similarity of characteristics of a real-world space used for augmented reality (AR) gameplay is disclosed. The method includes receiving sensor data captured from a real-world space used by a user for AR gameplay of a game. The sensor data provides data for identifying characteristics of physical objects in the real-world space. The method includes generating a user space score using the characteristics of the physical objects identified in the real-world space. The method includes comparing the user space score to a predefined game space score for the game. The comparison is used to generate a fitness handicap for the user's AR gameplay of the game in the real-world space. Gameplay parameters may be adjusted to compensate or adjust the fitness handicap.

Description

Technical Field

[0001] The present disclosure generally relates to enabling adjustment of gameplay parameters in an AR game and / or adapting to the uniqueness of a user's game space using the similarity of features in the real-world space.

Background Art

[0002] Description of Related Art Augmented Reality (AR) technology has achieved unprecedented growth over the years and is expected to continue growing at an average annual growth rate. AR technology is an interactive three-dimensional (3D) experience that combines a view of the real world with computer-generated elements (e.g., virtual objects) in real time. In an AR simulation, virtual objects are infused into the real world to provide an interactive experience. With the increasing popularity of AR technology, various industries have implemented AR technology to improve the user experience. Target industries include, for example, the video game industry, entertainment, and social media.

[0003] For example, in the video game industry, there is an increasing trend to improve the user's gaming experience by enhancing the reality of the experience. However, since the AR experience necessarily depends on what exists in the real world, even if the same media is used for the AR experience, one user's AR experience will be different from another user's AR experience.

[0004] In such a context, embodiments of the present disclosure arise.

Summary of the Invention

[0005] Embodiments of the present disclosure include methods and systems for generating and using feature similarity scores for gameplay in an AR environment. In one embodiment, the method includes scanning the environment using multiple sensors (such as vision, audio, etc.) and capturing the main aspects of the play environment. In one configuration, a scoring method is used to determine how well the user's real-world space conforms compared to an optimized type of gameplay. As an example, in scoring, the number of clean walls, the open floor area, the shape of objects or obstacles, the size or volume of objects or obstacles, the amount of smooth or rough surfaces, the number of windows, the type of furniture, the presence of other people or players, etc. can be taken into account. This scoring of the user's real-world space generates a user space score. The user's space score may be compared to a game space score (e.g., representing an optimized type of environment for playing a VR game) predefined by the game developer.

[0006] Once the user space score is known, it becomes possible to dynamically adjust the gameplay parameters to reduce or eliminate the user's fitness handicap. In some cases, the gameplay is adjusted to standardize the gameplay (such as how enemy types spawn, how scores are assigned, the difficulty of actions or tasks, etc.) compared to a group of other game players and what the game developer intended as the grand truth. In one embodiment, by capturing and calculating the user's space score, it becomes possible to predict and accommodate the uniqueness of the environment in AR game development.

[0007] In one embodiment, a method for processing the similarity of features of a real-world space used for augmented reality (AR) game play is disclosed. This method includes receiving sensor data captured from the real-world space used by a user for the aforementioned AR game play of the game. The sensor data provides data for identifying the characteristics of physical objects in the real-world space. This method includes using the characteristics of the physical objects identified in the real-world space to generate a user space score. This method includes comparing the user space score with a game space score predefined for the game. This comparison is used to generate a fitness handicap for the user's AR game play of the game in the real-world space.

[0008] In other embodiments, a method for processing the similarity of features of a real-world space used for augmented reality (AR) game play is disclosed. This method includes receiving sensor data captured from the real-world space used by a user for the AR game play of the game. The sensor data provides data for identifying the characteristics of physical objects in the real-world space. This method includes using the characteristics of the physical objects identified in the real-world space to generate a user space score. This method includes generating a fitness handicap for the user's AR game play of the game in the real-world space. This method includes adjusting the game play parameters of the game for the user. The adjustment changes the difficulty setting of the game and compensates for the fitness handicap.

[0009] In some embodiments, the fitness handicap is based on a comparison of the user space score with a game space score that defines an optimized baseline for the type of environment used to play the game during AR game play.

[0010] In some embodiments, this method further includes adjusting a game scoring threshold. Adjusting the scoring threshold reduces the skill requirements of in-game actions for obtaining scores within the game.

[0011] In some embodiments, the user space score includes a numerical count of each type of surface material within the interactive zone.

[0012] In some embodiments, each numerical count is scored when it meets a threshold size for the aforementioned type of surface material.

[0013] In some embodiments, the sensor data is captured by one or more of a camera, a microphone, or an ultrasonic sensor, and the sensor data is processed to identify a feature set labeled by a machine learning process, and the machine learning process is trained to identify the characteristics of physical objects within the real-world space.

[0014] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, which illustrates the principles of the present disclosure by way of example in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0015] The present disclosure can be better understood by referring to the following description in conjunction with the accompanying drawings.

[0016]

Figure 1

[0017]

Figure 2

[0018]

Figure 3

[0019]

Figure 4

[0020]

Figure 5

[0021]

Figure 6

MODE FOR CARRYING OUT THE INVENTION

[0022] The following embodiments of the present disclosure provide methods, systems, and devices for generating and using game play characteristic similarity scores in an AR environment. When playing a game using augmented reality (AR) glasses, a characteristic similarity score is generated in the form of a game space score associated with a user's interaction zone in the real-world space. The AR glasses used here can handle both AR and virtual reality. For example, in AR, virtual objects can be placed within or around the real-world space, while the VR environment is completely generated by a computing system. According to one embodiment, the computing system may be associated with the AR glasses. The AR glasses may be equipped with a processor that executes program instructions for rendering AR objects and / or VR content. In one embodiment, the AR glasses have a wireless communication function so that they can communicate directly with a server via the Internet. In other embodiments, the AR glasses communicate locally with a game console or a local computer, and the game console or the local computer can communicate with the server via the Internet.

[0023] In one embodiment, the method includes scanning the environment using multiple sensors (such as vision, audio, etc.) and capturing the main aspects of the play environment. The multiple sensors can be various, such as cameras, microphones, ultrasonic sensors, optical sensors, motion sensors, inertial sensors, and / or combinations thereof. In some embodiments, sensor fusion is used to collect data captured from various sensors when scanning a room, such as an interactive zone. In one configuration, a scoring method is used to determine how well the user's real-world space conforms compared to an optimized type of gameplay. As an example, in scoring, the number of clean walls, the open floor area, the shape of objects or obstacles, the size or volume of objects or obstacles, the amount of smooth or rough surfaces, the number of windows, the type of furniture, the presence of other people or players, etc. can be taken into account. This scoring of the user's real-world space generates a user space score. The user's space score may be compared to a game space score (e.g., representing an optimized type of environment for playing a VR game) predefined by the game developer.

[0024] Once the user space score is known, it becomes possible to dynamically adjust the gameplay parameters to reduce or eliminate the user's fitness handicap. In some cases, the gameplay is adjusted to standardize the gameplay (such as the way enemy types are generated, how scores are assigned, the difficulty of actions or tasks, etc.) compared to a group of other game players and what the game developer intended as the grand truth. In one embodiment, by capturing and calculating the user's space score, it becomes possible to predict and accommodate the uniqueness of the environment in AR game development.

[0025] Figure 1 shows an example of the real - world space 100 where user 104 is playing a game. In one embodiment, the user is playing an augmented reality game using AR glasses 103 in the interactive zone 102. The user's position may cover a wider area, but the interactive zone 102 is an area close to the user, for example, the place where the user is located and playing the game. Generally speaking, the real - world space 100 may not include a house or other rooms within the place where one is. This is because in other areas, the performance of the AR glasses 103 may not be provided when playing an AR game. For example, the AR glasses 103 may project virtual objects on or near real - world objects within the interactive zone.

[0026] If there are not enough surfaces or objects on which AR objects can be projected or moved according to the game, the performance of the projection, for example, how virtual objects are recognized within the real - world space 100, may be affected. As an example, if there are too many windows, the external ambient light may affect the performance of the projected AR objects. Similarly, if there are too many cluttered walls or spaces in the space, there may not be enough space to ensure the projection and / or movement of AR objects. In some cases, if the floor reflection is too strong or too rough, the projection of AR content may not be optimal.

[0027] When developing an AR game, the developer needs to make some baseline assumptions about how the real-world space looks and / or prepare a minimal level of space for placing, moving, or interacting with the projected and tracked AR content. If these baseline assumptions for the functions required for the AR game are lacking in the space, the user's game performance will decline during gameplay. According to one embodiment, a process is described that enables the interactive zone 102 to be scanned to generate a game space score. Next, the game space score of the user's real-world space 100 is compared with the game space score pre-defined for the game by the developer. For example, the game space score may define the baseline requirements for achieving an optimal game play experience using AR or VR.

[0028] In one embodiment, the scan can be performed using one or more sensors. The sensors can be various, such as those that capture sound, those that capture video, those that capture movement, those that capture ultrasonic waves, those that capture temperature, those that capture humidity, those that capture inertial data, etc. The sensors may be part of existing devices in the real-world space. For example, they may be one or more cameras S1, S2, microphones S3, microphones S4, cameras and microphones S5 on the AR glasses 103, microphones S6 that may be part of a door or doorbell, microphones S7 on a mobile phone, microphones S8 integrated into one or more controllers that the user 104 may hold, etc.

[0029] In one embodiment, these sensors may be configured to capture the characteristics of the interactive zone 102 and identify the type of object and / or the surface material. In the illustrated example, the user 104 is standing on the rug 108. The chair 106 and the chair 126 are identified within the space. A picture 110 is hung on the wall. A window 114 is identified on one of the walls. A television 116 is hung on another wall. A door 120 is arranged next to the television 116. The game console 118 is installed under the television 116. A table 112 exists and the mobile phone S7 is placed on the table. The dog 124 moving around within the interactive zone 102 is captured. A lamp 112 is placed on the small table 128 in the center of the room. In one embodiment, sensor fusion 130 represents capturing environmental data from the real-world space to identify objects, surfaces, materials, and other environmental features.

[0030] In one embodiment, machine learning is utilized to process the inputs received from various sensors. The machine learning algorithm uses a model trained to identify objects that may be present within the interactive zone. For example, the raw sensor data is first processed by the machine learning system and feature data is identified from the captured sensor data. Next, the feature data is processed using a labeling algorithm. The labeling algorithm is configured to assign meaning to the feature data in order to classify the labeled feature data. Over time, based on the training, the labeled feature data is properly classified and the machine learning model is improved. Once the machine learning model is trained, it can be used to process the sensor data captured in the interactive zone.

[0031] When processing sensor data using a machine learning model, it is possible to identify what real-world features exist in the real-world space. For example, using the sensor data, it is possible to identify a lamp 112, a rug 108, a dog 124, a window 114, a television 116, and other objects that may enter and exit the interactive zone. As will be described in more detail below, by identifying the functions present in the interactive zone and the properties of those functions, a game space score for the interactive zone can be generated.

[0032] FIG. 2 shows an example of a process of adjusting game play parameters of a game 202 using a game play normalization engine 200 according to one embodiment. In one example, user 104 may be playing game 202 or may have started playing game 202. The normalization engine 200 may be triggered (by a program or user input) to perform a scan of the interaction zone 102 before the game starts, whereby the AR game experience is preconfigured with adjusted game play parameters. In other embodiments, user 104 may start playing the game and, after the game starts, may adjust all of the game play parameters at once or gradually over time to achieve a smooth transition.

[0033] In operation 204, the user's real-world space represents a location where the user may be playing an AR game, and here it is referred to as the interactive zone 102. The interactive zone represents, as described above, an area where the user may interact with the AR game or the AR glasses 103. Sensors within the real space 100 may be utilized to scan the environment and generate an acoustic profile of the environment. The scan may perform sensor fusion 130 and utilize various sensors that may be present within the interactive zone 102 or that can be brought into the interactive zone 102 to perform an initial scan of the real-world space. In this way, the scan of the real-world space 100, specifically the interactive zone 102, can be performed once or periodically and safely stored for later retrieval for game play.

[0034] In this way, each time game play using the AR glasses 103 is executed, it is not necessary to continuously perform a scan. The scoring process 206 is executed using sensor fusion 130. The scoring process 206 is generally used to identify various types of objects within the real-world space 100 and surface materials associated with various types of materials present within and around the interactive zone 102. As will be described later, the scoring process can include identifying various types of materials of various types of objects present within the interactive zone 102. The scoring process can include identifying specific types of materials and determining whether those types of materials are present in the interactive zone 102.

[0035] The threshold is used to determine whether those surfaces exist, or whether there is a sufficient amount of those surfaces to be counted in a numerical count. Thus, the numerical count is used to score the types of surfaces present within the interactive zone and the acoustic absorption or reflection associated with the materials of those types. The numerical score is a score that combines various types of materials and surfaces and the associated amounts and absorption and reflectance rates of those materials. Combining these values generates a vector that defines various types of materials and the weights associated with those materials in the real world space.

[0036] Thus, the scoring process 206 generates a user space score 210. The user space score 210 represents the output of the scoring process 206. In one embodiment, a unique user space score 210 is set for each interactive zone 102 where the user is using the AR glasses 103. The game developer of the game 202 also generates an optimized game space score 208 that represents an optimal or optimized baseline score for the type of environment in which the game can be played using the AR glasses 103 or similar AR type glasses.

[0037] In one embodiment, the user space score can be further modified or adjusted using self-identification disabilities. These self-identification disabilities are specific to the user and can be adjusted based on input, settings, or learned settings. By way of example, and without limitation, self-identification disabilities may include, but are not limited to, color blindness, hearing impairment, sensitivity to motion sickness, the need to be seated, immobility, reduced motor ability, sensitivity to loud noises, difficulty seeing in the dark, discomfort with bright lights, or combinations thereof, and / or other self-identification disabilities or preferences. In one configuration, these self-identification disabilities are considered additional accessibility options and can be set using the user interface, voice input, controller input, or touch screen.

[0038] By comparing the game space score 208 with the user space score 210, a fitness handicap 212 can be generated for the user space score 210. The fitness handicap 212 represents how close the user space score 210 is to the game space score 208, or how far apart the scores are. The greater the deviation between the two, the more the surface characteristics of the interactive zone 102 indicate that the interactive zone 102 is not suitable for playing the AR game. As a result, the user 104 plane within the interactive zone 102 with a large deviation will be at a disadvantage compared to other users with a minimal deviation.

[0039] In one embodiment, a deviation of less than 10 percent is appropriate and does not affect the user 104 significantly when playing the AR game 202. A deviation of less than 30 percent is within the acceptable range, but the user 104 will play at a disadvantage compared to other players playing with a smaller deviation. For example, if the deviation exceeds 30 percent, the handicap will increase, and the user 104 may not be able to progress in the game or the gameplay may be hindered compared to other users.

[0040] In one embodiment, it is possible to adjust the game play parameters 214 and correct the deviation using the fitness handicap 212 of the user 104. As an example, the game play parameters may be dynamically adjusted to make it easier to obtain points in the game, easier to interact with the AR character, easier to see the AR character, change the parameters of the AR character to make it more prominent in the background, increase the sound of the AR character or object based on the acoustics of the user space, or make it easier to progress to other levels.

[0041] These parameters serve as inputs to the game engine and are used to play with a fitness handicap and adjust various difficulties, settings, and other parameters when adjusting according to that fitness handicap. In one embodiment, when one player is competing against another player in a multiplayer AR game, the game play parameters 214 of each player can be adjusted so that the fitness handicaps 212 are approximately equal. In this way, it is not necessary to completely eliminate the fitness handicap, and by simply adjusting the game play parameters of each player's respective game instance, the players can be kept on a nearly equal or equivalent footing. In a multiplayer game, by optimizing one or both players, or multiple players, the AR game and the game play experience are improved because, due to the characteristics of each real-world environment, one or several users do not have an advantageous position over other users.

[0042] Figure 3 shows an example embodiment according to one embodiment in which game play parameters can be adjusted based on a user's fitness handicap. In one embodiment, user 104 selects game 302. Game 302 is an AR game equipped by the developer to be able to adjust various parameters, and these parameters may be pre-defined parameters for removing the deviation of a particular player's handicap. When game 302 is selected, the real-world space of the user can be scanned in operation 304. As described above, the scan can be performed by one or more sensors. In a simple example, the sensors of the AR glasses themselves may be used. The sensors can include a microphone, a camera, an inertial sensor, an ultrasonic sensor, or a combination of these or multiple sensors. In other embodiments, the sensors can include other sensors that are part of the environment in which the player or user plays game 302.

[0043] As shown in FIG. 1, a large number of sensors can exist, and the outputs of the sensors can be utilized in the sensor fusion 130 process to accurately scan and quantify different materials, surfaces, absorption, reflectivity, and parameters of the user's real-world space. Using the scanned parameters (which may be in the form of vectors or matrices), data may be used to generate a user space score 306. In one embodiment, the user space score may be a normalized value. In another embodiment, the user space score may be a plurality of values in vector form with weights and magnitudes associated with each of the different types of parameters that make up the score. In another embodiment, the matrix can identify various types of parameters that define the surface and material of the space, and the magnitude or quantity of those materials and surfaces relative to the real-world space or the user's position within a particular interactive zone 102.

[0044] In operation 308, the user space score is compared to a pre-defined game space score of the game. As described above, the game space score is a score value signed by the game developer. These values represent the optimal values for the space in which the game can be played. For example, in an AR game where a large number of images need to be projected onto a wall, the playability of the AR game is improved by the amount of free space on the wall where projection is possible. Other parameters related to the space and the adaptability to the projected or displayed AR content are also considered. In operation 310, based on the comparison, a fitness handicap for the user is generated. As described above, the fitness handicap may identify the deviation from the optimal acquisition space score defined by the game developer.

[0045] In operation 312, the game play parameters of the game can be adjusted based on the fitness handicap. In one embodiment, the adjustment reduces the deviation from the optimized baseline of the game space score. In other embodiments, it may be desirable to increase the deviation of one player so that the deviations of two or more players are approximately equal. Thus, it should be understood that the deviation from the game space score may be tolerated in some cases, and in some cases, the deviation can be increased so that multiple players in a multiplayer game are on an approximately equal footing when the deviations are approximately or substantially equal.

[0046] FIG. 4 shows a process for generating a user space score according to one embodiment. As described above, there are several ways to generate a user space score, and the following example does not limit the other methods described above. In one embodiment, in operation 402, sensor data is captured by a plurality of sensors within the interactive zone of the real-world space. As described above, the sensor data can be captured using various types of sensors such as cameras, microphones, inertial sensors, ultrasonic sensors, temperature sensors, humidity sensors, etc. The types of devices that can be used to capture these sensor data inputs can include the AR glasses 103 and devices that may be present in the interactive zone 102 or the real-world space 100.

[0047] As an example, these types of devices can include, in addition to the AR glasses 103, television sensors, cameras, game consoles, controllers used for games, mobile phones, wristwatches, and other devices that may be present within and around the real-world space 100. In operation 404, the characteristics of the interactive zone are assembled. These characteristics identify various types of surface materials within the interactive zone and other objects that may be present or that define the real-world space. These objects may vary depending on what is present in the real-world space 100 of the user 104.

[0048] The example of FIG. 1 shows that various types of objects can exist in the interactive zone 102. It should be understood that the space varies by user, and there are various objects customized according to the user's preferences, wishes, decoration, and placement choices within the space. Therefore, the characteristics of the user space score vary depending on the location where the user decides to play an AR game and interact in a specific interactive zone. For this reason, the types of devices that define sensors and capture data in the form of sensor data of, for example, sensor fusion 130 vary greatly.

[0049] Still, sensor data is captured to scan and identify various types of objects that may be present in the space and the surface materials that define those types of objects. In some embodiments, a numerical count is performed to identify the amount of objects that meet the size of a threshold and the amount of the surfaces of those objects and materials. For example, if there is a clean and open wall in the interactive zone 102, that clean and open wall can represent a value based on the amount of square feet or square meters of that type of material. Therefore, different square feet or square meters can be calculated for each type of material, and a numerical value can be assigned to the amount of that type of material or surface object in the space.

[0050] If the amount of surface material is too small to count, such as less than 1 square foot, the material or object may not meet the threshold and may not be counted. In some embodiments, surface materials exceeding at least 1 square foot are counted to identify the number of that type of material within the interactive zone 102. In some embodiments, a machine learning model 406 can be utilized to process raw sensor data to identify various types of materials and characteristics present within the interactive zone. For example, the machine learning model can receive image data that can be processed to identify features in each image, classify those features, and identify the objects present in the space.

[0051] For example, using machine learning, it is possible to identify the actual features and characteristics present in an image captured by a camera. Features and characteristics can identify not only what an object is, but also attributes of the object such as texture, reflectivity, roughness, position, lighting, and other distinguishable or identifiable characteristics. Thus, a machine learning model can also incorporate other inputs such as sound, lighting, microphone arrays, and other data useful for identifying the characteristics of various types of objects and surface materials. Using this information and the numerical count generated in operation 408, a user space score is generated in operation 410.

[0052] FIG. 5 shows examples of various types of objects existing in a real-world space and the attributes and characteristics identified in the real-world space in response to the scan described above. As a simple example, the real-world space or interactive zone 102 includes walls 420, floors 422, furniture 424, and other objects 426. There are so many other objects that cannot be fully described here, but any type of object that may exist within the space can be included. In some embodiments, the other objects may be other types of moving objects such as people, pets, or robots.

[0053] In the illustrated example, the wall 420 can be defined by various types of surface materials such as rough, brick, cladding, smooth, window, etc. If a threshold for the amount of that material is met, a numerical count is assigned to that type of surface material. As an example, if the amount of the rough surface of the wall exceeds 2 square feet, at least 2 can be assigned to the numerical count. If the smooth surface exceeds at least 16 square feet, the numerical count can be set to 16. The numbers shown are provided only to illustrate a counting method that takes into account the type and quantity or amount of materials present in the space.

[0054] The same can be done for beds, furniture, and other objects. The result is a user space score 450 for the user's interactive zone 102 of user 104. The user space score can be defined as a vector, matrix, or other value or mathematical formula and can be compared to the game space score defined by the developer for the game. As described above, by utilizing or knowing the user space score to remove deviations or adjust the deviation for one or other players, the game play parameters of the game can be adjusted. As described above, the game play parameters can also include the adjustment of metadata that controls one or more difficulty settings or scoring thresholds when playing the game using a specific fitness handicap.

[0055] It should be understood that adjusting the game play parameters allows the difficulty or other parameters of the game to be removed or adjusted so that players can play the game on a substantially equal footing. Thus, scanning the environment generates a similarity of features between the user's real-world space and the features that provide a more optimal or optimized baseline for the type of environment used when playing the game during an AR game. By enabling dynamic adjustment of the game play parameters, the handicaps between players are reduced and the game can be enjoyed more considering the environment specific to each player when playing an AR game. Thus, the adjustment provides normalization or ground truth between two or more players even when the players are playing in different real-world environments (e.g., online games or cloud games during streaming).

[0056] FIG. 6 shows the components of an exemplary device 600 that can be used to execute aspects of various embodiments of the present disclosure. This block diagram shows a device 600 that can incorporate, or can be, a personal computer, a video game console, a personal digital assistant, a server, or other digital device suitable for implementing embodiments of the present disclosure. Device 600 includes a central processing unit (CPU) 602 for executing software applications and optionally an operating system. CPU 602 may be composed of one or more homogeneous or heterogeneous processing cores. For example, CPU 602 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs having a microprocessor architecture particularly adapted for high-parallel and compute-intensive applications, such as interpretation of queries, identification of contextually relevant resources, and immediate implementation and rendering of contextually relevant resources within a video game. Device 600 may be local to a player who plays a game segment (e.g., a game console), or remote from the player (e.g., a backend server processor), or one of many servers that uses virtualization in a game cloud system for remote streaming of game play to a client.

[0057] Memory 604 stores the applications and data used by CPU 602. Storage 606 provides non-volatile storage and other computer-readable media for applications and data, and may include a fixed disk drive, a removable disk drive, a flash memory device, and a CD-ROM, DVD-ROM, Blu-ray (registered trademark), HD-DVD, or other optical storage device, as well as signal transmission and storage media. User input device 608 communicates user input from one or more users to device 600, and examples of user input device 608 may include a keyboard, a mouse, a joystick, a touchpad, a touch screen, a still recorder / camera or a video recorder / camera, a tracking device that recognizes gestures, and / or a microphone. Network interface 614 enables device 600 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communication via a local area network or a wide area network such as the Internet. Audio processor 612 is adapted to generate analog or digital audio output from instructions and / or data provided by CPU 602, memory 604, and / or storage 606. The components of device 600 including CPU 602, memory 604, data storage 606, user input device 608, network interface 610, and audio processor 612 are connected via one or more data buses 622.

[0058] The graphics subsystem 620 is further connected to the data bus 622 and the components of the device 600. The graphics subsystem 620 includes a graphics processing unit (GPU) 616 and a graphics memory 618. The graphics memory 618 includes a display memory (e.g., frame buffer) used to store the pixel data of each pixel of the output image. The graphics memory 618 may be integrated into the same device as the GPU 608, may be connected as a separate device from the GPU 616, and / or may be incorporated within the memory 604. The pixel data can be provided directly from the CPU 602 to the graphics memory 618. Alternatively, the CPU 602 provides data and / or instructions defining the desired output image to the GPU 616, from which the GPU 616 generates the pixel data of one or more output images. The data and / or instructions defining the desired output image can be stored in the memory 604 and / or the graphics memory 618. In an embodiment, the GPU 616 includes a 3D rendering function for generating pixel data for the output image from instructions and data defining the geometry, lighting, shading, texturing, motion, and / or camera parameters of the scene. The GPU 616 can further include one or more programmable execution units capable of executing shader programs.

[0059] The graphics subsystem 614 periodically outputs the pixel data of the image from the graphics memory 618 for display on the display device 610. The display device 610 can be any device capable of displaying visual information in response to a signal from the device 600, including CRT, LCD, plasma, and OLED displays. The device 600 can provide, for example, an analog signal or a digital signal to the display device 610.

[0060] It should be noted that access services distributed over a wide area, such as providing access to the games of the present embodiment, often use cloud computing. Cloud computing is a computing paradigm in which dynamically scalable and often virtualized resources are provided as services via the Internet. Users do not need to be experts in the technical infrastructure of the "cloud" that supports them. Cloud computing can be divided into different services such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing services often provide common applications such as video games online, accessible from a web browser, but the software and data are stored on servers within the cloud. The term "cloud" is used as a metaphor for the Internet based on the way the Internet is depicted in a computer network diagram, and it is an abstract concept of the complex infrastructure it hides.

[0061] In some embodiments, a game server may be used to execute the operation of a duration information platform for video game players. Most video games played over the Internet operate via a connection to a game server. Typically, a game uses a dedicated server application that collects data from players and distributes the collected data to other players. In other embodiments, a video game may be executed by a distributed game engine. In these embodiments, the distributed game engine may be executed on a plurality of processing entities (PEs), and each PE executes a given functional segment of the game engine on which the video game is executed. Each processing entity is regarded as merely a computing node from the perspective of the game engine. The game engine typically performs a functionally diverse set of operations to execute a video game application along with additional services experienced by the user. For example, the game engine implements game logic and executes game calculations, physics, geometry transformations, rendering, lighting, shading, audio, as well as additional in-game or game-related services. The additional services may include, for example, messaging, social utilities, audio communication, game play replay functionality, help functionality, and the like. The game engine may be executed on an operating system virtualized by a hypervisor of a particular server, but in other embodiments, the game engine itself may be distributed across a plurality of processing entities, and each entity may reside on a different server unit of a data center.

[0062] According to this embodiment, for the execution of the methods / processing operations, each processing entity may be a server unit, a virtual machine, or a container, depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera conversion, that particular game engine segment will perform a number of relatively simple mathematical operations (e.g., matrix transformation), so it may be provisioned with a virtual machine associated with a graphics processing unit (GPU). Other game engine segments that require fewer but more complex operations may be provisioned with processing entities associated with one or more higher-powered central processing units (CPUs).

[0063] By distributing the game engine, the game engine has elastic computing characteristics that are not restricted by the capabilities of physical server units. Instead, the game engine is provisioned with more or fewer computing nodes as needed to meet the requirements of the video game. From the perspective of the video game and the video game player, a game engine distributed across multiple computing nodes is indistinguishable from a non-distributed game engine executed on a single processing entity because a game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the end user with the video game output components.

[0064] The user accesses the remote service through a client device that includes at least a CPU, a display, and I / O. The client device may be a PC, a mobile phone, a netbook, a PDA, etc. In one embodiment, the network executed on the game server recognizes the type of device used by the client and adjusts the communication method employed. In another case, the client device uses a standard communication method such as HTML to access the application on the game server via the Internet.

[0065] Of course, a given video game or game application may be developed for a specific platform and a specific associated controller device. However, when such a game becomes available via a game cloud system as described herein, a user can access the video game using a different controller device. For example, a game may have been developed for a game console and its associated controller, but a user can access a cloud-based version of the game from a personal computer using a keyboard and mouse. In such a scenario, through input parameter settings, a mapping can be defined from the inputs that can be generated by the controller devices available to the user (in this case, the keyboard and mouse) to inputs acceptable in the execution of the video game.

[0066] In another example, a user may access a cloud game system via a tablet computing device, a touch screen smartphone, or other touch screen-driven device. In this case, the client device and the controller device are integrated together within the same device, and the input is provided by the detected touch screen input / gesture. In such a device, through input parameter settings, specific touch screen inputs corresponding to the game inputs of the video game may be defined. For example, buttons, directional pads, or other types of input elements may be displayed or overlaid during the execution of the video game to indicate positions on the touch screen that the user can touch to generate game inputs. Gestures such as swipes in a specific direction, or specific touch motions may also be detected as game inputs. In one embodiment, a tutorial showing how to input into the game play via the touch screen can be provided to the user, for example, before starting the game play of the video game, to familiarize the user with the control operations on the touch screen.

[0067] In some embodiments, the client device functions as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection and sends inputs from the controller device to the client device. Next, the client device processes these inputs and can then send the input data via a network (e.g., a network accessible via a local network device such as a router) to a cloud gaming server. However, in other embodiments, the controller itself has the ability to communicate inputs directly to the cloud gaming server via the network, and it is not necessary to communicate such inputs through the client device first, and it can be a networked device. For example, the controller can connect to a local network device (such as the aforementioned router) to send and receive data with the cloud gaming server. Thus, although the client device is still required to receive video output from the cloud-based video game and render it on a local display, input latency can be reduced by enabling the controller to bypass the client device and send inputs directly to the cloud gaming server via the network.

[0068] In one embodiment, the networked controller and client device can be configured to send certain types of input directly from the controller to the cloud game server and other types of input via the client device. For example, apart from the controller itself, input that does not depend on any additional hardware or processing can bypass the client device and be sent directly from the controller to the cloud game server via the network. Such input can include button input, joystick input, embedded motion detection input (e.g., accelerometer, magnetometer, gyroscope), etc. However, input that utilizes additional hardware or requires processing by the client device can be sent to the cloud game server by the client device. These can include video or audio captured from the game environment that can be processed by the client device before being sent to the cloud game server. Additionally, input from the controller's motion detection hardware can be processed by the client device in conjunction with the captured video to detect the position and movement of the controller, and then communicated to the cloud game server by the client device. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud game server.

[0069] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Thus, the examples provided are only a part of the possible examples and do not limit the various embodiments that are possible by combining various elements to define more embodiments. In some examples, some embodiments may include fewer elements without departing from the spirit of the disclosed or equivalent embodiments.

[0070] Embodiments of the present disclosure may be practiced in various computer system configurations including, but not limited to, handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, and mainframe computers. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a wired or wireless network.

[0071] Although the method operations have been described in a particular order, other housekeeping operations may be performed between operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed within the system to enable the processing operations to occur at various processing-related intervals, as long as the telemetry and game state data processing for generating the modified game state is performed in the desired manner.

[0072] One or more embodiments may also be implemented as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that can store data and that can thereafter be read by a computer system. Examples of computer-readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tape, and other optical and non-optical data storage devices. A computer-readable medium may include a computer-readable tangible medium distributed over a network-connected computer system so that the computer-readable code is stored and executed in a distributed fashion.

[0073] In one embodiment, the video game is executed locally on a gaming machine, a personal computer, or a server. In some cases, the video game is executed by one or more servers in a data center. When the video game is executed, some instances of the video game can be simulations of the video game. For example, the video game can be executed by an environment or server that generates a simulation of the video game. The simulation is, in some embodiments, an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In any case, if the video game is represented as a simulation, the simulation can be executed to render interactive content that can be interactively streamed, executed, and / or controlled by user input.

[0074] The foregoing embodiments have been described in some detail for purposes of clarity of understanding, but it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the embodiments are to be regarded as illustrative rather than restrictive, and the embodiments are not to be limited to the details described herein but may be modified within the scope of the appended claims and the equivalents thereof.

Claims

**Claim 1** A method for processing the similarity of features of a real-world space used for augmented reality (AR) game play, comprising: receiving sensor data captured from the real-world space used by a user for the AR game play of the game, the sensor data providing data for identifying characteristics of physical objects within the real-world space; using the characteristics of the physical objects identified in the real-world space to generate a user space score; comparing the user space score with a pre-defined game space score for the game, the comparison being used to generate a fitness handicap for the AR game play of the game in the real-world space by the user; A method comprising the above steps. **Claim 2** The method according to claim 1, wherein the real-world space includes an interactive zone of the user during the AR game play, and the characteristics of the physical objects include the type of surface material. **Claim 3** The method according to claim 2, wherein the user space score includes a numerical count of each type of surface material within the interactive zone. **Claim 4** The method according to claim 3, wherein each numerical count is scored if it meets a threshold size for the type of surface material. **Claim 5** The method according to claim 1, wherein the sensor data is captured by one or more of a camera, a microphone, or an ultrasonic sensor, and the sensor data is processed to identify a feature set labeled by a machine learning process, the machine learning process being trained to identify the characteristics of the physical objects within the real-world space. **Claim 6** The method according to claim 1, wherein the game space score is correlated with interactive content developed for the game, and the game space score defines an optimized baseline for the type of environment used for playing the game during the AR game play. **Claim 7** The method according to claim 1, wherein the fitness handicap for the AR game play of the game in the real-world space by the user identifies a deviation from an optimized baseline for the type of environment used for playing the game during the AR game play. Claim 8 When playing the game, adjusting the game play parameters of the user to substantially eliminate the deviation from the optimized baseline, the adjustment being further configured to adjust metadata that controls one or more difficulty settings or scoring thresholds when playing using the fitness handicap, the method of claim 7. Claim 9 Substantially eliminating the deviation functions to place the user on a substantially equal footing with other users playing the game at a game space score that nearly reaches the optimized baseline, the method of claim 8. Claim 10 Each of the users and other users, when playing the game, to substantially eliminate the deviation from the optimized baseline, each of the users and the other users dynamically adjust their game play parameters so as to play the game on a substantially equal footing with each other, the method of claim 8. Claim 11 A method for processing the similarity of features of a real-world space used for augmented reality (AR) game play, Receiving sensor data captured from the real-world space used by a user for the AR game play of the game, the sensor data providing data for identifying the characteristics of physical objects within the real-world space, the receiving; Using the characteristics of the physical objects identified in the real-world space to generate a user space score; Generating a fitness handicap for the AR game play of the game in the real-world space by the user; Adjusting the game play parameters of the game for the user, the adjustment changing the difficulty setting of the game and compensating for the fitness handicap, the adjusting; comprising The fitness handicap is based on a comparison of the user space score and a game space score that defines an optimized baseline for the type of environment used to play the game during the AR game play. Claim 12 The method according to claim 11, further comprising adjusting a scoring threshold of the game, and reducing or enhancing a skill requirement of an action in the game for obtaining a score in the game by the adjustment of the scoring threshold.

13. Adjusting the game play parameters of the game of the user functions to reduce the deviation of the user in the real world space to a game space score at a baseline that is substantially optimized for the type of environment used to play the game during the AR game play. The method according to claim 11.

14. Reducing the deviation functions to place the user in a position substantially equivalent to that of other users playing the game at a game space score that substantially reaches the optimized baseline. The method according to claim 13.

15. The method according to claim 11, further comprising adjusting game play parameters of the game of the other user and arranging a fitness handicap of the other user to be substantially the same as the fitness handicap of the user.

16. The real world space includes an interactive zone of the user during the AR game play, and the characteristics of the physical object include the type of surface material. The method according to claim 11.

17. The user space score includes a numerical count of various types of surface materials within the interactive zone. The method according to claim 16.

18. Each numerical count is scored when it meets a threshold size for the type of surface material. The method according to claim 17.

19. The sensor data is captured by one or more of a camera, a microphone, or an ultrasonic sensor, the sensor data is processed to identify a feature set labeled by a machine learning process, and the machine learning process is trained to identify the characteristics of the physical object in the real world space. The method according to claim 11.

Images