Feature Similarity Scoring of Physical Environments for Augmented Reality Gameplay

By generating user space scores through sensor fusion and machine learning, the method adjusts AR gameplay parameters to address environmental uniqueness, enhancing gaming experience and fairness across varied real-world settings.

JP2024541423AActive Publication Date: 2024-11-08SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2024529659
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-18
Filing Date
2022-10-28
Publication Date
2024-11-08
Estimated Expiration
2042-10-28

Smart Images

  • Figure 2024541423000001_ABST
    Figure 2024541423000001_ABST
Patent Text Reader

Abstract

A method and system for processing similarity of characteristics of a real-world space used for augmented reality (AR) gameplay is disclosed. The method includes receiving sensor data captured from a real-world space used by a user for AR gameplay of a game. The sensor data provides data for identifying characteristics of physical objects in the real-world space. The method includes generating a user space score using the characteristics of the physical objects identified in the real-world space. The method includes comparing the user space score to a predefined game space score for the game. The comparison is used to generate a fitness handicap for the user's AR gameplay of the game in the real-world space. Gameplay parameters may be adjusted to compensate or adjust the fitness handicap.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure generally relates to using similarities of features in real-world spaces to enable adjustment of gameplay parameters in AR games and / or to adapt to the uniqueness of a user's game space. [Background technology]

[0002] 2. Description of Related Art Augmented reality (AR) technology has seen unprecedented growth over the years and is expected to continue to grow at a compound annual growth rate. AR technology is an interactive three-dimensional (3D) experience that combines a view of the real world with computer-generated elements (e.g., virtual objects) in real time. In an AR simulation, the real world is infused with virtual objects to provide an interactive experience. With the growing popularity of AR technology, various industries have implemented AR technology to improve user experience. Some of the industries include, for example, the video game industry, entertainment, and social media.

[0003] For example, in the video game industry, there is a growing trend to improve users' gaming experience by increasing the realism of the experience. However, because AR experiences necessarily rely on things that exist in the real world, one user's AR experience will be different from another user's AR experience, even if the same medium is used for the AR experience.

[0004] It is in this context that embodiments of the present disclosure arise. Summary of the Invention

[0005] Implementations of the present disclosure include methods and systems for generating and using feature similarity scores for gameplay in an AR environment. In one embodiment, the method includes scanning the environment using multiple sensors (visual, audio, etc.) to capture key aspects of the play environment. In one configuration, a scoring method is used to determine how well the user's real-world space fits compared to an optimized type of gameplay. As an example, the scoring can take into account the number of clean walls, open floor area, shape of objects or obstacles, size or volume of objects or obstacles, amount of smooth or rough surfaces, number of windows, type of furniture, presence of other people or players, etc. This scoring for the user's real-world space generates a user space score. The user's space score may be compared to a game space score predefined by the game's developer (e.g., representing an optimized type of environment for playing a VR game).

[0006] Knowing the user spatial score allows gameplay parameters to be dynamically adjusted to reduce or eliminate the user's fitness handicap. In some cases, gameplay is adjusted to standardize gameplay (e.g., how enemy types are spawned, how they are scored, the difficulty of actions or tasks, etc.) compared to the population of other game players and what the game developer intended as ground truth. In one embodiment, capturing and calculating the user's spatial score allows AR game development to predict and respond to the uniqueness of the environment.

[0007] In one embodiment, a method is disclosed for processing similarity of features of a real-world space used for augmented reality (AR) gameplay. The method includes receiving sensor data captured from a real-world space used by a user for said AR gameplay of a game. The sensor data provides data for identifying characteristics of physical objects in the real-world space. The method includes generating a user space score using the characteristics of the physical objects identified in the real-world space. The method includes comparing the user space score to a predefined game space score for the game. The comparison is used to generate a fitness handicap for the user's AR gameplay of the game in the real-world space.

[0008] In another embodiment, a method for processing similarity of features of a real-world space used for augmented reality (AR) gameplay is disclosed. The method includes receiving sensor data captured from a real-world space used by a user for AR gameplay of a game. The sensor data provides data for identifying characteristics of physical objects in the real-world space. The method includes generating a user space score using the characteristics of the physical objects identified in the real-world space. The method includes generating a fitness handicap for AR gameplay of the game by the user in the real-world space. The method includes adjusting gameplay parameters of the game for the user. The adjustment changes a difficulty setting of the game to compensate for the fitness handicap.

[0009] In some embodiments, the fitness handicap is based on a comparison of the user space score to a game space score that defines an optimized baseline for the type of environment used to play the game during AR gameplay.

[0010] In some embodiments, the method further includes adjusting a scoring threshold of the game, where adjusting the scoring threshold reduces the skill requirement of an action in the game to gain a score in the game.

[0011] In some embodiments, the user space score includes a numerical count of each type of surface material within the interactive zone.

[0012] In some embodiments, each numerical count is scored as meeting a threshold size for the aforementioned surface material type.

[0013] In some embodiments, sensor data is captured by one or more of a camera, a microphone, or an ultrasonic sensor, and the sensor data is processed to identify a set of labeled features by a machine learning process, and the machine learning process is trained to identify characteristics of physical objects in a real-world space.

[0014] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure. [Brief description of the drawings]

[0015] The present disclosure may be better understood by reference to the following description taken in conjunction with the accompanying drawings.

[0016] [Figure 1] 1 illustrates an example of a real-world space in which a user is playing a game, according to one embodiment.

[0017] [Diagram 2] 1 illustrates an example process for adjusting gameplay parameters of a game using a gameplay normalization engine according to one embodiment.

[0018] [Diagram 3]1 illustrates an example embodiment in which game play parameters may be adjusted based on a user's fitness handicap, according to one embodiment.

[0019] [Figure 4] 1 illustrates a process for generating a user space score according to one embodiment.

[0020] [Diagram 5] 1 illustrates examples of different types of objects that exist in a real-world space and the attributes and characteristics that are identified in the real-world space in response to a scan, according to one embodiment.

[0021] [Figure 6] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] The following embodiments of the present disclosure provide methods, systems, and devices for generating and using a characteristic similarity score for gameplay in an AR environment. When playing a game using augmented reality (AR) glasses, a characteristic similarity score is generated in the form of a game space score associated with a user's interaction zone in a real-world space. The AR glasses used herein can handle both AR and virtual reality. For example, AR allows virtual objects to be placed in and around a real-world space, while a VR environment is generated entirely by a computing system. According to one embodiment, a computing system may be associated with the AR glasses. The AR glasses may include a processor that executes program instructions for rendering AR objects and / or VR content. In one embodiment, the AR glasses are equipped with wireless communication capabilities so that they can communicate directly to a server over the Internet. In other embodiments, the AR glasses may communicate locally with a game console or a local computer, and the game console or local computer may communicate with a server over the Internet.

[0023] In one embodiment, the method includes scanning the environment using multiple sensors (visual, audio, etc.) to capture key aspects of the play environment. The multiple sensors may be various, such as cameras, microphones, ultrasonic sensors, light sensors, motion sensors, inertial sensors, and / or combinations thereof. In some embodiments, sensor fusion is used to collect data captured from various sensors when scanning a room, such as an interactive zone. In one configuration, a scoring method is used to determine how well the user's real-world space fits compared to an optimized type of gameplay. As an example, the scoring may take into account the number of clean walls, open floor area, shape of objects or obstacles, size or volume of objects or obstacles, amount of smooth or rough surfaces, number of windows, type of furniture, presence of other people or players, etc. This scoring of the user's real-world space generates a user space score. The user's space score may be compared to a game space score predefined by the game developer (e.g., representing an optimized type of environment for playing a VR game).

[0024] Knowing the user spatial score allows gameplay parameters to be dynamically adjusted to reduce or eliminate the user's fitness handicap. In some cases, gameplay is adjusted to standardize gameplay (e.g., how enemy types are spawned, how they are scored, the difficulty of actions or tasks, etc.) compared to the population of other game players and what the game developer intended as ground truth. In one embodiment, capturing and calculating the user's spatial score allows AR game development to predict and respond to the uniqueness of the environment.

[0025] FIG. 1 illustrates an example of a real-world space 100 in which a user 104 is playing a game. In one embodiment, the user is playing an augmented reality game using AR glasses 103 in an interactive zone 102. The interactive zone 102 is an area close to the user, e.g., where the user is located and playing the game, although the user's location may be more widespread. Generally speaking, the real-world space 100 may not include other rooms in the house or location. This is because other areas may not provide the performance of the AR glasses 103 when playing the AR game. For example, the AR glasses 103 may project virtual objects on or near real-world objects in the interactive zone.

[0026] If there are not enough surfaces or objects onto which AR objects can be projected or moved in response to a game, the performance of the projection, e.g., how the virtual object is perceived within the real-world space 100, may be affected. For example, having too many windows may cause external ambient light to affect the performance of the projected AR object. Similarly, having too many cluttered walls or spaces in a space may not provide enough space for the projection and / or movement of the AR object. In some cases, a floor that is too reflective or too rough may result in suboptimal projection of AR content.

[0027] When an AR game developer creates a game, the developer must make some baseline assumptions about how a real-world space looks and / or provide a minimum level of space to place, move, or interact with projected and tracked AR content. If the space lacks these baseline assumptions for functionality required for the AR game, the user's game performance will be reduced during gameplay. According to one embodiment, a process is described that allows the interactive zone 102 to be scanned to generate a game space score. The game space score of the user's real-world space 100 is then compared to a game space score predefined by the developer for the game. For example, the game space score may define a baseline requirement for an optimal gameplay experience using AR or VR.

[0028] In one embodiment, the scan can be performed using one or more sensors. The sensors can be various, such as those capturing audio, capturing video, capturing motion, capturing ultrasound, capturing temperature, capturing humidity, capturing inertial data, etc. The sensors can be part of existing devices in the real-world space, such as one or more cameras S1, S2, microphones S3, S4, a camera and microphone S5 on the AR glasses 103, a microphone S6 that may be part of a door or doorbell, a microphone S7 on a mobile phone, a microphone S8 integrated into one or more controllers that the user 104 may have, etc.

[0029] These sensors may be configured in one embodiment to capture characteristics of the interactive zone 102 and identify object types and / or surface materials. In the illustrated example, a user 104 is standing on a rug 108. A chair 106 and a chair 126 are identified in the space. A picture 110 is hung on the wall. A window 114 is identified on one of the walls. A television 116 is hung on another wall. A door 120 is located next to the television 116. A game console 118 is placed under the television 116. A table 112 is present and a mobile phone S7 is placed on the table. A dog 124 is captured moving around in the interactive zone 102. A lamp 112 is placed on a small table 128 in the center of the room. In one embodiment, sensor fusion 130 refers to capturing environmental data from a real-world space to identify objects, surfaces, materials, and other environmental features.

[0030] In one embodiment, machine learning is utilized to process inputs received from various sensors. The machine learning algorithm uses a trained model to identify objects that may be present within the interactive zone. For example, raw sensor data is first processed by a machine learning system to identify feature data from within the captured sensor data. The feature data is then processed using a labeling algorithm. The labeling algorithm is configured to assign meaning to the feature data to classify the labeled feature data. Over time, based on the training, the labeled feature data is properly classified and the machine learning model is improved. Once the machine learning model is trained, it can be used to process sensor data captured in the interactive zone.

[0031] The sensor data can be processed using machine learning models to identify what real-world features are present in the real-world space. For example, the sensor data can be used to identify lamps 112, rugs 108, dogs 124, windows 114, televisions 116, and other objects that may enter or exit the interactive zone. As described in more detail below, by identifying the features present in the interactive zone and the properties of those features, a game space score for the interactive zone can be generated.

[0032] 2 illustrates an example process for adjusting gameplay parameters of a game 202 using a gameplay normalization engine 200, according to one embodiment. In one example, a user 104 may be playing or have begun playing a game 202. The normalization engine 200 may be triggered (by programmatic or user input) to perform a scan of the interaction zone 102 before the game launches, such that the AR game experience is pre-configured with the adjusted gameplay parameters. In other embodiments, the user 104 may begin playing the game, and after the game starts, the gameplay parameters may be adjusted all at once or gradually over time to provide a smooth transition.

[0033] In operation 204, the user's real-world space represents where the user may be playing the AR game, referred to herein as the interactive zone 102. The interactive zone represents an area where the user may interact with the AR game or the AR glasses 103, as described above. Sensors in the real space 100 may be utilized to scan the environment and generate an acoustic profile of the environment. The scan may perform sensor fusion 130 to perform an initial scan of the real-world space utilizing various sensors that may be present in or that may be brought into the interactive zone 102. In this manner, a scan of the real-world space 100, and specifically the interactive zone 102, may be performed once or periodically and securely stored for later retrieval for gameplay.

[0034] In this manner, continuous scanning is not required each time a game is played using the AR glasses 103. Using the sensor fusion 130, a scoring process 206 is performed. The scoring process 206 is generally used to identify surface materials associated with different types of objects in the real-world space 100 and different types of materials present in and around the interactive zone 102. As described below, the scoring process may include identifying different types of materials for different types of objects present in the interactive zone 102. The scoring process may include identifying specific types of materials and determining whether those types of materials are present in the interactive zone 102.

[0035] A threshold is used to determine whether those surfaces are present, or whether there is a sufficient amount of those surfaces present to be counted in the numerical count. The numerical count is thus used to score the types of surfaces present in the interactive zone, and the acoustic absorption or reflection associated with those types of materials. The numerical score is a combined score of the various types of materials and surfaces, and their associated quantities and absorption and reflection rates. Taken together, these values ​​generate a vector that defines the various types of materials and the weights associated with those materials in real-world space.

[0036] Thus, the scoring process 206 generates a user space score 210. The user space score 210 represents the output of the scoring process 206. In one embodiment, each interactive zone 102 in which a user is using AR glasses 103 is assigned a unique user space score 210. The game developer of the game 202 also generates an optimized game space score 208 that represents an optimal or optimized baseline score for the type of environment in which the game may be played using the AR glasses 103 or similar AR type glasses.

[0037] In one embodiment, the user space score can be further modified or adjusted using self-identified disabilities. These self-identified disabilities are user specific and can be adjusted based on input, settings, or learned settings. By way of example and without limitation, self-identified disabilities may include color blindness, hearing impairment, susceptibility to motion sickness, need to be seated, mobility impairment, reduced motor skills, sensitivity to loud noises, difficulty seeing in the dark, dislike of bright light, or combinations thereof, and / or other self-identified disabilities or preferences. In one configuration, these self-identified disabilities are considered added accessibility options and can be set using the user interface, voice input, controller input, or touch screen.

[0038] By comparing the game space score 208 with the user space score 210, a fitness handicap 212 can be generated for the user space score 210. The fitness handicap 212 represents how close the user space score 210 is to the game space score 208 or how far the scores are apart. The greater the deviation between the two, the less suitable the interactive zone 102 is for playing the AR game due to the surface characteristics of the interactive zone 102. As a result, a user 104 plane in the AR game within an interactive zone 102 with a large deviation will be handicapped relative to other users with minimal deviation.

[0039] In one embodiment, a deviation of less than 10 percent is appropriate and does not significantly impact the user 104 when playing the AR game 202. A deviation of less than 30 percent is acceptable, but puts the user 104 at a disadvantage relative to other players playing with smaller deviations. For example, a deviation of more than 30 percent may result in a significant handicap that may prevent the user 104 from progressing in the game or hinder the user 104 from playing the game relative to other users.

[0040] In one embodiment, the fitness handicap 212 of the user 104 may be used to adjust game play parameters 214 to compensate for deviations. As an example, the game play parameters may be dynamically adjusted to make it easier to earn points in a game, easier to interact with the AR character, easier to see the AR character, change the parameters of the AR character to make it more prominent in the background, make the AR character or objects louder based on the acoustics of the user space, easier to progress to another level, etc.

[0041] These parameters become inputs into the game engine, which adjusts various difficulty levels, settings, and other parameters when playing with a fitness handicap and adjusts to that fitness handicap. In one embodiment, when one player is playing against another in a multiplayer AR game, the gameplay parameters 214 of each player can be adjusted to ensure that the fitness handicaps 212 are approximately equal. In this manner, the fitness handicap does not need to be eliminated entirely, but rather the gameplay parameters of each player's respective game instance can be simply adjusted to keep the players on an approximately equal or comparable footing. In a multiplayer game, optimizing for one or both of the players, or multiple players, improves the AR game and gameplay experience by ensuring that the characteristics of their respective real-world environments do not give one or a few users an advantage over the others.

[0042] FIG. 3 illustrates an example embodiment in which game play parameters may be adjusted based on a user's fitness handicap, according to an embodiment. In one embodiment, the user 104 selects a game 302. The game 302 is an AR game that has been outfitted by a developer to allow adjustment of various parameters, which may be predefined parameters to eliminate deviations of a particular player's handicap. Once the game 302 is selected, the user's real-world space may be scanned in operation 304. As previously mentioned, the scan may be performed by one or more sensors. In a simple example, sensors in the AR glasses themselves may be utilized. The sensors may include microphones, cameras, inertial sensors, ultrasonic sensors, or a combination of these or multiple sensors. In other embodiments, the sensors may include other sensors that are part of the environment in which the player or user plays the game 302.

[0043] As shown in FIG. 1, there can be many sensors, and the output of the sensors can be utilized in the sensor fusion 130 process to accurately scan and quantify different materials, surfaces, absorption, reflectance, and parameters of the user's real-world space. Using the scanned parameters (which may be in the form of a vector or matrix), the data can be utilized to generate a user space score 306. In one embodiment, the user space score can be a normalized value. In another embodiment, the user space score can be multiple values ​​in vector format with weights and magnitudes associated with each of the different types of parameters that make up the score. In another embodiment, the matrix can identify various types of parameters that define the surfaces and materials of the space, and the magnitude or amount of those materials and surfaces relative to the user's location in that real-world space or within a particular interactive zone 102.

[0044] In operation 308, the user space score is compared to a predefined game space score of the game. As mentioned above, the game space score is a score value signed by the developer of the game. These values ​​represent the optimal value of the space in which the game can be played. For example, if an AR game requires a large amount of projection of images on a wall, the amount of free space on the wall where the projection can be made improves the playability of the AR game. Other parameters related to the space and its suitability for the projected or displayed AR content are also taken into account. In operation 310, a fitness handicap is generated for the user based on the comparison. As mentioned above, the fitness handicap may identify deviations from the optimal acquisition space score defined by the developer of the game.

[0045] At operation 312, gameplay parameters of the game may be adjusted based on the fitness handicap. In one embodiment, the adjustment reduces deviations of the game space score from an optimized baseline. In other embodiments, it may be desirable to increase the deviation of one player so that the deviations of two or more players are approximately equal. Thus, it should be appreciated that deviations from the game space score may be tolerated in some cases, and may be increased in other cases, such that players in a multiplayer game are approximately on an equal footing when the deviations are approximately equal or substantially equal.

[0046] 4 illustrates a process for generating a user space score according to one embodiment. As previously mentioned, there are several ways to generate a user space score, and the following examples are not limiting of other methods previously mentioned. In one embodiment, in operation 402, sensor data is captured by multiple sensors in an interactive zone of the real-world space. As previously mentioned, the sensor data can be captured using various types of sensors, such as cameras, microphones, inertial sensors, ultrasonic sensors, temperature sensors, humidity sensors, etc. The types of devices that can be used to capture these sensor data inputs can include AR glasses 103, and devices that may be present in the interactive zone 102 or real-world space 100.

[0047] By way of example, these types of devices may include, in addition to the AR glasses 103, television sensors, cameras, game consoles, controllers used in games, mobile phones, watches, and other devices that may be present in and around the real-world space 100. In operation 404, properties of the interactive zone are assembled. These properties identify different types of surface materials within the interactive zone and other objects that may be present or that define the real-world space. These objects may change depending on what is present in the real-world space 100 of the user 104.

[0048] The example of FIG. 1 illustrates that different types of objects can be present in the interactive zone 102. It should be understood that different users have different spaces and different objects present within the space that are customized to the user's tastes, desires, decoration and placement choices. Thus, the characteristics of the user space score will vary depending on where the user decides to play the AR game and interact in a particular interactive zone. Thus, the types of devices that define the sensors and capture data, for example in the form of sensor data for sensor fusion 130, can vary widely.

[0049] Nevertheless, sensor data is captured to scan and identify various types of objects that may be present within the space and the surface materials that define those types of objects. In some embodiments, a numerical count is performed to identify the amount of objects that meet a threshold size and the amount of surfaces of those objects and materials. For example, if there is a clean, open wall in the interactive zone 102, that clean, open wall may represent a value based on the amount of square feet or square meters of that type of material. Thus, a different square foot or square meter can be calculated for each type of material and a numerical value can be assigned to the amount of that type of material or surface object in the space.

[0050] If the amount of surface material is too small to be counted, such as less than one square foot, the material or object may not be counted as it does not meet the threshold. In some embodiments, material surfaces greater than at least one square foot are counted to determine the number of materials of that type within the interactive zone 102. In some embodiments, a machine learning model 406 may be utilized to process the raw sensor data to identify different types of materials and characteristics present within the interactive zone. For example, the machine learning model may receive image data that can be processed to identify features within each image, and classify those features to identify objects present within the space.

[0051] For example, machine learning can be used to identify actual features and characteristics present in the image captured by the camera. The features and characteristics can identify not only what the object is, but also attributes of the object such as texture, reflectivity, roughness, position, lighting, and other distinguishable or identifiable characteristics. Thus, the machine learning model can also incorporate other inputs such as sound, lighting, microphone arrays, and other data that helps identify the characteristics of different types of objects and surface materials. Using this information and the numerical count generated in act 408, a user space score is generated in act 410.

[0052] 5 illustrates examples of different types of objects present in a real-world space and their attributes and characteristics that may be identified in the real-world space in response to the above-described scan. As a simple example, the real-world space or interactive zone 102 includes walls 420, floors 422, furniture 424, and other objects 426. The other objects are too numerous to mention here, but may include any type of object that may exist in a space. In some embodiments, the other objects may be people, pets, or other moving types of objects, such as robots.

[0053] In the illustrated example, wall 420 may be defined by various types of surface materials, such as rough, brick, cladding, smooth, window, etc. If a threshold is met for the amount of that material, a numerical count is assigned to that type of surface material. By way of example, if the amount of rough surface on the wall exceeds 2 square feet, the numerical count may be assigned at least 2. If the amount of smooth surface exceeds at least 16 square feet, the numerical count may be set to 16. The illustrated numbers are provided solely to illustrate a method of counting that takes into account the type and quantity or amount of materials present in the space.

[0054] The same can be done for floors, furniture, and other objects. The result is a user space score 450 for the interactive zone 102 for the user 104. The user space score can be defined as a vector, matrix, or other value or formula, and can be compared to the game space score defined by the developer for the game. As previously mentioned, utilizing or knowing the user space score allows adjustment of gameplay parameters of the game to remove deviations or adjust deviations for one or other players. As previously mentioned, the gameplay parameters can also include adjustments of metadata that control one or more difficulty settings or scoring thresholds when playing the game with a particular fitness handicap.

[0055] It should be appreciated that adjusting gameplay parameters may remove or adjust game difficulty or other parameters to allow players to play the game on a substantially equal footing. Thus, scanning the environment generates feature similarities between the user's real-world space and features during the AR game that provide a more optimal or optimized baseline for the type of environment used when playing the game. Allowing dynamic adjustment of gameplay parameters reduces handicaps between players and allows players to enjoy a more enjoyable game that takes into account each player's unique environment when playing the AR game. Thus, adjustment provides normalization or ground truth between two or more players, even when the players are playing in different real-world environments (e.g., online games or cloud games during streaming).

[0056] FIG. 6 illustrates components of an exemplary device 600 that can be used to implement aspects of various embodiments of the present disclosure. The block diagram illustrates device 600, which can incorporate or be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for implementing embodiments of the present disclosure. Device 600 includes a central processing unit (CPU) 602 for executing software applications and optionally an operating system. CPU 602 may be comprised of one or more homogeneous or heterogeneous processing cores. For example, CPU 602 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly adapted for highly parallel and computationally intensive applications, such as interpreting queries, identifying contextually relevant resources, and implementing and rendering contextually relevant resources in real time within a video game. Device 600 may be one that is localized to a player playing a game segment (e.g., a game console) or one that is remote from the player (e.g., a back-end server processor), or one of many servers that use virtualization in a game cloud system for remote streaming of game play to clients.

[0057] Memory 604 stores applications and data used by CPU 602. Storage 606 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices 608 communicate user input from one or more users to device 600, and examples of user input devices 608 may include a keyboard, mouse, joystick, touchpad, touch screen, still or video recorder / camera, gesture-recognizing tracking device, and / or microphone. Network interface 614 enables device 600 to communicate with other computer systems over an electronic communications network, which may include wired or wireless communications over a local area network or a wide area network such as the Internet. The audio processor 612 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 602, memory 604, and / or storage 606. The components of the device 600, including the CPU 602, memory 604, data storage 606, user input device 608, network interface 610, and audio processor 612, are connected via one or more data buses 622.

[0058] A graphics subsystem 620 is further coupled to the data bus 622 and the components of the device 600. The graphics subsystem 620 includes a graphics processing unit (GPU) 616 and a graphics memory 618. The graphics memory 618 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 618 may be integrated in the same device as the GPU 608, may be coupled as a separate device from the GPU 616, and / or may be incorporated within the memory 604. The pixel data may be provided to the graphics memory 618 directly from the CPU 602. Alternatively, the CPU 602 provides data and / or instructions defining a desired output image to the GPU 616, from which the GPU 616 generates pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in the memory 604 and / or the graphics memory 618. In an embodiment, GPU 616 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. GPU 616 may further include one or more programmable execution units capable of executing shader programs.

[0059] Graphics subsystem 614 periodically outputs pixel data of an image from graphics memory 618 for display on display device 610. Display device 610 may be any device capable of displaying visual information in response to signals from device 600, including CRT, LCD, plasma, and OLED displays. Device 600 may provide analog or digital signals to display device 610, for example.

[0060] It should be noted that access services distributed over a wide geography, such as providing access to games in the current embodiment, often use cloud computing. Cloud computing is a computing paradigm in which dynamically scalable, often virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure of the "cloud" that supports them. Cloud computing can be divided into different services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications, such as video games, online, accessed from a web browser, but the software and data are stored on servers in the cloud. The term cloud is used as a metaphor for the Internet based on how the Internet is depicted in computer network diagrams, and is an abstraction of the complex infrastructure it hides.

[0061] In some embodiments, a game server may be used to perform the operation of the duration information platform for video game players. Most video games played over the Internet operate through a connection to a game server. Typically, the game uses a dedicated server application that collects data from the player and distributes the collected data to other players. In other embodiments, the video game may be executed by a distributed game engine. In these embodiments, the distributed game engine may run on multiple processing entities (PEs), each PE executing a functional segment of the given game engine in which the video game is executed. Each processing entity is viewed by the game engine as simply a computational node. The game engine typically performs a functionally diverse set of operations to execute the video game application along with additional services experienced by the user. For example, the game engine implements game logic and performs game calculations, physics, geometry transformations, rendering, lighting, shading, audio, as well as additional in-game or game-related services. The additional services may include, for example, messaging, social utilities, audio communication, game play playback functions, help functions, and the like. While in some cases the game engine may run on an operating system virtualized by a hypervisor on a particular server, in other embodiments the game engine itself may be distributed across multiple processing entities, each of which may reside on a different server unit in a data center.

[0062] According to this embodiment, each processing entity for the execution of the method / processing operations may be a server unit, a virtual machine, or a container depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformation, that particular game engine segment may be provisioned with a virtual machine associated with a graphics processing unit (GPU) since it will be performing a lot of relatively simple mathematical operations (e.g., matrix transformations). Other game engine segments requiring fewer but more complex operations may be provisioned with processing entities associated with one or more higher powered central processing units (CPUs).

[0063] By distributing the game engine, the game engine has elastic computational characteristics that are not bound by the capabilities of a physical server unit. Instead, the game engine is provisioned with more or fewer computation nodes as needed to meet the demands of the video game. From the perspective of the video game and the video game player, a game engine that is distributed across multiple computation nodes is indistinguishable from a non-distributed game engine running on a single processing entity, since a game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output component to the end user.

[0064] Users access remote services through a client device that includes at least a CPU, a display, and I / O. The client device may be a PC, a cell phone, a netbook, a PDA, etc. In one embodiment, a network running on the game server recognizes the type of device used by the client and adjusts the communication method employed. In another case, the client device accesses the application on the game server over the Internet using a standard communication method such as HTML.

[0065] Of course, a given video game or game application may be developed for a particular platform and a particular associated controller device. However, when such games are made available through a game cloud system as described herein, users may access the video game using different controller devices. For example, a game may be developed for a game console and its associated controller, but a user may access a cloud-based version of the game from a personal computer utilizing a keyboard and mouse. In such a scenario, the input parameter settings may define a mapping from inputs that may be generated by the controller devices available to the user (keyboard and mouse in this case) to inputs that are acceptable for execution of the video game.

[0066] In another example, a user may access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen driven device. In this case, the client device and the controller device are integrated together in the same device, and input is provided by detected touchscreen input / gestures. In such a device, an input parameter setting may define a particular touchscreen input that corresponds to game input of a video game. For example, a button, a directional pad, or other type of input element may be displayed or overlaid during the execution of the video game to indicate a location on the touchscreen that the user can touch to generate game input. Gestures such as swipes in a particular orientation, or particular touch motions may also be detected as game input. In one embodiment, to familiarize the user with control operations on the touchscreen, a tutorial may be provided to the user showing how to input gameplay via the touchscreen, for example, before starting gameplay of the video game.

[0067] In some embodiments, the client device serves as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection and transmits inputs from the controller device to the client device. The client device may then process these inputs and then transmit the input data to the cloud gaming server over a network (e.g., a network accessed through a local network device such as a router). However, in other embodiments, the controller itself may be a networked device that has the ability to communicate inputs directly to the cloud gaming server over a network, without the need for such inputs to be communicated through the client device first. For example, the controller may connect to a local network device (such as the router mentioned above) to send and receive data to the cloud gaming server. Thus, while the client device is still required to receive video output from the cloud-based video game and render it on a local display, input latency may be reduced by allowing the controller to bypass the client device and transmit inputs directly over the network to the cloud gaming server.

[0068] In one embodiment, the networked controller and client devices can be configured to transmit certain types of inputs directly from the controller to the cloud gaming server and other types of inputs via the client device. For example, inputs whose detection does not rely on any additional hardware or processing, apart from the controller itself, can be transmitted directly from the controller to the cloud gaming server over the network, bypassing the client device. Such inputs can include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), and the like. However, inputs that utilize additional hardware or require processing by the client device can be transmitted by the client device to the cloud gaming server. These can include captured video or audio from the gaming environment that can be processed by the client device before transmission to the cloud gaming server. In addition, inputs from the controller's motion detection hardware can be processed by the client device in conjunction with the captured video to detect the position and movement of the controller, which is then communicated by the client device to the cloud gaming server. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0069] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations that use various features disclosed herein. Thus, the examples provided are only some of the possible examples and are not intended to limit the various implementations that are possible by combining various elements to define more implementations. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0070] Embodiments of the present disclosure may be practiced with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0071] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various processing-related intervals, so long as the processing of telemetry and game state data to generate modified game state is performed in the desired manner.

[0072] One or more embodiments may also be fabricated as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data and can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. A computer readable medium may include computer readable tangible media distributed across network-connected computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0073] In one embodiment, the video game is executed locally on a game console, personal computer, or server. In some cases, the video game is executed by one or more servers in a data center. When the video game is executed, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. A simulation, in some embodiments, is an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In either case, when the video game is represented as a simulation, the simulation may be executed to render interactive content that can be interactively streamed, executed, and / or controlled by user input.

[0074] Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A method for processing similarity of real-world spatial features used in augmented reality (AR) gameplay, comprising: receiving sensor data captured from the real-world space for use by a user for the AR gameplay of a game, the sensor data providing data for identifying characteristics of physical objects within the real-world space; generating a user space score using the characteristics of the identified physical objects in the real-world space; and comparing the user space score with a predefined game space score for the game, the comparison being used to generate a fitness handicap for the AR gameplay of the game in the real world space by the user; The method includes:

2. The method of claim 1 , wherein the real-world space includes an interactive zone for the user during the AR gameplay, and the properties of the physics objects include a surface material type.

3. The method of claim 2 , wherein the user space score comprises a numerical count of each type of surface material within the interactive zone.

4. The method of claim 3 , wherein each numerical count is scored as meeting a threshold size for the surface material type.

5. 2. The method of claim 1 , wherein the sensor data is captured by one or more of a camera, a microphone, or an ultrasonic sensor, and the sensor data is processed to identify a set of labeled features by a machine learning process, the machine learning process being trained to identify the characteristics of the physical object in the real-world space.

6. 2. The method of claim 1 , wherein the game space score is correlated to interactive content developed for the game, and the game space score defines an optimized baseline for the type of environment used to play the game during the AR gameplay.

7. 2. The method of claim 1 , wherein the fitness handicap for the AR gameplay of the game in the real-world space by the user identifies a deviation from an optimized baseline for a type of environment used to play the game during the AR gameplay.

8. 8. The method of claim 7, further comprising adjusting game play parameters of the user while playing the game to significantly eliminate the deviation from the optimized baseline, the adjustment being configured to adjust metadata controlling one or more difficulty settings or scoring thresholds while playing with the fitness handicap.

9. 9. The method of claim 8, wherein substantially eliminating the deviation serves to place the user on a generally equal footing with other users playing the game with game space scores that generally approach the optimized baseline.

10. 9. The method of claim 8, wherein each of the users and each of the other users dynamically adjusts their game play parameters as they play the game to substantially eliminate the deviations from the optimized baseline, such that each of the users and each of the other users play the game on a roughly equal footing with one another.

11. 1. A method for processing similarity of real-world spatial features used in augmented reality (AR) gameplay, comprising: receiving sensor data captured from the real-world space for use by a user for the AR gameplay of a game, the sensor data providing data for identifying characteristics of physical objects within the real-world space; generating a user space score using the characteristics of the identified physical objects in the real-world space; and generating a fitness handicap for the AR gameplay of the game in the real-world space by the user; and adjusting game play parameters of the game for the user, the adjustment changing a difficulty setting of the game to compensate for the fitness handicap; The method includes:

12. 12. The method of claim 11 , wherein the fitness handicap is based on a comparison of the user space score to a game space score that defines an optimized baseline for the type of environment used to play the game during the AR gameplay.

13. 13. The method of claim 12, further comprising adjusting a scoring threshold of the game, the adjustment of a scoring threshold reducing or increasing skill requirements for actions in the game to gain a score in the game.

14. 12. The method of claim 11, wherein adjusting gameplay parameters of the game of the user serves to reduce the user's deviation in the real-world space to a game space score that is approximately at a baseline optimized for the type of environment used to play the game during the AR gameplay.

15. 15. The method of claim 14, wherein reducing the deviation serves to place the user on a generally equal footing with other users playing the game with game space scores that generally approach the optimized baseline.

16. 12. The method of claim 11, further comprising adjusting game play parameters of the game of the other user to place the fitness handicap of the other user approximately equal to the fitness handicap of the user.

17. The method of claim 11 , wherein the real-world space comprises an interactive zone for the user during the AR gameplay, and the properties of the physics objects include a surface material type.

18. The method of claim 17 , wherein the user space score comprises a numerical count of each type of surface material within the interactive zone.

19. The method of claim 18 , wherein each numerical count is scored as meeting a threshold size for the surface material type.

20. 12. The method of claim 11, wherein the sensor data is captured by one or more of a camera, a microphone, or an ultrasonic sensor, and the sensor data is processed to identify a set of labeled features by a machine learning process, the machine learning process being trained to identify the characteristics of the physical object in the real-world space.

Citation Information

Patent Citations

  • Contextual Applications in Mixed Reality Environments

    JP2020520000A

  • Mixed Reality Instruments

    JP2021514082A

  • Runtime adaptation of augmented reality gaming content based on context of surrounding physical environment

    US20200218426A1