User emotion detection to identify user impairment during game amusement to provide automatic generation or modification of in-game effects

Abstract

A method includes executing game logic of a video game to generate a plurality of video frames for game amusement of the video game by a player, wherein game state data is generated during the execution of the game logic. The method includes determining a current context in the game amusement of the video game based on the game state data. The method includes determining a user emotion of the player for the current context in the game amusement of the video game. The method includes determining that the user emotion of the player is inconsistent with an expected user emotion for the current context. The method includes generating an in-game effect for the current context. The method includes simultaneously presenting the in-game effect with one or more video frames associated with the current context in the game amusement of the video game.

Description

Technical Field

[0001] The present disclosure relates to games, and more particularly to providing overlay logic to support a game engine, the overlay logic being used to detect user emotions during game play of a video game and automatically provide in-game effects when the user emotions are undesirable for the current context in the game play to make the video game more interesting or more intense. In addition, an artificial intelligence model configured to predict the user emotions of the corresponding player can be built for the player. Background Art

[0002] Video games and / or game applications and their related industries (e.g., playing video games) are very popular and account for a large percentage of the global entertainment market. Video games can be played anytime, anywhere using various types of platforms (including game consoles, desktop or laptop computers, mobile phones, etc.).

[0003] As is well known, video games provide immersive experience to players. For example, video games can provide audio effects in a given scene to enhance the player's experience. However, hearing-impaired players may not be able to fully experience and / or appreciate the audio effects. In addition, subtitles in video games may be insufficient or cumbersome to use during game play, making it impossible for players to understand in time what the subtitles are trying to convey. Therefore, hearing-impaired players will not have the same experience of playing video games as non-impaired players. For example, hearing-impaired players may not be able to feel the same urgency as non-impaired players who can hear audio effects when playing video games in a certain scene. In other cases, hearing-impaired players may not really understand what is happening during the game play of video games (for example, important events highlighted in the scene are happening in the background) or what players are expected to do (for example, moving in a certain direction).

[0004] It would be desirable to be able to increase the accessibility of video games to impaired players so that impaired players would have the same or similar experience as non-impaired players when playing the video games.

[0005] It is against this background that embodiments of the present disclosure are presented. Summary of the invention

[0006] Embodiments of the present disclosure relate to providing overlay logic that supports a game engine, and the overlay logic is used to detect user emotions during the game play of a video game, especially for players who may be physically impaired (e.g., hearing impaired, etc.). When the user emotion is undesirable for the current context of the game play, the overlay logic is configured to provide in-game effects to make the video game more interesting or more intense, so as to more accurately convey to the user what is happening during the game play. For example, the in-game effects can be transformed from subtitles to convey to the user that a violent explosion is occurring. Although hearing-impaired users may not usually be aware of the urgency presented in the game play, the newly generated in-game effects allow the player to be more fully aware of the explosions that are occurring, and the player can experience the urgency of the intention that these explosions should trigger. In this way, hearing-impaired players can enjoy a game experience similar to the game experience of non-impaired players who can hear sound effects.

[0007] In one embodiment, a method is disclosed. The method includes executing game logic of a video game to generate a plurality of video frames of a game play of the video game performed by a player, wherein game state data is generated during the execution of the game logic. The method includes determining a current context in the game play of the video game based on the game state data. The method includes determining a user emotion of the player for the current context in the game play of the video game. The method includes determining that the user emotion of the player is inconsistent with an expected user emotion for the current context. The method includes generating an in-game effect for the current context. The method includes presenting the in-game effect simultaneously with one or more video frames associated with the current context in the game play of the video game.

[0008] In another embodiment, a non-transitory computer-readable medium storing a computer program for implementing a method is disclosed. The computer-readable medium includes program instructions for executing the game logic of a video game to generate a plurality of video frames of a game play of the video game performed by a player, wherein game state data is generated during the execution of the game logic. The computer-readable medium includes program instructions for determining a current context in the game play of the video game based on the game state data. The computer-readable medium includes program instructions for determining a user emotion of the player for the current context in the game play of the video game. The computer-readable medium includes program instructions for determining that the user emotion of the player is inconsistent with an expected user emotion for the current context. The computer-readable medium includes program instructions for generating an in-game effect for the current context. The computer-readable medium includes program instructions for presenting the in-game effect simultaneously with one or more video frames associated with the current context in the game play of the video game.

[0009] In yet another embodiment, a computer system is disclosed, wherein the computer system includes a processor and a memory, the memory being coupled to the processor and having instructions stored therein, the instructions, if executed by the computer system, causing the computer system to perform a method. The method includes executing game logic of a video game to generate a plurality of video frames of a game play of the video game performed by a player, wherein game state data is generated during the execution of the game logic. The method includes determining a current context in the game play of the video game based on the game state data. The method includes determining a user emotion of the player for the current context in the game play of the video game. The method includes determining that the user emotion of the player is inconsistent with an expected user emotion for the current context. The method includes generating an in-game effect for the current context. The method includes presenting the in-game effect simultaneously with one or more video frames associated with the current context in the game play of the video game.

[0010] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present disclosure may be best understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1A system according to one embodiment of the present disclosure is illustrated, the system comprising overlay logic supporting a game engine, the overlay logic being used to detect user emotions during gameplay of a video game and to provide an in-game effect that automatically delivers to the player the correct emotion for the current context when the detected user emotion is undesirable for the current context during gameplay of the video game performed by the player.

[0013] Figure 2 An example of overlay logic supporting a game engine according to one embodiment of the present disclosure is provided, wherein the overlay logic is used to detect user emotions during gameplay of a video game and automatically provide in-game effects when the user emotions are undesirable for the current context to make the video game more interesting or more intense.

[0014] Figure 3 is a flow chart illustrating steps in a method according to one embodiment of the present disclosure, the method being configured to support overlay logic of a game engine, the overlay logic being used to detect user emotions during gameplay of a video game and to provide in-game effects that automatically convey to the player the correct emotion for the current context when the detected user emotion is undesirable for the current context during gameplay of the video game by the player.

[0015] FIG. 4A to FIG. 4D is an exemplary illustration of various techniques to convey to a player the correct emotion for the current context during game play of a video game played by the player, according to embodiments of the present disclosure.

[0016] Figure 5 Components of an example device that may be used to perform aspects of various embodiments of the present disclosure are illustrated. DETAILED DESCRIPTION

[0017] Although the following detailed description contains many specific details for illustrative purposes, those skilled in the art will appreciate that many variations and modifications of the following details are within the scope of the present disclosure. Therefore, the aspects of the present disclosure described below are set forth without causing any loss of generality to the appended claims of this specification and without imposing limitations on these claims.

[0018] In general, various embodiments of the present disclosure describe systems and methods for providing overlay logic that supports a game engine, the overlay logic is used to detect user emotions during the game play of a video game, especially for players who may be physically impaired (e.g., hearing impaired, etc.), and when the user emotions are undesirable for the current context of the game play, the overlay logic is configured to provide in-game effects to make the video game more interesting or more intense, so as to more accurately convey to the user what is happening during the game play. For example, the system is configured to identify when there is a problem of correctly conveying what is happening during the corresponding game play of the video game to a player (such as an impaired player (e.g., a hearing impaired player, other types of physical injuries, etc.)). As an example, a hearing-impaired player will miss a specific audio in a video game that the subtitles provided by the video game may not be able to fully compensate for, and the subtitles are intended to convey the full emotion of the current context of the video game. In this way, the system can be configured to change the subtitles or provide in-game effects that change or support the subtitles to correctly convey the correct emotion of the current context of the video game to the player. For example, in-game effects can make subtitles more flashy or unique, to draw players' attention to them, or provide an alternative form of communicating to players what the subtitles are trying to achieve. In this way, impaired players can enjoy a gaming experience similar to that of non-impaired players. In addition, artificial intelligence (AI) techniques can be implemented to predict the user emotions of corresponding players and determine which in-game effects each player prefers.

[0019] Advantages of methods and systems configured to provide overlay logic to support a game engine to help impaired players understand what is happening in a video game include providing support and / or alternative in-game effects to properly convey to the player the expected gaming experience of playing the video game. Thus, video game players who may be impaired to a certain extent and unable to have a full gaming experience may now have the same or similar gaming experience as unimpaired players. In this way, these impaired players who may have otherwise refused to play these video games may now be able to play the video games.

[0020] Throughout this specification, references to "games" or "video games" or "gaming applications" are intended to refer to any type of interactive application that is directed by executing input commands. For purposes of illustration only, interactive applications include applications for gaming, word processing, video processing, video game processing, and the like. Additionally, the terms "virtual world" or "virtual environment" or "metaverse" are intended to refer to any type of environment generated by one or more corresponding applications for interaction between multiple users in a multi-player session or multi-player gaming session. Further, the terms introduced above are interchangeable.

[0021] With the above general understanding of the various embodiments, example details of the embodiments will now be described with reference to the various figures.

[0022] Figure 1 A system 100 according to an embodiment of the present disclosure is illustrated, the system including an overlay logic 120 supporting a game engine 111, wherein the overlay logic is configured to detect user emotions during the game play of a video game and provide an in-game effect that automatically delivers the correct emotion for the current context to the player when the detected user emotion is undesirable for the current context during the game play of the video game performed by the player. The overlay logic 120 can be implemented to provide an alternative approach for delivering the correct emotion or understanding of the current context encountered during the game play to the player, such as by presenting in the scene but not usually generated during the execution of the video game. For example, the in-game effect may include modifying and / or generating subtitles, new animations, transforming subtitles into emoticons or animations, etc. In this way, impaired players can enjoy a game experience and / or understanding of the current context of the video game similar to the game experience and / or understanding of non-impaired players during the game play. In another example, the overlay logic can build and / or implement an artificial intelligence (AI) model that can predict the user emotions of the corresponding player and determine which in-game effects each player prefers. In this way, the application of newly generated and / or modified in-game effects can be personalized for each player through the corresponding AI model. In this way, the impaired player can enjoy a gaming experience and / or understanding of the current context of the video game similar to the gaming experience and / or understanding of the non-impaired player during game play.

[0023] As shown, the system 100 can provide games to one or more client devices 110 and between the one or more client devices via a network 150. Specifically, according to one embodiment of the present disclosure, the system 100 can be configured to provide games to users participating in a single-player or multi-player game session via a cloud gaming network 190, wherein the video game can be executed locally (e.g., on a local client device of the corresponding user) or can be executed remotely from the corresponding client device 110 of the corresponding user who is playing the video game (e.g., acting as a thin client). In at least one capability, the cloud gaming network 190 supports multi-player game sessions for user groups, including delivering and receiving game data of players for coordinating and / or aligning the actions of objects and players within a scene of a game world or metaverse, managing communications between users, etc., so that users participating in a multi-player game session in distributed locations can interact with each other in the game world or metaverse in real time.

[0024] Specifically, the system 100 may provide game control to one or more users playing one or more applications (e.g., video games) in one or more game sessions through a local instance operating on a client device or through a cloud-based instance operating in a cloud gaming network 190 via a network 150. In some embodiments, the cloud gaming network 190 may include multiple virtual machines (VMs) running on a host's hypervisor, wherein one or more virtual machines are configured to execute a game processor module using hardware resources available to the host's hypervisor. It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) typically use cloud computing. Cloud computing is a computing method in which dynamically scalable and usually virtualized resources are provided as a service over the Internet.

[0025] As shown, the cloud gaming network 190 includes a game server 160 that provides access to multiple video games. Video games played in corresponding single-player or multi-player sessions can be played through a network 150 connected to the game server 160. For example, in a multi-player session involving multiple instances of an application (e.g., generating a virtual environment, a game world, a metaverse, etc.), a dedicated server application (session manager) collects data from a user and distributes it to other users so that all instances are updated with respect to objects, characters, etc. to allow real-time interaction within the virtual environment of the multi-player session, where the user can execute a local instance or a cloud-based instance of the corresponding application. The game server 160 can be any type of server computing device available in the cloud, and can be configured as one or more virtual machines executed on one or more hosts. For example, the game server 160 can manage a virtual machine for a game processor that supports a cloud-based instance of an instantiated application for a user. Thus, multiple game processors of the game server 160 associated with multiple virtual machines are configured to execute multiple instances of one or more applications associated with the game play of multiple users. In this manner, the backend server supports streaming of media (e.g., video, audio, etc.) providing gaming entertainment for multiple applications (e.g., video games, gaming applications, etc.) to multiple corresponding users. That is, the game server 160 is configured to stream data (e.g., rendered images and / or frames of the corresponding gaming entertainment) back to the corresponding client devices 110 via the network 150. In this manner, computationally complex gaming applications can be executed at the backend servers in response to controller inputs received and forwarded by the client devices 110. Each server is capable of rendering images and / or frames, which are then encoded (e.g., compressed) and streamed to the corresponding client devices for display.

[0026] The cloud gaming network includes a plurality of user profiles 290A-N. Each user profile includes information related to a corresponding player or user. More specifically, the user profile may include rules for predicting user emotions and / or for defining preferred in-game effects (e.g., for one or more game contexts) of the corresponding player and / or corresponding AI models or parameters for AI models.

[0027] In a single-player or multi-player game session, an instance of the application may be executed locally on the client device 110 or at the cloud gaming network 190. In either case, the application as game logic 115 is executed by a game engine 111 (e.g., a game title processing engine). For the purpose of clarity and brevity, the implementation of game logic 115 and game engine 111 is described in the context of a cloud gaming network 190. Specifically, the application may be executed by a distributed game title processing engine (referred to herein as a "game engine"). Specifically, the game server 160 and / or the game title processing engine 111 include basic processor-based functions for executing applications and services associated with the application. For example, processor-based functions include 2D or 3D rendering, physics (physic), physics simulation, scripts, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadows, culling, transformation, artificial intelligence, etc. In this way, the game engine implements game logic, performs game calculations, physics, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. In addition, the application's services include memory management, multi-threading management, Quality of Service (QoS), bandwidth testing, social networking, social friend management, communication with friends on social networks, social utilities, communication channels, audio communication, texting, messaging, instant messaging, chat support, game entertainment replay function, help function, etc.

[0028] The user accesses the remote service using a client device 110, which includes at least a CPU, a display, and input / output (I / O). For example, the user can access the cloud gaming network 190 via the communication network 150 using a corresponding client device 110 configured to update a session controller (e.g., deliver and / or receive user game state data), receive streaming media, etc. The client device 110 can be a personal computer (PC), a mobile phone, a netbook, a personal digital assistant (PAD), a handheld device, etc.

[0029] In one embodiment, the client device 110 may be configured with a game title processing engine (also referred to as a "game engine") and game logic for at least some local processing of the application, and may be further utilized to receive streaming content generated by the application executed at the backend server, or other content provided by the backend server support. In still other embodiments, for independent local processing, the game title processing engine 111 at the client device 110 includes basic processor-based functions for executing the application and services associated with the application, as previously described. For local processing, the game logic 115 is stored on the local client device 110 and used to execute the application. For example, an instance of the application is executed by the game title processing engine 111 of the corresponding client device 110. The game logic 115 (e.g., executable code) that implements the application is stored on the corresponding client device 110 and used to execute the application. For illustration purposes, the game logic 115 can be delivered to the corresponding client device 110 via a portable medium (e.g., optical media) or via a network (e.g., downloaded from a game provider via the Internet).

[0030] In one embodiment, the client device 110 may be configured as a thin client that provides an interface with a backend server (e.g., a game server 160 of the cloud gaming network 190) configured to provide computing functionality (e.g., including a game title processing engine 111). Specifically, the client device 110 of a corresponding user (not shown) is configured to request access to an application through a communication network 150 such as the Internet, and to render images generated by a video game executed by the game server 160 for display, wherein the encoded images are delivered (i.e., streamed) to the client device 110 for display in association with the corresponding user. For example, a user may interact with an instance of an application associated with the game play of the corresponding user executed on a game processor of the game server 160 through the client device 110, such as through input commands for driving game play. The client device 110 may receive input from various types of input devices (such as a game controller, a tablet computer, a keyboard, gestures captured by a camera, a mouse, a touchpad, an audio input terminal, etc.). More specifically, an instance of an application is executed by the game title processing engine 111 and is configured to generate a rendered image that is delivered over the network 150 for display on a corresponding display associated with the client device 110. That is, the client device 110 is configured to receive an encoded image (e.g., encoded from a game rendered image generated by execution of a video game), and to display the image rendered for display. The game title processing engine 111 is capable of supporting multiple applications using multiple game logics, each of which is selectable by a user.

[0031] In addition, the system 100 includes an overlay logic 120 having such functionality as follows: the functionality is configured to provide an alternative way for conveying the correct emotion or understanding of the current context encountered during the game play to the player, such as by presenting in the scene but not usually generated during the execution of the video game. In this way, the impaired player can enjoy the game experience and / or understanding of the current context of the video game similar to the game experience and / or understanding of the non-impaired player during the game play, such as when supporting the game engine 111 for executing the video game. In another example, the overlay logic can build and / or implement an artificial intelligence (AI) model, which can predict the user emotion of the corresponding player and determine which in-game effects each player prefers. In various embodiments, the application of the newly generated and / or modified in-game effects can be personalized according to each player by rules and / or corresponding AI models. As shown, the overlay logic 120 can be located at the cloud game network 190 or at the client device 110, and can be configured to support the corresponding game engine 111. For example, when the game engine 111 executing the game logic of the video game is located at the client device 110, the overlay logic 120 is also located at the client device 110 and operates to support the game engine. In addition, when the game engine 111 is located at the cloud gaming network 190, the overlay logic 120 is also located at the cloud gaming network 190, thereby operating to support the game engine.

[0032] In some embodiments, the overlay logic 120 includes artificial intelligence (AI), which includes a deep / machine learning engine 130, which is configured to build or train and implement an AI model (e.g., AI model 135A-N) for each user and / or player in a plurality of users and / or players. In one embodiment, the AI ​​learning model is a machine learning model configured to apply machine learning to identify features from input data (e.g., gaze tracking data, game state data, etc.), which features can be classified and / or labeled to learn and / or predict user emotions and determine which in-game effects each player prefers for each type of game context in a plurality of types of game contexts. In another embodiment, the AI ​​learning model is a deep learning model configured to apply deep learning to identify features from input data (e.g., gaze tracking data, game state data, etc.), which features can be classified and / or labeled to learn and / or predict user emotions and determine which in-game effects each player prefers for each type of game context in a plurality of types of game contexts. Further, machine learning is a subclass of artificial intelligence, and deep learning is a subclass of machine learning. Thus, artificial intelligence is used to predict the user emotions of the corresponding players and automatically provide each player's preferred in-game effects when the user emotions are undesirable for the current context in game play to make the video game more interesting or intense.

[0033] For illustration only, according to one embodiment of the present disclosure, the deep / machine learning engine 190 may be configured as a neural network for training and / or implementing the AI ​​model 135. In general, a neural network represents a network of interconnected nodes that respond to inputs (e.g., extracted features) and generate outputs (e.g., learning a user's game usage and / or play behavior and providing recommendations). In one implementation, the AI ​​neural network includes a hierarchical structure of nodes. For example, there may be an input layer of nodes, an output layer of nodes, and an intermediate or hidden layer of nodes. The input nodes are interconnected to the hidden nodes in the hidden layer, and the hidden nodes are interconnected to the output nodes. The interconnections between the nodes may have digital weights, which can be used to link multiple nodes together between inputs and outputs, such as when defining the rules of the AI ​​model 135.

[0034] Figure 2The overlay logic 120 of the support game engine according to an embodiment of the present disclosure is illustrated, and the overlay logic is used to detect the user emotion during the game play of the video game and automatically provide the in-game effect when the user emotion is undesirable for the current context, so as to make the video game more interesting or more intense. As shown in the figure, the overlay logic 120 supports the game name processing engine 111, and the game name processing engine executes the game logic 115 of the corresponding video game, such as to support the game play of the corresponding player associated with the user profile 290A and the AI ​​model 135A. As previously described, the game engine 111 is configured to perform game calculations, physical effects, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services when executing the game logic. For example, the game engine 111 is configured to provide an output 220, and the output includes subtitles 221, video 222, audio 223, physical effects 224, communication 225 (game communication with other players), etc. to some extent, as previously described.

[0035] In addition, the game engine 111 is configured to generate game state data 205, wherein the game state data defines the state of the video game being executed at that time. For example, the game state data may include game characters, game objects, game object properties, game properties, game object states, graphic overlays, in-game effects generated for one or more video frames, etc. In this way, the game state data allows the generation of a game environment (e.g., including in-game effects, etc.) that exists at a corresponding point in the video game. The game state data may also include the state of each device used to render the game entertainment, such as CPU state, GPU state, memory state, register values, program counter values, programmable DMA state, DMA buffer data, audio chip state, CD-ROM state, etc. The game state data may also identify which parts of the executable code need to be loaded to start executing the video game from that point.

[0036] Analyzer 210 communicates with game engine 111. For example, output 220 is delivered to analyzer for further processing. Specifically, game state data 205 is delivered to analyzer 210 for processing by game metric generator 211, context identifier 213 and emotion predictor 215.

[0037] The game metrics generator 211 is configured to process the game state data 205 and provide additional information (e.g., game metrics) that can be used by the overlay logic 120. For example, the game metrics may include the number of lives of a character, the number of attempts to complete a task, the time a player spends within a scene, the progress of gameplay of the video game, and other information.

[0038] In addition, the context identifier 213 is configured to process the game state data 205 and determine a current game context for the corresponding game play of the corresponding video game. For example, the context identifier may compare and analyze the game state data of the current game play with previously collected game state data and previous determinations of game contexts to determine the current game context. That is, the game state data of the corresponding game play may be used and compared with previously collected game state data corresponding to a known game context to determine the current game context for the corresponding game play.

[0039] Further, the emotion predictor 215 is configured to process the game state data 205 and determine a predicted user emotion for the identified current game context. For example, the current game context of a scene of game play of a video game may be a chaotic battle scene, which includes explosions and enemy combatants attacking characters. Therefore, the scene is intended to convey and / or trigger a sense of urgency to the corresponding player. This sense of urgency may define at least a portion of the player's user emotion for the current game context. Other user emotions may be defined for various game contexts. Therefore, the emotion predictor is configured to process the current game context and determine the predicted user emotion that the player should show and / or feel during game play.

[0040] The user profile 290A corresponds to a player playing a video game in the game play. As shown, the user profile 290A may include a corresponding AI model 135A and / or parameters defining the AI ​​model 135A, as previously described. For example, the information in the user profile is delivered to the overlay logic 120 for further processing. In one embodiment, the user profile 290A is used to help identify the user emotion of the player for the current game context that has been previously identified.

[0041] In addition, the gaze detector 295 is configured to collect information related to the player's face and / or body. For example, the gaze detector collects facial information of the player during game play. This facial information can be processed by the overlay logic 120 for processing to determine, to some extent, the user emotion of the player during game play for the current game context that has been previously identified, in conjunction with other information (e.g., user profile information) or independently of the other information.

[0042] As shown and as previously described, the overlay logic 120 supports the game engine 111. Specifically, the game state data 205 is delivered to the overlay logic and / or accessed by the overlay logic (e.g., as represented by path 280) for analysis to detect user emotions for the current context of game play of the video game and automatically provide in-game effects to convey the correct emotions for the current context when the user emotions are undesirable (e.g., the player does not understand what is happening in the video game), such as to make the video game more interesting or more intense. The overlay logic 120 includes a user emotion identifier 230, a comparator 232, a transformer 234, and an in-game effect generator / modifier 235.

[0043] Specifically, the overlay logic 120 may be configured to provide additional functionality in support of the game engine 111. That is, additional elements may be generated by the overlay logic and provided to support and / or in conjunction with video frames generated by the game engine 111 when executing the game logic 115 of the video game. For example, the overlay logic 120 may provide these additional elements as outputs 270, which may include, in part, additional in-game effects 280, additional subtitles 271, video 272, audio 273, physics effects 274, communications 275 (in-game communications with other players), and the like.

[0044] In one implementation, based on one or more pieces of information obtained from the analyzer 210, the user profile 290A, and the gaze detector 295, the user emotion identifier 230 of the overlay logic 120 is configured to identify the user emotion for the current context of the video game during the user's game play based on the rules or implementation of the AI ​​model 135A. For example, the user emotion includes an understanding of what is happening in the game play of the video game, or the general opinions, feelings, or emotions of the player during the game play of the video game. Specifically, the user emotion identifier 230 analyzes the information from the gaze detector 295 to determine the facial features of the player, which are processed to determine the user emotion.

[0045] In addition, the user emotion identifier analyzes and processes the information (e.g., game metrics) from the game metrics generator 211 to determine the user emotion for the current context. Specifically, the game metrics and / or game state data of the current game play are analyzed to determine the progress of the player's game play. For example, the analysis of the game state data and / or game metrics can show that the user plays poorly (i.e., does not advance the game play) in the part of the game that occurs during the current context. In addition, the analysis of the game state data and / or game metrics can show that the character controlled by the player does not act in a predicted manner. For example, the analysis includes one or more actions of the identification character, and determines that the character is inconsistent with the current context. As an example, the character should march through the doorway, or locate the object to be occupied, but the game metrics show that these actions do not occur in the game play. Therefore, the user emotion of the player can show that the user is confused and does not understand what is happening exactly during the game play of the video game. By way of example, in order to progress in a video game, a task must be completed within the current context (obtain an object, pass through a door, etc.); however, the player does not understand this task and wanders around aimlessly without advancing the game.

[0046] In one embodiment, the user emotion identifier 290A may implement the AI ​​model 135A of the user profile 290A for processing one or more pieces of information obtained from the analyzer 210, the user profile 290A, and the gaze detector 295 to identify user emotions. For example, the AI ​​model 135A may be implemented to determine the facial features of the player, and process these facial features to determine the user emotions consistent with the identified facial features. For example, the AI ​​model may collect multiple gazes of the player while playing multiple video games. The multiple gazes are processed to identify facial features, which are labeled and classified for propagation through a deep learning engine to build an AI model of the player, wherein the AI ​​model is configured to identify the user emotions of the player. Specifically, the AI ​​model is configured to detect and / or identify the gaze and / or facial features of the player, and further process the information to identify the user emotions of the player. The AI ​​model 135A may also be implemented to process game metrics from the generator 211 in order to determine user emotions, as previously described. Thus, the AI ​​model 135A is able to process one or more of information from the analyzer, the user profile 290A, and the gaze detector to determine the user's emotion.

[0047] In addition, the comparator 232 of the overlay logic is configured to compare the player's user emotion for the current context (i.e., as determined by the user emotion recognizer 230) with the predicted user emotion from the predictor 215 of the analyzer 210. When the recognized user emotion (i.e., for the current context of the game play) is not aligned with the predicted user emotion, this may indicate that the player does not fully understand what is happening in the video game of the game play. Additional operations may be performed by the overlay logic 120 to correct this misalignment, such as those performed by the in-game effect generator / modifier 235 and / or the transformer 234.

[0048] For example, the in-game effect generator / modifier 235 is configured to modify effects generated for a player's game play during execution of a video game. For example, a hearing-impaired player may turn on a subtitle feature. The player may not fully understand what the subtitles are trying to convey because the game may move too fast, or the subtitles may be too lengthy, or the player may be overwhelmed while trying to read the subtitles and play the video game, etc.

[0049] Therefore, the in-game effector generator / modifier 235 can modify the subtitles to better convey to the player what the set-up-titling is intended to be. For example, the text of the subtitles can be modified, including changing the font, making the text larger, changing the color of the text, modifying the format of the text of the subtitles, highlighting the text or window for the subtitles, etc. In this way, the player's attention can be actively directed to the subtitles.

[0050] In addition, the in-game effector generator / modifier 235 can generate new in-game effects to be displayed together with the video frame of the game associated with the current context. For example, subtitles can be highlighted, as previously described. In addition, subtitles can be moved in the scene displayed in the game, so that the subtitles move back and forth between the vertical planes in the scene, or move left and right or diagonally in the horizontal plane of the scene, or any other intention to bring the player's attention to the movement on the subtitles. In addition, the in-game effector generator / modifier 235 can generate animations that better convey correct user emotions to the user for the current context, including emoticons or animated emoticons, or animations, or graphical user interfaces.

[0051] Further, the converter 234 of the effector generator / modifier 235 in the game can generate new text of subtitles, or generate new subtitles when subtitles were not presented before.For example, generator 234 is configured to generate new text of current subtitles for current game context, such as when user emotion shows that the player is confused or does not get full experience (for example, sense of urgency or high fear, etc.) from playing video games.As an example, new subtitles can be more concise, and replace old subtitles, or are newly presented in the scene of current game context.New subtitles are better to pass the correct user emotion for current game context to the user.In addition, converter 234 can be configured to transform old subtitles, so as to understand what the subtitles are trying to pass to the player.In this way, converter 234 can be able to generate or determine appropriate in-game effects for old subtitles.For example, converter 234 can transform subtitles into emoticons, or animated emoticons, or animations, or graphical user interfaces, or new texts, etc. presented in the scene of game entertainment.

[0052] Through Figure 1 System 100 and Figure 2 A detailed description of the overlay logic 120 is provided. Figure 3 300 is a flowchart 300 illustrating steps in a method according to one embodiment of the present disclosure, the method being configured to support overlay logic of a game engine, the overlay logic being used to detect user emotions during game play of a video game and to provide in-game effects that automatically deliver to the player the correct emotions for the current context when the detected user emotions are undesirable for the current context during game play of the video game by the player. The operations performed in flowchart 300 may be performed by one or more of the previously described components as well as by Figure 1 and Figure 2 300 is implemented by the system 100 including the overlay logic 120 described in . Specifically, the method of flowchart 300 can be implemented so as to provide an alternative way for delivering the correct emotion or understanding of the current context encountered during the game play to the player, such as by in-game effects that are presented in the scene but are not usually generated during the execution of the video game. For example, the in-game effects may include modifying and / or generating subtitles, new animations, transforming subtitles into emoticons or animations, etc. In this way, impaired players can enjoy the game experience and / or understanding of the current context of the video game similar to the game experience and / or understanding of non-impaired players during the game play. For illustration, hearing-impaired players who have difficulty hearing the sound effects generated by the corresponding video game and / or reading subtitles can obtain the same game experience and / or understanding of the current game context similar to the game experience and / or understanding of non-impaired players by modifying and / or generating the corresponding in-game effects.

[0053] At 310, the method includes executing the game logic of the video game to generate a plurality of video frames of the game play of the video game performed by the player. For example, the game engine is configured to execute the game logic of the corresponding video game to support the game play. During the execution of the game logic, game state data is also generated. As previously described, the game state data defines the state of the video game being executed at that time, and to some extent, the game state data allows the generation of a game environment existing at a corresponding point in the video game. For example, the game state data can be analyzed to generate game metrics, and further analyzed to determine the user emotion of the player for the current context of the game play, and to determine the appropriate in-game effects presented in association with the current context in the scene of the game play.

[0054] At 320, the method includes determining a current context in a game play of the video game based on the game state data. For example, the game state data for the current game play may be analyzed and compared with previous game state data for other game plays of the video game (e.g., by multiple players) to determine the current game context. That is, the game state data for the current game context should be consistent with previous game state data for previous game plays and previous determinations of game context for those game plays.

[0055] At 330, the method includes determining the user emotion of the player for the current context in the game play of the video game. For example, as previously described, the user emotion of the player can be determined by analyzing the game state data, game metrics, information from the gaze detector 295, the user profile 290A and / or implementing the AI ​​model 135A. For example, the game metrics of the game play based on the game state data can be analyzed to determine the progress of the game play of the video game performed by the player. The analysis can determine that the progress of the game play is not good (that is, the player does not advance in the video game as expected). Therefore, it can be determined that the user emotion indicates that the player is confused and / or does not fully understand what is happening in the video game. In another example, one or more actions of the character controlled by the player in the game play of the video game are analyzed to determine that the action is inconsistent with the current context. For example, the character does not act as expected, which indicates that the player is confused and / or and the player is confused and does not fully understand what is happening in the video game. For illustration, the player may not understand that in order to advance the game play in the current context, an object must be picked up and / or obtained (for example, the player does not pick up an axe in the scene).

[0056] As previously described, an AI model of a player may be implemented to determine a user emotion for a current game context. Specifically, an AI model is constructed by collecting multiple gazes of a player while playing multiple video games. The gazes are input into a deep learning engine and processed to identify facial features, which are labeled and classified for propagation through the deep learning engine to construct an AI model of the user. The AI ​​model is configured to predict multiple user emotions of the user. For example, during implementation of the AI ​​model, information from a gaze detector is input into the AI ​​model and processed by the AI ​​model to identify the user emotion of the player.

[0057] At 340, the method includes determining that the player's user emotion is inconsistent with the expected user emotion for the current context. As previously described, the predicted user emotion can be determined based on the current game context and the game state data. That is, the previous game play of multiple players can help identify the user emotion corresponding to the specific game context within the video game. In addition, the predicted user emotion can be decoded into the video game by the developer. The predicted user emotion for the current game context is compared with the player's recognized user emotion for the current game context in the player's game play. Therefore, the method can determine when the player's user emotion identified for the current game context is inconsistent with the predicted user emotion. Inconsistency can indicate that the player is confused and / or does not fully understand what is happening in the video game.

[0058] At 350, the method includes generating an in-game effect for the current context. That is, when it is determined that the player's user emotion is inconsistent with the expected user emotion for the current context, additional actions may be taken to correct the inconsistency, including generating an in-game effect, as previously described. The in-game effect is presented simultaneously with one or more video frames associated with the current context in the game play of the video game.

[0059] For illustration purposes, FIG. 4A to FIG. 4D is an exemplary illustration of various techniques for providing in-game effects to convey to a player the correct emotion for the current context during game play of a video game played by the player, in accordance with embodiments of the present disclosure. Figure 4A Screen shot of 400A, Figure 4B Screen shot 400B, Figure 4C Screen shot of 400C and Figure 4DScreenshots 400D of 400D each show a scene 403, which includes a soldier or character 405 walking through a forest area. One or more video frames of scene 403 are intended by the video game to convey a sense of urgency when the corresponding player is playing the video game, such as when engaging in combat, or when experiencing a high sense of danger when it is expected that enemy combatants are just off-screen, etc. Impaired players (e.g., hearing-impaired, or players who are unwilling or unable to process the large amount of information presented in the video game, etc.) may not experience a full sense of urgency in the current game context of the game. That is, the player may feel confused and / or not fully understand what is happening in the video game. In some embodiments, the AI ​​model of the corresponding player is implemented to determine the user emotion of the player during the current game context, and also determine the preferred in-game effects to be displayed in association with the current game context during the player's game play of the video game.

[0060] For example, in Figure 4A In the screenshot 400A of , subtitles 410 can be generated by the video game during the execution of the video game. Subtitles may not be enough to convey the full experience of the current game context to the player. Specifically, subtitles 410 may be troublesome to read, and in addition, may appear dull to the reader. In other words, subtitles 410 may be too direct (e.g., "Currently in Pollock Forest ..."), and therefore will not produce any emotions or at least the correct emotions in the player. In this case, additional in-game effects 420 can be presented to the player in the scene 103. For example, the original effect of the scene is intended to trigger and / or convey the player's sense of urgency, because there may be ordnance or enemy combatants coming, especially from the rear of the character 403. Specifically, the additional text 420A (i.e., "Look behind!") in bold, italic and large fonts is presented to the player in a way that allows the player to quickly notice and understand the severity and seriousness of the current game context. In addition to or independently of the text 420A, additional in-game effects 420B (e.g., emoticons or icons, or animations) may be presented to convey the correct user emotions to the player. In-game effects 420B may be designed to allow the player to understand the situation in the current game context at a glance. For example, large eyes and arrows pointing backwards may convey to the player that danger is approaching and to look back cautiously. Additional in-game effects may include modifying the color of the text of subtitles 410; and modifying the format of the text of subtitles 410; and highlighting subtitles 410; and moving subtitles 410 within a scene displayed in one or more video frames.

[0061] Figure 4BScreen shot 400B of exemplifies an in-game effect that includes generating new text for the subtitles for the current game context. Specifically, new text may be generated that better conveys the correct user emotion for the current context to the player. For example, the new text may simply include the words "Beware!!" in bold and enlarged form. This new text is sufficient to convey to the player a sense of danger and urgency in the current game context. As shown, the new text in subtitles 430 replaces Figure 4A The old text in the subtitles 410 of the game is shown, where the subtitles 430 are now displayed within the scene 403 in one or more video frames associated with the current game context.

[0062] In some embodiments, the in-game effect includes Figure 4A The text of the subtitles 410 shown in the video game is transformed into a newly generated in-game effect, as previously described. The newly generated in-game effect is presented in one or more video frames in the game play of the video game in association with the current game context. For example, the newly generated in-game effect includes an emoticon, an animated emoticon, an animation, a graphical user interface, newly generated text, etc. Generally speaking, the newly generated in-game effect can better convey the correct emotion for the current context to the player during the game play of the corresponding video game. FIG. 4A to FIG. 4B In addition to the newly generated in-game effects presented in Figure 4C and Figure 4D Other examples of newly generated in-game effects are also presented in.

[0063] Specifically, Figure 4C Screenshot 400C is shown, and the screenshot includes one or more examples of animated in-game effects. For example, scene 403 is intended to convey a sense of urgency as a user emotion in the player. To this end, animation 440 is an animation 440 including a moving hand pointing to the right side of the display, which can be repeated. Animation 440 is trying to convey to the player to move character 405 in the direction of the moving hand, or to move the character to the left side of character 405. In addition, animation 450 can be used to convey additional urgency to the player, wherein a running icon moving or running across the bottom of screenshot 400C is shown. One or both of animation 440 and animation 450 can be presented in scene 403 to convey the correct emotion of the current game content in the player's game entertainment to the player.

[0064] also, Figure 4DScreen shot 400D is shown, and the screen shot includes additional in-game effects 470A and 470B. For example, a converter (e.g., converter 234) may be configured to convert the text of subtitles 410. The conversion is intended to determine what subtitles 410 is trying to convey to the player and / or the correct emotion to be conveyed to the player. For example, the video game in scene 403 may be taking place and produce sound effects, which are inaudible to hearing-impaired players. These sound effects may include explosions occurring in the entire background of scene 403, and are intended to convey a sense of danger (enemy combatants or ordnance incoming) and a sense of urgency (e.g., get out of there). However, hearing-impaired players may not fully understand or understand what is happening in scene 403. Therefore, additional in-game effects are provided to convey correct user emotions to the player. For example, in-game effect 470A may be an icon or text ("BOOM!! (boom!!)") that is trying to convey to the player that the left side of character 405 is exploding. In-game effect 470A may include or be represented by an animated explosion or any other effect that indicates that an explosion is occurring in that portion of scene 403. Additionally, in-game effect 470B may be an icon or text ("KA-BOOM!!") that is also attempting to convey to the player that an explosion is occurring to the right of character 405. In this manner, in-game effects 470A and 470B may be displayed in one or more video frames of scene 403 to convey to the player that multiple explosions are occurring in scene 403 over a period of time and that the player should have a sense of urgency or be aware of the dangerous situation in the current game context of the game play of the video game.

[0065] In another embodiment, a video game may include an effect (e.g., subtitle) that is unable to effectively transmit the correct user emotion to the player. For example, as previously described, subtitles as an effect may be dull or too lengthy for the player to understand what is being conveyed. The player playing the video game may provide new effects, such as new and / or revised subtitles, to the back-end server and / or system that serves as a collector of recommendations and / or user comments. After review, the subtitles of the suggestion may be used in the video game (e.g., by a software patch) for subsequent game play performed by one or more different players. Then, the system may provide a reward (e.g., a trophy, or a player identifier that promotes the player to an expert or community helper who assists in transforming the subtitles in the video game or the exclusive content associated with the video game, etc.) to provide the player who provides the suggestion.

[0066] Figure 5Components of an example device 500 that can be used to perform aspects of various embodiments of the present disclosure are illustrated. This block diagram illustrates a device 500 that can be combined with or can be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for practicing embodiments of the present disclosure. The device 500 includes a central processing unit (CPU) 502 for running software applications and optionally an operating system. The CPU 502 can be composed of one or more homogeneous or heterogeneous processing cores. For example, the CPU 502 is one or more general-purpose microprocessors with one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly suitable for highly parallel and computationally intensive applications such as processing operations for interpreting queries, which identify context-related resources and immediately implement and render context-related resources in a video game. The device 500 can be localized to a player playing a game segment (e.g., a game console), or remote from the player (e.g., a back-end server processor), or one of many servers in a game cloud system that uses virtualization to remotely stream game play to a client or to implement additional services such as supervisor functionality.

[0067] Specifically, CPU 502 can be configured to implement the superposition logic 120 with the following functionality: the functionality is configured to detect the user's emotion during the game play of the video game and provide the in-game effect of automatically delivering the correct emotion for the current context to the player when the detected user's emotion is undesirable for the current context during the game play of the video game performed by the player. In some embodiments, the superposition logic is configured to build and / or implement an artificial intelligence model, which is configured to predict the user's emotion of the corresponding player and / or determine which in-game effects each player prefers. In this way, by modifying and / or generating in-game effects, it is possible to receive the expected game experience for the current context and / or the game experience of the impaired player who is impaired in understanding when playing the video game, and the game experience similar to that of the non-impaired player who hears the sound effect can be enjoyed, including delivering what is happening during the corresponding game play of the video game to the impaired player. For example, when the user's emotion is undesirable for the current context in the game play, the in-game effect is configured to make the video game more interesting or more intense.

[0068] Memory 504 stores applications and data for use by CPU 502. Storage 506 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other optical storage devices, as well as signal transmission and storage media. User input device 508 transmits user input from one or more users to device 500, examples of which may include keyboards, mice, joysticks, touch pads, touch screens, static or video recorders / cameras, tracking devices for gesture recognition, and / or microphones. Network interface 514 allows device 500 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communications through local area networks and wide area networks (such as the Internet). Audio processor 512 is adapted to generate analog or digital audio output using instructions and / or data provided by CPU 502, memory 504, and / or storage device 506. The components of device 500 , including CPU 502 , memory 504 , data storage 506 , user input device 508 , network interface 510 , and audio processor 512 , are connected via one or more data buses 522 .

[0069] The graphics subsystem 520 is further connected to the data bus 522 and the components of the device 500. The graphics subsystem 520 includes a graphics processing unit (GPU) 516 and a graphics memory 518. The graphics memory 518 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory 518 may be integrated with the GPU 516 in the same device, connected to the GPU 516 as a separate device, and / or implemented in the memory 504. The pixel data may be provided directly from the CPU 502 to the graphics memory 518. Alternatively, the CPU 502 may provide the GPU 516 with data and / or instructions defining the desired output image, and the GPU 516 generates pixel data for one or more output images according to the data and / or instructions. The data and / or instructions defining the desired output image may be stored in the memory 504 and / or the graphics memory 518. In one embodiment, the GPU 516 includes a 3D rendering capability that generates pixel data for an output image from instructions and data defining the geometry, lighting, shading, texture, motion, and / or camera parameters of a scene. The GPU 516 may further include one or more programmable execution units capable of executing shader programs. In one embodiment, the GPU 516 may be implemented within an AI engine (e.g., machine learning engine 190) to provide additional processing capabilities, such as for AI, machine learning functionality, or deep learning functionality.

[0070] Graphics subsystem 520 periodically outputs pixel data for an image to be displayed on display device 510 from graphics memory 518. Display device 510 may be any device capable of displaying visual information in response to a signal from device 500, including CRT, LCD, plasma, and OLED displays. For example, device 500 may provide analog or digital signals to display device 510.

[0071] In other embodiments, the graphics subsystem 520 includes multiple GPU devices that are combined to perform graphics processing for a single application executed on a corresponding CPU. For example, multiple GPUs may perform alternating forms of frame rendering, where GPU 1 renders the first frame, and GPU 2 renders the second frame, and so on, in sequential frame cycles, until the last GPU is reached, whereupon the initial GPU renders the next video frame (e.g., if there are only two GPUs, GPU 1 renders the third frame). That is, the GPUs rotate when rendering frames. Rendering operations may overlap, where GPU 2 may start rendering the second frame before GPU 1 finishes rendering the first frame. In another implementation, different shader operations in rendering and / or graphics pipelines may be assigned to multiple GPU devices. The master GPU performs master rendering and compositing. For example, in a group including three GPUs, master GPU 1 may perform a primary rendering (e.g., a first shader operation) and synthesize outputs from slave GPU 2 and slave GPU 3, where slave GPU 2 may perform a second shader (e.g., a fluid effect such as a river) operation, and slave GPU 3 may perform a third shader (e.g., a particle smoke) operation, where master GPU 1 synthesizes the results from each of GPU 1, GPU 2, and GPU 3. In this way, different GPUs may be assigned to perform different shader operations (e.g., flag waving, wind, smoke generation, fire, etc.) to render a video frame. In yet another embodiment, each of the three GPUs may be assigned to a different object and / or scene portion corresponding to a video frame. In the above embodiments and implementations, these operations may be performed in the same frame cycle (simultaneous parallelism) or in different frame cycles (sequential parallelism).

[0072] Thus, in various embodiments, the present disclosure describes systems and methods configured to provide overlay logic supporting a game engine to detect user emotions during game play of a video game and provide in-game effects that automatically deliver the correct emotion to the player for the current context when the detected user emotion is undesirable for the current context during game play of the video game by the player. In other words, various embodiments enable the overlay logic to build and / or implement an artificial intelligence model that is configured to predict the user emotions of the corresponding players and / or determine which in-game effects each player prefers.

[0073] It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) typically use cloud computing. Cloud computing is a computing method in which dynamically scalable and usually virtualized resources are provided as services over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing services typically provide common applications (such as video games) accessed from a web browser online, while software and data are stored on servers in the cloud. Based on the way the Internet is depicted in computer network diagrams, the term cloud is used as a metaphor for the Internet and is an abstraction of the complex infrastructure that it hides.

[0074] In some embodiments, a game server can be used to perform the operation of a duration information platform for video game players. Most video games played through the Internet are operated by connecting to a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, video games can be performed by a distributed game engine. In these embodiments, a distributed game engine can be performed on multiple processing entities (PEs) so that each PE performs a functional fragment of a given game engine on which a video game runs. The game engine simply regards each processing entity as a computing node. The game engine typically performs a series of operations with diverse functions to perform video game applications and additional services for user experience. For example, the game engine implements game logic, performs game calculations, physical effects, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communications, game entertainment replay functions, help functions, etc. Although the game engine can sometimes be performed on an operating system virtualized by a hypervisor of a specific server, in other embodiments, the game engine itself is distributed among multiple processing entities, and each of the multiple processing entities can reside on different server units in a data center.

[0075] According to this embodiment, the corresponding processing entity used to perform the operation can be a server unit, a virtual machine, or a container, depending on the needs of each game engine fragment. For example, if a game engine fragment is responsible for camera transformations, a virtual machine associated with a graphics processing unit (GPU) can be provided to this particular game engine fragment because it will perform a large number of relatively simple mathematical operations (e.g., matrix transformations). Other game engine fragments that require fewer but more complex operations can be provided with processing entities associated with one or more higher-powered central processing units (CPUs).

[0076] By distributing the game engine, the game engine is provided with elastic computing properties that are not constrained by the capabilities of the physical server units. Instead, more or fewer computing nodes are provided to the game engine as needed to meet the needs of the video game. From the perspective of the video game and the video game player, a game engine distributed across multiple computing nodes is no different than a non-distributed game engine executed on a single processing entity, because the game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output components to the end user.

[0077] The user uses a client device to access the remote service, and the client device includes at least a CPU, a display, and an I / O. The client device can be a PC, a mobile phone, a notebook computer, a PDA, etc. In one embodiment, the network identification client used on the game server performs and adjusts the communication method adopted. In other cases, the client device uses a standard communication method such as html to access the application on the game server through the Internet. It should be understood that a given video game or game application can be developed for a specific platform and a specific associated controller device. However, when such a game is made available via a game cloud system as presented herein, the user may access the video game with different controller devices. For example, a game may have been developed for a game console and its associated controller, and the user may access the cloud-based version of the game from a personal computer using a keyboard and mouse. In such scenarios, the input parameter configuration can define the mapping of the input generated from the available controller device (in this case, keyboard and mouse) of the user to the execution of the video game that is acceptable.

[0078] In another example, a user may access the cloud gaming system via a tablet computing device, a touch screen smart phone, or other touch screen driven device. In this case, the client device and the controller device are integrated together in the same device, where input is provided by means of a detected touch screen input / gesture. For such a device, an input parameter configuration may define a specific touch screen input corresponding to a game input of a video game. For example, during the running of a video game, a button, a directional key, or other type of input element may be displayed or overlaid to indicate a position on the touch screen where the user may touch to generate a game input. Gestures (such as swipes or specific touch movements in a particular direction) may also be detected as game inputs. In one embodiment, guidance may be provided to the user, indicating, for example, how to provide input for game play via a touch screen before starting the game play of a video game, so as to adapt the user to the operation of the controls on the touch screen.

[0079] In some embodiments, the client device is used as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device can then process these inputs and then transmit the input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller itself can be a networked device with the ability to directly communicate input to the cloud game server via a network without first communicating such input through a client device. For example, the controller can be connected to a local networking device (such as the above-mentioned router) to send data to and receive data from the cloud game server. Therefore, although the client device may still be required to receive video output from a cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send input directly to the game cloud server over the network, thereby bypassing the client device.

[0080] In one embodiment, the networked controller and client device may be configured to send certain types of input directly from the controller to the cloud gaming server, and to send other types of input via the client device. For example, inputs whose detection does not depend on any additional hardware or processing other than the controller itself may be sent directly from the controller to the gaming cloud server via the network, thereby bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or need to be processed by the client device may be sent by the client device to the cloud gaming server. These may include video or audio captured from the gaming environment, which may be processed by the client device before being sent to the cloud gaming server. In addition, inputs from the motion detection hardware of the controller may be processed by the client device in conjunction with the captured video to detect the position and motion of the controller, which is then communicated to the cloud gaming server by the client device. It should be understood that the controller device according to various embodiments may also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0081] The access of the client device to the cloud gaming network can be achieved through a communication network that implements one or more communication technologies. In some embodiments, the network may include a 5th generation (5G) network technology with an advanced wireless communication system. 5G is the fifth generation of cellular network technology. The 5G network is a digital cellular network in which the service area covered by the provider is divided into small geographical areas called cells. The analog signals representing sound and images are digitized in the phone, converted by an analog-to-digital converter and transmitted as a bit stream. All 5G wireless devices in the cell communicate with the local antenna array and low-power automatic transceiver (transmitter and receiver) in the cell through radio waves on the frequency channel allocated by the transceiver from the frequency pool reused in other cells. The local antenna is connected to the telephone network and the Internet through a high-bandwidth optical fiber or wireless backhaul connection. As in other cell networks, a mobile device that crosses from one cell to another automatically transfers to the new cell. It should be understood that the 5G network is only an exemplary type of communication network, and the embodiments of the present disclosure can utilize earlier generations of wireless or wired communications, as well as later generations of wired or wireless technologies that appear after 5G.

[0082] In one embodiment, various technical examples can be implemented using a virtual environment through a head-mounted display (HMD). The HMD may also be referred to as a virtual reality (VR) head-mounted device. As used herein, the term "virtual reality" (VR) generally refers to the interaction of a user with a virtual space / environment, which involves viewing the virtual space in a real-time response to the movement of the HMD (controlled by the user) through the HMD (or VR head-mounted device) to provide the user with a feeling of being in a virtual space or metaverse. For example, a user can see a three-dimensional (3D) view of the virtual space when facing a given direction, and when the user turns to one side and thus turns the HMD in the same manner, the view of the side in the virtual space is then rendered on the HMD. The HMD can be worn in a manner similar to glasses, goggles, or a helmet, and is configured to display video games or other metaverse content to the user. The HMD can provide a very immersive experience to the user by means of its display mechanism being set close to the user's eyes. Therefore, the HMD can provide a display area that occupies most of the user's or even the entire field of view to each eye of the user, and can also provide viewing with three-dimensional depth and perspective.

[0083] In one embodiment, the HMD may include a gaze tracking camera and / or system configured to capture images of the user's eyes as the user interacts with the VR scene. The gaze information captured by the gaze tracking camera may include information related to the user's gaze direction and specific virtual objects and content items in the VR scene that the user is paying attention to or interested in interacting with. Thus, based on the user's gaze direction, the system can detect specific virtual objects and content items that may be of potential focus to the user (where the user is interested in interacting with and engaging with them), such as game characters, game objects, game props, etc. In addition, the gaze tracking system can be configured to capture the user's facial features for analysis purposes to determine the user's emotions while playing a video game.

[0084] In some embodiments, the HMD may include an outward-facing camera configured to capture images of the user's real-world space, such as the user's body movements and any real-world objects that may be located in the real-world space. In some embodiments, the images captured by the external camera may be analyzed to determine the position / orientation of the real-world objects relative to the HMD. Using the known position / orientation of the HMD, real-world objects, and inertial sensor data from the user's posture and movement, the user's posture and movement during the user interaction with the VR scene can be continuously monitored and tracked. For example, when interacting with a scene in a game, a user may make various gestures, such as pointing and walking toward a specific content item in the scene. In one embodiment, the gestures may be tracked and processed by the system to generate predictions of interactions with specific content items in the game scene. In some embodiments, machine learning may be used to facilitate or assist predictions.

[0085] During the use of the HMD, various single-handed and two-handed controllers can be used. In some implementations, the controller itself can be tracked by tracking the lights included in the controller or tracking the shapes, sensors, and inertial data associated with the controller. Using these different types of controllers or even just using gestures made and captured by one or more cameras, the virtual reality environment or metaverse rendered on the HMD can be docked, controlled, manipulated, interacted with, and participated in the virtual reality environment or metaverse. In some cases, the HMD can be wirelessly connected to the cloud computing and gaming system through a network. In one embodiment, the cloud computing and gaming system maintain and execute the video game played by the user. In some embodiments, the cloud computing and gaming system is configured to receive input from the HMD and interface objects through the network. The cloud computing and gaming system is configured to process input to affect the game state of the video game being executed. Output (such as video data, audio data, and tactile feedback data) from the video game being executed is transmitted to the HMD and the interface object. In other implementations, the HMD can communicate wirelessly with the cloud computing and gaming system through an alternative mechanism or channel such as a cellular network.

[0086] In addition, although the implementations in the present disclosure may be described with reference to a head-mounted display, it should be understood that in other implementations, a non-head-mounted display may be substituted, including but not limited to a portable device screen (e.g., a tablet, a smart phone, a laptop, etc.) or any other type of display that can be configured to render video and / or provide a display of an interactive scene or virtual environment according to the implementation of the present invention. It should be understood that the various embodiments defined herein may be combined or assembled into a specific implementation using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that can define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0087] Embodiments of the present disclosure may be practiced with various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through wired or wireless networks.

[0088] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed in a system that allows processing operations to occur at various intervals associated with the processing as long as the processing of telemetry and game state data used to generate the modified game state is performed in the desired manner.

[0089] In view of the above embodiments, it should be understood that the embodiments of the present disclosure may employ various computer-implemented operations involving data stored in a computer system. These operations are those operations that require physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. The embodiments of the present disclosure also relate to devices or equipment for performing these operations. The device may be specially constructed for the desired purpose, or the device may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used together with computer programs written according to the teachings of this article, or it may be more convenient to construct a more specialized device to perform the desired operations.

[0090] One or more embodiments may also be manufactured as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. A computer readable medium may include a computer readable tangible medium distributed on a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0091] In one embodiment, the video game is executed locally on a game console, a personal computer, or on a server. In some cases, the video game is executed by one or more servers in a data center. When executing the video game, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. In some embodiments, the simulation is an instance of the video game. In other embodiments, the simulation may be produced by an emulator. In either case, if the video game is represented as a simulation, the simulation may be executed to render interactive content that may be interactively streamed, executed, and / or controlled by a user input.

[0092] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it is apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments are to be considered illustrative rather than restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method comprising: executing game logic of a video game to generate a plurality of video frames of game play of the video game by a player, wherein game state data is generated during said executing the game logic; determining a current context in the game play of the video game based on the game state data; determining a user sentiment of the player with respect to the current context in the game play of the video game; determining that the user emotion of the player is inconsistent with an expected user emotion for the current context; generating an in-game effect for the current context; as well as The in-game effect is presented simultaneously with one or more video frames associated with the current context in the game play of the video game.

2. The method of claim 1, wherein determining the user emotion comprises: analyzing a plurality of game metrics of the game play based on the game state data; determining a progress of said game play of said video game; as well as It is determined that said progress is poor.

3. The method of claim 1, wherein determining the user emotion comprises: identifying one or more actions of a character controlled by the player in the game play of the video game; as well as It is determined that the one or more actions are inconsistent with the current context.

4. The method of claim 1, further comprising: collecting a plurality of gazes of the player while playing a plurality of video games; as well as The plurality of gazes are processed to identify facial features, the facial features are labeled and classified for propagation through a deep learning engine to build an AI model of the user, the AI ​​model being configured to predict a plurality of user emotions of the user.

5. The method of claim 4, wherein the determining the user emotion comprises: detecting a gaze of the player; as well as The AI ​​model is executed to identify the user emotion of the player.

6. The method of claim 1, wherein said generating said in-game effect comprises at least one of the following: Change the color of the subtitle text; and modifying the format of the text of the subtitles; and highlighting the subtitle; and The subtitles are caused to move within a scene displayed in the one or more video frames.

7. The method of claim 1, wherein generating the in-game effect comprises: Generating new text for subtitles for the current game context; replacing old text of the subtitle with the new text of the subtitle; as well as The subtitles having the new text are displayed within the scene displayed in the one or more video frames.

8. The method of claim 1, further comprising: Transforming text of subtitles displayed in the one or more video frames into the in-game effect; as well as The in-game effect is presented in the one or more video frames.

9. The method of claim 8, wherein the in-game effect comprises at least one of: Emoticons; and animation; and Graphical user interface; and text.

10. A non-transitory computer-readable medium storing a computer program for executing a method, the computer-readable medium comprising: program instructions for executing game logic of a video game to generate a plurality of video frames of game play of said video game by a player, wherein game state data is generated during said executing said game logic; program instructions for determining a current context in the game play of the video game based on the game state data; program instructions for determining a user sentiment of the player with respect to the current context in the game play of the video game; program instructions for determining that the user emotion of the player is inconsistent with an expected user emotion for the current context; program instructions for generating an in-game effect for said current context; as well as Program instructions for presenting the in-game effect concurrently with one or more video frames associated with the current context in the game play of the video game.

11. The non-transitory computer readable medium of claim 10, wherein the program instructions for determining the user emotion include: program instructions for analyzing a plurality of game metrics of said game play based on said game state data; program instructions for determining progress in said game play of said video game; as well as Program instructions for determining said poor progress.

12. The non-transitory computer-readable medium of claim 10, wherein the program instructions for determining the user emotion include: program instructions for collecting a plurality of gazes of the player while playing a plurality of video games; program instructions for processing the plurality of gazes to identify facial features, the facial features being labeled and classified for propagation through a deep learning engine to build an AI model of the user, the AI ​​model being configured to predict a plurality of user emotions of the user; program instructions for detecting a gaze of the player; as well as Program instructions for executing the AI ​​model to identify the user emotion of the player.

13. The non-transitory computer-readable medium of claim 10, wherein the program instructions for generating the in-game effect include at least one of: Program instructions for modifying the color of the text of the subtitles; as well as program instructions for modifying the format of said text of said subtitles; as well as program instructions for highlighting said subtitles; as well as Program instructions for causing the subtitles to be within a scene displayed in the one or more video frames.

14. The non-transitory computer readable medium of claim 10, further comprising: Program instructions for transforming text of subtitles displayed in the one or more video frames into the in-game effects; as well as Program instructions for presenting the in-game effect in the one or more video frames.

15. The non-transitory computer-readable medium of claim 14, wherein the in-game effect in the program instructions comprises at least one of: Emoticons; and animation; and Graphical user interface; and text.

16. A computer system comprising: processor; a memory coupled to the processor and having instructions stored therein, the instructions, if executed by the computer system, causing the computer system to perform a method comprising: executing game logic of a video game to generate a plurality of video frames of game play of the video game by a player, wherein game state data is generated during said executing the game logic; determining a current context in the game play of the video game based on the game state data; determining a user sentiment of the player with respect to the current context in the game play of the video game; determining that the user emotion of the player is inconsistent with an expected user emotion for the current context; generating an in-game effect for the current context; and The in-game effect is presented simultaneously with one or more video frames associated with the current context in the game play of the video game.

17. The computer system of claim 16, wherein in the method, determining the user emotion comprises: analyzing a plurality of game metrics of the game play based on the game state data; determining a progress of said game play of said video game; as well as It is determined that said progress is poor.

18. The computer system of claim 16, wherein in the method, determining the user emotion comprises: collecting a plurality of gazes of the player while playing a plurality of video games; processing the plurality of gazes to identify facial features, the facial features being labeled and classified for propagation through a deep learning engine to build an AI model of the user, the AI ​​model being configured to predict a plurality of user emotions of the user; detecting a gaze of the player; as well as The AI ​​model is executed to identify the user emotion of the player.

19. The computer system of claim 16, wherein in the method, the generating the in-game effect comprises at least one of: Change the color of the subtitle text; as well as Modifying the format of the text of the subtitle; as well as highlighting the subtitle; and The subtitles are caused to move within a scene displayed in the one or more video frames.

20. The computer system of claim 1, the method further comprising: Transforming text of subtitles displayed in the one or more video frames into the in-game effect; as well as The in-game effect is presented in the one or more video frames. The in-game effect includes at least one of the following: Emoticons; and animation; and Graphical user interface; and text.

Images