Method of checking game scenario to determine voice command of user

Abstract

There is provided a method for executing a session of a video game, comprising the operations of: recording voice of a player participating in game progress of the session of the video game; analyzing a game state generated by execution of the session of the video game, wherein analyzing the game state identifies a scenario in which the game is performed; analyzing the recorded speech using the recognized scenario of the game and the speech recognition model to recognize text content of the recorded speech; the identified textual content is applied as a game input of a session of the video game.

Description

Technical Field

[0001] The present disclosure generally relates to examining game context to determine voice commands of a user in a video game.

[0002] background

[0003] 2. Description of related technologies

[0004] The video game industry has gone through many changes over the years. As technology advances, video games continue to enable greater immersion through sophisticated graphics, realistic sounds, engaging soundtracks, haptics, and more. Players are able to enjoy immersive gaming experiences where they engage and become part of the virtual environment, and seek out new ways to interact.

[0005] It is in this context that the implementation of the present disclosure is proposed. Summary of the invention

[0006] Implementations of the present disclosure include methods, systems, and apparatuses directed to examining game context to determine voice commands of a user in a video game.

[0007] In some implementations, a method for executing a session of a video game is provided, comprising the following operations: recording speech of players participating in a game play of the session of the video game; analyzing a game state generated by the execution of the session of the video game, wherein analyzing the game state identifies a context of the game play; analyzing the recorded speech using the identified context of the game play and a speech recognition model to identify text content of the recorded speech; and applying the identified text content as game play input for the session of the video game.

[0008] In some implementations, the method is performed in substantially real time, such that analyzing the game state is performed in substantially real time with the game state being continuously updated through the execution of the session, and such that the analysis to identify the textual content of the recorded speech is responsive to changes in the game state in substantially real time.

[0009] In some implementations, the identified context of game play includes an identified scene or stage of a video game.

[0010] In some implementations, the identified context of game play includes an identified virtual location in a virtual environment defined by execution of the session of the video game.

[0011] In some implementations, the identified context of the game play includes identified attributes of a player avatar in the video game.

[0012] In some implementations, analyzing the game state includes identifying activities occurring during the game play, the identified activities at least partially defining the identified context of the game play.

[0013] In some implementations, analyzing the game state includes predicting future game play activity, the predicted future game play at least partially defining the identified context of the game play.

[0014] In some implementations, a method for conducting a session of a video game is provided, the method comprising: recording speech of a player participating in a game play of the session of the video game; analyzing the recorded speech using a speech recognition model, the speech recognition model identifying a plurality of candidate words as possible interpretations of the player's speech; presenting the plurality of candidate words to the player during the session of the video game; receiving selection input from the player, identifying one of the candidate words as a correct interpretation of the player's speech; and applying, by the video game, the selected one of the candidate words as a gameplay input for the video game.

[0015] In some implementations, presenting the plurality of candidate words is in response to a recognition confidence of the speech recognition model falling below a predefined threshold.

[0016] In some implementations, presenting the candidate word to the player includes presenting the candidate word in a video generated from the session of the video game.

[0017] In some implementations, presenting the candidate word pauses the game play until the selection input has been received.

[0018] In some implementations, applying the selected one of the candidate words includes triggering a command for the game play of the video game.

[0019] In some implementations, the selected one of the candidate words is used as feedback to improve the speech recognition model.

[0020] In some implementations, receiving the selection input is through an input device of the controller.

[0021] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present disclosure may be better understood with reference to the following description in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A user is shown engaging in game play of a video game according to implementations of the present disclosure.

[0024] Figure 2 The embodiment conceptually illustrates the dynamic adjustment of the speech recognition model based on the changed game scenario according to the implementation of the present disclosure.

[0025] Figure 3 A player providing feedback regarding speech recognition applied to his or her voice during game play of a video game is conceptually illustrated in accordance with an implementation of the present disclosure.

[0026] Figure 4 Game context prediction for improving speech recognition during game play of a video game according to implementations of the present disclosure is conceptually illustrated.

[0027] Figure 5 A method of applying speech recognition during video gaming according to an implementation of the present disclosure is conceptually illustrated.

[0028] Figure 6 Components of an example device that may be used to perform aspects of various embodiments of the present disclosure are shown. DETAILED DESCRIPTION

[0029] The following implementations of the present disclosure provide methods, systems, and apparatus for examining game context to determine voice commands of a user in a video game.

[0030] Broadly speaking, the implementation of the present disclosure relates to a method and system for providing situational automatic speech recognition in a video game. When a player plays a given video game, the scenario of the game played by the player and the virtual environment of the video game is constantly changing. However, the scenario of the game played can provide important clues, indicating that the player may say what at any given moment. Therefore, according to the implementation of the present disclosure, the game playing scenario is used to enhance the player's speech recognition. For example, various aspects of the game playing such as scenes or settings, the characteristics of the player's avatar, the actions of the player's avatar, etc. are all used to improve speech recognition. In some implementations, speech is used to provide commands for the game playing, so it is important to realize highly accurate speech recognition for providing high-quality game playing experience for the player. In some implementations, if the system cannot determine the correct interpretation of the given voice of the player with sufficient confidence, candidate words or phrases can be presented to the player for selection, and thus indicate which is the correct interpretation of the user's voice.

[0031] With the above overview in mind, several example figures are provided below to facilitate an understanding of the exemplary embodiments.

[0032] Figure 1 A user is shown engaging in game play of a video game according to implementations of the present disclosure.

[0033] In the illustrated implementation, a user / player 100 participates in interactive game play of a video game executed by a computing device 102. By way of example (but not limitation), the computing device 102 may be a game console, a personal computer, a laptop, a set-top box, or other general or special purpose computer having a processor and memory and capable of executing program instructions of a video game. Furthermore, while user interaction with a video game is described for purposes of illustrating embodiments of the present disclosure, it should be understood that the principles of the present disclosure may be applied to other types of interactive applications that may be executed by a computing device 102 and with which a user 100 may interact. Furthermore, it should be understood that in other implementations, at least some of the functionality attributed to the computing device 102 may be cloud-executed by a remote cloud processing resource 132 accessed via a network 130 (e.g., including the Internet).

[0034] In some implementations, computing device 102 executes a session 120 of a video game, which includes generating a game state 122 and rendering video and audio of a virtual environment of the video game. The video is presented on a display 104 (e.g., a television, monitor, LED / LCD display, projector screen, etc.) for viewing by user 100. In some implementations, display 104 takes the form of a head-mounted display (HMD) worn by a user. When user 100 engages in interaction with the video game, user 100 can operate a user input device (such as controller 106 in the illustrated implementation) that generates user input for the video game. It should be understood that the executing video game responds to the user input generated due to the user's interaction, and continuously updates its game state based on the user input to drive the execution of the video game.

[0035] The audio generated by the video game may be presented through an audio device such as a speaker (which may be included in the display 104 or, as shown in the illustrated implementation, a headset 108 worn by the user 100). The headset 108 may be wired or wireless and may be connected directly or indirectly to the computing device 102. For example, in some implementations, the headset 108 is connected to the controller 106, and the controller 106 communicates with the computing device 102 to receive the audio, which is then passed to the headset 108 for presentation.

[0036] Broadly speaking, implementations of the present disclosure relate to methods and systems for providing context-based speech recognition in video games. Thus, the speech of the user 100 during game play is captured by a microphone 110. In the illustrated implementation, the microphone 110 is connected to a headset 108. However, in other implementations, the microphone may be a separate device, incorporated into the controller 106, as part of another device for sensing the local environment (e.g., a device having a camera and a microphone for detecting the user and the local environment), as part of an HMD worn by the user, or otherwise provided in a manner that enables the capture of the user's speech during game play.

[0037] The voice of user 100 is captured and recorded as audio data. The recorded audio data is analyzed using speech recognition model 124 to identify the text content of the user's voice. More specifically, in order to improve speech recognition, the current scenario of the user's game is identified and applied to the speech recognition model. It is understandable that in a given game scenario, the user is more likely to say certain words or phrases compared to another game scenario. For example, in a combat scenario, the user is more likely to say combat-specific words or phrases, such as issuing commands for shooting weapons or launching attacks on enemies. Therefore, by understanding the game scenario, the automatic speech recognition of the user's voice can be improved.

[0038] In some implementations, the game state 122 is analyzed to identify the context of the user's game play. It is understandable that the game play is dynamic, and therefore the context may change at any time. Therefore, in some implementations, in order to optimize speech recognition, a real-time or substantially real-time understanding of the game context is applied. In this way, the ability of the speech recognition model 124 to determine the text content of the user's (recorded) speech responds to the real-time changes that occur during the game play, which is reflected in the game state, which is constantly updated during the interactive game play.

[0039] In some implementations, the game state 122 is analyzed using a trained machine learning model configured to recognize or identify game play contexts. In some implementations, such a machine learning model may be configured to predict future game play activities / contexts based on the current game state, and this information may also be used to enhance automatic speech recognition. Broadly speaking, the speech recognition model 124 is adjusted based on the determined game play context, for example, to bias or weight text determinations that are more likely based on the game play context.

[0040] Once the textual content (words or phrases) of the user's speech is determined, this can be applied to the video game session 120. For example, the textual content can be interpreted to trigger one or more commands applied to the game play. In this way, improved speech recognition is achieved to better recognize voice commands used during game play of the video game.

[0041] Although in the illustrated implementation, as shown, the video game and speech recognition are performed by the computing device 102, in other implementations, either or both of the video game and speech recognition may be remotely performed / accomplished by a cloud processing resource 132 accessed via the network 130. In some implementations, recorded audio of the user's speech may be transmitted to the cloud processing resource 132 for analysis using the speech recognition model and the game context to determine the textual content of the user's speech.

[0042] Figure 2 The embodiment conceptually illustrates the dynamic adjustment of the speech recognition model based on the changed game scenario according to the implementation of the present disclosure.

[0043] It will be appreciated that the scenario of the video game's gameplay changes as the user progresses during a session. For example, in the illustrated implementation, at an early point in time, the player may be engaged in gameplay in a first scene 200 of the video game, where the player may control an avatar 202. While at a later point in time in the session, the player may be engaged in gameplay in a second scene 204 of the video game, where the player may control an airplane 206, as examples (but not limitations). It will be appreciated that the types of speech that the player may speak during these scenes may differ due to the nature of the scenes and the requirements for the gameplay activities (e.g., controlling an avatar versus controlling an airplane / vehicle).

[0044] Therefore, a dynamic automatic speech recognition (ASR) adjustment process 208 is performed, which adjusts the speech recognition model 210 based on the game scenario that occurred at that time. In some implementations, the dynamic ASR process includes analyzing the game state of the video game to determine the current game play scenario. In some implementations, the game state is analyzed using a machine learning model that is trained to identify or recognize the scenario of the game play of the video game. For example, (but not limited to), the game scenario may include identifying various aspects of the type of game play that describes the game play that is taking place, such as settings, scenes, stages, parts, virtual / spatial positions in the virtual environment, progress, timelines / time positions in the campaign storyline, current goals, role / avatar attributes, abilities, inventories, achievements, skill levels, etc., or other types of information about the game play that may affect the likelihood of the player saying a specific word or phrase during the game play.

[0045] For example, during scenario 200, the player controls the avatar 202 and is therefore more likely to utter words / phrases related to controlling the avatar 202. While in scenario 204, the player controls the airplane 206 and is therefore more likely to utter words / phrases related to controlling the airplane 206. Thus, during each of scenarios 200 and 204, the dynamic ASR adjustment process 208 adjusts the speech recognition model 210 in a different manner to favor relevant and more likely spoken words, such as by adjusting / modulating weights or variables of the speech recognition model 210.

[0046] In various implementations, the speech recognition model can be based on various recognition models known in the art, and can include acoustic and language models. In some implementations, the speech recognition model is based on a hidden Markov model, an artificial neural network, a deep neural network, an AI / machine learning model, or other types of suitable modeling mechanisms.

[0047] Figure 3 Conceptually illustrated is an implementation of a player providing feedback regarding speech recognition applied to their voice during gameplay of a video game in accordance with the present disclosure.

[0048] In the illustrated implementation, the player 100 is participating in a game play of a video game, wherein a video of the game play is rendered on the display 104. As already discussed, automatic speech recognition is applied to the voice of the player 100. Also as described above, the game play context may be determined and used to improve the results of the speech recognition process. However, in some cases, the voice of the player 100 may not be recognized with sufficient confidence.

[0049] Therefore, in some implementations, a method for requesting additional feedback from a player is implemented. For example, at operation 300, a threshold determination is performed on the confidence level of speech recognition. If the speech recognition model recognizes the player's voice, and its confidence level reaches or exceeds a predefined threshold, the recognized text content of the player's voice will be applied in the manner discussed above. However, if the confidence level is lower than the predefined threshold, at method operation 302, the system identifies potential candidate words / phrases that may interpret the player's recorded voice. These candidates are presented to the player 100, such as by presenting a dialog window 308 on the display 104, asking the player 100 whether to say one of the candidate words / phrases.

[0050] At operation 304, the player 100 may respond by indicating a selection of one of the candidates as the correct interpretation. For example, the dialogue window 308 may indicate that a particular button on the controller 106 operated by the player 100 is associated with the presented candidate word / phrase, such that the player may indicate a selection of a given candidate word / phrase by pressing the corresponding controller button.

[0051] At operation 306, the word / phrase indicated as the correct interpretation of the player's speech is applied to the video game, such as by triggering a command for the game. In addition, information about the correct interpretation of the player's speech can be used as feedback or further training data for the speech recognition model to improve the accuracy of the speech recognition model in determining the text content of the player's speech.

[0052] Figure 4 Game context prediction for improving speech recognition during game play of a video game according to implementations of the present disclosure is conceptually illustrated.

[0053] In the illustrated implementation, a player 400 is experiencing a video game through a VR headset (or head mounted display) 402. The player can operate controllers 404a and 404b, and can also issue voice commands or other speech related to the video game. The player 400 views a virtual environment 410, which can be seen from the perspective of the player's avatar 412 in the virtual environment. It can be appreciated that the VR headset 402 provides the player 400 with an immersive view of the virtual environment 410.

[0054] In some implementations, in addition to determining game play scenarios to improve speech recognition, future game play activities and scenarios can also be predicted based on current game state data, and the future game play activities / scenarios can also be used to improve speech recognition. For example, the player 400 may be looking in a given direction 414 in the virtual environment 410, or moving in a direction 414. Based on such viewing direction or movement, the system (e.g., including the machine learning model for analyzing the game state described above) can predict that the player's future game scenario will include a virtual location within an area 416 of the virtual environment 410. This data can be used to adjust the speech recognition model before the player reaches the area 416, for example, to favor words or phrases that may be spoken when in the area 416.

[0055] While predicted future positions within a virtual environment are described, it will be appreciated that other types of game play contextual information (such as the examples described above) may be predicted and used to enhance the speech recognition process for a video game player during game play.

[0056] Figure 5 A method of applying speech recognition during video gaming according to an implementation of the present disclosure is conceptually illustrated.

[0057] At method operation 500, execution of a session of a video game is initiated. At method operation 502, player speech occurring during game play of the video game is recorded.

[0058] At method operation 504, the current context of the game play in which the player speech occurs is determined, such as by analyzing the game state of the session being executed. At method operation 506, the determined context of the game play is applied to the speech recognition process, and speech recognition incorporating the context of the game play is performed on the recorded speech to determine the textual content of the player speech.

[0059] At method operation 508, a confidence level of the speech recognition result is determined. If the confidence level exceeds a predefined threshold, indicating that the speech recognition result has a sufficiently high confidence level, then at method operation 510, the textual content of the player's speech determined by the speech recognition is applied to the game play of the video game, such as by triggering a command or some other action in the video game session.

[0060] If the confidence level does not exceed the predefined threshold, then at method operation 512, the candidate words / phrases determined by the speech recognition process are presented to the player as possible interpretations of the player's speech. At method operation 514, player input is received and one of the candidates is identified as the correct interpretation. Once the correct interpretation is identified, the relevant text content is applied to the video game at method operation 510, as described above. In addition, at method operation 516, the identification of the correct interpretation of the player's speech is used to improve speech recognition, such as by updating / training / adjusting the speech recognition model.

[0061] In some implementations described herein, analysis of game scenarios and speech recognition are performed by separate models. However, in other implementations, a single speech recognition model is configured to evaluate game scenarios together and perform speech recognition. That is, in some implementations, the speech recognition model can be configured to accept game state data (indicating game scenarios) and player's recorded voice, and perform situational speech recognition.

[0062] In other implementations, there may be multiple speech recognition models specific to different game scenarios. For example, a first speech recognition model may be trained and configured to recognize speech occurring during a first stage / scene of a video game, and a second speech recognition model may be trained and configured to recognize speech occurring during a second stage / scene of the video game, and so on.

[0063] Figure 6Components of an example device 600 that can be used to perform various aspects of various embodiments of the present disclosure are shown. This block diagram shows a device 600 that can be combined with or can be suitable for practicing a personal computer, video game console, personal digital assistant, server, or other digital device of an embodiment of the present disclosure. Device 600 includes a central processing unit (CPU) 602 for running software applications and optionally an operating system. CPU 602 can be composed of one or more isomorphic or heterogeneous processing cores. For example, CPU 602 is one or more general-purpose microprocessors with one or more processing cores. Other embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly suitable for highly parallel and computationally intensive applications, such as processing interpretation queries, identifying context-related resources, and immediately implementing and presenting context-related resources in a video game. Device 600 can be localized to a player playing a game segment (e.g., a game console), or away from a player (e.g., a back-end server processor), or one of many servers that use virtualization in a game cloud system to remotely stream a game process to a client.

[0064] Memory 604 stores applications and data for use by CPU 602. Storage device 606 provides non-volatile storage devices and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other optical storage devices, as well as signal transmission and storage media. User input device 608 transmits user input from one or more users to device 600, examples of which may include keyboards, mice, joysticks, touch pads, touch screens, static or video recorders / cameras, tracking devices for identifying gestures, and / or microphones. Network interface 614 allows device 600 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communications through local area networks and wide area networks (such as the Internet). Audio processor 612 is suitable for generating analog or digital audio output from instructions and / or data provided by CPU 602, memory 604 and / or storage device 606. The components of device 600 , including CPU 602 , memory 604 , data storage device 606 , user input device 608 , network interface 610 , and audio processor 612 , are connected via one or more data buses 622 .

[0065] The graphics subsystem 620 is also connected to the data bus 622 and the components of the device 600. The graphics subsystem 620 includes a graphics processing unit (GPU) 616 and a graphics memory 618. The graphics memory 618 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory 618 may be integrated into the same device as the GPU 608, connected to the GPU 616 as a separate device, and / or implemented in the memory 604. The pixel data may be provided directly from the CPU 602 to the graphics memory 618. Alternatively, the CPU 602 provides the GPU 616 with data and / or instructions defining the desired output image, and the GPU 616 generates pixel data for one or more output images based on the data and / or instructions. The data and / or instructions defining the desired output image may be stored in the memory 604 and / or the graphics memory 618. In one embodiment, GPU 616 includes 3D rendering capabilities that generate pixel data for output images from instructions and data that define the geometry, lighting, shading, textures, motion, and / or camera parameters of a scene. GPU 616 may also include one or more programmable execution units capable of executing shader programs.

[0066] Graphics subsystem 614 periodically outputs pixel data of an image from graphics memory 618 for display on display device 610. Display device 610 may be any device capable of displaying visual information in response to a signal from device 600, including CRT, LCD, plasma, and OLED displays. For example, device 600 may provide analog or digital signals to display device 610.

[0067] It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) often use cloud computing. Cloud computing is a computing method in which dynamically scalable and usually virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services typically provide common applications (such as video games) accessed online from a web browser, while the software and data are stored on servers in the cloud. Based on the way the Internet is depicted in computer network diagrams, the term cloud is used as a metaphor for the Internet and is an abstraction of the complex infrastructure that it hides.

[0068] In some embodiments, a game server can be used to perform the operation of a duration information platform for video game players. Most video games played over the Internet are operated by connection with a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, video games can be performed by a distributed game engine. In these embodiments, a distributed game engine can be performed on multiple processing entities (PEs) so that each PE performs a functional fragment of a given game engine on which a video game runs. The game engine simply regards each processing entity as a computing node. The game engine typically performs a series of operations with diverse functions to perform video game applications and additional services for user experience. For example, a game engine implements game logic, performs game calculations, physical effects, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communications, game play replay functions, help functions, etc. Although a game engine can sometimes be performed on an operating system virtualized by a hypervisor of a specific server, in other embodiments, the game engine itself is distributed between multiple processing entities, each of which can reside on different server units in a data center.

[0069] According to this embodiment, the corresponding processing entity for performing the operation can be a server unit, a virtual machine, or a container, depending on the needs of each game engine fragment. For example, if a game engine fragment is responsible for camera transformations, a virtual machine associated with a graphics processing unit (GPU) can be provided to this particular game engine fragment, because it will perform a large number of relatively simple mathematical operations (e.g., matrix transformations). Other game engine fragments that require fewer but more complex operations can be provided with processing entities associated with one or more higher-powered central processing units (CPUs).

[0070] By distributing the game engine, the game engine is provided with elastic computing properties that are not constrained by the capabilities of the physical server units. Instead, more or fewer computing nodes are provided to the game engine as needed to meet the needs of the video game. From the perspective of the video game and the video game player, a game engine distributed across multiple computing nodes is no different than a non-distributed game engine executed on a single processing entity, because the game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output components to the end user.

[0071] The user uses a client device to access the remote service, and the client device includes at least a CPU, a display, and an I / O. The client device can be a PC, a mobile phone, a notebook computer, a PDA, etc. In one embodiment, the network identification client used on the game server performs the device type and adjusts the communication method adopted. In other cases, the client device uses a standard communication method such as html to access the application on the game server through the Internet. It should be understood that a given video game or game application can be developed for a specific platform and a specifically associated controller device. However, when such a game is made available via a game cloud system as presented herein, the user may access the video game with different controller devices. For example, a game may have been developed for a game console and its associated controller, and the user may use a keyboard and mouse to access the cloud-based version of the game from a personal computer. In this scenario, the input parameter configuration can define the mapping of the input generated from the user's available controller device (in this case, i.e., keyboard and mouse) to the execution of the video game acceptable input.

[0072] In another example, the user can access the cloud gaming system via a tablet computing device, a touch screen smart phone, or other touch screen driven device. In this case, the client device and the controller device are integrally formed in the same device, wherein input is provided by means of a detected touch screen input / gesture. For such a device, the input parameter configuration can define a specific touch screen input corresponding to the game input of the video game. For example, during the running of the video game, buttons, directional pads, or other types of input elements may be displayed or overlaid to indicate the position on the touch screen where the user can touch to generate game input. Gestures (such as swipes or specific touch movements in a specific direction) may also be detected as game inputs. In one embodiment, guidance may be provided to the user, indicating, for example, how to provide input for the game to be played via the touch screen before starting to play the video game, so as to adapt the user to the operation of the controls on the touch screen.

[0073] In some embodiments, the client device is used as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device can process these inputs in turn, and then transmit the input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller itself can be a networked device with the ability to directly communicate input to the cloud game server via a network without first communicating such input through a client device. For example, the controller can be connected to a local networking device (such as the above-mentioned router) to send data to and receive data from the cloud game server. Therefore, although the client device may still be required to receive video output from a cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send input directly to the game cloud server over the network to bypass the client device.

[0074] In one embodiment, the networked controller and client device may be configured to send certain types of input directly from the controller to the cloud gaming server, and to send other types of input via the client device. For example, inputs whose detection does not depend on any additional hardware or processing other than the controller itself can be sent directly from the controller to the cloud gaming server via the network, thereby bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or require client devices to process can be sent by client devices to the cloud gaming server. These may include video or audio captured from a game scenario, which can be processed by the client device before being sent to the cloud gaming server. In addition, inputs from the motion detection hardware of the controller can be processed by the client device in conjunction with the captured video to detect the position and motion of the controller, which is then communicated to the cloud gaming server by the client device. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0075] In one embodiment, various technical examples can be implemented using a virtual environment through a head-mounted display (HMD). The HMD may also be referred to as a virtual reality (VR) head-mounted device. As used herein, the term "virtual reality" (VR) generally refers to the interaction of a user with a virtual space / environment, which involves viewing the virtual space in a real-time response to the movement of the HMD (controlled by the user) through the HMD (or VR headset) to provide the user with a feeling in a virtual space or metaverse. For example, a user can see a three-dimensional (3D) view of the virtual space when facing a given direction, and when the user turns to one side and thus turns the HMD in the same manner, the view of the side in the virtual space is presented on the HMD. The HMD can be worn in a manner similar to glasses, goggles, or a helmet, and is configured to display video games or other metaverse content to the user. The HMD can provide a very immersive experience to the user by means of its display mechanism being close to the user's eyes. Therefore, the HMD can provide a display area that occupies most of the user's or even the entire field of view to each eye of the user, and can also provide viewing with three-dimensional depth and perspective.

[0076] In one embodiment, the HMD may include a gaze tracking camera configured to capture images of the user's eyes as the user interacts with the VR scene. The gaze information captured by the gaze tracking camera may include information related to the user's gaze direction and specific virtual objects and content items in the VR scene that the user is paying attention to or interested in interacting with. Thus, based on the user's gaze direction, the system can detect specific virtual objects and content items that may be of potential interest to the user (where the user is interested in interacting with and engaging with them), such as game characters, game objects, game props, etc.

[0077] In some embodiments, the HMD may include an outward-facing camera configured to capture images of the user's real-world space, such as the user's body movements and any real-world objects that may be located in the real-world space. In some embodiments, the images captured by the external camera may be analyzed to determine the position / orientation of the real-world objects relative to the HMD. Using the known position / orientation of the HMD, real-world objects, and inertial sensor data from the user's posture and movement, the user's posture and movement during the user interaction with the VR scene can be continuously monitored and tracked. For example, when interacting with a scene in a game, a user may make various gestures, such as pointing and walking toward a specific content item in the scene. In one embodiment, the system may track and process gestures to generate predictions of interactions with specific content items in the game scene. In some embodiments, machine learning may be used to facilitate or assist the predictions.

[0078] During the use of the HMD, various single-hand and two-hand controllers can be used. In some implementations, the controller itself can be tracked by tracking the lights included in the controller or tracking the shapes, sensors and inertial data associated with the controller. Using these different types of controllers or even just using gestures made and captured by one or more cameras, the virtual reality environment or metaverse presented on the HMD can be docked, controlled, manipulated, interacted with and participated in the virtual reality environment or metaverse. In some cases, the HMD can be wirelessly connected to the cloud computing and game system through a network. In one embodiment, the cloud computing and game system maintain and execute the video game played by the user. In some embodiments, the cloud computing and game system are configured to receive input from the HMD and interface objects through the network. The cloud computing and game system are configured to process input to affect the game state of the video game being executed. Output (such as video data, audio data and tactile feedback data) from the video game being executed is transmitted to the HMD and the interface object. In other implementations, the HMD can communicate wirelessly with the cloud computing and game system through an alternative mechanism or channel such as a cellular network.

[0079] In addition, although the implementations in the present disclosure may be described with reference to head-mounted displays, it should be understood that in other implementations, non-head-mounted displays may be substituted, including but not limited to portable device screens (e.g., tablets, smart phones, laptops, etc.) or any other type of display that can be configured to render video and / or provide a display of an interactive scene or virtual environment according to the present implementation. It should be understood that the various embodiments defined herein can be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that can define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0080] Embodiments of the present disclosure may be practiced with various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through wired or wireless networks.

[0081] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed in a system that allows processing operations to occur at various intervals associated with the processing as long as the processing of telemetry and game state data used to generate the modified game state is performed in the desired manner.

[0082] One or more embodiments may also be manufactured as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage devices (NAS), read-only memories, random access memories, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. A computer readable medium may include a computer readable tangible medium distributed on a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0083] In one embodiment, the video game is executed locally on a game console, a personal computer, or on a server. In some cases, the video game is executed by one or more servers in a data center. When executing the video game, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. In some embodiments, the simulation is an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In either case, if the video game is represented as a simulation, the simulation can be executed to render interactive content that can be interactively streamed, executed, and / or controlled by a user input.

[0084] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it is apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments are to be considered illustrative rather than restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method for executing a session of a video game, the method comprising: recording speech of players participating in game play of said session of said video game; analyzing a game state resulting from said execution of said session of said video game, wherein analyzing said game state identifies a context in which said game play is performed; analyzing the recorded speech using the recognized context and speech recognition model performed by the game to recognize textual content of the recorded speech; The recognized textual content is applied as game play input for the session of the video game.

2. A method according to claim 1, wherein the method is performed in substantially real time, such that analysis of the game state is performed in substantially real time with the game state being continuously updated through the execution of the session, and such that the analysis to identify the textual content of the recorded speech is a substantially real-time response to changes in the game state.

3. The method of claim 1, wherein the identified context of the game play comprises an identified scene or stage of the video game.

4. The method of claim 1, wherein the identified context of the game play comprises an identified virtual location in a virtual environment defined by the execution of the session of the video game.

5. The method of claim 1, wherein the identified context of the game play includes identified attributes of the player's avatar in the video game.

6. The method of claim 1, wherein analyzing the game state comprises identifying activities occurring during the game play, the identified activities at least partially defining the identified context of the game play.

7. The method of claim 1, wherein analyzing the game state comprises predicting future game play activity, the predicted future game play at least partially defining the identified scenario of the game play.

8. A method for executing a session of a video game, the method comprising: recording speech of players participating in game play of said session of said video game; analyzing the recorded speech using a speech recognition model, the model identifying a plurality of candidate words as possible interpretations of the speech of the player; during the session of the video game, presenting the plurality of candidate words to the player; receiving a selection input from the player and identifying one of the candidate words as a correct interpretation of the player's speech; A selected one of the candidate words is applied by the video game as an input for gameplay of the video game.

9. The method of claim 8, wherein presenting the plurality of candidate words is in response to a recognition confidence of the speech recognition model falling below a predefined threshold.

10. The method of claim 8, wherein presenting the candidate word to the player comprises presenting the candidate word in a video generated from the session of the video game.

11. The method of claim 8, wherein presenting the candidate word pauses the game play until the selection input has been received.

12. The method of claim 8, wherein applying the selected one of the candidate words comprises triggering a command for the game play of the video game.

13. The method of claim 8, wherein the selected one of the candidate words is used as feedback to improve the speech recognition model.

14. The method of claim 8, wherein receiving the selection input is through an input device of the controller.

15. A non-transitory computer readable medium having program instructions embodied thereon, the program instructions, when executed by at least one computing device, causing the at least one computing device to perform a method for executing a session of a video game, the method comprising the following operations: recording speech of players participating in game play of said session of said video game; analyzing a game state resulting from said execution of said session of said video game, wherein analyzing said game state identifies a context in which said game play is performed; analyzing the recorded speech using the recognized context and speech recognition model performed by the game to recognize textual content of the recorded speech; The recognized textual content is applied as game play input for the session of the video game.

16. A non-transitory computer-readable medium according to claim 15, wherein the method is performed in substantially real time, such that analysis of the game state is performed in substantially real time with the game state being continuously updated through the execution of the session, and such that the analysis to identify the textual content of the recorded speech is responsive to changes in the game state in substantially real time.

17. The non-transitory computer-readable medium of claim 15, wherein the identified context of the game play comprises an identified scene or stage of the video game.

18. The non-transitory computer-readable medium of claim 15, wherein the identified context of the game play comprises an identified virtual location in a virtual environment defined by the execution of the session of the video game.

19. The non-transitory computer-readable medium of claim 15, wherein the identified context of the game play includes identified attributes of the player's avatar in the video game.

20. The non-transitory computer-readable medium of claim 15, wherein analyzing the game state comprises identifying activities occurring during the game play, the identified activities at least partially defining the identified context of the game play.

Images