Inference window for gesture input

Abstract

A system is configured to determine (520) a time window for executing (540) a particular gesture or voice command to a computer game. Otherwise, the system does not execute the command outside the time window, thereby helping to reduce false alarms caused by accidental inputs.

Description

TECHNICAL FIELD

[0001] The following disclosure relates generally to an inferred time window during which gesture input to a computer game is monitored. BACKGROUND

[0002] As described herein, one of the technical challenges faced by computer game developers and game console manufacturers in common is the need to correctly identify gesture and voice input to a computer game in order to distinguish that input from other natural gestures and voices that can not be related to the game. For example, a player can make an unrelated gesture or speak to a non-player during the time that the game is executing, but the game system will still capture the gesture or voice and execute the corresponding game command, even though the player did not intend to actually control the game. This in turn can have an adverse effect on the game play, sometimes even a considerable one. There is currently no adequate solution to the foregoing computer-related technical problem. SUMMARY

[0003] Accordingly, in one aspect, an apparatus includes at least one processor assembly programmed with instructions to execute a computer game and receive game state data related to execution of the computer game. The at least one processor assembly is further programmed with instructions to identify, based on the game state data and during execution of the computer game, a first time window during which commands are executed in accordance with first non-computer game controller input to the computer game. The at least one processor assembly is further programmed with instructions to identify the first non-computer game controller input during the first time window and execute a command related to the computer game based on the identification of the first non-computer game controller input during the first time window. The command is executed in accordance with the first non-computer game controller input.

[0004] Accordingly, in one example implementation, the game state data can be first game state data and the at least one processor assembly can be programmed with instructions to receive second game state data related to execution of the computer game. Here, the at least one processor assembly can be programmed with instructions to then identify, based on the second game state data and during execution of the computer game, a second time window during which commands are not executed in accordance with the first non-computer game controller input to the computer game. Accordingly, the at least one processor assembly can also be programmed with instructions to not execute a command related to the computer game during the second time window. Not executing the command can include not processing the first non-computer game controller input to the computer game and / or can include processing the first non-computer game controller input to the computer game but not executing the command in accordance with the first non-computer game controller input.

[0005] In various examples, the first non-computer game controller input can include gesture input. Thus, the device can include a camera, and the gesture input can be identified using one or more images from the camera.

[0006] Also in various examples, the first non-computer game controller input can include voice input. Thus, the device can also include a microphone, and the voice input can be identified using voice data from the microphone.

[0007] Additionally, if desired, the game state data can indicate a first context associated with the execution of the computer game, and the at least one processor complex can be programmed with instructions to identify the first time window based on identifying the first context from the game state data. Thus, for example, the at least one processor complex can be programmed with instructions to: provide the game state data as input to a machine learning (ML) model; receive an inference indicating the first context as output from an activation layer of the ML model; and identify the first context from the inference. The first time window can be a dynamic time window, such as a time window that is present as long as the first context persists during the execution of the computer game.

[0008] Also, if desired, the at least one processor complex can be programmed with instructions to: provide the game state data as input to a machine learning (ML) model; and receive an inference indicating the first time window itself as output from an activation layer of the ML model. The ML model can be trained using at least one dataset of game state data and true time windows during which non-computer game controller input was monitored.

[0009] In another aspect, a method includes executing a computer game and receiving game state data related to the execution of the computer game. The method also includes identifying a first time window during which commands are executed in accordance with a first non-computer game controller input to the computer game based on the game state data and during the execution of the computer game. The method also includes identifying the first non-computer game controller input during the first time window and executing a command related to the computer game based on the identification of the first non-computer game controller input during the first time window. Thus, the command is executed in accordance with the first non-computer game controller input.

[0010] In various examples, the first non-computer game controller input can include gesture input of a predetermined gesture and / or voice input of a predetermined voice command.

[0011] In yet another aspect, a system includes at least one computer medium that is not a transitory signal. The computer medium includes instructions executable by at least one processor assembly to infer a first time window during which, for a first execution instance of a computer game, to execute a non-computer game controller command for the first execution instance of the computer game. The first time window does not span all of the first execution instance of the computer game. The instructions are also executable to execute the non-computer game controller command during the first time window based on the inferring.

[0012] Thus, in certain specific non-limiting examples, the first time window can not span all of a certain segment or level of the computer game or a particular gameplay sequence thereof, but can span some of the particular gameplay sequence of the segment or level of the computer game.

[0013] Also, in certain non-limiting implementations, the instructions can be executable to infer the first time window using game state data and a machine learning (ML) model. If desired, the ML model can be trained using at least one dataset of game state data, player outcomes from previous gameplay instances, and real time windows in which corresponding non-computer game controller commands were executed.

[0014] The details of the application, both as to its structure and operation, can be gleaned in part by reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a block diagram of an example system in accordance with the principles of the application; Figure 2 An example schematic is shown with a screen notification that can be presented to a player during live gameplay of a computer game, where the notification is presented based on the system inferring a dynamic time window during which the player can provide particular gesture- or voice-based input for execution in accordance with the principles of the application; Figure 3 An example schematic is shown with another screen notification informing a player that a suitable gesture or voice command has been detected and executed in accordance with the principles of the application during a time window; Figure 4 Yet another example schematic is shown with yet another screen notification informing a player that execution of a particular gesture- or voice-based command has been denied in accordance with the principles of the application based on the command being received outside of a suitable time window; Figure 5is an example flow diagram of example overall logic that can be performed by one or more devices in accordance with the principles of the application; Figure 6 and Figure 7 different example artificial intelligence software architectures that can be used in accordance with the principles of the application are shown; Figure 8 example training logic that can be performed in accordance with the principles of the application to train Figure 6 ML models is shown; Figure 9 example training logic that can be performed in accordance with the principles of the application to train Figure 7 ML models is shown; and Figure 10 example settings graphical user interface (GUI) that can be used to configure one or more settings of a device to operate in accordance with the principles of the application is shown. DETAILED DESCRIPTION

[0016] The present disclosure relates generally to computer ecosystems, including aspects of consumer electronics (CE) device networks, such as but not limited to computer gaming networks. The systems herein can include server components and client components, which can be connected by a network such that data can be exchanged between the client components and the server components. The client components can include one or more computing devices including game consoles (such as a Sony PlayStation® or a game console made by Microsoft or Nintendo or other manufacturers), virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart TVs, Internet-enabled TVs), portable computers (such as laptop computers and tablet computers), and other mobile devices including smartphones and additional examples discussed below. These client devices can operate in a variety of operating environments. For example, some client computers can employ a Linux operating system, an operating system from Microsoft, or a Unix operating system, or an operating system produced by Apple, Inc. or Google. These operating environments can be used to execute one or more browsing programs, such as a browser made by Microsoft or Google or Mozilla, or other browser programs that can access websites hosted by Internet servers discussed below. Also, one or more computer game programs can be executed using operating environments in accordance with the principles of the application.

[0017] The server and / or gateway can include one or more processors that execute instructions that configure the server to receive and transmit data over a network, such as the Internet. Alternatively, the client and server can be connected through a local intranet or virtual private network. The server or controller can be instantiated by a game console (such as a Sony PlayStation®, a personal computer, etc.).

[0018] Information can be exchanged between the client and the server over a network. To this end, and for security, the server and / or client can include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers can form an apparatus that implements a method of providing a secure community, such as an online social website or network of game players, to members of a network.

[0019] The processor can be a single-chip or multi-chip processor that executes instructions to perform logic by means of various lines, such as address lines, data lines, and control lines, as well as registers and shift registers. The processor assembly can include one or more processors that can act independently or in concert with one another to execute algorithms, whether those processors are located in one device or in more than one device.

[0020] The components included in one implementation can be used in other implementations in any appropriate combination. For example, any of the various components described herein and / or depicted in the drawings can be combined, interchanged or excluded from other implementations.

[0021] A "system having at least one of A, B, and C" (likewise, "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes: a system having A only; a system having B only; a system having C only; a system having both A and B; a system having both A and C; a system having both B and C; and / or a system having all of A, B, and C, etc.

[0022] Reference will now be made to specific implementations Figure 1An example system 10 is illustrated, which may include one or more of the example devices mentioned above and further described below according to the principles of the invention. The first device among the example devices included in system 10 is a consumer electronics (CE) device, such as an audio-visual device (AVD) 12, such as, but not limited to, an internet-enabled TV with a TV tuner (equivalently, a set-top box controlling a TV). Alternatively, the AVD 12 may also be a computerized internet-enabled 5G (“smart”) phone, tablet computer, laptop computer, head-mounted device (HMD) (such as smart glasses or other wearable computerized devices (e.g., AR or VR headsets)), computerized internet-enabled music player, computerized internet-enabled headphones, computerized internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to implement the principles of the invention (e.g., communicate with other CE devices to implement the principles of the invention, perform the logic described herein, and perform any other functions and / or operations described herein).

[0023] Therefore, in order to implement this principle, AVD 12 can be... Figure 1 Some or all of the components shown are constructed. For example, AVD 12 may include one or more displays 14, which may be implemented by a high-definition or ultra-high-definition "4K" or higher flat screen and may be touch-enabled for receiving user input signals via touch on the display. AVD 12 may include one or more speakers 16 for outputting audio according to the principles of the invention and includes at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to control AVD 12. Example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to implement the principles of the invention, including controlling other elements of AVD 12 described herein, such as controlling display 14 to present images on said display and to receive input from said display. In addition, it should be noted that network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above.

[0024] In addition to the foregoing, the AVD 12 can also include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI) port or a Universal Serial Bus (USB) port that physically connects to another CE device and / or a headphone port that connects headphones to the AVD 12 to present audio from the AVD 12 to a user through the headphones. For example, the input port 26 can connect via a wired or wireless means to a cable or satellite source 26a of audio video content. Thus, the source 26a can be a separate or integrated set-top box, or satellite receiver. Alternatively, the source 26a can be a game console or disk player that contains content. The source 26a when implemented as a game console can include some or all of the components described below with respect to the CE device 48.

[0025] The AVD 12 can also include one or more computer memories / computer readable storage media 28 that are not transitory signals, such as disk-based storage or solid state storage, which in some cases are embodied as a separate device in the housing of the AVD, or as a personal video recording device (PVR) or video disk player inside or outside the housing of the AVD for playing back AV programs, or as a removable memory media or a server described below. Further, in some embodiments, the AVD 12 can include a location or positioning receiver, such as but not limited to a cellular telephone receiver, a GPS receiver, and / or an altimeter 30 configured to receive geographic position information from satellites or cellular telephone base stations and provide the information to the processor 24 and / or determine an altitude at which the AVD 12 is located in conjunction with the processor 24. The component 30 can also be implemented by an inertial measurement unit (IMU), which typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions, or by an event-based sensor.

[0026] Continuing the description of the AVD 12, in some embodiments, the AVD 12 can include one or more cameras 32, which can be thermal imaging cameras, digital cameras such as webcams, event-based sensors, and / or cameras integrated into the AVD 12 and controllable by the processor 24 to capture pictures / images and / or video in accordance with the principles of the application. Also included on the AVD 12 can be a Bluetooth transceiver 34 and other near field communication (NFC) elements 36 to communicate with other devices using Bluetooth and / or NFC technology, respectively. An example NFC element can be a radio frequency identification (RFID) element.

[0027] Further, the AVD 12 can include one or more auxiliary sensors 38 (e.g., motion sensors such as accelerometers, gyroscopes, odometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)) that provide input to the processor 24. The AVD 12 can include a wireless TV broadcast port 40 to provide input to the processor 24 for receiving OTA TV broadcasts. In addition to the foregoing, it is noted that the AVD 12 can also include an infrared (IR) emitter and / or IR receiver and / or IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) can be provided to power the AVD 12, as can be a kinetic energy harvester that can convert kinetic energy into electrical power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 can also be included. One or more haptic / vibration generators 47 can be provided for generating haptic signals that can be sensed by a person holding or touching the device. Thus, the haptic generators 47 can use electric motors connected via a rotatable shaft of the motor to an eccentric and / or unbalanced weight, such that the shaft can be rotated under the control of the motor (which in turn can be controlled by a processor such as the processor 24) to produce vibrations of various frequencies and / or amplitudes and force simulations in various directions.

[0028] Still referring to Figure 1 In addition to the AVD 12, the system 10 can include one or more other CE device types. In one example, a first CE device 48 can be a computer game console that can be used to send audio and video of a computer game to the AVD 12 via commands sent directly to the AVD 12 and / or through a server described below, while a second CE device 50 can include similar components as the first CE device 48. In the example shown, the second CE device 50 can be configured as a computer game controller manipulated by a player or a head-mounted display (HMD) worn by a player. The HMD can include a heads-up transparent or opaque display for presenting AR / MR content or VR content, respectively (more generally, extended reality (XR) content). The HMD can be configured as eyeglass-type displays or as VR-type displays sold by computer game equipment manufacturers.

[0029] In the example shown, only two CE devices are shown, it being understood that fewer or more devices can be used. The devices herein can implement some or all of the components shown for the AVD 12. Any of the components shown in the following figures can be combined with some or all of the components shown in the case of the AVD 12.

[0030] Referring now to the at least one server 52 mentioned above, the at least one server includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based storage or solid state storage, and at least one network interface 58 that, under control of the server processor 54, allows communication over the network 22 with other devices of the system 10, and indeed can facilitate communication between the server and client devices in accordance with the principles of the application. It should be noted that the network interface 58 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other appropriate interface, such as, for example, a wireless telephony transceiver. Figure 1

[0031] Thus, in some embodiments, the server 52 can be an Internet server or a field of servers, and in an exemplary embodiment for a networked play gaming application, the server can include and perform "cloud" functionality, such that the devices of the system 10 can access a "cloud" environment via the server 52. Alternatively, the server 52 can be implemented by one or more game consoles or other computers that are in the same room as the other devices shown or nearby. Figure 1

[0032] The components shown in the following figures can include some or all of the components shown. The user interfaces (UIs) described herein can be incorporated, extended, and UI elements can be mixed and matched between UIs. Figure 1

[0033] The principles of the application can employ various machine learning models, including deep learning models. Machine learning models in accordance with the principles of the application can use various algorithms trained in ways including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms, which can be implemented by computer circuitry, include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN called a long short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks can also be considered examples of machine learning models. In addition to the network types mentioned above, the models herein can also be implemented by classifiers.

[0034] As understood herein, thus, performing machine learning can involve accessing training data and then training a model on the training data to enable the model to process further data to make inferences. Thus, an artificial neural network / artificial intelligence model trained by machine learning can include an input layer, an output layer, and a plurality of hidden layers between them, configured and weighted to make inferences about appropriate outputs. ​​​

[0035] In view of the foregoing, the present principles relate to apparatus and computer-implemented methods for determining a reasoning time window for performing game actions from gesture or voice input during computer game play. Outside the time window, even if the system recognizes a gesture or voice command, the command is not executed.

[0036] Thus, a game player can use buttons, analog sticks, and triggers on a wireless video game controller to input less complex commands to a computer game. More complex actions can then be provided through gesture or voice input, where the gesture or voice input can be translated into a macro, where the gesture or voice command is mapped to a series of video game controller inputs (e.g., button presses, joystick operations) that are input to the game engine to execute the gesture or voice command itself. But to minimize false positives, where the player can make a physical movement or gesture that is not related to the game, or where the player can speak to another person or even a separate digital assistant, but not intending to provide voice input to the game, the apparatus can also use a reasoning window that opens and closes based on the game state. Thus, the system can only look for certain gesture / voice inputs to execute at certain times. Otherwise, the system can ignore those particular gesture / voice inputs, or ignore all gesture / voice inputs generally.

[0037] Thus, for example, the game engine can know that the player's virtual character has low health, and then open a dynamic reasoning window whenever the player's character health is below a certain threshold level. The game engine can then look for appropriate player gestures / physical actions during the reasoning window that can be interpreted as drinking a health potion (to improve health). In response to the character health rising above the threshold level, the game engine can close the reasoning window, and no longer look for those same player actions (at least until the health drops below the threshold again).

[0038] Accordingly, various artificial intelligence (AI)-based machine learning (ML) models and techniques can be used to reduce false positives of game execution commands. For example, a system / model can observe outcomes of other players playing the same computer game and find out what causes the game to end or the player’s mission to fail, and conversely, what causes the player to succeed. The system / model can then extrapolate a time window for future game play instances in similar situations, using past player instances to determine when to look again for the same action that caused other players to succeed. For example, a classifier can identify gesture / speech inputs that cause positive game impact, and then look again for the same action in similar situations in the future, even if the positive game impact output is not necessarily the highest ranked likelihood from the model output. Or, the model can filter out negative impact inputs, or be trained to change weights based on which gesture / speech inputs are likely to cause positive game impact, and then look for those specific positive inputs again when the same context occurs in the game (while not looking for other types of gesture / speech commands that are not relevant to the context, or at least reducing their weights on a likelihood scale, during that time).

[0039] In terms of gestures, specifically, these gestures can be non-computer game controller inputs (e.g., gestures that do not control the following controller 220 or otherwise do not provide input to the input device). And whether the non-computer game controller inputs are voice-based or free-space gesture-based inputs, the input actions commanded by the player can themselves relate to (as non-exhaustive examples) moving a virtual character, having the character shoot another virtual character, selecting an option from a screen menu, and / or drinking a health potion.

[0040] Additionally, it should be noted that the computer game itself can be a two-dimensional (2D) game presented on a television or computer monitor, or can be a virtual reality or augmented reality (more generally, extended reality) computer game presented on a headset or smartphone. However, it should also be noted that the principles of the present invention can also be applied to other technical fields, such as non-game extended reality and other computing environments where user inputs, while not used for games themselves, are still used.

[0041] Further, it should also be noted that if the same command is received more than once during the inference window, the principles of the present invention can also be used to ignore intentional or accidental / faulty repeated commands. Thus, the command can be executed during the inference window based on the first time the command is received, and then, if the same command is received again: outside the inference window, or within the inference window, but still within the threshold time range of the initial command reception, the system can ignore / discard the same command (e.g., if a repeated command is received within five seconds of the initial input command, the command is ignored). This can occur, for example, in the case of detecting a gesture, such as when a child repeatedly makes the same gesture to have a character drink a potion of life. Here, even though the player / child has multiple potions of life in their inventory - which could have been consumed within the inference window based on the repeated commands - the potion of life can only be consumed once within the window.

[0042] Reference is now made to the following drawings, in which Figure 2 As shown, a display 200 (such as a computer monitor, television, or smartphone display) is presenting visual game content 210, such as a first-person perspective of a player exploring a virtual world or battling a computer game "boss" character. Also shown is a wireless video game controller 220 with buttons, analog sticks, and trigger buttons. The controller 220 can be used to provide controller-based commands to a game engine rendering a current instance of a computer game. Thus, the display 200 itself can include hardware components for receiving commands wirelessly from the controller 220 and executing the computer game / engine in accordance with the commands. Additionally or alternatively, the display 200 can be in communication with a cloud-based server executing the computer game / engine and receiving the commands in order to present the game content 210 on the display 200; and / or can be in communication with a local video game console (not shown) that can be executing the computer game / engine and receiving the commands in order to present the game content 210 on the display 200.

[0043] As for the non-computer game controller commands themselves, they can include free space gestures recognized, e.g., via gesture recognition using input from a camera. The camera can be in communication with the device executing the game engine, such as a camera 230 on the display 200 itself. As an example, a free space gesture can include mimicking a drinking motion, with the player first making a "C" shape with the fingers and thumb, and then holding the same hand to the mouth, mimicking drinking from a cup. The non-computer game controller commands can also include voice commands recognized, e.g., via speech recognition using input from a microphone. The microphone can be in communication with the device executing the game engine, such as a microphone 240 on the display 200 itself. For example, a voice input can be the command "drink potion."

[0044] Accordingly, cross-referencing Figures 2 to 4 the game engine infers that the player is engaged in a boss battle or other combat with a non-player antagonist or other enemy in the computer game, and that the player’s own character has low health (e.g., below a threshold health level, such as 20% of remaining health). Based on the inference, Figure 2 A first indicator 250 is shown overlaid on the game content 210 as part of a graphical user interface (GUI). The indicator 250 can provide a notification of the circumstances that led to the inference (here, boss battle and low health), and can also notify the player that the game engine is currently looking for a gesture that the player can make that can be translated into a command to drink health potion.

[0045] Then, once the game engine detects / infer a free space gesture that maps to a drink health potion command (e.g., using input from the camera 230), the indicator 250 can be removed. Figure 3 Another indicator 300 is shown as it can be presented. The indicator 300 indicates that a drink potion action has been detected. This can serve as a notification that any body pose based action that the player just made in the real world has been translated into a gesture command to the computer game, and that the game engine is executing the game in compliance with the command.

[0046] However, Figure 4 In a different situation in the same game instance, if a drink gesture is recognized, but the game engine has not inferred a context in which a drink health potion voice and gesture command should be executed, a different notification 400 can be presented instead, as shown. The notification 400 indicates that the received gesture or voice command to drink health potion has been rejected. The notification 400 can also include a prompt asking whether the player really intended to provide this input, and the notification 400 indicates that the player’s character does not need health potion at this time (thereby helping the player to conserve virtual items / assets for future use to improve game play).

[0047] As also Figure 4 shown, the prompt can be accompanied by a “yes” selector 410 and a “no” selector 420. The “yes” selector 410 can be selectable to provide a positive input in response to the prompt, indicating that the player did intend to provide the command to drink potion. Accordingly, selection of the selector 410 can be used in one or both of the following operations: (A) commanding the player’s character to drink health potion in this instance regardless (the command being executed in response to selection of the selector 410), and (B) commanding the device to perform additional training on the ML model that erroneously inferred that the inference window for gesture and voice based health potion commands was not applicable.

[0048] Selection of selector 420 can also serve one or both of two different functions. Specifically, selection of selector 420 can perform one or both of the following operations: (1) provide a command to confirm that the device rejected the potion of life potion; and (2) provide a command to the device to perform additional training of the ML model that reinforces the ML model to correctly infer that the reasoning window is not applicable to the current game context. Thus, selection of either selector 410, 420 can not only provide input to the game engine related to execution of this particular game instance, but also be used to generate a label to use game context data from the current game situation for additional training to help improve the model itself.

[0049] Before describing Figure 5 it should also be noted that while notifications 400 and corresponding selectors 410, 420 can be presented in real-time as a player plays a computer game, in other instances, the notifications and selectors can not be presented during gameplay, but rather can be presented in response to a pause command that pauses the computer game. This can help minimize clutter on the screen that can distract the player and have an adverse effect on gameplay.

[0050] Reference is now made to Figure 5 which illustrates example logic that one or more devices can perform in accordance with the present principles. For example, Figure 5 the logical steps of Figure 5 may be performed by one or more of a personal computer, a game console, and / or an internet-based cloud gaming server, either individually or in any suitable combination. It should be noted that while the logic of is illustrated in the form of a flowchart, other suitable logic can also be used.

[0051]

[0052] At block 510, the device can receive first game state data related to execution of the computer game from the game engine. The first game state data can be indicative of a first context related to execution of the computer game, such as being low on health while fighting a boss, as described above. Or as another example, the game state data can be indicative that a weapon switch can be helpful because the player’s character is at a disadvantage while fighting a boss using a certain combat weapon (and so the device would infer a time window for executing a gesture / voice command to switch virtual weapons). The game state data can also include data regarding the current game level or stage the player character is in, menu items that are operable, items in the player’s game inventory or arsenal, remaining health, etc. The first context itself can be identified using a rules-based algorithm, and / or by providing the first game state data to an ML model to receive an output of the first context from an activated layer of the ML model. From block 510, the logic can then proceed to block 520.

[0053] At block 520, the device can identify a first time window during which to execute a command in accordance with a first non-computer game controller input (e.g., a gesture or voice input) to the computer game based on the game state data (e.g., particularly the first context) and during execution of the computer game. Here, it should be noted that the first time window can span some but not all of a first instance of execution of the computer game. For example, the first time window can not span all of a certain segment or level of the computer game, or a particular gameplay sequence in the segment or level (e.g., a boss fight), although it can span some of the particular gameplay sequence (e.g., only the last third of the boss fight itself).

[0054] From block 520, the logic can then proceed to block 530. At block 530, during the first time window, the device can identify the first non-computer game controller input itself. Based on identifying the first non-computer game controller input during the first time window, the logic can proceed to block 540, where the device can execute a command related to the computer game in accordance with the first non-computer game controller input itself. It should again be noted that gesture inputs can be identified using one or more images from a camera and execution of computer vision algorithms, such as gesture recognition algorithms and / or motion recognition algorithms. Voice inputs can be identified using voice data from a microphone and execution of speech recognition algorithms and / or speech-to-text algorithms.

[0055] After block 540, the logic can proceed to block 550. At block 550, the device can receive second game state data related to the execution of the computer game. The second game state data can indicate a second context associated with the execution of the computer game, such as when in combat with a boss or non-boss in the same game instance, the health is not at a low value. The second context can also be identified using a rules-based algorithm, and / or by providing the second game state data to the same ML model to receive an output of the second context from an activated layer of the ML model.

[0056] From block 550, the logic can proceed to block 560, where the device can identify a second time window during which the (same) command is not performed in accordance with the first non-computer game controller input to the computer game based on the second game state data and during the execution of the computer game. Thus, at block 560, the first time window can dynamically end based on the identified rules-based context or inferred context changing from the first context to the second context.

[0057] The logic can then proceed to block 570, where the device can not perform the same command related to the computer game that the device attempted to perform during the first time window during the second time window. Thus, not performing the command can include not processing the first non-computer game controller input to the computer game, such as by refusing to receive and / or process input from a camera or microphone, or even by turning off the camera or microphone itself. Not performing the command can also include, in some examples, processing the first non-computer game controller input to the computer game to identify the command, but not actually performing the command in accordance with the command.

[0058] Additionally, it should be noted that in some examples, the device can not process / perform any gesture or voice input during the second time window. However, in other examples, the device can still process / perform other gesture or voice inputs during the second time window (regardless of the context or for a different identified context), but can not perform the particular gesture or voice command that was inferred as being specifically applicable during the first time window. Thus, for example, a gesture input of holding up and swinging an axe can be inferred at any point during the execution instance of the computer game, but a similar body motion of the player holding up a virtual cup to drink potion can only be inferred if the player’s health is below a certain threshold level while in combat with a boss. This can help reduce false positives and corresponding negative game events, such as when the gesture of holding up an axe and swinging is misinterpreted as a command to drink potion when the character’s health is not below the threshold health level.

[0059] Referring now to Figure 6 , a system is shown that can incorporate the principles of the application, including Figure 5FIG. 6 illustrates an example artificial intelligence architecture of an ML model used in accordance with the logic of the present principles. The architecture includes a context identifier / classifier 600 into which game state data 610 from a game engine can be input such that a context 620 of an instance of a computer game can be inferred. In various non-limiting examples, the identifier / classifier 600 can be established by one or more convolutional neural networks and / or feedforward neural networks, such as in particular, feedforward backpropagation neural networks (FFBPNNs).

[0060] Accordingly, cross-referencing Figure 5 and Figure 6 At block 520, the apparatus can provide the first game state data received at block 510 as input 610 to an ML model established at least in part by the identifier / classifier 600. The apparatus can then receive as output from an activation layer of the identifier / classifier 600 an inference indicative of a first context, whereby the first context is identified from the inference. Also as Figure 6 illustrated, the identified first context can then be provided at block 520 to a rules-based software algorithm, which can then be executed to identify a first time window using one or more rules based on the first context. Using the example described above, the first context can thus include both the occurrence of a boss battle and the character’s health being below a threshold of 20%, and the rules-based algorithm can use a relational database to determine that, in these circumstances, a time window should be initiated during which the apparatus processes gesture or voice inputs to drink potions that improve health.

[0061] With continued reference to Figure 7 described in detail, FIG. 7 illustrates an example artificial intelligence architecture of another ML model used in accordance with the logic of the present principles (including Figure 5 The architecture includes a time window generator 700 into which game state data 710 from a game engine can be input such that a time window 730 consistent with the present principles can be inferred. In various non-limiting examples, the time generator 700 can be established by one or more convolutional neural networks and / or recurrent neural networks, such as in particular, long short-term memory (LSTM) neural networks.

[0062] Accordingly, cross-referencing Figure 5 and Figure 7 At block 520, the apparatus can provide the first game state data received at block 510 as input 710 to an ML model established at least in part by the window generator 700. The apparatus can then receive as output from an activation layer of the generator 700 an inference indicative of a first time window, whereby the first time window is identified at block 520.

[0063] Thus, using either or both of the ML models from Figure 6 and Figure 7 , a first time window can be established as a dynamic time window. Moreover, the first time window can exist as long as the first context persists during the same game instance; and if the first context occurs more than once, the first time window can occur more than once, at different parts of the game / single game instance. The dynamic time window can also end each time the first context ends in the game instance.

[0064] Now will be described Figure 8 and Figure 9 , which show the logic for training the models of Figure 6 and Figure 7 prior to deployment and / or for additional training after deployment. Starting first with the logic of Figure 8 , at block 800, the logic can provide at least one dataset as training input, the at least one dataset comprising game state data and corresponding real game context. From block 800, the logic can proceed to block 810 to actually train the model using the at least one dataset. The ML model can comprise a neural network, a decision tree, and / or a support vector machine, and the training itself can comprise supervised learning, unsupervised learning, and / or reinforcement learning.

[0065] As an example, for supervised learning, the ML model can be trained using labeled data (labeled real context), where each training set comprises one or more different types of game state data and their corresponding real context. The model can learn to map game state data to context by minimizing the difference between its predictions and the real context.

[0066] As another example, for reinforcement learning, the ML model can learn to map game state data to context by interacting with the game environment and receiving feedback in the form of rewards or penalties for correctly or incorrectly inferring the context. Thus, the model can learn to optimize its mapping strategy to maximize cumulative rewards over time, resulting in a model that produces positive game impact as described above. Thus, the inferred context can be used to establish a time window during which specific gestures or voice commands are performed for that context - potentially producing positive game impact - and within the same time window, the device can also not perform other specific gestures and voice commands for the same context - potentially producing negative game impact.

[0067] Now continuing with the detailed description of Figure 9 , which also shows additional logic that can be used for training, this time with respect to Figure 7The window generator is 700. Therefore, at box 900, the logic can provide at least one dataset as training input, said at least one dataset comprising game state data and a corresponding real-time window during which non-computer game controller input is monitored. In some examples, at least one dataset may also include player results from previous game execution instances in the same or similar context, to be trained on previous commands that had a positive game impact in the previous game execution instances.

[0068] Starting at box 900, the logic can proceed to box 910 to actually train the model using at least one dataset. This ML model may also include neural networks, decision trees, and / or support vector machines, and the training itself may include supervised learning, unsupervised learning, and / or reinforcement learning. As an example, for supervised learning, labeled data (labeled real time windows) can be used to train the ML model, where each training set includes one or more different types of game state data, previous player outcome data, and their corresponding real time windows. The model learns to map game state data to time windows by minimizing the difference between its predictions and real time windows.

[0069] As another example, in reinforcement learning, ML models can learn to map game state data to dynamic time windows by interacting with the game environment and receiving feedback in the form of rewards or penalties for correctly or incorrectly inferring time windows. Thus, the model may learn to optimize its mapping strategy to maximize cumulative rewards over time, thereby producing a model that generates positive game impact by correctly inferring the time window for a given gesture / voice command.

[0070] Continue to refer to Figure 10 For detailed description, the figures illustrate an example setup of a graphical user interface (GUI) 1000, which can be displayed on a monitor to configure one or more settings for client devices, consoles, computer games, etc., to implement the principles of the present invention. For example, GUI 1000 can be presented based on the user's navigation through device or game menus.

[0071] like Figure 10 As shown, the GUI 1000 may include a first option 1010, which can be selected once to set / configure the device to control a computer game in multiple future instances by inferring time windows for processing and executing certain non-computer game controller inputs. Therefore, it should be noted again that the input may include one or more specific predetermined gestures, and / or one or more specific predetermined voice commands. Thus, for example, selecting option 1010 can configure the device to perform the aforementioned actions in the future, including presentation... Figures 2 to 4 GUI, executionFigure 5 logic and / or using Figure 6 and Figure 7 ML models.

[0072] While specific embodiments are shown and described in detail herein, it will be appreciated that the subject matter encompassed by the present application is not limited to the specific embodiments shown and described.

Claims

1. A device comprising: at least one processor complex programmed with instructions to: execute a computer game; receive game state data related to the execution of the computer game; identify, based on the game state data and during the execution of the computer game, a first time window during which to execute a command in accordance with a first non-computer game controller input to the computer game; identify, during the first time window, the first non-computer game controller input; and execute the command related to the computer game based on the identification of the first non-computer game controller input during the first time window, the command being executed in accordance with the first non-computer game controller input.

2. The device of claim 1, wherein the game state data is first game state data, and wherein the at least one processor complex is programmed with instructions to: receive second game state data related to the execution of the computer game; identify, based on the second game state data and during the execution of the computer game, a second time window during which to not execute the command in accordance with the first non-computer game controller input to the computer game; and not execute the command related to the computer game during the second time window.

3. The device of claim 2, wherein not executing the command comprises not processing the first non-computer game controller input to the computer game.

4. The device of claim 2, wherein not executing the command comprises processing the first non-computer game controller input to the computer game, but not executing the command in accordance with the first non-computer game controller input to the computer game.

5. The device of claim 1, wherein the first non-computer game controller input comprises a gesture input.

6. The device of claim 5, comprising a camera, wherein the gesture input is identified using one or more images from the camera.

7. The device of claim 1, wherein the first non-computer game controller input comprises a voice input.

8. The device of claim 7, comprising a microphone, wherein the voice input is identified using voice data from the microphone.

9. The device of claim 1, wherein the game state data indicates a first context associated with the execution of the computer game, and wherein the at least one processor complex is programmed with instructions to: identify the first time window based on identifying the first context from the game state data.

10. The device of claim 9, wherein the at least one processor complex is programmed with instructions to: provide the game state data as input to a machine learning (ML) model; receive, as output from an activation layer of the ML model, an inference indicating the first context; and identify the first context from the inference. ​ 11. The device of claim 9, wherein the first time window is a dynamic time window.

12. The device of claim 11, wherein the dynamic first time window is always present as long as the first context persists during execution of the computer game.

13. The device of claim 1, wherein the at least one processor complex is programmed with instructions to: provide the game state data as input to a machine learning (ML) model; and receive an inference indicating the first time window as output from an activation layer of the ML model.

14. The device of claim 13, wherein the ML model is trained using at least one dataset of game state data and true time windows during which non-computer game controller inputs were monitored.

15. A method comprising: executing a computer game; receiving game state data related to the execution of the computer game; identifying a first time window during which commands are executed in accordance with first non-computer game controller inputs to the computer game based on the game state data and during the execution of the computer game; identifying the first non-computer game controller inputs during the first time window; and executing the commands related to the computer game based on the identification of the first non-computer game controller inputs during the first time window, the commands being executed in accordance with the first non-computer game controller inputs.

16. The method of claim 15, wherein the first non-computer game controller inputs comprise one or more of: gesture inputs of predetermined gestures, voice inputs of predetermined voice commands.

17. A system comprising: at least one computer medium that is not a transitory signal and includes instructions executable by at least one processor complex to: infer a first time window during which, for a first execution instance of a computer game, non-computer game controller commands to the first execution instance of the computer game are executed, the first time window not spanning all of the first execution instance of the computer game; and execute the non-computer game controller commands during the first time window based on the inference.

18. The system of claim 17, wherein the first time window does not span all of a certain gameplay sequence of a level or episode of the computer game, but does span some of the certain gameplay sequence of the level or episode of the computer game.

19. The system of claim 17, wherein the instructions are executable to infer the first time window using game state data and a machine learning (ML) model. ​ 20. The system of claim 19, wherein the ML model is trained using at least one dataset of game state data, player outcomes from instances of prior game play, and real-time windows of performance of corresponding non-computer game controller commands.

Images