Brain-Computer Interfaces for Computing Systems

Abstract

Various embodiments are directed to using one or more body sensors placed on or near a video game player to obtain biofeedback measurement data that can be used to dynamically adjust the state of video game play. The sensors may or may not be connected to the game player and may replace or augment a traditional physical game controller. The sensors collect various biofeedback measurement data and provide such measurement data to a biofeedback application programming interface (API). Before and / or during video game play, the video game queries the biofeedback API to request inferences regarding the game player's internal state. The response is then used to adjust the state of video game play. If the video game is a multiplayer video game, biofeedback measurement data from other game players may also be obtained and used to further adjust the state of video game play.

Description

[Technical field]

[0001] The present disclosure relates generally to interactive video games, and more particularly, but not by way of limitation, to brain-computer interfaces for computing systems. [Background technology]

[0002] Today, the computer gaming industry is a multi-billion dollar industry. Its growth may be due in part to faster computing devices, higher quality graphics, and higher quality games. Many of today's video games offer a variety of different input / output devices that game players can use to interact with the game. For example, many video games allow players to interact with a keyboard and / or a mouse. Although such input / output controllers allow game players to interact with the game, the game player may not "feel" immersed in the game. Thus, many video games have been redesigned to provide a way to immerse the video game player, such as with game pads, joysticks, trackballs, and game paddles. Some joysticks and / or paddles are configured to resemble a type of device that is in keeping with the video game being played. For example, in some flight simulation games, the joystick may be designed to provide a throttle quadrant, throttle level, throttle wheel, and handheld stick that give the game player the illusion of being seated in the cockpit of an aircraft and flying.

[0003] By adjusting the input device, the video game player becomes more engaged with the video game and therefore more likely to enjoy the video game. Thus, the video game player is more likely to continue playing the game, share the game with others, and perhaps purchase similar games in the future. This trend of adjusting the input device to increase the game player's engagement has become even more evident with the advent of wireless controllers. For example, in one popular video game, the game's input controller is a wireless handheld controller that may include a built-in accelerometer, infrared detector, or similar components. Such components are used to detect the position of the controller in three-dimensional space when aimed at light-emitting diodes (LEDs) in a remote sensor bar. The game player then controls the game using physical gestures or traditional buttons to play games such as bowling, virtual musical instruments, or boxing games.

[0004] However, while many game players may perceive this as providing a higher level of engagement with the video game, other game players may feel that their engagement with the video game is still insufficient.It is with respect to these considerations and more that the present disclosure is made. Summary of the Invention

[0005] A video game device may be summarized as including: one or more physical biofeedback sensors, at least one non-transitory processor-readable storage medium storing at least one of data and instructions, and at least one processor operatively coupled to the at least one non-transitory processor-readable storage medium and the one or more physical biofeedback sensors, where in operation the at least one processor provides a gameplay to a video game player via a user interface providing video game functionality, the gameplay having a plurality of individual components, receives biofeedback measurement data of the video game player from the one or more physical biofeedback sensors while the video game player is playing the video game, processes the biofeedback measurement data to determine a response of the video game player to the plurality of individual components during gameplay of the video game, and adjusts or enhances the gameplay of the video game based at least in part on the determined response of the video game player.

[0006] To process the biofeedback measurement data, the at least one processor may apply at least one of a Fourier transform or a spectral density analysis. The at least one learning model may be trained to determine a particular subset of the plurality of individual components that causes the video game player to have a particular cognitive state. The plurality of individual components may include at least one of a game character, a chat message, a weapon, a character selection, a character action, an event associated with a character, or a characteristic of another video game player. The one or more physical biofeedback sensors may include one or more electroencephalography (EEG) electrodes, and the biofeedback measurement data may include an EEG signal. The one or more physical biofeedback sensors may include one or more electrodes, and the biofeedback measurement data may include a neural signal. The biofeedback measurement data may include at least one of a neural signal, an EEG signal, an EMG signal, an EOG signal, an fNIR signal, a signal indicative of blood flow, a functional near infrared spectroscopy (fNIR) spectroscopic signal, a force sensitive resistor (FSR) signal, a facial expression detection signal, a pupil dilation display signal, an eye movement signal, or a gestural motion signal. The at least one processor may determine a relative weighting of the contribution of each individual component to the determined response. At least one of the one or more physical biofeedback sensors may be incorporated into a head mounted display (HMD) device.

[0007] The video game system may be summarized as including at least one non-transitory processor-readable storage medium storing at least one of data and instructions, and at least one processor operatively coupled to the at least one non-transitory processor-readable storage medium, wherein, in operation, the at least one processor provides game play to a population of video game players via respective user interfaces providing video game functionality, receives biofeedback measurement data of the video game players from a physical biofeedback sensor proximate to the video game players while the video game players are playing the video game, the biofeedback measurement data being captured during a display of a plurality of individual components, analyzes the biofeedback measurement data to determine a subset of the plurality of individual components that contribute to an overall emotion or impression of the population of video game players, and adjusts or enhances the video game in response to the analysis of the biofeedback measurement data.

[0008] The plurality of individual components may include at least one of a character of the game, a chat message, a weapon, a selection of the character, an action of the character, an event associated with the character, or a characteristic of another video game player. The one or more physical biofeedback sensors may include one or more electroencephalography (EEG) electrodes, and the biofeedback measurement data may include an EEG signal. To analyze the biofeedback measurement data, the at least one processor may implement at least one model operable to isolate individual components of the plurality of individual components that contribute to an overall emotion or impression of the video game player. The at least one processor may receive class information for each video game player, and may analyze the biofeedback measurement data and the class information to determine how various classes of video game players respond differently to each individual component of the video game. The at least one processor may estimate an opinion about the video game based on the received biofeedback measurement data. The at least one processor may estimate a life cycle of the video game based on the received biofeedback measurement data. The at least one processor may determine similarities between different portions of the video game based on the received biofeedback measurement data.

[0009] A video game device may be summarized as including: one or more physical biofeedback sensors, at least one non-transitory processor-readable storage medium storing at least one of data and instructions, and at least one processor operatively coupled to the at least one non-transitory processor-readable storage medium and the one or more physical biofeedback sensors, where in operation the at least one processor provides gameplay to a video game player via a user interface providing video game functionality, receives biofeedback measurement data of the video game player from the one or more physical biofeedback sensors while the video game player is playing the video game, processes the biofeedback measurement data to determine an internal state of the video game player during gameplay of the video game, and adjusts or enhances gameplay of the video game based at least in part on the determined internal state of the video game player.

[0010] The at least one processor may utilize the determined internal state to predict that the video game player is likely to stop playing the video game. The at least one processor may utilize the determined internal state to determine the video game player's impression of at least one of a weapon, a character, a map, a game mode, a tutorial, a game update, a user interface, a teammate, or a game environment. The biofeedback measurement data may include at least one of a neural signal, an EEG signal, an EMG signal, an EOG signal, an fNIR signal, a signal indicative of blood flow, a functional near-infrared spectroscopy (fNIR) spectroscopy signal, a force sensitive resistor (FSR) signal, a facial expression detection signal, a pupil dilation display signal, an eye movement signal, or a gesture motion signal. At least one of the one or more physical biofeedback sensors may be incorporated into a head mounted display (HMD) device.

[0011] The video game device may further include a head mounted display (HMD) device that includes at least one of the one or more physical biofeedback sensors.

[0012] A video game device may be summarized as including: one or more physical neurostimulators, at least one non-transitory processor-readable storage medium storing at least one of data and instructions, and at least one processor operatively coupled to the at least one non-transitory processor-readable storage medium and the one or more physical neurostimulators, where in operation, the at least one processor provides game play to a video game player via a user interface that provides video game functionality, and provides neurostimulation to the video game player via the one or more physical neurostimulators while the video game player is playing the video game to provide an enhanced experience to the video game player.

[0013] The neurostimulation may provide at least one of: improving the concentration of the video game player, improving the memory of the video game player, improving the learning ability of the video game player, changing the emotional arousal of the video game player, adjusting the visual cognition of the video game player, or adjusting the auditory cognition of the video game player. The one or more physical neurostimulators may include at least one of a non-invasive neurostimulator or an invasive neurostimulator. The one or more physical neurostimulators may include at least one of a transcranial magnetic stimulation device, a transcranial electrical stimulation device, a microelectrode-based device, or an implantable device. The one or more physical neurostimulators may function to provide at least one of a sensory system stimulation or a motor system stimulation. [Brief description of the drawings]

[0014] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the drawings listed below, in which like reference numerals refer to like parts throughout the various views unless otherwise specified. For a full understanding of the present disclosure, reference is made to the following detailed description, which should be read in connection with the accompanying drawings, in which:

[0015] [Figure 1] FIG. 1 shows a pictorial block diagram illustrating one embodiment of a suitable environment for implementing one or more features of the present disclosure.

[0016] [Diagram 2] 2 illustrates one embodiment of a client device for use in the environment of FIG. 1.

[0017] [Diagram 3] 2 illustrates one embodiment of a network device for use in the environment of FIG. 1.

[0018] [Figure 4] 1 illustrates a flow chart of one embodiment of a process for adjusting gameplay states of a video game using biofeedback measurements from a game player.

[0019] [Diagram 5] 1 illustrates a flow chart of one embodiment of a process for analyzing biofeedback measurement data from a game player for use in a video game.

[0020] [Figure 6] 1 illustrates one embodiment of a non-exhaustive, non-limiting example of queries that may be used to query a biofeedback application programming interface (API) for biofeedback measurement data.

[0021] [Figure 7]1 illustrates one embodiment of a non-exhaustive, non-limiting example of using biofeedback measurement data for use in adjusting game play conditions in an arena combat video game.

[0022] [Figure 8] 1 illustrates one embodiment of a non-exhaustive, non-limiting example of using biofeedback measurement data for use in adjusting gameplay states in a space video game.

[0023] [Figure 9] 1 illustrates a flow chart of one embodiment of a process for dynamically adjusting or enhancing gameplay of a video game based on tracking of a video game player's gaze position.

[0024] [Figure 10] 1 shows a flow chart of one embodiment of a process for detecting upcoming movements of a user of a user interface.

[0025] [Figure 11] 1 illustrates a flowchart of one embodiment of a process for updating or training a model that functions to detect upcoming movements of a user of a user interface.

[0026] [Figure 12] FIG. 1 shows a pictorial block diagram illustrating one embodiment of a suitable environment for implementing one or more features of the present disclosure.

[0027] [Figure 13] 1 shows a flow chart of one embodiment of a process for adapting a user interface to ameliorate a user's difficulty in operating the user interface by analyzing biofeedback measurement data.

[0028] [Figure 14]1 illustrates a flowchart of one embodiment of a process for analyzing biofeedback measurement data from a user operating a video game device to ascertain the user's responses to multiple individual components during gameplay of a video game.

[0029] [Figure 15] 1 illustrates a flow chart of one embodiment of a process for analyzing biofeedback measurement data from a population of users operating a video game system to adjust or enhance the video game.

[0030] [Figure 16] 1 illustrates a flow chart of one embodiment of a process for analyzing biofeedback measurement data from a user operating a video game system to ascertain the user's internal state and adjust or enhance the video game.

[0031] [Figure 17] 1 illustrates a flow chart of one embodiment of a process for providing neurostimulation to a user during video game play on a video game system to enhance the user's gaming experience.

[0032] [Figure 18] FIG. 1 is an illustrative diagram showing non-limiting exemplary mechanisms for inducing, writing, or generating signals in a user's (e.g., video game player's) brain to enhance the user's experience.

[0033] [Figure 19] FIG. 1 is an illustrative diagram showing various potential features of a brain-computer interface (BCI) according to embodiments of the present disclosure.

[0034] [Figure 20] FIG. 1 illustrates the inputs that cause neurons to fire, including sensory perception, internal perception, and external influences.

[0035] [Figure 21] FIG. 1 illustrates a BCI having various features of the present disclosure that may be implemented to provide an enhanced experience to a video game player. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] One or more implementations of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, which form a part of this disclosure and which illustrate certain exemplary embodiments. However, implementations of the present disclosure may be embodied in many different forms and should not be construed as being limited to the embodiments described herein, but rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. In particular, one or more implementations may be embodied as a method or device. Thus, these embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Thus, the following detailed description should not be taken in a limiting sense.

[0037] Throughout this specification and claims, the following terms have the meanings clearly associated therewith, unless the context clearly dictates otherwise. The phrase "in one embodiment" as used herein does not necessarily refer to the same embodiment, but may. Additionally, the phrase "in another embodiment" as used herein does not necessarily refer to different embodiments, but may. Thus, as described below, various embodiments may be readily combined without departing from the scope or spirit of the present disclosure.

[0038] Additionally, as used herein, the term "or" is an inclusive "or" operator and is equivalent to the term "and / or," unless the context clearly indicates otherwise. The term "based on" is not exclusive and allows for other factors not stated, unless the context clearly indicates otherwise. Additionally, throughout this specification, the meanings of "a," "an," and "the" include the plural. The meaning of "in" includes "in" and "on."

[0039] As used herein, the terms "biofeedback" and "physiological" refer to measurement data of certain quantifiable body functions of a game player. Such biofeedback measurement data are also commonly referred to as measurements of unconscious or involuntary body functions. Such biofeedback measurement data may include, but are not limited to, blood pressure, heart rate, eye movement, pupil dilation, skin temperature, sweat gland activity, muscle tone, neuronal activity, and other measurement data discussed herein. As further described herein, such measurement data can be used to make inferences regarding the game player's state of arousal or emotional state. It is noted that state of arousal includes physiological states as well as emotional states. Furthermore, as used herein, state of arousal further includes determination of engagement, valence, and / or other user states based on physiological measurements. As used herein, brain-computer interface (BCI) refers to a communication pathway that converts neuronal signals into actionable inputs for an external system.

[0040] The following is a sequential, simplified description of several embodiments in order to provide a basic understanding of some aspects of the disclosure. This brief description is not intended to be an extensive overview, nor is it intended to identify key or critical elements, nor is it intended to be delineating or limiting in scope. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0041] Briefly, various embodiments are directed to obtaining biofeedback measurement data about a game player using one or more body sensors placed on or near the video game player that can be used to dynamically adjust video game play or provide other functions. In one embodiment, such adjustments may be made substantially in real time. In another embodiment, such adjustments may be made for use in the next game play. The body sensors may be connected to the game player and, in some implementations, may replace and / or otherwise augment a conventional physical game controller. In another embodiment, the body sensors need not be connected to the game player but may instead be placed near the game player. Non-limiting examples of such non-physically connected sensors include video cameras, index tracking systems, weight / position sensor pads on which the game player may stand, and the like. These sensors are positioned to collect various biofeedback measurement data, such as cardiac activity, galvanic skin response, body temperature, eye movement, head or other body movement, and to provide such measurement data to a biofeedback application programming interface (API). Prior to and / or during video game play, the video game may query the biofeedback API for inferences regarding the game player's state of emotional arousal, affective state, or cognitive state, etc., based on biofeedback measurement data, as further described below. The video game then adjusts the state of video game play based on the response to this query. In this manner, the video game may determine whether the game player's current physiological state is consistent with the type and / or level of experience the video game may seek to provide. For example, if the game player's state of stress or emotional arousal is determined to be above a given threshold, the video game may adjust the state of game play to provide the game player an opportunity to unwind and / or decompress.In another embodiment, if the game player's state of stress or emotional arousal is determined to be below another threshold, the video game may adjust gameplay conditions to increase the game player's level of excitement.

[0042] In one embodiment, the threshold may be based on past biofeedback measurement data and / or inferences regarding a particular game player. In another embodiment, the threshold may be based on an analysis of a particular game player relative to current video game play. In yet another embodiment, the threshold may be based on a statistical analysis of multiple game players.

[0043] In one embodiment, if the video game is configured as a multiplayer video game, biofeedback measurement data from other game players may also be obtained and used to further adjust the state of the video game play.

[0044] [Example operating environment]

[0045] FIG. 1 shows a block diagram that broadly illustrates an overview of one embodiment of a system in which one or more features of the present disclosure may be implemented. The system 100 may include fewer or more components than those shown in FIG. 1. However, the components shown are sufficient to disclose an exemplary embodiment. As shown in the figure, the system 100 includes a local area network ("LAN") / wide area network ("WAN"), i.e. (network) 105, a wireless network 111, a client device 101, a game server device (GSD) 110, and a biofeedback sensor 120.

[0046] One embodiment of a client device that can be used as client device 101 is described in more detail below in connection with FIG. 2. Briefly, however, client device 101 may include substantially any mobile computing device capable of sending and receiving messages over a network, such as network 111. Such devices include portable devices such as radio frequency (RF) devices, infrared (IR) devices, personal digital assistants (PDAs), game consoles, handheld computers, laptop computers, wearable computers, tablet computers, or all-in-one devices that combine one or more of the foregoing devices. Client device 101 may also include substantially any computing device, such as a personal computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics device, or a network PC, that typically connects using a wired communication medium, such as network 105. Thus, in one embodiment, client device 101 may be configured to operate over a wired network and / or a wireless network.

[0047] Client devices 101 typically span a wide range in terms of capabilities and functionality. For example, a handheld device may have a numeric keypad and a few lines of a monochrome LCD display capable of displaying only text. In another example, a web-enabled client device may have a touch-sensitive screen, a stylus, and a few lines of a color LCD display capable of displaying both text and graphics.

[0048] A web-enabled client device may include a browser application configured to receive and send web pages, web-based messages, or the like. The browser application may be configured to receive and display graphics, text, multimedia, or the like using virtually any web-based language, including wireless application protocol messages (WAP), and the like. In one embodiment, the browser application is capable of displaying and transmitting information using Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SGML), Hypertext Markup Language (HTML), Extensible Markup Language (XML), or the like.

[0049] The client device 101 may also include at least one application configured to receive content from another computing device. The application may include functionality to provide and receive text content, multimedia information, or components to a computer application such as a video game. The application may further provide information that identifies itself (including type, function, or name, etc.). In one embodiment, the client device 101 may uniquely identify itself by any of a variety of mechanisms including a telephone number, a mobile identification number (MIN), an electronic serial number (ESN), a mobile device identifier, a network address, or other identifier. The identifier may be provided, such as in a message sent to another computing device.

[0050] The client device 101 may be configured to communicate messages to and from another computing device via email, short message service (SMS), multimedia message service (MMS), instant messaging (IM), Internet Relay Chat (IRC), Mardam-Bey's IRC (mIRC), Jabber, or the like. However, the present disclosure is not limited to these message protocols and virtually any other message protocol may be used. Thus, in one embodiment, the client device 101 may enable a user to participate in one or more messaging sessions, such as a chat session or a gaming with messaging session. Such messaging sessions may be text-oriented in that communication is accomplished using text. However, other messaging sessions may occur using the client device 101 using other communication mechanisms, including, but not limited to, audio, graphics, video, and / or a combination of text, audio, graphics, and / or video.

[0051] The client device 101 may be configured to receive messages, images, and / or other biofeedback measurement data from various biofeedback sensors 120. Non-limiting and non-exhaustive examples of possible physical biofeedback sensors 120, which may or may not be connected to a user, are shown in FIG. 1, which replace and / or otherwise augment a conventional physical game controller. Thus, as shown, the biofeedback sensors 120 may be integrated into the game controller (sensors 123) or into one or more keys or wheels on a keyboard, etc. (sensors 124). In one embodiment, the game controller may include modular and / or pluggable components, which may include modular and / or pluggable sensors (123).

[0052] Similarly, biofeedback sensors 120 may include cameras 121, touch pads 122, and even head devices 125 (e.g., incorporated into a head mounted display (HMD) device). However, as previously mentioned, other biofeedback sensors 120 may also be used, including glasses, wristbands, finger sensor attachments, sensors integrated within or on a computer mouse, or microphones for measuring various voice patterns, etc. Thus, it should be apparent to one skilled in the art that various embodiments may use virtually any mechanism configurable to obtain biofeedback measurement data of a game player.

[0053] The biofeedback sensor 120 may be positioned to collect various measurement data of the game player before, after, and / or during video game play. Such measurement data may include, but is not limited to, heart rate and / or heart rate variability, galvanic skin response, body temperature, eye movement, head, face, hand, or other body movement, gestures, position, facial expression, posture, or facial tension, etc. Additionally, the biofeedback sensor 120 may collect other measurement data including blood oxygen concentration, other forms of skin conductance level, respiration rate, skin tension, voice stress level, voice recognition, blood pressure, electroencephalography (EEG) measurement data, electromyography (EMG) measurement data, response time, electrooculography (EOG), blood flow (e.g., by an IR camera), functional near infrared spectroscopy (fNIR) spectroscopy, or force sensitive resistor (FSR), etc.

[0054] The biofeedback sensor 120 may provide these measurement data to the client device 101. In one embodiment, these measurement data may be provided to the client device 101 via any of a variety of wired and / or wireless connections. Thus, the biofeedback measurement data may be transmitted via various cables or wires, and other information may also be transmitted using various cables or wires for gameplay. For example, the biofeedback measurement data may be transmitted via a USB cable or a coaxial cable, and a mouse, keyboard, or game controller, etc. may also be coupled to the client device 101 using a USB cable or a coaxial cable, etc. However, in other embodiments, other wired connections may be used. Similarly, the biofeedback sensor 120 may transmit the biofeedback measurement data using various wireless connections. Furthermore, any of a variety of communication protocols may be used to transmit the measurement data. Thus, the present disclosure should not be construed as limited to any particular wired or wireless communication mechanism and / or communication protocol.

[0055] In one embodiment, the client device 101 may include a biofeedback device interface (BFI) configured to determine whether one or more body sensors 120 are available and to manage the receipt of biofeedback measurement data from the body sensors 120. One embodiment of the BFI is described in more detail below in connection with FIG. 2. Briefly, however, the BFI may further add a timestamp to the received biofeedback measurement data, buffer at least some of the measurement data, and / or forward the measurement data to the GSD 110 for use in adjusting the state of current or future video game play. Buffering the received biofeedback measurement data may enable the BFI to perform a quality analysis of the received measurement data and issue an alert message based on the analysis results.

[0056] Wireless network 111 is configured to couple client device 101 to network 105. Wireless network 111 may include any of a variety of wireless sub-networks that may further overlay standalone, ad-hoc networks, and the like, to provide infrastructure-oriented connectivity to client device 101. Such sub-networks may include mesh networks, wireless LAN (WLAN) networks, cellular networks, and the like.

[0057] Wireless network 111 may further include an autonomous system of terminals, gateways, routers, etc., connected by wireless radio links, etc. These connected devices may be configured to move freely and randomly and organize themselves arbitrarily, such that the topology of wireless network 111 may change rapidly.

[0058] The wireless network 111 may further use multiple access technologies including second generation (2G), third generation (3G), fourth generation (4G) wireless access such as for cellular systems, WLAN, or wireless router (WR) mesh. Access technologies such as 2G, 2.5G, 3G, 4G, and future access networks may enable wide area coverage for client devices such as the client device 101 with various degrees of mobility. For example, the wireless network 111 may enable wireless connectivity through wireless network access such as Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), or Bluetooth, etc. In essence, the wireless network 111 may include virtually any wireless communication mechanism by which information may travel between the client device 101 and another computing device or network, etc.

[0059] The network 105 is configured to couple computing devices such as the GSD 110 to other computing devices, possibly including coupling to the client device 101 through a wireless network 111. However, as shown, the client device 101 may also be connected to the GSD 110 through the network 105. In any case, the network 105 may use any form of computer-readable media to convey information from one electronic device to another. Also, the network 105 may include the Internet, as well as a local area network (LAN), a wide area network (WAN), a direct connection (such as through a universal serial bus (USB) port), other forms of computer-readable media, or any combination thereof. In a collection of interconnected LANs, including LANs based on different architectures and protocols, a router acts as a link between the LANs, allowing messages to be sent from one to another. Additionally, while LAN-to-LAN communication links typically include twisted pair wires or coaxial cable, network-to-network communication links may utilize analog telephone lines, fully or partially dedicated digital lines (including T1, T2, T3, and T4), Integrated Services Digital Networks (ISDN), Digital Subscriber Lines (DSL), wireless links (including satellite links), or other communication links known to those skilled in the art. Additionally, remote computers and other associated electronic devices may be remotely connected to either the LAN or WAN via modems and temporary telephone links. In essence, network 105 includes any communication method by which information may travel between computing devices.

[0060] One embodiment of the GSD 110 is described in more detail below in connection with Figure 3. Briefly, however, the GSD 110 may include any computing device capable of connecting to a network 105 to enable a user to participate in one or more online games (including but not limited to multiplayer games, including single player games). Thus, although Figure 1 illustrates a single client device 101 having a biofeedback sensor 120, the disclosure is not so limited and multiple similar client devices having biofeedback sensors may be deployed in the system 100.

[0061] Thus, the GSD 110 is configured to receive various biofeedback measurement data from one or more game players and adjust the state of the video game using the received measurement data. The GSD 110 may use the biofeedback to dynamically adjust the difficulty of the game play and / or other aspects of the video game based on the biofeedback measurement data. For example, in one embodiment, if it is determined that the user is experiencing a level of stress that is deemed excessive, based on a certain threshold, the video game of the GSD 110 may provide an alternative gameplay to alleviate the determined stress level.

[0062] The GSD 110 may also enable a video game to provide a special experience based on the game player's biofeedback measurement data each time the video game is played. For example, in one embodiment, the color of an object or the size, shape, and / or behavior of a character in the game may be adjusted based on the biofeedback measurement data. That is, various aspects of the scenery displayed in the game background may be adjusted based on an analysis of the biofeedback measurement data.

[0063] In one embodiment, past measurements may be stored and analyzed to enable the GSD 110 to detect a particular game player or to tailor current game play for a particular game player. Such stored measurements may then be used to customize game play for a particular game player, identify changes in game play by a particular game player based on determined trend determinations, etc. In one embodiment, past measurements may be used in conjunction with analysis of biofeedback measurement data to determine whether a game player is still associated with a previous user profile, i.e., whether the current game player is someone who has played before. The GSD 110 may also tailor the type of game play provided based on a determination of the game player's level of engagement during game play, or based on past patterns, etc.

[0064] The GSD 110 may further assign opponents based in whole or in part on the physiological or emotional state of a game player who may be requesting a multiplayer game session. In yet other embodiments, the GSD 110 may dynamically adjust game play instructions, tutorials, etc. based on the received biofeedback measurement data. For example, if it may be determined that a game player is bored or not interested in instructions, tutorials, etc., the GSD 110 may enable the display of the material to be sped up, skipped, etc. Alternatively, if it may be determined that a game player is confused or having trouble making a decision based on the biofeedback measurement data, a tutorial or other guidance may be provided to assist the game player.

[0065] However, the GSD 110 is not limited to these examples of how the biofeedback measurement data may be used, and other ways of using the biofeedback measurement data to adjust gameplay states may also be used. For example, the biofeedback measurement data may be used to directly control aspects of gameplay. One such non-limiting example is described in more detail below in connection with FIG. 8.

[0066] In yet other embodiments, the GSD 110 may depict the game player's emotions, physiological state, and / or other aspects of the game player's facial expression in a character depicted in the game. For example, based on the received game player biofeedback measurement data, the game player's avatar may be adjusted to display a heart beating at the game player's heart rate, the avatar may be depicted breathing at the game player's rate, or sweating, or even exhibiting facial expressions or postures. Thus, the GSD 110 may use the biofeedback measurement data in any of a variety of ways to adjust the state of game play.

[0067] Devices that may operate as a GSD 110 include personal computers, desktop computers, multiprocessor systems, video game consoles, microprocessor-based or programmable consumer electronics devices, network PCs, server devices, and the like.

[0068] Further, although the GSD 110 is shown as a single network device, the disclosure is not so limited. For example, one or more of the functions associated with the GSD 110 may be implemented in multiple different network devices distributed throughout a peer-to-peer system architecture, or the like, without departing from the scope or spirit of the disclosure. Thus, as described below in connection with FIG. 3, a network device 300 may be configured to manage game play using biofeedback measurement data to adjust game state. However, other configurations are contemplated.

[0069] For example, in another embodiment, the client device 101 may be configured to include components of the GSD 110 such that the client device 101 may operate independently of the GSD 110. That is, in one embodiment, the client device 101 may include gaming software with biofeedback and a biofeedback application programming interface (API), etc., and operate without a network connection to the GSD 110. Thus, the client device 101 may operate as a substantially standalone gaming device with interfaces to biofeedback sensors and other input / output devices for the user's enjoyment. Thus, the present disclosure is not constrained or limited by the configurations shown in these figures.

[0070] 1 illustrates a single client device 101 with one game player and a "single set" of biofeedback sensors 120, other embodiments are contemplated. For example, in one embodiment, multiple game players, each with their own biofeedback sensors, may interact and play the same video game together through the same client device 101 or through multiple client devices connected together over a network. Thus, a multiplayer configuration may include variations such as multiple game players using the same or different client devices. Thus, FIG. 1 should not be construed as limited to a single game player configuration.

[0071] [Exemplary Client Device]

[0072] 2 illustrates one embodiment of a client device 200 that may be included in a system implementing the present disclosure. Client device 200 may include more or fewer components than those illustrated in FIG. 2. For example, client device 200 may be configured with a reduced set of components for use as a standalone video game device. However, the components illustrated are sufficient to disclose an exemplary embodiment. Client device 200 may represent, for example, client device 101 of FIG. 1.

[0073] As shown in FIG. 2, the client device 200 includes a processing unit (CPU) 222 in communication with a mass memory 230 via a bus 224. The client device 200 also includes a power source 226, one or more network interfaces 250, an audio interface 252 that may be configured to receive audio input and provide audio output, a display 254, a keypad 256, an illuminator 258, an input / output interface 260, a haptic interface 262, and a global positioning system (GPS) receiver 264. The power source 226 provides power to the client device 200. A rechargeable or non-rechargeable battery may be used to provide power. Power may also be provided from an external power source, such as an AC adapter or a powered docking cradle that supplements and / or charges the battery. The client device 200 may also include a graphical interface 266 that may be configured to receive image input, such as through a camera or scanner.

[0074] Network interface 250 includes circuitry that interfaces client device 200 with one or more networks and is configured for use with one or more communication protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), SMS, General Packet Radio Service (GPRS), WAP, Ultra Wideband (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMAX), SIP / RTP, Bluetooth, Wi-Fi, Zigbee, UMTS, HSDPA, WCDMA, WEDGE, or any of a variety of other wired and / or wireless communication protocols. Network interface 250 may be known as a transceiver, a transceiver device, or a network interface card (NIC).

[0075] Audio interface 252 is provided for generating and receiving audio signals, such as the sound of a human voice. For example, audio interface 252 may be coupled to a speaker and microphone (not shown) to enable remote communication with others and / or to generate audio acknowledgments for some actions. Display 254 may be a liquid crystal display (LCD), a gas plasma display, a light emitting diode (LED) display, or any other type of display used with computing devices. Display 254 may also include a touch-sensitive screen arranged to receive input by an object such as a stylus or the finger of a person's hand.

[0076] The keypad 256 may include any input device arranged to receive input from a user. For example, the keypad 256 may include a push button numeric dial or keyboard. The keypad 256 may also include command buttons associated with the selection and transmission of images, game play, messaging sessions, etc. In one embodiment, the keypad 256 may include various biofeedback sensors arranged to obtain various measurement data including, but not limited to, the amount of pressure, the length of response time, the amount of sweating, etc.

[0077] The illuminator 258 may provide a status indication and / or provide illumination. The illuminator 258 may remain enabled for a particular period of time or in response to an event. For example, when the illuminator 258 is enabled, it may backlight the buttons of the keypad 256 and remain in that state while the client device is powered. Also, when certain actions are taken, such as dialing another client device, the illuminator 258 may backlight those buttons in various patterns. In response to an action, the illuminator 258 may illuminate a light source disposed in a transparent or translucent case of the client device.

[0078] The client device 200 also has an input / output interface 260 for communicating with external devices such as a headset or other input or output devices (including but not limited to a joystick or mouse, etc.). As described above in connection with FIG. 1, the client device 200 may be configured to communicate with one or more biofeedback sensors through the input / output interface 260. The input / output interface 260 may utilize one or more communication technologies such as USB, infrared, or Bluetooth, etc. The haptic interface 262 is arranged to provide haptic feedback to a user of the client device. For example, the haptic interface may be used to make the client device 200 vibrate in a particular manner when a call is coming in from another user of the computing device.

[0079] The GPS transceiver 264 can measure the physical coordinates of the client device 200 on the surface of the Earth. This GPS transceiver typically outputs a location as latitude and longitude values. The GPS transceiver 264 can also use other geographic positioning mechanisms (including but not limited to triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, or BSS, etc.) to further measure the physical location of the client device 200 on the surface of the Earth. It should be understood that in other circumstances, the GPS transceiver 264 can measure the physical location of the client device 200 to within millimeters, and in other cases, the measured physical location can be an inaccurate measurement, such as within a meter or a significant distance away. However, in one embodiment, the client device 200, through other components, may provide other information (including, for example, a MAC address, IP address, or other network address) that can be used to ascertain the geographic physical location of the device.

[0080] Mass memory 230 includes RAM 232, ROM 234, and / or other storage devices. Mass memory 230 represents another example of a computer storage medium for storing information such as computer readable instructions, data structures, program modules, or other data. Mass memory 230 stores a basic input / output system ("BIOS") 240 that controls the low level operation of client device 200. Mass memory also stores operating system 241 that controls the operation of client device 200. It will be appreciated that this component may include a general purpose operating system such as a UNIX or Linux version, or a dedicated client communications operating system such as Windows Mobile, Symbian operating systems, or any of the operating systems for various video game consoles. The operating system may include or interface with a Java virtual machine module that allows control of hardware components and / or operating system operation via Java application programs.

[0081] The memory 230 further includes one or more data stores 244 that may be utilized by the client device 200 to store, among other things, applications and / or other data. For example, the data storage 244 may be used to store information describing various capabilities of the client device 200, a device identifier, and the like. This capability information may also be provided to another device based on any of a variety of events, including as part of a header in a communication, or upon request, and the like. The data storage 244 may be used to buffer one or more measurement data received from a biofeedback sensor.

[0082] In one embodiment, data storage 244 may also include cookies, portions of computer applications, user settings, game play data, messaging data, and / or other digital content. At least a portion of the stored data may also be stored on optional hard disk drive 272, optional portable storage medium 270, or other storage medium (not shown) within client device 200.

[0083] Applications 242 may include computer-executable instructions that, when executed by client device 200, send, receive, and / or otherwise process messages (e.g., SMS, MMS, IMS, IM, email, and / or other messages), audio, video, and enable remote communication with other users of other client devices. Other examples of application programs include calendars, browsers, email clients, IM applications, VOIP applications, contact managers, task managers, database programs, word processing programs, security applications, spreadsheet programs, search programs, and the like. Applications 242 may further include a browser 245, a messenger 243, a game client 248, and a biofeedback device interface (BFI) 249.

[0084] Messenger 243 may be configured to initiate and manage messaging sessions using any of a variety of messaging communications including, but not limited to, email, short message service (SMS), instant message (IM), multimedia message service (MMS), Internet Relay Chat (IRC), mIRC, or VOIP. For example, in one embodiment, messenger 243 may be configured as an IM application such as AOL Instant Messenger, Yahoo! Messenger, .NET Messenger Server, or ICQ. In one embodiment, messenger 243 may be configured to include a mail user agent (MUA) such as Elm, Pine, MH, Outlook, Eudora, Mac Mail, or Mozilla Thunderbird. In another embodiment, messenger 243 may be a client application configured to integrate and use a variety of messaging protocols. Additionally, messenger 243 may be configured to simultaneously manage multiple messaging sessions, allowing a user to communicate with multiple different other users in different messaging sessions and / or the same messaging session. As used herein, the term "active messaging session" refers to a messaging session in which a user can communicate with another user without having to restart and / or re-establish the messaging session. Thus, maintaining a messaging session actively indicates that the messaging session has been established and has not been terminated, or is otherwise in a sleep or other dormant mode such that messages cannot be actively sent and / or received.

[0085] Browser 245 may include virtually any client application configured to receive and display graphics, text, multimedia, and the like, and to use virtually any web-based language. In one embodiment, the browser application is capable of displaying and sending messages using Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SGML), Hypertext Markup Language (HTML), Extensible Markup Language (XML), and the like. However, any of a variety of other web-based languages ​​may also be used.

[0086] Game client 248 represents a game application component configured to enable a user to select one or more games to play, register for access to the one or more games, and / or launch the one or more games for online interactive play. In one embodiment, game client 248 may establish communication with a network device, such as GSD 110, over a network to enable registration, purchase, access, and / or play of one or more computer games.

[0087] The game client 248 may receive instructions from a user to launch a computer game via various user input devices, including but not limited to those described above. The game client 248 may then enable communication of game data between the client device 200, the GSD 110, another client device, etc.

[0088] In one embodiment, game client 248 represents a computer game application. However, game client 248 is not limited to game applications and may represent substantially any interactive computer application or other interactive digital content. Thus, although biofeedback measurement data is described herein as being used to adjust the state of video game play, the disclosure should not be construed as being limited to video game play, and other applications may also be adjusted. For example, displays, tutorials, etc. may be adjusted based on the biofeedback measurement data.

[0089] Thus, in one embodiment, game engine 248 represents a client component that can be used to enable online multi-user gameplay and / or single game player use. Non-exhaustive, non-limiting examples of such computer games include, but are not limited to, Half-Life, Team Fortress, Portal, Counter-Strike, Left 4 Dead, and Day of Defeat, developed by Valve Corporation (Bellevue, Washington, USA).

[0090] The BFI 249 is configured to detect the connection of one or more biofeedback sensors and collect measurement data received from such sensors. In one embodiment, the BFI 249 may provide information to the remote network device and / or the game client 248 indicating that a connection with a biofeedback sensor has been detected. The BFI 249 may further buffer at least a portion of the received measurement data. In another embodiment, the BFI 249 may instead choose not to buffer the received measurement data and provide it to the remote network device substantially in real time. In one embodiment, the BFI 249 may convert the measurement data into a format and / or protocol that can be used to communicate the measurement data to the remote network device over the network. In another embodiment, the BFI 249 may choose not to communicate the measurement data over the network, such as when the client device 200 may be configured as a standalone video game console. In one embodiment, the BFI 249 may add a timestamp to the received measurement data so that the measurement data can be easily associated. Additionally, the BFI 249 may provide a sensor source identifier to the measurement data so that the measurement data can be differentiated based on its sensor source.

[0091] The BFI 249 may further perform one or more analyses on the received measurement data to determine whether a sensor is showing an erroneous indication, has lost connectivity, or the like. Such a determination may be based on a comparison of multiple received measurement data over time for a given sensor to detect a change from an expected range of values ​​of the received measurement data. For example, if the BFI 249 detects that the sensor measurement is a heart rate sensor, and the measurement data indicates, for example, a heart rate of 2 beats per minute or as many as 100 beats per second, the BFI 249 may determine that the sensor measurement data is erroneous. However, it should be apparent that the BFI 249 may use other range values ​​and is not bound by these exemplary range values. Additionally, the BFI 249 may use different range values ​​for different sensors. In one embodiment, the BFI 249 may provide the determined erroneous measurement data over the network for at least a certain period of time under the assumption that the game player is temporarily calibrating the sensor. However, in another embodiment, if a sensor is determined to be erroneous for more than a certain period of time, the BFI 249 may choose to cease transmitting measurement data and / or send a message to a remote network device.

[0092] 1, client device 200 may be configured to include components of network device 300 (described below in connection with FIG. 3), including a biofeedback API and a game server component, etc. In such an embodiment, client device 200 may operate substantially as a standalone game console without communicating with network device 300. In such a configuration, client device 200 may be referred to as a standalone video game device.

[0093] [Example Network Device]

[0094] 3 illustrates one embodiment of a network device according to one embodiment. The network device 300 may include more or fewer components than shown. However, the components shown are sufficient to disclose an exemplary embodiment. The network device 300 may represent, for example, the GSD 110 of FIG. 1.

[0095] Network device 300 includes a processing unit 312, a video display adapter 314, and mass memory, all of which communicate with each other via a bus 322. The mass memory generally includes RAM 316, ROM 332, one or more persistent mass storage devices (such as a hard disk drive 328), and a removable storage device 326, which may represent a tape drive, an optical drive, and / or a floppy disk drive. The mass memory stores an operating system 320 that controls the operation of network device 300. Any general-purpose operating system may be used. A basic input / output system ("BIOS") 318 is also provided to control low-level operation of network device 300. As shown in FIG. 3, network device 300 can also communicate with the Internet or some other communication network via a network interface unit 310. The network interface unit is configured for use with a variety of communication protocols, including TCP / IP protocol, Wi-Fi, Zigbee, WCDMA, HSDPA, Bluetooth, WEDGE, EDGE, or UMTS, etc. The network interface unit 310 may be known as a transceiver, a transceiver device, or a network interface card (NIC).

[0096] The mass memory mentioned above illustrates another type of computer-readable medium, i.e., computer storage media. Computer-readable storage media may include volatile, non-volatile, removable, and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include memory devices, such as RAM, ROM, EEPROM, or flash memory, optical storage devices, such as CD-ROMs or digital versatile disks (DVDs), magnetic storage devices, such as magnetic cassettes, magnetic tapes, or magnetic disk storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computing device.

[0097] The mass memory also stores program code and data. In one embodiment, the mass memory may include a data store 356. The data store 356 includes substantially any component configured and arranged to store data including, but not limited to, game player settings, game play state and / or other game play data, messaging data, and biofeedback measurement data. The data store 356 also includes substantially any component configured and arranged to store and manage digital content such as computer applications and video games. Thus, the data store 356 may be implemented using a database, file, directory, or the like. At least a portion of the stored data may also be stored on the hard disk drive 328, a portable device (such as the CD-ROM / DVD-ROM drive 326), or other storage medium (not shown) included in the network device 300, or may be stored remotely on yet another network device.

[0098] One or more applications 350 are loaded into the mass memory and run on the operating system 320. Examples of application programs may include transcoders, schedulers, calendars, database programs, word processing programs, HTTP programs, customizable user interface programs, IPSec applications, computer games, encryption programs, security programs, VPN programs, SMS message servers, IM message servers, email servers, account management, etc. Applications 350 may also include web services 346, a message server 354, a game server with biofeedback (GSB) 352, and a biofeedback API (BAPI) 353.

[0099] Web service 346 represents any of a variety of services configured to provide content to another computing device over a network. Thus, Web service 346 may include, for example, a web server, a messaging server, a File Transfer Protocol (FTP) server, a database server, or a content server. Web service 346 may provide content over a network in any of a variety of formats, including, but not limited to, WAP, HDML, WML, SGML, HTML, XML, cHTML, or xHTML.

[0100] Message server 354 may include substantially any computing component or group of computing components configured and arranged to manage messages from message user agents and / or other message servers or to deliver messages to message applications (i.e., to each other's network devices). Message server 354 is not limited to a particular type of messaging. Thus, message server 354 may provide functionality for such messaging services, including, but not limited to, email, SMS, MMS, IM, IRC, mIRC, Jabber, VOIP, and / or a combination of one or more messaging services.

[0101] The GSB 352 is configured to manage the delivery and playing of a video game using biofeedback information obtained from one or more client devices, such as the client device 101 of FIG. 1. Typically, the GSB 352 may provide components for an application, such as a game application, to the client device over a network. In one embodiment, at least one of the provided components is encrypted using any of a variety of encryption mechanisms. For example, in one embodiment, Crypto++, an open source class library of encryption techniques, is used to encrypt and decrypt the components of the application. However, virtually any other encryption and decryption mechanism may be used.

[0102] The GSB 352 may also receive and / or authenticate requests from client devices to access applications. The GSB 352 may facilitate purchase of applications, such as computer games, enable registration for play of the applications, and / or enable download access for the applications.

[0103] The GSB 352 may further enable communication between client devices participating in a multiplayer application by receiving and / or providing various data, messages, and the like between the client devices.

[0104] The GSB 352 may query a biofeedback API (BAPI) 353 for information regarding the state or emotional arousal of one or more game players and / or other information regarding the game players. The GSB 352 may then adjust the state of the video game play based on responses to the received queries. Non-limiting and non-exhaustive examples of queries that the GSB 352 may pose to the BAPI 353 are described below in connection with FIG. 6. Non-limiting and non-exhaustive examples of possible ways in which video game play may be adjusted are described below in connection with FIGs. 7-8. In one embodiment, the GSB 352 may generally perform at least some of its actions using processes such as those described below in connection with FIGs. 5-6.

[0105] The BAPI 353 is configured to perform various analyses on the received biofeedback measurement data and provide responses to various queries from the GSB 352. In one embodiment, the BAPI 353 may collect and store the received biofeedback measurement data in a data store 356, or examine historical data collected and analyzed over a period of time to perform the analysis, etc., to enable data analysis to be performed. In one embodiment, the BAPI 353 may perform at least some analyses on the received biofeedback measurement data substantially in real time, i.e., at least some analyses are performed on the measurement data as soon as the BAPI 353 receives the measurement data.

[0106] As previously mentioned, the BAPI 353 may receive biofeedback measurement data from a variety of different biofeedback sensors, including but not limited to those described above in connection with FIG. 1. In one embodiment, the received measurement data may be identified as a sensor source, such as a heart rate sensor or a galvanic skin sensor.

[0107] As previously mentioned, the BAPI 353 may perform analysis on the received measurement data. For example, the BAPI 353 may receive “raw” biofeedback measurement data and, based on the measurement data, ascertain a heart rate from the measurement data. In another embodiment, the BAPI 353 may use one or more measurement data to ascertain other physiological information about the associated game player. For example, the BAPI 353 may calculate heart rate variability from the measurement data of the cardiac sensor. Similarly, the BAPI 353 may calculate a standard deviation of heart rate activity over a period of time to ascertain trends in heart rate over time and / or ascertain other heart rate patterns. The BAPI 353 may analyze the frequency spectrum of the heart rate data (e.g., including decomposing the heart rate cycle into various frequencies using a Fourier transform or similar analysis technique). The BAPI 353 may also use various measurement data to ascertain other physiological information about the game player (including, but not limited to, respiration rate, relaxation level, or fight-or-flight response data, etc.).

[0108] The BAPI 353 may store the results of the analysis for use in subsequent gameplay or may review and use the results substantially in real-time. The BAPI 353 may further perform various recalibration activities, including incremental recalibration activities, etc. In one embodiment, recalibration activities may be performed on the sensors and / or to account for physiological changes over time.

[0109] Similarly, the BAPI 353 may use historical data based on the biofeedback measurement data to recognize a particular game player or profile, etc. through various mechanisms including pattern matching, etc. The BAPI 353 may further recognize when a game player has fallen off the sensor and / or been replaced by another game player based on activity such as loss and / or corruption of biofeedback measurement data, pattern changes, etc.

[0110] The BAPI 353 may also be configured to detect certain patterns or situations, etc., from an analysis of the received biofeedback measurement data. For example, in one embodiment, the BAPI 353 may be able to detect and / or even predict the onset of motion sickness, for example, based on causal coherence between heart rate, blood pressure, and / or other measurement data. However, the BAPI 353 may also detect other situations that may be serious enough to warrant sending an alert message to the video game player and / or the GSB 352 to stop game play. However, the BAPI 353 is not constrained to these actions and other actions may also be taken.

[0111] As described above, the BAPI 353 is further configured to make inferences regarding the game player's state of emotional arousal, emotional state, etc., based on analysis of the received biofeedback measurement data. Such inferences may be made based on the received measurement data and / or based on historical data regarding the game player and / or other game players. The GSB 352 may query the BAPI 353 for information regarding the state or emotional arousal of one or more game players and / or other information regarding the game players based in part on the inferences.

[0112] In one embodiment, the GSB 352 may send a query request for information regarding the game player's state of emotional arousal. In response, the BAPI 353 may provide a qualitative response, such as "happy," "sad," "stressed," "lying," "bored," or "excited." However, in another embodiment, the response may be a quantitative response indicating a level of satisfaction, such as 0-10. However, the disclosure is not limited to these values, or even to this exemplary range, and it will be apparent that other values ​​and / or ranges may be used. For example, the quantitative response indicating a level of satisfaction may be a letter grade.

[0113] In any event, FIG. 6 illustrates one embodiment of a non-exhaustive, non-limiting example of queries that the GSB 352 may send to the BAPI 353. For example, as shown, the GSB 352 may send a query that seeks to ascertain whether the game player is "frustrated." Similarly, the GSB 352 may send a query that seeks to ascertain whether the game player is "bored," "relaxed," or "spaced out" (indicating that the game player is not concentrating on the game play), etc. The GSB 352 may also query whether the game player is "anticipating" some action. Such information may be based, for example, on skin conductance levels or heart rate measurement data, etc.

[0114] The GSB 352 may also send queries for specific biofeedback information, such as "check heart rate trend" or "check SCL trend" (for skin conductance level). The GSB 352 may further query for information about the game player's past state, such as "player was surprised."

[0115] As shown in Figure 6, the GSB 352 may also send query requests that provide information about a game player to be compared with other information. For example, as shown, the GSB 352 may perform queries to obtain a comparison of a game player's current state to a previous state, may also perform comparisons of the game player to other game players, baselines, norms, etc. While Figure 6 provides numerous examples of queries that can be performed, it should be apparent that other queries may also be performed. Thus, this disclosure is not limited to these examples.

[0116] In any event, the GSB 352 may then use the results of the query to adjust the gameplay state in any of a variety of ways. In one embodiment, as used herein, the results of a query to the GSB 352 may then provide what may be referred to as biofeedback information or "biometric characteristics." Using such biometric characteristics obtained from the game player's biofeedback is done toward providing a more immersive gameplay experience beyond traditional gameplay. For example, the gameplay state may be adjusted by allowing an avatar to mimic the player's emotional state. For example, if the player is determined to be happy, the player's avatar may also be adjusted to appear happy. Similarly, if the player is determined to be angry, the game state may be adjusted to present the player with a different set of gameplay experiences than if the player was determined to be happy.

[0117] Additionally, in at least one embodiment, a biometric characteristic, such as the game player's state of emotional arousal, may be used to adjust characteristics of an input and / or input / output user device. For example, the color of a joystick or the level of resistance of a joystick may be adjusted as a result of the game player's state of emotional arousal. Similarly, the color of some other input / output user device may change based on heart rate, intensity level and / or color may change based on heart rate, stress or boredom level, or other biometric characteristic indicative of the game player's state of emotional arousal.

[0118] It should be noted that although GSB 352 and BAPI 353 are shown as residing in network devices separate from the client device (such as client device 101 of FIG. 1), the disclosure is not so constrained. Thus, in alternative embodiments, GSB 352 and / or BAPI 353 may reside in the client device, in multiple different client devices, and / or across one or more different network devices. Similarly, BAPI 353 may reside in GSB 352 without departing from the scope of the disclosure.

[0119] [General behavior]

[0120] The operation of certain aspects of the present disclosure will now be described. Figure 4 shows a flow chart of one embodiment of a process for adjusting gameplay state of a video game using biofeedback measurements from a game player. In one embodiment, the process 400 of Figure 4 may be implemented in combination with the GSB 352 and BAPI 353 of Figure 3.

[0121] Process 400 of FIG. 4 begins after a start block with decision block 402 where a determination is made whether a biofeedback sensor is connected. Such a determination may be based on a flag or switch received from a client device or a game server application, or the like. In another embodiment, the determination may be made based on receiving biofeedback measurement data from one or more biofeedback sensors, and the measurements are determined to be within expected ranges. For example, if measurement data is received of a heart rate sensor that appears to indicate background noise measurements, it may be determined that the sensor is erroneous and / or otherwise not connected, and the like. In any event, if it is determined that a biofeedback sensor is not connected for purposes of adjusting a gameplay state, processing proceeds to block 420; otherwise, processing proceeds to block 404.

[0122] At block 420, other user input is received. Such other user input may include, but is not limited to, joystick, game controller, keyboard, mouse input, or audio input. Such input is typically considered to be the result of voluntary or deliberate activity by the game player's body parts, as opposed to input of biofeedback measurement data. Processing then continues to block 422, where the state of game play is adjusted based on such other user input. Processing then proceeds to decision block 416, where a determination is made whether game play should continue. If game play should continue, processing returns to decision block 402. Otherwise, processing proceeds to block 418, where game play ends. Processing then returns to the calling process to perform other actions.

[0123] Alternatively, if at decision block 402 it is determined that a biofeedback sensor is connected, processing proceeds to block 404 where biofeedback measurement data is received from one or more biofeedback sensors. In one embodiment, receiving the biofeedback sensor includes performing a quality analysis on the measurements, adding a timestamp to the measurements, or identifying the source of the biofeedback sensor, etc. Additionally, receiving such biofeedback measurement data may include transmitting the measurements over a network to a biofeedback API, as described above. Processing then proceeds to block 406 where other user inputs are received, including voluntary or intentional user inputs as described in connection with block 420. It should be noted that blocks 406 and 408 may occur in different orders or even simultaneously.

[0124] Processing then continues to block 408, which is described in more detail below in conjunction with FIGURE 5. Briefly, however, analysis is performed on the biofeedback measurement data to generate historical data and / or perform other analysis to ascertain the game player's state of arousal or other biometric characteristics. In one embodiment, block 408 may be performed substantially in real time upon receiving the biofeedback measurement data.

[0125] Processing continues to block 410, where queries may be made before, during, and / or after by the gaming application (or other interactive application). Such queries may include, but are not limited to, those discussed above in connection with FIG.

[0126] Continuing next to block 411, the gameplay state may be adjusted based on other user input, such as joystick input, game controller input, keyboard input, audio input, or mouse input. Processing then proceeds to block 412, where the gameplay state may be adjusted based on the results of the query to obtain the game player's biometric characteristics. Examples of gameplay state adjustments include, but are not limited to, adjusting the type and / or number of opponents in the game, adjusting the pace or tempo of the game, increasing / decreasing the time limit of a game event, adjusting the difficulty of a fight, puzzle, or other challenge, adjusting the availability of items, power-ups, and / or other aspects of the game, adjusting the volume and / or type of sounds, music, and / or other audio features, adjusting the colors or other aspects of the game, including background features of the game, adjusting lighting, weather effects, and / or other environmental aspects of the game, adjusting the dialogue of various characters in the game (including, in some cases, adjusting an avatar representing the game player), providing or disallowing game hints, suggestions, or adjusting the appearance or functionality of the application. For example, in one embodiment, the user interface may be adjusted based on various biometric characteristics. Similarly, tutorials or instructions, etc. may be adjusted by skipping or slowing / speeding up the display, etc. It should be apparent to one skilled in the art that other ways of adjusting the game state based on the biometric characteristics resulting from the query may be used. Processing then continues to decision block 416, where a determination is made as described above as to whether to continue game play.

[0127] 5 illustrates a flow chart of one embodiment of a process for analyzing biofeedback measurement data from a game player for use in a video game. In one embodiment, the process 500 of FIG. 5 may be implemented in the BAPI 353 of FIG. 3.

[0128] After a start block, process 500 begins at block 502 where biofeedback measurement data is received. Continuing to block 504, other user input, such as voluntary or conscious user input, is received. In at least one embodiment, analysis of the biofeedback measurement data may use or be supplemented with information obtained from the voluntary or conscious user input. For example, if a user is typing a particular command, text, or the like into a keyboard, the text or command may be used to supplement the interpretation of the heart rate variability, or the like. Similarly, proceeding to block 506, other game state data may be selectively received and used to further assist in the analysis of the biofeedback measurement data. For example, such game state data may indicate that the game is presenting the game player with a very difficult challenge, or the like. However, the heart rate measurement data may be determined to be a standard adult male heart rate at rest.

[0129] Thus, proceeding to block 508, a first analysis may be performed on the received biofeedback measurement data to determine if there is missing and / or corrupted data. In one embodiment, such a determination may indicate that the biofeedback sensor is erroneous, or that the game player has repositioned the sensor, etc. In one embodiment, if the measurement data is determined to be corrupted or erroneous in a first time period, but not corrupted or erroneous in a second time period, an interpolation may be performed to "smooth" the received measurement data. In another embodiment, the sensor associated with the corrupted / erroneous measurement data may be recorded as corrupt or identified as corrupt. In either case, in one embodiment, the recorded sensor measurement data may be ignored. In yet another embodiment, recent data known to be correct from recent historical circumstances may be used to replace the data determined to be corrupted / erroneous or missing, etc., to "fill" periods between sensor recalibrations and / or other data fluctuations, for example.

[0130] Processing then proceeds to block 510, where a second analysis is performed on the received biofeedback measurement data, in part using other received data, to ascertain the game player's state of emotional arousal and / or other biometric characteristics. Using a combination of such information, block 510 may determine that the game player is bored, absent-minded, etc. In any event, it should be noted that blocks 502, 504, 506, and 508 may be performed in a different order or even simultaneously.

[0131] As described herein, various mechanisms may be used to infer biometric and / or other physiological characteristics of a game player, including performing statistical analysis, pattern matching, etc. In one embodiment, historical information about one or more game players may be used to assist in performing analyses to infer various biometric characteristics of a game player, including the game player's state of emotional arousal.

[0132] Processing then proceeds to block 512, where in one embodiment, at least a portion of the inferences, the measurement data, and / or other data may be used to update a user profile. Processing then proceeds to block 514, where a selected priority event based on the inferences, the biofeedback measurement data, and / or other data may be identified. For example, in one embodiment, if it is determined that the measurement data of the game player can be used to infer that the game player is unwell, such a event may be identified for further action. Accordingly, processing then proceeds to decision block 516, where a determination is made whether any such priority event is identified. If a priority event is identified, processing then proceeds to block 520, where an alert may be sent to the game player, an administrator, or the like. In one embodiment, game play may end. Processing then proceeds to decision block 518.

[0133] However, if no priority circumstances are identified, processing continues to decision block 518 where a determination is made whether to continue analysis of the received biofeedback measurement data. If analysis continues, processing returns to block 502; otherwise, processing may return to the call processing.

[0134] Some possible use cases illustrating the use of biofeedback measurement data to adjust gameplay states are described below, however, it should be noted that the present disclosure is not constrained to these use cases and other use cases may also be used.

[0135] As mentioned above, FIG. 6 illustrates one embodiment of a non-exhaustive and non-limiting example of queries that may be used to query the biofeedback application programming interface (API) for biofeedback measurement data. It is noted that the disclosure is not limited to these example queries illustrated in FIG. 6, and other example queries may be used. However, as illustrated, a variety of different queries may be made, including, but not limited to, determining the player's emotional arousal level and / or emotion level. In one embodiment, specific queries regarding emotional arousal may include whether the player is "happy," "sad," "frustrated," "energetic," "engaged," "bored," "relaxed," or "spaced out." Queries may also be made regarding whether the player is determined to be anticipating an action, is surprised, or is surprised, etc. Similarly, specific biofeedback may be obtained, including, for example, heart rate trends, SCL trends, or any other signal trends. In some embodiments, a time period over which the trend is determined may be provided along with the query.

[0136] Queries are not limited to these examples, and other queries may include comparative information regarding the player and / or other players. In one embodiment, any query may be generated. For example, a particular formula, equation, or combination of biofeedback measurement data may be presented.

[0137] FIG. 7 illustrates one embodiment of a non-exhaustive, non-limiting example of using biofeedback measurement data for use in adjusting game play conditions in an arena combat video game.

[0138] As shown, process 700 of FIG. 7 begins after a start block at block 702 where a computer game configured to provide a fighting scenario is executed. Execution of the computer game places a player in a fighting arena. That is, in one embodiment, an avatar or mechanism may be used to represent the player in the computer game. The player uses one or more biofeedback sensors (such as those described above).

[0139] Processing then proceeds to block 704 where a request may be made during the computer game for the BAPI to establish baseline values ​​for a player's biofeedback measurement data. In one embodiment, the biofeedback measurement data may include baseline heart rate values, skin conductance levels, or other biofeedback measurement data, which may then be analyzed to ascertain a baseline state of the player's emotional arousal or biological characteristics.

[0140] Processing then proceeds to block 706 where an opponent is introduced into the playing field to play against the player. In one embodiment, the selection of the opponent is based on the determined baseline state of emotional arousal. In one embodiment, the baseline value may be used to detect whether the player is associated with a user profile that indicates that the player has previously played this game or a similar game. Based on the user profile, opponents may also be selected at a level determined to sufficiently stimulate the player without boring or frustrating the player.

[0141] Processing then proceeds to block 708, where a combat is conducted between the player and a given game opponent. As the combat is conducted, various biofeedback measurement data is collected, recorded, and / or analyzed.

[0142] In one embodiment, processing may then proceed to decision block 710 where a determination is made as to whether the fight has been concluded, i.e., whether either the player or the game opponent has won. If the fight has been concluded, processing may proceed to decision block 712; otherwise, processing may return to block 708.

[0143] At decision block 712, a determination may be made as to whether the player has defeated the game opponent. If so, processing continues to block 714; otherwise, processing continues to block 722.

[0144] Note that in another embodiment, decision block 710 may be removed so that a determination can be made during the same fight. That is, with decision block 710 removed, decision block 712 may be adjusted, thereby determining whether the player has defeated the game opponent (i.e., won the game opponent). In this manner, changes to the game state may dynamically adjust the same game fight.

[0145] In any event, at block 722, a query may be provided to the BAPI to analyze the biofeedback measurement data obtained during the fight at block 708. In one embodiment, this analysis may include comparing the state of arousal at block 708 to the state of arousal determined from the player's baseline at block 704.

[0146] Processing then proceeds to decision block 724, where a determination is made whether the player's state of arousal during the fight is low. Such a determination may be based on whether the difference from the comparison at block 722 exceeds a defined threshold. In another embodiment, a statistical analysis may be performed to determine whether it can be statistically determined, within some confidence level, that the player's arousal is significant. In any case, if the player's arousal is determined to be significant, processing proceeds to block 728, where another enemy may be introduced into the game having a similar level of power or difficulty as the previous enemy. Processing then returns to block 708.

[0147] However, if the state of arousal is determined to be statistically insignificant or is below some threshold, processing continues to block 726 where an enemy that is weaker than the previous enemy is introduced. Processing then returns to block 708.

[0148] However, if at decision block 712 it is determined that the player has lost or is about to lose, processing continues to block 714 where a query substantially similar to that of block 722 is made. Then, at decision block 716, a determination is made substantially similar to the determination at decision block 724 of whether the player has a low state of arousal. If the state of arousal is low, processing continues to block 718; otherwise, processing continues to block 720.

[0149] At block 718, an enemy is introduced that is stronger than the previous enemy. Processing then returns to block 708. At block 720, an enemy may be introduced that has similar strength to the previous enemy. Processing then again returns to block 708.

[0150] As will be apparent, process 700 may be adapted to dynamically adjust enemy strength during the same fight, where enemy substitution may take several forms including, for example, simply increasing the strength of a current enemy or removing some strength from a current enemy, or introducing and / or removing additional enemies, etc.

[0151] FIG. 8 illustrates another embodiment of another non-exhaustive, non-limiting example of using biofeedback measurement data to adjust game play states. In process 800 of FIG. 8, the game shown is a space video game. In this exemplary game, the player is required to try to conserve oxygen by trying to control his air consumption. For example, in this game, the player may be introduced to a situation where he is to be rescued after a certain period of time, such as 5 minutes. However, the player's space suit contains 6 minutes worth of oxygen if consumed at a predetermined "standard" consumption rate, for example, 1 unit of oxygen per second. The player is then introduced to various situations that may be adjusted based on the player's biofeedback measurement data. Thus, in one embodiment, based on the player's biofeedback measurement data, the game state may be adjusted to make the game more complex or easier, introducing more or fewer activities that the player needs to perform. During game play, the player is further expected to manage his oxygen consumption. Thus, in one embodiment, a player is required to control his or her air consumption by attempting to maintain low levels of physiological arousal (which may be related to oxygen consumption by the video game avatar) while engaging in various stressful missions within the video game, such as combating enemies, solving difficult puzzles, or other challenges.

[0152] As shown, in this example, process 800 then begins after a start block at block 804 where various game variables may be set including, for example, game time, oxygen level, and consumption rate. Proceeding then to block 806, instructions or similar information may be displayed to the player. During the display of instructions or similar, various biofeedback measurement data may be received and analyzed to ascertain player metrics. For example, in one embodiment, the biofeedback measurement data may include the player's heart rate measurement data.

[0153] Processing continues at block 808, where the BAPI may be queried to ascertain the player's average heart rate over a period of time. As shown in FIG. 8, a period of time is 30 seconds. However, it should be apparent that the game duration and other parameters herein are merely for illustration, and other values ​​may be used. In any case, the result of the query may then be used as the baseline heart rate.

[0154] Processing then proceeds to block 812 where game play may begin. Continuing to block 814, the player is introduced to various game states of play. These game states may include having the player move, fight, play music and / or otherwise repair items, or communicate with other players. Proceeding to block 816, during game play, the game makes additional query requests to collect additional heart rate measurement data. An average heart rate may then be determined over a period of time, such as the last 10 seconds of game play. Continuing to block 818, an oxygen consumption rate may further be determined, for example, based on the rate at which the player consumes oxygen as determined based on the biofeedback measurement data. In one embodiment, the oxygen consumption rate may be derived or inferred from a ratio of the player's current heart rate to the player's average baseline heart rate.

[0155] Continuing to block 820, the time of game play is decremented. Continuing to block 822, the amount of oxygen remaining is determined based on the determined player's consumption rate. Moving to decision block 824, a determination may be made whether more oxygen remains. If so, processing continues to decision block 828; otherwise, processing continues to block 826.

[0156] At block 826, it is determined that the player's character has run out of oxygen and has died of suffocation. The game may then end and return to the calling process. Alternatively, at decision block 828, a determination is made as to whether the game has run out of time. If so, then at block 830, the game is determined to have ended and the player's character is determined to have survived. Then the process returns. However, if there is more time, the process returns to block 814 to continue the game.

[0157] It will be understood that each block of the flowchart illustrations, and combinations of blocks in the flowchart illustrations, can be embodied with computer program instructions. These program instructions can be provided to a processor to create a machine, such that the instructions, when executed by the processor, create means for implementing the actions specified in one or more of the flowchart blocks. The computer program instructions can be executed by the processor to cause the processor to perform a series of operational steps to create a computer-implemented process, such that the instructions, when executed by the processor, provide steps for implementing the actions specified in one or more of the flowchart blocks.

[0158] Thus, each block of the flowchart diagrams supports a combination of means for performing the specified actions, a combination of steps for performing the specified actions, and program instruction means for performing the specified actions. It will also be understood that each block of the flowchart diagrams and combinations of blocks in the flowchart diagrams can be realized by dedicated hardware-based systems that perform the specified actions or steps, or a combination of dedicated hardware and computer instructions.

[0159] As can be seen from the above game example, biofeedback measurement data may be used in various ways to adjust the state of game play. However, the variations in the ways are not limited to those mentioned above. For example, in a variation to the above game, biofeedback measurement data may be used to control inputs to the game. For example, if a large creature is trying to find the game player's character, the player may be expected to maintain or reduce stress levels so as to avoid the creature's awareness of his / her location. In a similar game, the player may be required to exhibit a sudden physiological arousal in order to escape handcuffs or other restraints or to break through a locked door to escape danger.

[0160] In another game scenario, a sprite character that gives cookies to the user may only appear when the player is calm and will not approach if the user is determined to be discaled. A player who desires cookies (or other rewards) must achieve a state of calm in order to attract the sprite character. In yet another game scenario, in a forest adventure, green trees and clear skies may occur when the player is determined to be in a particular state of physiological arousal. If the player deviates from this state, the sky may darken, trees may wither and / or turn black, and / or various colors, music, and / or other sounds may change. Thus, various background aspects within the game may be dynamically adjusted based on the player's biofeedback measurement data.

[0161] Similarly, based on biofeedback measurement data from the player, various characters other than the player may make dialogue choices or change their appearance, including, for example, commenting directly on the user's inferred state of emotional arousal or other biometric characteristics.

[0162] In yet another example, the user's avatar may exhibit a visible heart, brain, or other body parts that may be adjusted based on biofeedback measurement data. For example, the heart may change color to indicate boredom, anger, contentment, etc. Similarly, the heart may beat to match the player's heart rate. In yet another embodiment, the avatar's heart rate may be adjusted to be slightly slower than the player's heart rate, i.e., to try to induce the player to be calm. The avatar's facial expression may also change as a result of the player's inferred state of emotional arousal, including, for example, smiling, frowning, or expressing anger.

[0163] Additionally, user interface devices or screen displays, etc. may also be adjusted based on the player's inferred state of emotional arousal. Thus, if the player is determined to be stressed, the user interface may display a help function to guide the player to a solution to the gameplay problem the player is experiencing. There are still multiple other ways in which biofeedback measurement data may adjust the gameplay state. Thus, as already mentioned, the present disclosure is not limited to the above.

[0164] 9 shows a flow chart of one embodiment of a process for analyzing biofeedback measurement data from a game player indicative of gaze position for use in a video game, and for adjusting or enhancing such video game in response to the analysis of the biofeedback measurement data. In one embodiment, process 900 may be performed in one or more computing devices, such as one or both of devices 200 and 300 of FIGS. 2 and 3, respectively (generally referred to as "video game devices").

[0165] Process 900 begins after a start block at block 902, where a video game device provides gameplay to a video game player via a user interface that provides video game features. At 904, the video game device receives biofeedback measurement data of a video game player from one or more physical biofeedback sensors while the video game player is playing a video game. The one or more biofeedback sensors may function to provide index tracking of one or both of the video game player's eyes while the player is playing a video game. The one or more physical biofeedback sensors may include at least one optical sensor, such as one or more optical sensors (e.g., IR sensor, video camera) coupled to a head-mounted device (e.g., a head-mounted display device). In at least some implementations, the one or more physical biofeedback sensors may include at least one infrared light source and at least one infrared sensor.

[0166] At 906, the video game device processes the biofeedback measurement data to track the video game player's point of gaze during gameplay of the video game. As an example, the biofeedback measurement data may be used to determine a location on a display of the video game device where the video game player is looking when the user plays the video game.

[0167] As described herein, various mechanisms may be used to determine gaze location, including, for example, performing statistical analysis, pattern matching, using one or more models, etc. In one embodiment, historical information about one or more game players may be used to assist in performing the gaze location function.

[0168] At 908, the video game device dynamically adjusts or enhances gameplay of the video game based at least in part on the tracked point of gaze of the video game player. As one example, the video game device may cause featured characters or other objects to appear in an area where the video game player is not currently gazed upon, thereby creating an element of surprise for the video game player. As another example, the video game device may cause featured characters or other objects to appear in an area where the video game player is currently gazed upon, thereby causing such objects to appear in a path where the video game player intends to travel.

[0169] In at least some implementations, the video game device may provide hints or other assistance to the video game player based on the tracked gaze location. For example, if the video game player stares at a door or wall for an extended period of time, the video game device may provide a visual and / or audible notification to the video game player to provide hints on how to improve at the video game. For example, if the video game device recognizes that the player is lost based on the tracked gaze location, the video game device may provide the player with a map or directions.

[0170] As another example, a video game device may present a tutorial to a video game player based on tracked gaze positions. For example, a video game player may gaze at a display in a pattern that is determined to indicate that the video game player needs help. In response to detecting such a pattern, the video game device may provide a tutorial or other help to the video game player to help the player learn how to play or improve at a video game.

[0171] 10 illustrates a flow chart of one embodiment of a process for analyzing biofeedback measurement data from a game player and determining the video game player's next move in response to the analysis of the biofeedback measurement data for use in a video game. Process 1000 may, for example, in one embodiment be performed in one or both of devices 200 and 300 of FIGS. 2 and 3, respectively.

[0172] Process 1000 begins after a start block at block 1002, where a video game device provides gameplay to a video game player via a user interface that provides video game features. At 1004, the video game device receives biofeedback measurement data of a video game player from one or more physical biofeedback sensors while the video game player is playing a video game. The one or more physical biofeedback sensors may include one or more electroencephalography (EEG) electrodes, and the biofeedback measurement data may include EEG signals. Additionally or alternatively, the one or more physical biofeedback sensors may include one or more electrodes, and the biofeedback measurement data may include neural signals. In such cases, the one or more electrodes may be positionable on the video game player's neck, back, chest, shoulder, arm, wrist, hand, etc. By way of non-limiting example, the biofeedback measurement data may include one or more of neural signals, EEG signals, EMG signals, EOG signals, fNIR signals, signals indicative of blood flow (e.g., from an IR camera), functional near-infrared spectroscopy (fNIR) spectroscopic signals, force sensitive resistor (FSR) signals, facial expression detection signals, pupil dilation indication signals, eye movement signals, gesture motion signals, and the like.

[0173] At 1006, the video game device analyzes the biofeedback measurement data to determine a next or upcoming movement of the video game player during gameplay of the video game. The analysis may include utilizing one or more learned or trained models, such as one or more models utilizing one or more neural networks. Additionally or alternatively, the analysis may include using one or more other signal processing techniques to sense the data, such as Fourier transform, spectral density analysis, and the like. For example, the video game device may determine that the video game player is about to provide input to an input device (such as a mouse, keyboard, handheld controller, etc.) of the video game device based on the received biofeedback measurement data. The input may be actuating a button, key, wheel, trigger, or other input of the input device. The next movement may be a physical movement of the input device (e.g., controller). In some implementations, the next movement may be a physical movement of the video game player, such as moving an arm, moving a leg, making a gesture, standing up, sitting down, changing facial expression, changing gaze position, or any other physical movement.

[0174] At 1008, the video game device initiates an action resulting from the determined next movement of the video game player. In at least some implementations, the video game device may initiate the action before the video game player initiates the next movement, such that the next movement is predicted by the video game device. For example, the video game device may analyze a biofeedback signal (e.g., neural signals, EEG signals) to determine that the video game player is about to click a mouse button. In response to such a determination, the video game device may initiate a mouse click before the video game player actually clicks the mouse button, thereby providing a user with a much faster reaction time than was previously possible. As another example, the video game device may detect that the video game player is about to move based on the biofeedback signal, and the video game device may move an object (e.g., a character corresponding to the game player, a virtual weapon) before the video game player actually moves. Such functionality reduces the delay that occurs between the video game player deciding to move and the actual movement occurring.

[0175] In at least some implementations, the video game device may receive an indication as to whether the video game player actually performed the determined next move. For example, the video game device may receive an indication as to whether the player actually clicked a mouse button. In response to receiving an indication that the video game player did not perform the next move, the video game device may adjust or undo the initiated action (e.g., a mouse click, a character movement, etc.) to "undo" or minimize the effect of the incorrectly predicted move.

[0176] 10 is described in the context of a video game, it should be understood that the disclosure is not so limited. In general, the features discussed herein may be used in numerous applications, such as a variety of applications in which a user interacts with a user interface of a computing device.

[0177] 11 shows a flow chart of one embodiment of a process for analyzing biofeedback measurement data from a user and updating or training a model operable to predict user movements. Process 1100 may be performed by a computing device such as, for example, devices 200 and 300 of FIGS. 2 and 3, respectively.

[0178] Process 1100 begins after a start block at block 1102 where a computing device provides a user interface to a user. The user interface may include one or more input devices such as a mouse, a keyboard, a controller, a microphone, a video camera, etc.

[0179] At 1104, the computing device receives biofeedback measurement data of the user from one or more physical biofeedback sensors while the user interacts with the user interface. As described above, the one or more physical biofeedback sensors may include one or more EEG electrodes for acquiring EEG signals or one or more electrodes for measuring neural signals. The one or more electrodes may be positionable on the video game player's neck, back, chest, shoulder, arm, wrist, hand, etc. In general, the biofeedback measurement data may include one or more of neural signals, EEG signals, EMG signals, EOG signals, fNIR signals, signals indicative of blood flow (e.g., from an IR camera), functional near-infrared spectroscopy (fNIR) spectroscopy signals, force sensitive resistor (FSR) signals, facial expression detection signals, pupil dilation display signals, eye movement signals, gesture motion signals, etc.

[0180] At 1106, the computing device may analyze the biofeedback measurement data based on one or more learned models to predict the user's interaction with at least one input device. The learned or trained models may include, for example, one or more models utilizing one or more neural networks. As described above, the analysis may additionally or alternatively include using one or more other signal processing techniques to sense the data, such as Fourier transform, spectral density analysis, etc. For example, the video game device may determine that the video game player intends to provide input to an input device (such as a mouse, keyboard, handheld controller, etc.) of the video game device based on the received biofeedback measurement data. The input may be actuating a button, key, wheel, trigger, or other input of the input device. The next movement may also be a physical movement of the input device (e.g., controller). In some implementations, the next movement may be a physical movement of the video game player, such as moving an arm, moving a leg, making a gesture, standing up, sitting down, changing facial expression, changing gaze position, or any other physical movement.

[0181] At 1108, the computing device may detect whether the user actually interacted with the at least one input device as predicted. For example, the computing device may determine whether the user actually performed a mouse click if the computing device so predicted.

[0182] At 1110, the computing device may update the learned model based on detecting whether the user actually interacted with the at least one input device as predicted. In other words, the computing device may use the feedback to provide new labeled samples that can be used in the supervised learning process to update (e.g., tune, train, retrain) or otherwise improve the model's ability to predict future movements of the user or other users.

[0183] FIG. 12 illustrates a schematic diagram that broadly outlines one embodiment of a system 1200 in which one or more features of the present disclosure, such as any of the processes described herein, may be implemented. The system 1200 may include fewer or more components than those illustrated in FIG. 12. As illustrated in FIG. 12, the system 1200 includes a client computing device 1204 operated by a user 1202. The client computing device may be similar to or identical to the client device 101 of FIG. 1. Although not illustrated in FIG. 12, it should be understood that the system 1200 may also include one or more wired or wireless networks, one or more game server devices, and the like, as illustrated in the system 100 of FIG. 1.

[0184] The client device 1204 may be configured to receive messages, signals, images, and / or other biofeedback measurement data from various biofeedback sensors 1208. FIG. 12 shows non-limiting, non-exhaustive examples of possible physical biofeedback sensors 1208 that may or may not be connected to the user 1202, and that replace and / or otherwise augment a conventional physical game controller. In the illustrated embodiment, the biofeedback sensor 1208 includes a head-mounted biofeedback sensor 1208c, which may be used to measure EEG or other signals. More generally, the head-mounted biofeedback sensor 1208c may function to directly measure neural signals, which may then be translated into something meaningful or useful, such as emotions, decisions, intentions, thoughts, something else, or any combination thereof. System 1200 may alternatively or additionally include sensor 1208a or 1208b, which may include one or more electrodes that may be placed on the back, shoulder, arm, wrist, hand, finger, etc. of user 1202 and may function to measure neural signals to predict user movement. Biofeedback sensor 1208 may be integrated into a game controller, or into one or more keys or wheels on a keyboard, etc. In one embodiment, a game controller may include modular and / or pluggable components that may include modular and / or pluggable sensors.

[0185] The biofeedback sensor 1208 may include a camera, a touchpad, or a head device (e.g., a sensor integrated into an HMD device). However, as previously mentioned, other biofeedback sensors 1208 may also be used, including glasses, wristbands, finger sensor attachments, sensors integrated into or on a computer mouse, or microphones for measuring various voice patterns, etc. Thus, it should be apparent to one skilled in the art that various embodiments may use virtually any mechanism configurable to obtain biofeedback measurement data of a game player.

[0186] The biofeedback sensor 1208 may be positioned to collect various measurement data of the user before, after, and / or during interaction with the computing device (e.g., video game play). Such measurement data may include, but is not limited to, neural signals, EEG signals, heart rate and / or heart rate variability, galvanic skin response, body temperature, eye movements, head, face, hand, or other body movements, gestures, position, facial expression, or posture, etc. Additionally, the biofeedback sensor 1208 may collect other measurement data including blood oxygen levels, other forms of skin conductance levels, respiration rate, skin tension, voice stress levels, voice recognition, blood pressure, EEG measurement data, electromyography (EMG) measurement data, response time, electro-oculography (EOG), blood flow (e.g., via an IR camera), fMRI, functional near-infrared spectroscopy (fNIR) spectroscopy, or force-sensitive resistors (FSR), etc.

[0187] The biofeedback sensor 1208 may provide these measurement data to the client device 1204. In one embodiment, these measurement data may be provided to the client device 1204 via any of a variety of wired and / or wireless connections. Thus, the biofeedback measurement data may be transmitted via various cables or wires, and other information may also be transmitted using various cables or wires (e.g., for gameplay). For example, the biofeedback measurement data may be transmitted via a USB cable or a coaxial cable, and a mouse, keyboard, or game controller, etc. may also be coupled to the client device 1204 using a USB cable or a coaxial cable, etc. However, in other embodiments, other wired connections may be used. Similarly, the biofeedback sensor 1208 may transmit the biofeedback measurement data using various wireless connections. Furthermore, any of a variety of communication protocols may be used to transmit the measurement data. Thus, the present disclosure should not be construed as limited to any particular wired or wireless communication mechanism and / or communication protocol.

[0188] 13 illustrates a flow chart of one embodiment of a process for analyzing biofeedback measurement data from a user operating a user interface to remedy the user's difficulties or other problems. Process 1300 may be performed by a computing device, such as devices 200 and 300 of FIGS. 2 and 3, respectively.

[0189] Process 1300 begins after a start block at block 1302, where a computing device provides a user interface to a user. The user interface may include one or more input devices, such as a mouse, keyboard, controller, microphone, video camera, etc. At 1304, the computing device receives biofeedback measurement data of the user from one or more physical biofeedback sensors while the user interacts with the user interface. As mentioned above, the one or more physical biofeedback sensors may include, for example, any of the biofeedback sensors discussed elsewhere herein.

[0190] At 1306, the computing device analyzes the received biofeedback measurement data to determine whether the user is having difficulty with the user interface or decision making. For example, the computing device may analyze the received biofeedback measurement data to determine that the user is having difficulty selecting an object, is frustrated, is having trouble, etc.

[0191] At 1308, in response to determining that the user is so struggling, the computing device adapts the user interface to ameliorate the user's difficulty. For example, in a video game, the computing device may determine, based on the biofeedback measurement data, that the user is experiencing frustration while learning how to play the game, and in response to this determination, may provide guidance to the user. As another example, the computing device may determine, based on the user's gaze position, that the user is struggling to select an object, such as a weapon, in a video game. In response to such a determination, the computing device may provide the user with suggestions regarding objects to select.

[0192] In at least some implementations, the computing device may utilize data from one or more input devices, alone or in conjunction with a biofeedback sensor, to determine how the user is attempting to acquire a skill (e.g., acquiring skill to play a video game, acquiring skill to operate a software program, etc.). The computing device may use the input device data and / or biofeedback data to determine when the user is struggling and may adapt the user interface to assist the user. As one example, in a video game, the computing device may determine that the user is struggling with a particular skill and may provide training or a tutorial to assist the user. As another example, the computing device may determine, based on user input and / or biofeedback measurement data, that the user is struggling with a user interface (e.g., is struggling with the complexity of the user interface) and may simplify the user interface in response to such a determination.

[0193] 14 illustrates a flow chart of one embodiment of a process 1400 for analyzing biofeedback measurement data from a user operating a video game device to ascertain the user's responses to multiple distinct components during gameplay of the video game. Process 1400 may be performed by a computing device, such as, for example, devices 200 and 300 of FIGS. 2 and 3, respectively.

[0194] Process 1400 begins at 1402, where at least one processor operatively coupled to one or more physical biofeedback sensors provides gameplay to a video game player via a user interface that provides video game functionality. The gameplay may include a number of individual components. By way of non-limiting example, the number of individual components may include at least one of a game character, chat messages, weapons, character selection, character actions, events associated with the character, characteristics of another video game player, audio (e.g., music, voices, sound effects), or other individual components.

[0195] At 1404, the at least one processor receives biofeedback measurement data of the video game player from one or more physical biofeedback sensors while the video game player is playing the video game. The one or more physical biofeedback sensors may include one or more electroencephalography (EEG) electrodes, and the biofeedback measurement data includes an EEG signal. The one or more physical biofeedback sensors may include one or more electrodes, and the biofeedback measurement data includes a neural signal. In at least some implementations, the biofeedback measurement data may include at least one of a neural signal, an EEG signal, an EMG signal, an EOG signal, an fNIR signal, a signal indicative of blood flow, a functional near-infrared spectroscopy (fNIR) spectroscopy signal, a force sensitive resistor (FSR) signal, a facial expression detection signal, a pupil dilation display signal, an eye movement signal, a gesture motion signal, and the like.

[0196] At 1406, the at least one processor may process the biofeedback measurement data to determine a response of the video game player to the plurality of individual components during gameplay of the video game. In at least some implementations, the at least one processor may apply at least one learning model (e.g., a deep learning model) to process the biofeedback measurement data. The at least one learning model may be trained to determine a particular subset of the plurality of individual components that cause the video game player to have a particular cognitive state. In at least some implementations, the at least one processor determines a relative weighting of the contribution of each individual component to the determined response. In at least some implementations, this analysis may include using one or more other signal processing techniques to sense the data, such as Fourier transforms, spectral density analysis, etc.

[0197] At 1408, the at least one processor adjusts or enhances the gameplay of the video game based at least in part on the determined response of the video game player, as discussed elsewhere herein.

[0198] 15 illustrates a flow chart of one embodiment of a process 1500 for performing an analysis of biofeedback measurement data from a population of users operating a video game system to adjust or enhance the video game. Process 1500 may be performed by a computing device, such as devices 200 and 300 of FIGS. 2 and 3, respectively.

[0199] Process 1500 begins at 1502 where at least one processor of a video game system provides game play to a population of video game players via respective user interfaces that provide video game functionality.

[0200] In 1504, the at least one processor receives biofeedback measurement data of the video game player from a physical biofeedback sensor proximate to the video game player while the video game player is playing the video game. The biofeedback measurement data may be captured while a plurality of distinct components of the video game are displayed. The plurality of distinct components may include at least one of a character of the game, chat messages, weapons, character selections, character actions, events associated with the character, characteristics of another video game player, or other components. For example, the one or more physical biofeedback sensors may include one or more electroencephalography (EEG) electrodes, and the biofeedback measurement data may include an EEG signal.

[0201] In 1506, the at least one processor analyzes the biofeedback measurement data to determine a subset of the plurality of individual components that contribute to an overall emotion or impression of the population of video game players. In at least some implementations, to analyze the biofeedback measurement data, the at least one processor may implement at least one model (e.g., a deep learning model) or other signal processing technique that operates to isolate the individual components of the plurality of individual components that contribute to the overall emotion or impression of the video game players. The at least one processor may receive class information for each video game player and analyze the biofeedback measurement data and the class information to determine how various classes of video game players respond differently to each individual component of the video game. In at least some implementations, the at least one processor may estimate an opinion about the video game based on the received biofeedback measurement data, estimate a life cycle of the video game based on the received biofeedback measurement data, or determine a similarity between different portions of the video game based on the received biofeedback measurement data.

[0202] At 1508, the at least one processor adjusts or enhances the video game in response to the analysis of the biofeedback measurement data.

[0203] 16 illustrates a flow chart of one embodiment of a process 1600 for analyzing biofeedback measurement data from a user operating a video game system to ascertain the user's internal state and adjust or enhance the video game. Process 1600 may be performed by a computing device such as, for example, devices 200 and 300 of FIGS. 2 and 3, respectively.

[0204] Process 1600 begins at 1602, where at least one processor coupled to one or more physical biofeedback sensors provides game play to a video game player via a user interface that provides video game functionality.

[0205] In 1604, the at least one processor receives biofeedback measurement data of the video game player from one or more physical biofeedback sensors while the video game player is playing the video game. The biofeedback measurement data may include, for example, at least one of neural signals, EEG signals, EMG signals, EOG signals, fNIR signals, signals indicative of blood flow, functional near-infrared spectroscopy (fNIR) spectroscopy signals, force sensitive resistor (FSR) signals, facial expression detection signals, pupil dilation indication signals, eye movement signals, gesture motion signals, or other measurement data.

[0206] In 1606, the at least one processor processes the biofeedback measurement data to determine an internal state of the video game player during gameplay of the video game. In at least some implementations, the at least one processor utilizes the determined internal state to predict that the video game player is likely to stop playing a session of the video game or to stop playing the video game altogether. As another example, the at least one processor may utilize the determined internal state to determine the video game player's impression of at least one of a weapon, a character, a map, a game mode, a tutorial, a game update, a user interface, teammates, a game environment, or the player's impression of another object, interface, or other feature of the video game.

[0207] At 1608, the at least one processor adjusts or enhances gameplay of the video game based at least in part on the determined internal state of the video game player.

[0208] Figure 17 illustrates a flow chart of one embodiment of a process 1700 for providing neurostimulation to a user during video game play on a video game system to enhance the user's gaming experience. Process 1700 may be performed by a computing device such as, for example, devices 200 and 300 of Figures 2 and 3, respectively. Figures 18-21, described below, provide further details of one or more implementations of the present disclosure.

[0209] Process 1700 begins at 1702 where at least one processor coupled to one or more physical neurostimulators provides game play to a video game player via a user interface that provides video game functionality.

[0210] In 1704, the at least one processor provides neural stimulation to the video game player via the one or more physical neural stimulation devices while the video game player is playing the video game to provide an enhanced experience to the video game player. The neural stimulation may provide at least one of improving the video game player's concentration, improving the video game player's memory, improving the video game player's learning ability, changing the video game player's emotional arousal, modulating the video game player's visual perception, modulating the video game player's auditory perception, or other experience enhancing phenomenon for the video game player.

[0211] The one or more physical neurostimulators may include at least one of a non-invasive neurostimulator or an invasive neurostimulator. Non-limiting examples of physical neurostimulators include at least one of a transcranial magnetic stimulation device, a transcranial electrical stimulation device, a microelectrode-based device, an implantable device, or other stimulation device. The one or more physical neurostimulators may function to provide at least one of a sensory system stimulation, a motor system stimulation, or other type of stimulation.

[0212] FIG. 18 is an illustration (1800) showing a non-limiting exemplary mechanism for inducing, writing, or generating signals in the brain 1802 of a user 1804 (e.g., a video game player) to enhance the user's experience. The mechanism may include one or more non-invasive techniques, such as EEG 1806 or MEG 1806. Additionally or alternatively, invasive techniques may be used, such as electrocorticography (eCoG) 1810, stereoelectroencephalography (SEEG) 1812, or intracortical implants 1814. These or other techniques may be used to detect or induce brain activity from inside or outside the skull of the user 1804. As discussed elsewhere herein, the collected data may be processed using various signal processing techniques, machine learning (e.g., deep learning), analysis of time versus space information, etc.

[0213] Using the techniques discussed herein, various internal states of a game player may be measured, including learning, surprise / novelty, excitement, relaxation, emotion (positive or negative feelings), attention, engagement, boredom, learning ability, response to in-game stimuli, and other internal states.

[0214] As mentioned above, many of the features of the present disclosure can help improve playtesting. Traditional playtesting methods focus on direct observation, question and answer, surveys, game metrics, usability, etc. Potential drawbacks of such methods are that people justify, invent excuses, and make up stories. Further drawbacks include that playtesters can only measure what they can record, the lack of moment-by-moment insight, and the impracticality of large sample sizes.

[0215] FIG. 19 is an illustration (1900) depicting various potential features of a brain-computer interface (BCI) 1902 according to embodiments of the present disclosure. The depicted BCI 1902 provides one or more of the following non-limiting features: moment-to-moment insight 1904, more objective data 1906, scaled-up playtesting 1908, new data 1910, time variation 1912, and signal convergence 1914. The moment-to-moment insight 1904 provides real-time understanding of the player's emotional state on a moment-to-moment (e.g., second-by-second) basis. This allows the system to obtain responses to each individual component of gameplay. As an example, the system can understand how the player reacts to a particular enemy, chat message, bullet fire, character death, character killing, character selection, art assets, etc. The system can also use the obtained insight to determine which components add up to an overall impression. Advantageously, this data can be obtained in real-time without disrupting the player's experience.

[0216] Obtaining more objective Data 1906 allows for undistorted creativity and justification in responses, and avoids misremembering and fabrication, as internal sensations are not interpreted by the player.

[0217] Scaled playtesting 1908 provides a much larger data set that can be obtained through internal testing, and also allows for more effective isolation of each individual component that contributes to an overall emotion or impression. As discussed above, such isolation can be achieved using a variety of techniques, such as machine learning, Fourier transforms, spectral density analysis, etc. Additionally, because the system can continuously collect data from a population of users, new information can be learned and used to improve the player's experience.

[0218] As discussed herein, the new data 1910 allows for inferences of the rationale behind player actions. The new data allows for more accurate measurements of overall sentiment, more granular measurements of individual sentiment, and also more granular measurements of gameplay components. Some example inferences or questions that can be answered include predicting when players are about to quit (for a session or permanently), determining if responses in a forum correlate with the sentiment of the overall player base, understanding impressions of new features, determining which parts of updates players enjoy, determining how players feel about the game's interface, determining which players are good or bad teammates, determining which aspects of gameplay are most and least satisfying, determining how helpful a tutorial was, or other evaluation criteria.

[0219] The time change 1912 may be used to compare responses over time. For example, responses may be compared before and after an update to see how the response changes with the update. The time change 1912 may also be used to assess sentiment or estimate the life cycle of a game based on the change in responses over time.

[0220] Signal convergence 1914 may include combining data obtained from the BCI and physiological sensors with other data sources, which may allow the system to ascertain why something is about to occur. Such signal convergence 1914 may be correlated with player retention, engagement, playtime, etc.

[0221] Using the features discussed herein, gameplay may be adapted, adjusted, or enhanced to enhance the player's experience. For example, the game may be designed to allow for adaptive enemies or opponents, teammates, rewards, weapons, difficulty levels, pairing with other users, and the like. With respect to adaptive enemies, the game may determine what types of enemies the player likes or dislikes to play against, what types of enemies are challenging, what types of enemies the player finds boring, and may select or design enemies accordingly, which may be human-controlled (opponents) or artificial intelligence (AI)-controlled enemies. One or more characteristics (e.g., difficulty level) of these enemies may be dynamically adjusted, for example, based on biofeedback during gameplay. As another example, the player may only be able to engage enemies (e.g., "kill them") when in a particular cognitive state (e.g., relaxed, focused). Similar adaptive techniques may be used to select or adjust teammates (e.g., human teammates, AI teammates). With respect to rewards, the system may determine which rewards are liked or disliked by a particular player or group of players and may provide rewards based on such determination.

[0222] 20 is a diagram (2000) showing inputs that cause a neuron to fire 2002. As discussed herein, a neuron may fire (e.g., produce an electrical signal) due to a sensory perception 2006, an internal perception 2004, or an external influence 2008.

[0223] 21 is a diagram 2100 illustrating a BCI 2102 providing one or more various functions 2104-2112 that may be used to provide an augmented experience to a player. In the illustrated example, the BCI 2102 may provide one or more of the following exemplary functions: neuroprosthesis 2104, restricted intent 2106, augmented cognition 2108, augmented awareness 2110, and simulated reality 2112.

[0224] Neural prostheses 2104 may include sensory or motor system replacements. Vision and movement are generated by neuronal firing. In at least some implementations, each approach may be used to replace or supplement the player's visual or motor functions by firing appropriate neurons in a prescribed manner. Limited will 2106 may be used to allow the player to control gameplay with their own thoughts, which may replace one or more of a gamepad, keyboard, or mouse. Augmented cognition 2108 may be used to provide various non-traditional motor and sensory processing functions. For example, augmented cognition may be used to allow the player to see infrared, have increased contrast sensitivity, or access other spatial information (e.g., echolocation, etc.).

[0225] Augmented cognition 2110 may be used to provide various improvements, such as improved focus of attention or learning ability. For example, certain areas of the brain may be stimulated to reduce activation or dominance of neurons focused on processing something (e.g., sunlight) and allow the brain to focus on other tasks (e.g., playing a game, learning, etc.).

[0226] In the foregoing detailed description, various implementations of devices and / or processes have been described using block diagrams, schematics, and examples. To the extent that such block diagrams, schematics, and examples include one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation included in such block diagrams, flow charts, or examples may be implemented individually and / or collectively by a wide variety of hardware, software, firmware, or substantially any combination thereof. In one implementation, the subject matter may be implemented by an application specific integrated circuit (ASIC). However, those of ordinary skill in the art will recognize that the implementations disclosed herein can be equivalently implemented, in whole or in part, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers), as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as substantially any combination thereof, on standard integrated circuits, and that designing circuitry and / or writing software and / or firmware code is well within the skill of those in the art in light of the present disclosure.

[0227] Those skilled in the art will recognize that many of the methods or algorithms described herein may employ additional operations, omit certain operations, and / or perform operations in an order other than that specified.

[0228] Moreover, those skilled in the art will appreciate that the mechanisms taught herein can be distributed in a variety of forms as a program product, and that the exemplary implementations apply equally regardless of the particular type of signal-bearing medium actually used to effect the distribution, including, but not limited to, floppy disks, hard disk drives, CD-ROMs, digital tape, and recordable media such as computer memory.

[0229] The various implementations described above may be combined to provide further implementations. All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent literature referenced herein, including U.S. Provisional Patent Application No. 62 / 821,839, filed March 21, 2019, U.S. Non-Provisional Patent Application No. 16 / 220,432, filed December 14, 2018, U.S. Non-Provisional Patent Application No. 15 / 369,625, filed December 5, 2016, and U.S. Non-Provisional Patent Application No. 12 / 501,284, filed July 10, 2009, are incorporated herein by reference in their entirety, unless those implementations are inconsistent with the specific teachings and definitions herein. If necessary, aspects of each implementation may be adjusted to provide further implementations using systems, circuits, and concepts from various patents, applications, and publications.

[0230] In light of the above description, these and other modifications can be made to each implementation. In general, in the following claims, the terms used should not be construed to limit the scope of the claims to the specific implementations disclosed in the specification and claims, but should be construed to include all possible implementations along with the full scope of equivalents to which such claims are entitled. Thus, the claims are not limited by this disclosure.

Claims

1. one or more physical biofeedback sensors; at least one non-transitory processor-readable storage medium for storing at least one of data and instructions; at least one processor operatively coupled to the at least one non-transitory processor-readable storage medium and to the one or more physical biofeedback sensors; wherein, in operation, the at least one processor: providing a gameplay to a video game player via a user interface that provides video game functionality, the gameplay having a plurality of distinct components; receiving biofeedback measurement data of the video game player from the one or more physical biofeedback sensors while the video game player is playing the video game; processing the biofeedback measurement data to determine a response of the video game player to the plurality of discrete components during gameplay of the video game, wherein said processing of the biofeedback measurement data includes analyzing the biofeedback measurement data to determine one or more biometric characteristics of the video game player, the one or more biometric characteristics of the video game player including a state of emotional arousal of the video game player, and the plurality of discrete components including at least one of a character featured in the video game and an action of the character featured; adjusting the gameplay of the video game based at least in part on the determined one or more biometric characteristics of the video game player, wherein adjusting the gameplay includes (i) adjusting a volume and / or type of sound or music, and / or (ii) adjusting a color or background feature of the video game; adjusting a characteristic of the user device based at least in part on the one or more biometric characteristics; A video game device.

2. The video game device of claim 1 , wherein the at least one processor applies at least one learning model to process the biofeedback measurement data.

3. 3. The video game device of claim 1 or 2, wherein the at least one processor applies at least one of a Fourier transform or a spectral density analysis to process the biofeedback measurement data.

4. 3. The video game device of claim 2, wherein the at least one learning model is trained to determine a particular subset of the plurality of individual components that causes the video game player to have a particular cognitive state.

5. 5. The video game device of claim 1, wherein the plurality of individual components further includes at least one of chat messages, weapons, a selection of the character, an event associated with the character, or a characteristic of another video game player.

6. 6. The video game device of claim 1, wherein the one or more physical biofeedback sensors include one or more electroencephalography (EEG) electrodes and the biofeedback measurement data includes an EEG signal.

7. The video game device of claim 1 , wherein the one or more physical biofeedback sensors include one or more electrodes and the biofeedback measurement data includes neural signals.

8. 8. The video game device of claim 1, wherein the biofeedback measurement data includes at least one of a neural signal, an EEG signal, an EMG signal, an EOG signal, an fNIR signal, a signal indicative of blood flow, a functional near-infrared spectroscopy (fNIR) spectroscopic signal, a force sensitive resistor (FSR) signal, a facial expression detection signal, a pupil dilation indication signal, an eye movement signal, or a gestural motion signal.

9. 9. The video game device of claim 1, wherein the at least one processor determines a relative weighting of the contributions of the plurality of individual components to the determined one or more biological characteristics.

10. 10. The video game device of claim 1, wherein at least one of the one or more physical biofeedback sensors is incorporated into a head mounted display (HMD) device.

11. 11. The video game device of claim 1, wherein the processing of the biofeedback measurement data further comprises analyzing the biofeedback measurement data to determine whether any data is lost and / or corrupted.

12. at least one non-transitory processor-readable storage medium for storing at least one of data and instructions; at least one processor operatively coupled to the at least one non-transitory processor-readable storage medium; wherein, in operation, the at least one processor: providing game play to a population of video game players via respective user interfaces that provide video game functionality; receiving, from a physical biofeedback sensor proximate to the video game player, biofeedback measurement data of the video game player while the video game player is playing the video game, the biofeedback measurement data being captured during the display of a plurality of distinct components; analyzing the biofeedback measurement data to determine a subset of the plurality of individual components that contribute to an overall emotion or impression of the population of video game players, wherein analyzing the biofeedback measurement data includes determining one or more biometric characteristics of the video game players, the one or more biometric characteristics of the video game players including a state of emotional arousal of the video game players, and the plurality of individual components includes at least one of a character appearing in the video game and an action of the character appearing in the video game; adjusting the video game based at least in part on the determined one or more biometric characteristics of the video game player, wherein adjusting the video game includes (i) adjusting a volume and / or type of sound or music, and / or (ii) adjusting a color or background feature of the video game; adjusting a characteristic of the user device based at least in part on the one or more biometric characteristics; A video game system.

13. 13. The video game system of claim 12, wherein the plurality of individual components further comprises at least one of chat messages, weapons, a selection of the character, an event associated with the character, or a characteristic of another video game player.

14. 14. The video game system of claim 12 or 13, wherein the one or more physical biofeedback sensors include one or more electroencephalography (EEG) electrodes and the biofeedback measurement data includes EEG signals.

15. 15. A video game system as claimed in any one of claims 12 to 14, wherein to analyse the biofeedback measurement data, the at least one processor implements at least one model operable to identify individual components among the plurality of individual components that contribute to an overall emotion or impression of the video game player.

16. 16. The video game system of claim 12, wherein the at least one processor receives class information for each video game player and analyzes the biofeedback measurement data and the class information to determine how various classes of video game players respond differently to the plurality of individual components of the video game.

17. 17. The video game system of claim 12, wherein the at least one processor estimates a rating of the video game by the video game player based on the received biofeedback measurement data.

18. A method for manufacturing a computer-implemented computer-implemented system comprising: at least one non-transitory processor-readable storage medium for storing at least one of data and instructions; and at least one processor operatively coupled to the at least one non-transitory processor-readable storage medium; wherein, in operation, the at least one processor: providing game play to a population of video game players via respective user interfaces that provide video game functionality; receiving, from a physical biofeedback sensor proximate to the video game player, biofeedback measurement data of the video game player while the video game player is playing the video game, the biofeedback measurement data being captured during the display of a plurality of distinct components; analyzing the biofeedback measurement data to determine a subset of the plurality of individual components that contribute to an overall emotion or impression of the population of video game players, wherein analyzing the biofeedback measurement data includes determining one or more biological characteristics of the video game players; adjusting or enhancing the video game in response to the analysis of the biofeedback measurement data; adjusting a characteristic of the user device based at least in part on the one or more biometric characteristics; determining a response of the video game player to the plurality of individual components based on the received biofeedback measurement data, and estimating a life cycle of the video game based on changes in the determined responses over time. A video game system.

19. 19. The video game system of claim 12, wherein the analyzing the biofeedback measurement data further comprises determining whether there is lost and / or corrupted data.

20. one or more physical biofeedback sensors; at least one non-transitory processor-readable storage medium for storing at least one of data and instructions; at least one processor operatively coupled to the at least one non-transitory processor-readable storage medium and to the one or more physical biofeedback sensors; wherein, in operation, the at least one processor: providing game play to a video game player via a user interface that provides video game functionality; receiving biofeedback measurement data of the video game player from the one or more physical biofeedback sensors while the video game player is playing the video game; processing the biofeedback measurement data to determine an internal state of the video game player during the gameplay of the video game; adjusting the gameplay of the video game based at least in part on the determined internal state of the video game player; said processing said biofeedback measurement data includes analyzing said biofeedback measurement data to determine one or more biometric characteristics of said video game player, said one or more biometric characteristics of said video game player including a state of arousal for a plurality of distinct components of said video game player, said plurality of distinct components including at least one of a character appearing in said video game and an action of said character appearing; adjusting the game play includes (i) adjusting a volume and / or type of sound or music, and / or (ii) adjusting a color or background feature of the video game based at least in part on the determined one or more biometric characteristics of the video game player; In operation, the at least one processor further adjusts a characteristic of the user device based at least in part on the one or more biometric characteristics. Video game device.

21. 21. The video game device of claim 20, wherein the at least one processor utilizes the determined internal state to predict when the video game player is likely to stop playing the video game.

22. 22. The video game device of claim 20 or 21, wherein the at least one processor utilizes the determined internal state to determine the video game player's impression of at least one of weapons, the characters, a map, a game mode, a tutorial, game updates, a user interface, teammates, or a game environment.

23. 23. The video game device of claim 20, wherein the biofeedback measurement data includes at least one of a neural signal, an EEG signal, an EMG signal, an EOG signal, an fNIR signal, a signal indicative of blood flow, a functional near-infrared spectroscopy (fNIR) spectroscopic signal, a force sensitive resistor (FSR) signal, a facial expression detection signal, a pupil dilation indication signal, an eye movement signal, or a gestural motion signal.

24. 24. The video game device of claim 20, wherein at least one of the one or more physical biofeedback sensors is incorporated into a head mounted display (HMD) device.

25. 25. The video game device of claim 20, further comprising a head mounted display (HMD) device that includes at least one of the one or more physical biofeedback sensors.

26. 26. The video game device of claim 20, wherein the processing of the biofeedback measurement data further comprises analysing the biofeedback measurement data to determine whether any data is lost and / or corrupted.

Images