Using gaze tracking in multi-player interactive narrative to affect player selection

Abstract

The present disclosure is directed to controlling results in a game that includes a plurality of different users that play respective roles of a particular virtual character. When tracking movement of eyes of a plurality of different users, the plurality of different users may be located at the same physical location, or the different users may be located at different physical locations. Here, the user may select one of a set of provided picks by seeing only the selected pick for a period of time or by seeing the selected pick and performing an action or gesture. This functionality allows multiple different users to control actions performed by different particular figures via an online multiplayer system. Depending on the content viewed by the first user, a corresponding set of selected or audio / visual content may be provided to the first user or the second user via respective gaming devices operated by the first user and the second user.

Description

Background Art 1. Technical Field

[0001] The present invention relates generally to digital interactive content. More particularly, the present invention relates to using gaze tracking in multi-player digital games. 2. Background technology

[0003] There are various interactive content titles that include different possible narratives that are shaped based on the user's actions. For example, an interactive game may allow a user to control a character (or avatar) and provide input that specifies how the character interacts with objects (including other characters) in a virtual environment. For example, a character may be located at a particular location within a virtual environment and encounter an attacker character. The user may choose to have the character fight the attacker or flee the scene by inputting instructions via a controller (e.g., pressing the X button to start a fight, pressing the O button to flee, or other button press combinations). The interactive content title may respond to such input by updating the display of the virtual environment to include a fight scene or a scene of the character fleeing.

[0004] The virtual environment may be implemented in virtual reality (VR) and generated by a head mounted display (HMD) or one or more projectors. Navigation and character control in the virtual environment may be implemented using a variety of controllers, including handheld controllers, voice or gesture based controllers, wearable sensors and sensor based controls, and other input devices and systems. The variety of different controllers may present an obstacle to newer, less experienced players who have not yet acquired or developed muscle memory, reflexes, or basic knowledge of how to effectively use each controller.

[0005] Therefore, there is a need in the art for improved systems and methods that allow users to control characters and interactions within a virtual environment based on eye movements. Summary of the invention

[0006] Embodiments of the present invention may relate to methods, non-transitory computer-readable storage media, and systems for identifying content to be provided to different specific users playing games based on the items that different specific users are looking at when playing games. In a first embodiment, a method may include identifying that a first player is focusing on an object displayed on a display of a device of the first player. After the first player has been associated with a first game character interacting with a second character of the game, this identification may be performed by evaluating the received image data of the eyes of the first player. Next, a first indication may be sent to a second player device based on the identification that the eyes of the first player are focusing on the object displayed on the display of the first player device. The sending of this first indication may command the second player device to display a first group of selections, and then data corresponding to the first selection in the first group of selections may be received from the second player device. This method may also include providing a first group of audio-visual content to the first player via the first player device based on the received data associated with a first group of audio / visual content. The first group of audio-visual content may include audio of a second game character speaking to a first game character. After providing the first set of audio-visual content via the first player device, the method may continue by identifying that the first player has made a second selection in the second set of selections based on the first player focusing on the second selection, and the method may also include sending a second indication to the second player device to command the second player device to provide a second set of audio-visual content to the second player. The second set of audio-visual content may correspond to the second selection.

[0007] A method consistent with the present disclosure may be implemented as a non-transitory computer-readable storage medium, wherein a processor executes instructions from a memory. Again, the method may include identifying that a first player is focusing on an object displayed on a display of the first player's device. After the first player has been associated with a first game character interacting with a second character of the game, such identification may be performed by evaluating the received image data of the first player's eyes. Next, a first indication may be sent to a second player device based on the identification that the eyes of the first player are focusing on the object displayed on the display of the first player device. The sending of the first indication may command the second player device to display a first set of selections, and data corresponding to the first selection in the first set of selections may then be received from the second player device. This method may also include providing a first set of audio-visual content to the first player via the first player device based on the received data associated with a first set of audio / visual content. The first set of audio-visual content may include audio of the second game character speaking to the first game character. After providing the first set of audio-visual content via the first player device, the method may continue by identifying that the first player has made a second selection in the second set of selections based on the first player focusing on the second selection, and the method may also include sending a second indication to the second player device to command the second player device to provide a second set of audio-visual content to the second player. The second set of audio-visual content may correspond to the second selection.

[0008] A system consistent with the present disclosure may include a first gaming device and a second gaming device, wherein each of these respective gaming devices includes a respective memory and a respective processor that executes instructions stored at the respective memory. The processor of the first player device may execute instructions to identify that a first player associated with a first gaming character has focused on an object displayed on a display of the first player device. The processor of the first player device may then prepare to send a first indication to the second player device based on the identification that the first player is focusing on the object displayed on the display of the first player device. The first indication may be associated with a first set of selections. The processor of the second player device may execute instructions to display the first set of selections on the display of the second player device based on receiving the first indication. The processor of the second player device may then prepare to send data to the first player device identifying that the second player has made the first selection of the first set of selections by focusing on the first selection. Such a system may then provide the first player with a first set of audio / visual content corresponding to the first selection via a speaker and display of the first player device based on receiving the data sent from the second player device. Again, the first set of audio-visual content including the audio of the second gaming character may be provided via the speaker of the first player device. Then, after identifying that the first player made a second selection in the second set of selections by focusing on the second selection, the processor of the first player device may prepare to send a second indication to the second player device. Next, when the processor of the second player device executes additional instructions, the system may provide a second set of audio-visual content to the second player based on the receipt of the second indication sent from the first player device. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An exemplary network environment is shown in which a system for gaze-based control in a multi-player game may be implemented.

[0010] Figure 2 is a flow chart illustrating an exemplary method for gaze-based control in a multi-player game.

[0011] Figure 3 is a flow chart illustrating another exemplary method for gaze-based control in a multi-player game.

[0012] Figure 4 is a flow chart illustrating yet another exemplary method for gaze-based control in a multi-player game.

[0013] Figure 5A computing system is shown that can be used to implement embodiments of the invention. DETAILED DESCRIPTION

[0014] The present disclosure is directed to controlling outcomes in a game that includes multiple different users (i.e., players) playing respective roles of specific game characters. When the movement of the eyes of multiple different users is tracked, the multiple different users may be located at the same physical location, or the different users may be located at different physical locations (or both). Here, a user may select one of a set of provided choices by simply looking at the selected choice for a period of time or by looking at the selected choice and performing an action or gesture (e.g., a gesture, a blink, or a head movement). This functionality allows multiple different users to control actions performed by different specific characters via an online multiplayer system. Depending on what the first user is looking at, a corresponding set of choices or audio / visual content may be provided to the first user or the second user via a corresponding gaming device operated by the first user and the second user.

[0015] When several players (users) are at the same physical location, each of those several players may interact via one or more gaming devices, one or more sets of electronics, displays, individual head mounted displays (HMDs), or some combination thereof. When different players are at different physical locations, each of those players may interact with a corresponding set of electronics, HMDs, or gaming systems.

[0016] Each of the respective players of the game may have the opportunity to select a game character that they wish to "play as." Once a particular player selects a character, that player will assume the role of that character while playing the game. For example, a first player may select to play the role of a customer, a second player may select to play the role of a store owner, and a third player may select to play the role of the customer's wife. Such games may also include "non-player characters," where the gaming device automatically assumes the role of one or more characters.

[0017] As the players play the game, actions performed by one player can be used to identify selections that may be offered to the other player. Such actions may be related simply to the fact that the first player has focused on an object shown on a display. In an example, imagine a scene in an equipment store with two characters playing a game. These characters may include a customer and a store manager. In one instance, the customer may be a "player character" (i.e., a character that acts according to the choices made by the player who assumes the role of the character), and the store manager may be a "non-player character." Here, the customer may choose to "buy the item" or "negotiate a better price." Depending on the player's selection, the store manager will react in a pre-programmed manner (e.g., sell the item or give a discount).

[0018] Alternatively, the store manager may be controlled by another player who has his own sets of store manager specific choices. These choices may be determined by the choices made by the customer. For example, if the customer chooses "Negotiate for a better price", the player controlling the store manager may be presented with his own options: "Offer a lower price" or "Smile and say no".

[0019] Figure 1 An exemplary network environment in which a system for gaze-based control in a multi-player game may be implemented is shown. As shown, the network environment may include different gaming devices 120 / 150 / 170 communicating via a communication network (e.g., cloud / internet 110). Each of the gaming devices 120 / 150 / 170 may be associated with one or more players located in the same or different physical geographic locations. Figure 1 It is also shown that the gaming device 120 / 150 / 170 may also communicate with one or more head mounted displays 130 / 140 / 160 or a game controller 180. The gaming devices 120, 150, and 170 may be gaming systems (e.g., Sony TM PlayStation), computer, handheld device, or another computing device.

[0020] As shown, a player 1 head mounted display (HMD) 130 and a player 4 HMD 140 may communicate with a first gaming device 120. Here, player 1 and player 4 may be located in close proximity to the first gaming device 120 with which their respective HMDs 130 / 140 may communicate. Figure 1 Each of the different gaming devices (120, 150, and 170) may be located at different physical locations, and each of these different gaming devices may communicate with each other through the cloud or the Internet.

[0021] Figure 1 Player 2 HMD 160 is also shown communicating with a second gaming device 150 , and player 3 game controller is shown communicating with a third gaming device 170 . Figure 1 Different gaming devices in the game can be configured to associate a specific game character with a specific user or player of the game. When using a head mounted display, the player can experience the game in a virtual reality environment, and depending on the capabilities of the specific HMD, the user can provide commands to the gaming device via the specific HMD. When using a game controller, the player (e.g., Figure 1 Player 3) in the game can view the game on a standard display (e.g., a flat-panel display) and can provide commands to the game using their game controller.

[0022] The functionality of a particular head mounted display may allow a user to provide control inputs to control actions performed by a game character associated with a particular player HMD. The HMD may receive inputs based on eye tracking data, user gestures (e.g., hand movements or blinks), or a combination of eye tracking and user gestures. In some implementations, eye tracking and gesture data may also be obtained from cameras and other sensors in the surrounding real world environment of the respective user. Commands for performing specific types of actions in a virtual environment may be identified based on an analysis of where the user is looking. For example, a specific action may be indicated for a direction of gaze that is maintained for a threshold period of time.

[0023] The equipment of the present disclosure may also include multiple speakers that provide direction-based noise. Here, sound can be provided to the user via speakers located on the left, right, front, rear, top or bottom of the person. This three-dimensional (3D) speaker system can be incorporated into the HMD, or can be arranged in the room of the game player. When using this 3D speaker system, the noise provided to the user can be directional. For example, the ringing of the bell on the left side of the player, the hissing rattlesnake under the player's feet, the sound of the aircraft flying above the player or the loading of the equipment behind the player. Although such 3D speaker systems can be supported by specific games, HMDs, speaker systems and / or game systems, even if some players do not have available 3D speaker systems, the game can still run. Game controllers (such as player 3 game controllers 180) can be any type of controllers known in the art (for example, joysticks, two-handed controllers or one or more single-handed controllers). Even when using a game controller, the game device or the controller itself can also include a camera dedicated to tracking the movement of the user's eyes.

[0024] The method of the present disclosure can provide various types of realism to the user based on the limitations of the type of device used to receive user input and provide content (e.g., sound, image, or video) to the user. Figure 1 The head mounted display, game controller, and gaming device are shown as separate physical devices, but the functionality of each of these devices can be incorporated into a single device. For example, a head mounted display (HMD) can also perform the functionality of a gaming device. In such an example, two HMDs located at the same physical location can act as separate gaming devices, or one of these HMDs can act as a master HMD that receives input from a second HMD.

[0025] In some instances, graphics or audio may be generated on a cloud server or cloud computing device, which may send the generated graphics images or audio content to a particular user device. This may allow computing devices with limited graphics generation capabilities to receive and display high-quality graphics images. In addition, many of the functions described herein may be performed in the cloud by one or more cloud servers or cloud computing devices based on cloud-based services associated with the current session. For example, associated data and content may be stored in one or more cloud storage devices, and such data may be processed using cloud computing resources in conjunction with data received from one or more user devices (e.g., devices 120 / 150 / 170) in the current session.

[0026] Figure 2 is a flow chart illustrating an exemplary method for gaze-based control in a multi-player game. Such a method may be performed by a gaming device for multi-player game play. Here, a first player and a second player may be located at different locations and may use a gaze-based control method. Figure 1 The communication is sent to the corresponding gaming device through the Internet (e.g., the cloud or Internet 110) to control the gaming action or sequence.

[0027] As previously mentioned, the scenario described above may occur in a virtual reality (VR) environment in which a player wears a head mounted display (HMD). In one example, when a player controlling a store manager has chosen to set the store manager as a woman wearing a low-cut shirt, and when a player controlling a customer looks at the store manager from a first-person perspective, when the customer focuses on the store manager's face, a first set of optional choices may be provided to the store manager player, and when the customer focuses on the store manager's low-cut shirt, a second set of optional choices may be provided to the store manager player. In the first example, when the customer player's gaze rests on the store manager's face (e.g., maintaining "eye contact") while choosing to negotiate a better price, the store manager's response selection choice may be "offer a lower price" or "reject with a smile."

[0028] Figure 2The method begins at step 210, where the gaze direction or eye focus / focal point of a player (e.g., a first player or a second player) can be identified. As described above, this can include identifying whether the customer player (first player) is looking at the store manager's face or at the store manager's collarbone, and the corresponding set of player choices discussed above can be provided to the store manager player (second player). After determining step 210 identifies where the first player (customer player) is looking, determining step 220 can identify whether a set of choices should be provided to the second player (store manager player). Determining step 220 can identify that a set of choices should not be provided to the second (store manager) player because the first (customer) player is not looking at the store manager's face or collarbone. In this example, the program flow can move to determining step 250 discussed below.

[0029] When the determination step 220 identifies that a choice should be provided to the second (store manager) player, an appropriate set of choices may be provided to the second (store manager) player. This may include displaying text in a display viewed by the second player and the second player may select a response from a set of responses. Here, the selection may be made by the second (store manager) player by staring at a particular line of text for a threshold amount of time (e.g., 5 seconds), or may include the second (store manager) player staring at a particular line of text and making a gesture (e.g., a hand movement, an eye movement, or a blink). This may include Figure 2 The second player's gaze direction or focus / focal point may be identified in step 230, after which the first (customer) player may be provided with content corresponding to the selection made by the second (store manager) player. Step 240 may include playing a video of the store manager character laughing and saying "No, my prices are fixed, no discounts".

[0030] After step 240, program flow may move to a determination step 250 to identify whether a set of response choices should be provided to the first (customer) player. For example, these choices may include "OK, I'll pay your price" or "I can only spend $200 and am willing to make up the difference with this knife."

[0031] As described above, determination step 250 may be performed after determining step 220 identifies that the second (store manager) player should not be offered a choice. In such an example, the first player may be offered a choice that says "Please show me that gear" or may include "What parts do you have that I can buy for $200?"

[0032] A first (customer) player may be offered a set of choices and may be based on Figure 2The selection may be identified by the first player's gaze direction or focus / focal point in step 260 of FIG. After step 260, content may be provided to the second (store manager) in step 270 of FIG. The content may include audio of the first (customer) player stating "What parts do you have that I can buy for $200?"

[0033] Alternatively, when the determination step 250 identifies that the first player should not be provided with a first player pick, program flow may move back to step 210 where a gaze direction or focus of the first player and / or the second player may be identified.

[0034] After step 270, the program flow may move to a determination step 280 to identify whether a third character should participate in the next scene of the interaction between the first player and the second player, and when yes, the program flow may move to step 290 where the third character is introduced. Here, the third player or character may be controlled by the third player, or the actions performed by the third character may be automatically performed by the gaming device selecting the actions that the third character should perform. Such automated characters may be referred to as non-player characters because the actions of the character are not controlled by the player.

[0035] When the determination step 280 identifies that the third character should not participate in the next scene of the game, the program flow can move back to step 210 where the gaze direction or focus of the first user and / or the second user can be identified. This can result in another sequence of interactions between the actions of the first player controlling the customer character and the actions of the second player controlling the store manager player. Depending on where each respective player is looking, different choices can be provided to the respective players, and additional content can be provided based on the choices made by those respective players. For example, such interactions can continue until the customer makes a purchase or leaves the equipment store.

[0036] Figure 3 is a flow chart illustrating another exemplary method for gaze-based control in a multi-player game. Such a method may be performed by a gaming device when a third character or player participates in a gaming session. Figure 2 After the determination step 280 identifies that the third person should participate in the game, the following may be performed: Figure 3 steps. Figure 3 Beginning with a determination step 310 of identifying whether a third player exists to control the actions of a third character. When determination step 310 identifies that a third player does not exist, program flow may move to step 340 where the actions are performed by the gaming device that automatically selects the actions that the third character should perform.

[0037] When the determination step 310 identifies that a third player exists, the program flow can move to step 320 where the gaze direction or focus / focal point of any or all players can be identified. Next, the player to whom a set of selection options can be provided can be identified. Then, the determination step 330 can identify whether a specific set of selection options should be provided to the identified player. Again, the different sets of selection options provided to a particular player can be based on the location where the particular player is looking. The determination step 330 can be performed after step 320, or can be performed after step 340, depending on whether the third player can play the game. When the determination step 330 identifies that selection should not be provided, the program flow can move back to step 320 where the player's gaze direction is evaluated again. This can cause the program flow to repeat the loop between steps 320 and 330 until the determination step 330 identifies that a set of selection options should be provided to the player.

[0038] When the determination step 330 identifies that pick options should be provided to the players, a set of pick options corresponding to the gaze direction of one player may be provided to any player in the game. In some instances, rather than the gaze direction of the first player causing the pick options to be sent to the second player, the pick options may be sent to the first player. For example, the first player may be looking at a box of ammunition, and the first (customer) player may be provided with a set of different statements (such as comments or questions) that the first (customer) player may ask the second (store manager) player. Such a comment may be "I like that plus P hollow point bullet," or a question may be "What's the price of that box of ammunition?"

[0039] Next, in step 350, the identified player may be provided with the identified set of selection options, and then in step 360, content corresponding to the selection made by the identified player may be provided. This may result in the first (customer) player asking the second (store manager) player "What is the price of that box of ammunition?"

[0040] After step 360, program flow may move to a determination step 370 where a non-player character action may be performed, and when so, program flow may move to step 340 where the non-player character action is performed by a third character. This may include the third character speaking a phrase automatically selected by the gaming device. When determination step 370 identifies that the non-player character action should not be performed, program flow may move back to step 320 where the gaze direction or focus / focal point of one or more players is again identified.

[0041] Figure 3 The steps of show that the actions performed by multiple characters can be performed based on the gaze direction or focus / focus point of a specific user playing the game. This can be the case even if the actions of some characters are performed automatically. Figure 3The steps can also be based on the execution Figure 2 steps to execute.

[0042] In yet another example related to the store manager scenario, when a customer is facing the store manager behind the counter, a bell to the customer's left rings (discernible via 3D audio) as the store door opens and a third person enters. In one instance, when the bell rings, the customer's wife (e.g., Figure 3 When a third player / character) enters a store, different selection options can be provided to a specific player based on the direction of the customer player's gaze.

[0043] In one example, when the customer turns around and sets his gaze on his wife, the selection options that may be offered to the customer player may include "What are you doing here?" and "You are late." At the same time, the store manager may become inactive, allowing a conversation between the customer and his wife.

[0044] In one example, when the customer does not turn toward the ringing bell, his wife may enter out of his field of vision. The scene between the customer and the store manager may then continue as before, with the customer attempting to negotiate a better price and the store manager making the selections described above. At the same time, the player playing the wife may observe the scene from different perspectives, including her husband (the customer) and the store manager, and the direction of the game may be based on the direction of the wife's gaze.

[0045] In another example, the first player can play the role of a detective character, investigating the crime scene in the presence of a suspect character controlled by a suspect player. Here, the actions performed by the detective player and the suspect player can be controlled by the actions performed by different players. In an example, when the detective player's gaze falls on a key piece of evidence (such as a murder weapon), the suspect player may face an action selection. In an example, when the detective's gaze never touches the evidence, the suspect player may not receive a selection. In each of these scenarios, the actions performed by certain players may result in providing a selection option to any other player, wherein each action and the resulting selection will cause the game to move in different directions, thereby resulting in different results.

[0046] In some instances, the player's gaze may trigger an action performed by a non-player character (NPC): a customer player's gaze on a price tag may be used to trigger the NPC store manager to automatically say "Can I help you find something?" In one instance, the NPC store manager may not ask the question "Can I help you find something?" when the player is not looking at the price. Instead, other phrases may be provided for the NPC store manager to say when the customer is looking at another item.

[0047] The duration of the gaze can be used to vary the intensity of the response. A quick glance at the store manager's collarbone can elicit a set of choices (or no choices at all), while a longer gaze can elicit a different set of more aggressive choices. The gaze threshold time can correspond to a number of seconds, such as two, five, or seven seconds.

[0048] Figure 4 is a flow chart showing another exemplary method for gaze-based control in a multi-player game. Such a method may be performed by a gaming device to identify a player's perspective or emotion. Figure 4 Beginning at step 410, where player pupil dilation data may be evaluated. This may include collecting a first set of eye image data of the player at a first point in time, evaluating the eye image data to identify the degree of dilation of the player's eyes at the first point in time, collecting a second set of image data at a second point in time, and identifying the degree of dilation of the player's eyes at the second point in time. Then, a determination step 420 may identify whether the player's eye dilation has changed. When the player's eye dilation has not changed, program flow may move to step 450, where the player's facial expression data is evaluated.

[0049] When determination step 420 identifies that the player's eye dilation has changed, program flow can move to step 430 where the player's perspective can be identified. For example, changes in the player's eye dilation can be used to identify an "interested" or "suspicious" perspective. When the player's pupils dilate (become larger), the game system can identify that the player is "interested" in the object or person the player is currently looking at. In one instance, when the player's pupils constrict (become smaller), the game system can identify that the player is "suspicious" about what they are looking at. Therefore, the method of the present disclosure can make a judgment based on the observed changes in the player's pupil dilation. In step 440, when the player appears to be interested in an object, content (audio and / or visual content) associated with the interest can be provided to the player. For example, when the player's pupils dilate when looking at a car, a set of selection options or audio questions can be provided to the player, asking the player if he is willing to drive the car.

[0050] In one example, when the player's pupils shrink, based on the game system recognizing that the player looks suspicious due to the shrinking of the player's pupils, another character can Figure 4 In step 440, an audio message is provided saying "Don't worry, you can trust me".

[0051] After step 440, the program flow can move to step 450 where the player's facial expression data can be evaluated. Here, the interpreted player's facial expressions and / or emotions can be used to change the reactions of specific characters. Here, a full-face camera or a specific HMD camera can be used to collect images so that the player's expressions and / or emotions can be detected. The evaluation performed in step 450 may include collecting multiple images of the player's face over time and may include identifying that the player's facial expression has changed. Alternatively, the facial expressions in a single image can be evaluated to see if the single facial expression corresponds to an emotion. Then, determination step 460 can identify whether the current facial expression or change in facial expression may correspond to an emotion. When determination step 460 identifies that the facial expression data does not correspond to an emotion, the program flow can move to step 490 where other actions can be performed.

[0052] When determination step 460 identifies that the facial expression data corresponds to an emotion, program flow can move to step 470 where a specific emotion (e.g., confusion, anger, happiness, excitement, or nervousness) can be identified. For example, the emotion of confusion can be identified by the player's eyes slightly squinting. The emotion of anger can be identified based on the following: the eyes are more obviously squinting, wrinkles appear between the player's eyebrows, the player's mouth is distorted, or some combination of these factors. The emotion of happiness can be identified based on identifying that the player is smiling, and the emotion of excitement can be identified based on identifying that the size of the opening of the player's eyelids increases. The emotion of nervousness can be identified by smiling and giggling or noticing that the player moves his body in a jerky manner.

[0053] Next, in Figure 4 In step 480, actions may be initiated based on the identified emotions. These expressions or detected emotions may be used as information, which may be used to change the narrative of the game. For example, in one instance, when the player's face indicates confusion, the store manager may be given additional explanatory lines of dialogue about the product. In one instance, when the player's expression indicates happiness, the store manager's reaction may be more cheerful than when the player is angry. Again, the store manager's statements may be based on user-selected options of the player playing the role of the store manager, or may be made through the gaming device, which may be programmed to automatically provide statements corresponding to the player's expression or identified emotional state. After step 480, program flow may move to step 490 where other actions are performed. These other actions may include performing actions regarding Figure 2 or Figure 3 The steps discussed.

[0054] Figure 5 A computing system is shown that can be used to implement embodiments of the invention. Figure 5The computing system 500 includes one or more processors 510 and a main memory 520. The main memory 520 stores instructions and data in part for execution by the processor 510. The main memory 520 can store executable code when operating. Figure 5 The system 500 also includes a mass storage device 530 , a portable storage media drive 540 , an output device 550 , a user input device 560 , a graphics display 570 , peripheral devices 580 , and a network interface 595 .

[0055] Figure 5 The components shown in the figure are depicted as being connected via a single bus 590. However, the components may be connected by one or more means of data transport. For example, the processor unit 510 and the main memory 520 may be connected via a local microprocessor bus, and the mass storage device 530, the peripheral device 580, the portable storage device 540, and the display system 570 may be connected via one or more input / output (I / O) buses.

[0056] Mass storage 530, which may be implemented as a magnetic disk drive or optical disk drive, is a non-volatile storage device for storing data and instructions for use by processor unit 510. Mass storage 530 may store system software for implementing embodiments of the present invention for loading into main memory 520.

[0057] Portable storage device 540 operates in conjunction with a portable non-volatile storage medium such as a flash memory, an optical disk, or a digital video disk to store data. Figure 5 The computer system 500 inputs data and codes and receives Figure 5 The computer system 500 outputs data and codes. System software for implementing embodiments of the present invention may be stored on such portable media and input into the computer system 500 via the portable storage device 540.

[0058] Input device 560 provides a portion of the user interface. Input device 560 may include an alphanumeric keypad, such as a keyboard, or a pointing device, such as a mouse, trackball, stylus, or cursor direction keys, for inputting alphanumeric and other information. Figure 5 The system 500 shown in FIG. 5 includes an output device 550. Examples of suitable output devices include speakers, printers, network interfaces, and monitors.

[0059] The display system 570 may include a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, an electronic ink display, a projector-based display, a holographic display, or another suitable display device. The display system 570 receives text and graphic information and processes the information for output to the display device. The display system 570 may include multi-touch touch screen input capabilities, such as capacitive touch detection, resistive touch detection, surface acoustic wave touch detection, or infrared touch detection. Such touch screen input capabilities may or may not allow variable pressure or force detection.

[0060] Peripheral device 580 may include any type of computer support device to add additional functionality to a computer system. For example, peripheral device 580 may include a modem or a router.

[0061] The network interface 595 may include any form of computer interface of the computer, whether a wired network or a wireless interface. Thus, the network interface 595 may be an Ethernet network interface, a Bluetooth TM Wireless interface, 802.11 interface, or cellular telephone interface.

[0062] Figure 5 The components included in computer system 500 are those commonly found in computer systems that may be suitable for use with embodiments of the present invention and are intended to represent a broad category of such computer components that are well known in the art. Figure 5 The computer system 500 may be a personal computer, a handheld computing device, a phone ("smart" or otherwise), a mobile computing device, a workstation, a server (on a server rack or otherwise), a minicomputer, a mainframe computer, a tablet computing device, a wearable device (such as a watch, a ring, a pair of glasses, or another type of jewelry / clothing / accessory), a video game console (portable or otherwise), an e-book reader, a media player device (portable or otherwise), a vehicle-based computer, some combination thereof, or any other computing device. The computer may also include different bus configurations, networked platforms, multi-processor platforms, etc. The computer system 500 may in some cases be a virtual computer system executed by another computer system. Various operating systems may be used, including Unix, Linux, Windows, Macintosh OS, Palm OS, Android, iOS, and other suitable operating systems.

[0063] The present invention may be implemented in applications that may be operated using a variety of devices. Non-transitory computer-readable storage media refers to any one or more media that participate in providing instructions to a central processing unit (CPU) for execution. Such media may take many forms, including but not limited to non-volatile and volatile media, such as optical or magnetic disks and dynamic memory, respectively. Common forms of non-transitory computer-readable media include, for example, flash memory / flash disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, digital video disks (DVDs), any other optical media, RAM, PROMs, EPROMs, FLASH EPROMs, and any other memory chips or cartridges.

[0064] Although the various flow charts provided and described above may show a specific order of operations performed by certain embodiments of the present invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc.).

Claims

1. A method for providing gaze-based control, the method comprising: After a first character associated with a first player interacts with a second character in a virtual environment, identifying based on image data of one or more eyes of the first player that a gaze of the first player is currently focused on a virtual object in the virtual environment; sending a first command to a second user device based on the identified gaze being focused on the virtual object, wherein the command is executable by the second user device to display a first set of options, and receiving data corresponding to a selection from the first set of options from the second user device; providing a first set of audio-visual content to a first user device based on the selection from the first set of options, the first set of audio-visual content including audio of the second character speaking within the virtual environment; identifying, based on the updated location of the identified gaze within the virtual environment, that the first player has made a selection from a second set of options; as well as A second command is sent to the second user device, wherein the second command is executable by the second user device to display a second set of audio-visual content corresponding to the selection from the second set of options. 2 . The method of claim 1 , wherein the audio of the second character speaking includes a set of words as part of a conversation between the first character and the second character.

3. The method of claim 1, wherein the second set of audio-visual content includes a set of words spoken by the first character, the set of words being part of a conversation between the first character and the second character.

4. The method of claim 1, wherein at least one of the first set of audio-visual content and the second set of audio-visual content comprises three-dimensional (3D) noise initially associated with a direction.

5. The method according to claim 1, further comprising: identifying that the identified gaze is moving in a direction within the virtual environment; as well as A display of an image of a third person is generated based on the movement of the identified gaze in the identified direction.

6. The method according to claim 5, further comprising: receiving data from a third user device associated with the third persona; as well as A third set of audio-visual content is provided to the first user device based on the data received from the third user device.

7. The method according to claim 1 further includes tracking the identified gaze of the first player to the updated location within the virtual environment, wherein text associated with the selection from the second set of options is displayed at the updated location, wherein the selection from the second set of options is identified based on the identified gaze pointing to the text at the updated location. The method of claim 7 , further comprising identifying that the identified gaze is focused on the text for at least a predetermined period of time.

9. The method of claim 1, wherein identifying that the first player has made the selection from the second set of options is also based on a detected gesture of the first player.

10. The method of claim 9, wherein the gesture comprises a blink of the first player's eye. The method of claim 10 , wherein the blinking comprises at least two different eye closings and eye openings.

12. The method according to claim 1, further comprising: identifying that a size of a pupil of at least one of the first player or the second player has changed; as well as An action is performed based on the identified change in pupil size.

13. The method of claim 1, wherein identifying that the first player has made the selection from the second set of options is further based on an expression of the first player.

14. The method of claim 13, further comprising matching the expression of the first player to one or more indicators of one or more predetermined emotions.

15. The method of claim 14, wherein the second set of options is further based on one of the predetermined emotions that matches the expression of the first player.

16. A non-transitory computer-readable storage medium having embodied thereon a program executable by a processor to implement a method for providing gaze-based control, the method comprising: After a first character associated with a first player interacts with a second character in a virtual environment, identifying based on image data of one or more eyes of the first player that a gaze of the first player is currently focused on a virtual object in the virtual environment; sending a first command to a second user device based on the identified gaze being focused on the virtual object, wherein the command is executable by the second user device to display a first set of options, and receiving data corresponding to a selection from the first set of options from the second user device; providing a first set of audio-visual content to a first user device based on the selection from the first set of options, the first set of audio-visual content including audio of the second character speaking within the virtual environment; identifying, based on the updated location of the identified gaze within the virtual environment, that the first player has made a selection from a second set of options; as well as A second command is sent to the second user device, wherein the second command is executable by the second user device to display a second set of audio-visual content corresponding to the selection from the second set of options.

17. A system for providing gaze-based control, the system comprising: a memory storing information about audio-visual content associated with the virtual environment; A communication interface that communicates with at least a first user device and a second user device via a communication network, wherein the communication interface: receiving data from one or more eye tracking sensors associated with the first user device, wherein the data from the eye tracking sensors indicates that after a first character associated with a first player interacts with a second character in the virtual environment, based on image data from one or more eyes of the first player, a gaze of the first player is currently focused on a virtual object in the virtual environment, sending a first command to a second user device based on the gaze being focused on the virtual object, wherein the command is executable by the second user device to display a first set of options, and receiving data corresponding to a selection from the first set of options from the second user device, providing a first set of said audio-visual content to said first user device based on said selection from said first set of options, said first set of said audio-visual content comprising audio of said second character speaking within said virtual environment; as well as a processor that executes instructions stored in the memory, wherein the processor executes the instructions to: identifying that the gaze of the first player is at an updated location within the virtual environment, and recognizing that the first player has made a selection from a second set of options based on the updated location of the identified gaze within the virtual environment, wherein the communication interface sends a second command to the second user device, and wherein the second command is executable by the second user device to display a second set of the audio-visual content corresponding to the selection from the second set of options.

18. The system of claim 17, wherein the memory comprises a cloud storage device, and wherein the first set of the audio-visual content and the second set of the audio-visual content are provided by the cloud storage device.

Images