Group control of computer games using aggregated gaze areas

Abstract

A group of humans use teamwork to control a computer game. This can be done by eye tracking (400) each human to detect where on the screen each human is looking at an object, such as a game control object. Control actions for the object seen by the most humans (404) are implemented by the game in a "heat map" style of data collection (408).

Description

[Technical Field]

[0001] This application relates generally to group control of computer simulations, such as computer games, using aggregated gaze areas. [Background technology]

[0002] As understood herein, computer simulations, such as computer games, can be increasingly group-oriented. Summary of the Invention

[0003] As will be further appreciated herein, fun and intuitive group control of computer games can be facilitated.

[0004] Thus, the apparatus includes at least one processor configured with executable instructions to identify a gaze direction of each of a plurality of viewers of the computer simulation and generate an input signal to the computer simulation based at least in part on the gaze direction.

[0005] The input signal may be a single input signal determined by gaze direction, or the input signal may be a selection signal that identifies one of multiple options presented in a computer simulation.

[0006] In some embodiments, the input signal may include a character control signal for controlling a character in a computer simulation, or the input signal may include a vote, or the input signal may include a navigation command.

[0007] In a non-limiting implementation, a first set of gaze directions is directed toward a first object in a computer, a second set of gaze directions is directed toward a second object in a computer simulation, the input signal corresponds to the first object, and the input signal is generated based on the first set of gaze directions being greater than the second set of gaze directions.

[0008] The computer simulation may be a 3D simulation, or it may be a legacy 2D computer simulation. A gaze direction may be counted only in response to the gaze being at a location for at least a threshold duration.

[0009] In another aspect, a method includes imaging eyes of a plurality of players of a computer simulation and identifying crowd-based input commands for the computer simulation based on the imaging.

[0010] In another aspect, the device is not a transitory signal, and then includes at least one computer storage device containing instructions, the instructions executable by at least one processor to track the gazes of multiple users of the computer simulation and implement community-based control of the computer simulation based on the gazes.

[0011] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram of an example system in accordance with present principles; [Figure 2] Specific systems of embodiments consistent with the present principles are illustrated. [Figure 3] 1 illustrates a screenshot of a user interface (UI) of an embodiment. [Figure 4] 1 illustrates the overall logic of an embodiment in flowchart format. [Figure 5] 1 illustrates the specific logic of the first embodiment in flowchart format of an embodiment. [Figure 6] 1 illustrates the specific logic of the first embodiment in flowchart format of an embodiment. [Figure 7] 10 illustrates a screenshot of an example that schematically illustrates a group gaze point. DETAILED DESCRIPTION OF THE INVENTION

[0013] The present disclosure generally relates to computer ecosystems, including, but not limited to, aspects of consumer electronics (CE) device networks such as computer gaming networks. Systems herein may include server and client components that may be connected over a network so that data may be exchanged between the client and server components. The client components may include one or more computing devices, including game consoles such as Sony PlayStation® or game consoles made by Microsoft, Nintendo, or other manufacturers; extended reality (XR) headsets such as virtual reality (VR) headsets, augmented reality (AR) headsets; portable televisions (e.g., smart TVs, Internet-enabled TVs); portable computers such as laptops and tablet computers; and smartphones and other mobile devices, including additional examples discussed below. These client devices may operate in a variety of operating environments. For example, some of the client computers may employ, by way of example, the Linux® operating system, an operating system from Microsoft, or a Unix® operating system, or an operating system produced by Apple, Inc., or Google, or the Berkeley Software Distribution or BSD OS, including Descendants of Berkeley Standard Distribution (BSD). These operating environments may be used to run one or more browsing programs, such as browsers created by Microsoft, Google, or Mozilla, or other browser programs capable of accessing websites hosted by the Internet servers discussed below. Also, operating environments according to present principles may be used to run one or more computer game programs.

[0014] Servers and / or gateways may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, clients and servers may be connected through a local intranet or virtual private network. The server or controller may be instantiated by a game console such as a Sony PlayStation®, personal computer, or the like.

[0015] Information may be exchanged between clients and servers over a network. To this end, and for security, the servers and / or clients may include firewalls, load balancers, temporary storage, proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing network members with a secure community, such as an online social website or gamer network.

[0016] The processor may be a single-chip processor or a multi-chip processor capable of executing logic through various lines such as address lines, data lines, and control lines, as well as registers and shift registers. Processors including digital signal processors (DSPs) may be circuit embodiments.

[0017] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described and / or illustrated herein may be combined, interchanged, or eliminated from other embodiments.

[0018] A "system having at least one of A, B, and C" (and similarly, "a system having at least one of A, B, or C" and "a system having at least one of A, B, and C") includes systems having A alone, having B alone, having C alone, having A and B together, having A and C together, having B and C together, and / or having A, B, and C together.

[0019] 1 , an example system 10 is shown that may include one or more of the example devices referenced above and further described below in accordance with the present principles. A first example device included in system 10 is a consumer electronics (CE) device, such as an audio-video device (AVD) 12, which may be, but is not limited to, a projector-based TV with a TV tuner or an Internet-enabled TV (equivalently, a set-top box that controls the TV), a theater display system, etc. AVD 12 may alternatively be a computer-controlled Internet-enabled (“smart”) phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or headset such as smart glasses or a VR headset, another wearable computer-controlled device, a computer-controlled Internet-enabled music player, computer-controlled Internet-enabled headphones, a computer-controlled Internet-enabled embedded device such as an embedded skin device, etc. It will be understood that AVD 12 is nevertheless configured to embody the present principles (e.g., to embody the present principles, execute the logic described herein, and communicate with other CE devices to perform any other functions and / or operations described herein).

[0020] Accordingly, AVD 12 can be established by some or all of the components shown to undertake such principles. For example, AVD 12 can include one or more touch-enabled displays 14, which may be implemented with high-resolution or ultra-high-resolution "4K" or higher flat screens. Touch-enabled display(s) 14 can include, for example, a capacitive or resistive touch-sensing layer having a grid of electrodes for touch sensing consistent with the present principles.

[0021] The AVD 12 may also include one or more speakers 16 for outputting audio in accordance with the present principles and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to the AVD 12 to control it. The example AVD 12 may also include one or more network interfaces 20 for communication over at least one network 22, such as the Internet, a WAN, or a LAN, under the control of one or more processors 24. Thus, the interface 20 may be, without limitation, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as a mesh network transceiver. It will be understood that the processor 24 controls the AVD 12 to implement the present principles, including other elements of the AVD 12 described herein, such as controlling the display 14 to present images thereon and receiving input therefrom. It will further be noted that the network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephony transceiver, or the Wi-Fi transceiver mentioned above.

[0022] In addition to the foregoing, AVD 12 may also include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI) port or a Universal Serial Bus (USB) port for physically connecting to another CE device, and / or a headphone port for connecting headphones to AVD 12 for presentation of audio from AVD 12 to a user through headphones. For example, input port 26 may be connected via wire or wirelessly to a cable or satellite source 26a of audio-video content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing the content. When implemented as a game console, source 26a may include some or all of the components described below in connection with CE device 48.

[0023] AVD 12 may further include one or more computer memory / computer-readable storage media 28, such as a non-transitory, disk-based, or solid-state storage device, in some cases embodied within the AVD's chassis as a standalone device, or as a personal video recording device (PVR) or video disc player either internal or external to the AVD's chassis for playing AV programs, or as a removable memory medium or server as described below. Also, in some embodiments, AVD 12 may include a position or location receiver, such as, but not limited to, a cellular receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular tower and provide the information to processor 24 and / or to determine, in conjunction with processor 24, the altitude at which AVD 12 is located.

[0024] Continuing with the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be a thermal imaging camera, a digital camera, such as a webcam, an IR sensor, an event-based sensor, and / or a camera integrated into AVD 12 and controllable by processor 24 to collect pictures / images and / or video in accordance with present principles. Also included in AVD 12 may be a Bluetooth transceiver 34 and another NFC element 36 for communicating with other devices using Bluetooth and / or near field communication (NFC) technology, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0025] Additionally, AVD 12 may include one or more auxiliary sensors 38 that provide input to processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors that form a layer of touch-enabled display 14 itself, and may be, without limitation, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc. Examples of other sensors include pressure sensors, motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands). Thus, sensors 38 may be implemented by one or more motion sensors such as individual accelerometers, gyroscopes, and magnetometers, and / or by an inertial measurement unit (IMU) that typically includes a combination of accelerometers, gyroscopes, and magnetometers for determining the location and orientation of AVD 12 in three dimensions, or by an event-based sensor such as an event detection sensor (EDS). An EDS consistent with the present disclosure provides an output indicative of a change in light intensity sensed by at least one pixel of the light-sensing array. If the light sensed by the pixel is decreasing, the output of the EDS can be −1, and if it is increasing, the output of the EDS can be +1. No change in light intensity below a certain threshold can be indicated by an output binary signal of 0.

[0026] The AVD 12 may also include an over-the-air TV broadcast port 40 for receiving OTA TV broadcasts, which provides input to the processor 24. In addition to the foregoing, it will be noted that the AVD 12 may also include an infrared (IR) transmitter and / or receiver and / or transceiver 42, such as an infrared data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, which may be a kinetic energy harvester that can convert kinetic energy into electrical power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more haptic / vibration generators 47 may be provided for generating tactile signals that can be sensed by a human holding or otherwise interacting with the device. Thus, the haptic generator 47 may vibrate all or portions of the AVD 12 using an electric motor connected to an off-center and / or unbalanced weight via a rotatable shaft of the motor, such that the shaft may rotate under the control of the motor (which may in turn be controlled by a processor such as processor 24) to produce force simulations in various directions with vibrations of various frequencies and / or amplitudes.

[0027] A light source such as a projector, such as an infrared (IR) projector, may also be included.

[0028] In addition to AVD 12, system 10 may include one or more other CE device types. In one embodiment, first CE device 48 may be a computer game console that can be used to transmit computer game audio and video to AVD 12 via commands sent directly to AVD 12 and / or through a server, described below, and second CE device 50 may include similar components as first CE device 48. In the illustrated embodiment, second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a heads-up see-through or non-see-through display for presenting AR / MR content or VR content (more generally, extended reality (XR) content), respectively. The HMD may be configured as a glasses-type display sold by computer game equipment manufacturers or as a bulkier VR-type display.

[0029] In the example shown, only two CE devices are shown, but it will be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown in the case of AVD 12.

[0030] Referring now to the at least one server 52 described above, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as a disk-based or solid-state storage device, and at least one network interface 58 that, under the control of the server processor 54, enables communication with other illustrated devices over the network 22, and may indeed facilitate communication between the server and client devices in accordance with the present principles. It will be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as, for example, a wireless telephony transceiver.

[0031] Thus, in some embodiments, server 52 may be an Internet server or an entire server "farm," e.g., in an example embodiment for a network gaming application, may include or perform "cloud" functionality such that devices of system 10 may access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in or near the same room as the other devices shown.

[0032] The components shown in the following figures may include some or all of the components shown in this specification. Any user interface (UI) described in this specification may be considered and / or extended, and UI elements may be blended and matched between UIs.

[0033] The present principles may employ a variety of machine learning models, including deep learning models. Machine learning models consistent with the present principles may use a variety of algorithms trained in methods including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms that may be implemented by computer circuitry include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and the type of RNN known as a long-short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models. In addition to the types of networks listed above, the models herein may be implemented by classifiers.

[0034] As understood herein, performing machine learning may therefore involve accessing the model against training data and then training the model to enable the model to process further data to make inferences. Thus, an artificial neural network / artificial intelligence model trained through machine learning may include an input layer, an output layer, and multiple hidden layers between them, configured and weighted to make inferences regarding appropriate outputs.

[0035] Referring now to FIG. 2, a computer simulation, such as a computer game, such as a legacy 2D game, may be transmitted from a computer game console 200 or computer game server 202 to a display device 204, such as a TV, for presentation of the computer simulation under the control of one or more computer simulation controllers 206, such as, but not limited to, a PlayStation® controller or other controller.

[0036] One or more haptic generator speakers 208 may play audio provided by the game console 200 or the server 202. One or more of the components shown in Figure 2 may include one or more cameras 210, including a camera on a head-mounted device 212 that may be worn by a player 214 of a virtual reality (VR) or augmented reality (AR) computer simulation, such as a computer game. The elements of the system shown in Figure 2 may incorporate some or all of the suitable devices and components described above with reference to Figure 1.

[0037] 3 illustrates a UI 300 that may be presented on a display 302, such as any of the displays described herein. The UI 300 in FIG. 3 includes a selector area 304 containing a plurality of selectors 306 (labeled A-D in FIG. 3). A prompt 308 may be provided for a player to view the selectors 306 that correspond to the player's choices. The choices represented by the selectors 306 may include, for example, game character actions, non-character choices such as voting, navigation commands, etc.

[0038] Figure 4 illustrates further: Beginning at block 400, for each of a plurality of players, their selections are detected by using eye tracking on images from any of the cameras herein to detect which selector the player is looking at, e.g., from Figure 3. In some embodiments, the player's gaze must dwell on the selector for at least the threshold period for gaze to count as a selection.

[0039] Proceeding to block 402, data regarding player selections as detected by eye tracking in block 400 is received in aggregate, and the selections are tallied by the number of players who selected each selection in block 404. If desired only for gaming systems that have opted in to community control, the selections with the highest number of player gaze selections are returned in block 406, and the associated action is implemented in block 408.

[0040] Figures 5 and 6 illustrate a particular use case. In Figure 5, at block 500, gaze-based group or community choices from Figure 4 are received and a character in a simulation, such as a player character in a computer game, is controlled by the choice.

[0041] 6, in contrast, gaze-based group or community selections are received in block 600, and non-character options are controlled by the selections in block 602. For example, in cases where a screen cursor or other object may move upward on a display, the up arrow may be gazed upon by most players. Or, votes may be recorded based on the most popular group or community selection from a set of options for which players can vote.

[0042] 7 illustrates a UI 700 that may be presented on a display 702 to present an at-a-glance heat map for directing community gaming action. In FIG. 7, it will be understood that more players may be considered, assuming that region 704 indicates the area of ​​gaze of a first player and area 708 indicates the area of ​​gaze of a second player.

[0043] Option 706 (labeled "D" in FIG. 7) is within the region where the gaze areas of two players overlap (in some embodiments, during the required gaze dwell time), and is therefore determined to be a community selection. The selection is recorded, and thus the most popular gaze-based selection within the community is identified as triggering the action associated with the most popular location / selection.

[0044] Similarly, region 710 indicates the area of ​​gaze of a first player and area 714 indicates the area of ​​gaze of a second player, although it will be appreciated that more players may be considered.

[0045] Option 712 (labeled "A" and "D" in FIG. 7) is within the region where the gaze areas of two players overlap (in some embodiments, during the required gaze dwell time), and is therefore determined to be a community selection. The selection in this case is recorded, and thus the selection based on the most popular gaze within the community is identified as triggering the most popular location / selection. Thus, multiple input commands may be associated with a single region where most players' gazes overlap.

[0046] Although particular embodiments have been shown and described in detail herein, the subject matter encompassed by the present invention is limited only by the claims.

Claims

1. identifying a gaze direction of each of a plurality of viewers of the computer simulation; generating an input signal to the computer simulation based at least in part on the gaze direction; 10. An apparatus comprising: at least one processor configured with executable instructions to:

2. The apparatus of claim 1 , wherein the input signal is a single input signal determined by the gaze direction.

3. The apparatus of claim 1 , wherein the input signal is a selection signal that identifies one of a plurality of options to be presented in the computer simulation.

4. 2. The device of claim 1, wherein a first set of gaze directions is directed toward a first object in the computer and a second set of gaze directions is directed toward a second object in the computer simulation, the input signal corresponds to the first object, and the input signal is generated based on the first set of gaze directions being greater than the second set of gaze directions.

5. The apparatus of claim 1 , wherein the input signal is a character control signal for controlling a character in the computer simulation.

6. The apparatus of claim 1 , wherein the computer simulation is a 2D computer simulation.

7. The apparatus of claim 1 , wherein the input signal comprises a vote.

8. The apparatus of claim 1 , wherein a gaze direction is counted only in response to gaze being at a location for at least a threshold duration.

9. The device of claim 1 , wherein the input signal comprises a navigation command.

10. imaging the eyes of a plurality of computer-simulated players; identifying cloud-based input commands for the computer simulation based on the imaging; and A method comprising:

11. The method of claim 10 , wherein the input command is a single input command determined by gaze direction.

12. The method of claim 10 , wherein the input command is a selection signal that identifies one of a plurality of options to be presented in the computer simulation.

13. The method of claim 10 , wherein the input commands include character control signals for controlling a character in the computer simulation.

14. The method of claim 10 , wherein the input command comprises a vote.

15. The method of claim 10 , wherein the input command comprises a navigation command.

16. and at least one computer storage device containing instructions, rather than transient signals, the instructions including: Tracking the gaze of multiple users of a computer simulation; implementing community-based control of the computer simulation based on the gaze; A device, wherein the method is executable by at least one processor to:

17. 17. The device of claim 16, comprising the at least one processor.

18. The device of claim 16 , wherein the instructions are executable to control at least one character in the computer simulation based on the gaze.

19. The device of claim 16 , wherein the instructions are executable to identify at least one vote based on the gaze.

20. The device of claim 16 , wherein the instructions are executable to input at least one screen navigation command based on the gaze.

Images