Group control of computer games using aggregated gaze regions

Abstract

A group of people uses team cooperation to control a computer game. This may be accomplished by tracking (400) eye movements of each person to detect a location at which each person views an object, such as a game control object, on the screen. Control actions of an object viewed by a maximum number of persons (404) in data collection in a "heat map" style are implemented (408) through a game.

Description

Technical Field

[0001] The present application relates generally to group control of computer simulations, such as computer games, using aggregated gaze regions. Background Art

[0002] As understood herein, computer simulations, such as computer games, are increasingly becoming group-oriented. Summary of the invention

[0003] As further understood herein, enjoyable and intuitive group control of computer games may be facilitated.

[0004]

[0013] Accordingly, an apparatus includes at least one processor configured with instructions executable to: identify respective gaze directions of a plurality of viewers of a computer simulation, and generate an input signal to the computer simulation based at least in part on the gaze directions.

[0005] The input signal may be a single input signal determined by gaze direction. Alternatively, the input signal may be a selection signal identifying one of a plurality of options presented in a computer simulation.

[0006] In some examples, the input signal includes a character control signal for controlling a character in a computer simulation. Alternatively, the input signal may include a vote. Still further, the input signal may include a navigation command.

[0007] In a non-limiting implementation, a first set of gaze directions are directed toward a first object in a computer, a second set of gaze directions are directed toward a second object in a computer simulation, an input signal corresponds to the first object, and the input signal is generated based on the first set of gaze directions being greater than the second set of gaze directions.

[0008] The computer simulation may be a 3D simulation, or it may also be a conventional 2D computer simulation.Gaze directions may be counted only in response to gaze being at a certain location for at least a threshold duration.

[0009] In another aspect, a method includes imaging eyes of a plurality of players of a computer simulation and identifying crowd-based input commands for the computer simulation based on the imaging.

[0010] In another aspect, an apparatus includes at least one computer storage device that is not a transient signal and further includes instructions executable by at least one processor to track gaze of multiple users of a computer simulation and to enable community-based control of the computer simulation based on gaze.

[0011] The details of the present application, both as to its structure and operation, may be best understood by reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an exemplary system in accordance with the principles of the present invention;

[0013] Figure 2 An exemplary specific system consistent with the principles of the invention is shown;

[0014] Figure 3 Screen shots of example user interfaces (UIs) are shown;

[0015] Figure 4 An example overall logic is shown in an example flow chart format;

[0016] Figure 5 A first example specific logic is shown in an example flow chart format;

[0017] Figure 6 A first example specific logic is shown in an example flow chart format; and

[0018] Figure 7 An example screenshot is shown schematically illustrating a group gaze point. DETAILED DESCRIPTION

[0019] The present disclosure relates generally to computer ecosystems that include aspects of a network of consumer electronics (CE) devices, such as, but not limited to, a computer gaming network. The system herein may include a server component and a client component that may be connected via a network so that data may be exchanged between the client component and the server component. The client component may include one or more computing devices, including a gaming console (such as a Sony or game consoles made by Microsoft or Nintendo or other manufacturers), extended reality (XR) headsets (such as virtual reality (VR) headsets, augmented reality (AR) headsets), portable televisions (e.g., smart TVs, Internet-enabled TVs), portable computers (such as laptops and tablets), and other mobile devices (including smartphones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some client computers may use, for example, a Linux operating system, an operating system from Microsoft, or a Unix operating system, or an operating system produced by Apple, Inc. or Google or a Berkeley Software Distribution or Berkeley Standard Distribution (BSD) OS (including descendants of BSD). These operating environments can be used to execute one or more browsing programs, such as browsers made by Microsoft or Google or Mozilla or other browser programs that can access websites hosted by Internet servers discussed below. In addition, an operating environment according to the principles of the present invention can be used to execute one or more computer game programs.

[0020] A server and / or gateway may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may be connected via a local intranet or a virtual private network. The server or controller may be provided by a game console (such as a Sony personal computer, etc.) is instantiated.

[0021] Information may be exchanged between the client and the server over the network. To this end, and for security purposes, the server and / or client may include firewalls, load balancers, temporary storage devices and proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method of providing a secure community (such as an online social networking site or a gamer network) to network members.

[0022] The processor may be a single chip or a multi-chip processor that may perform logic with the aid of various lines such as address lines, data lines, and control lines, as well as registers and shift registers. A processor including a digital signal processor (DSP) may be one embodiment of the circuit system.

[0023] The components included in one embodiment may be used in other embodiments in any appropriate combination. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged or excluded from other embodiments.

[0024] “A system having at least one of A, B, and C” (similarly, “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes the following systems: having only A; having only B; having only C; having both A and B; having both A and C; having both B and C; and / or having A, B, and C simultaneously.

[0025] Reference now Figure 1 , an example system 10 is shown, which may include one or more of the example devices mentioned above and further described below according to the principles of the present invention. The first of the example devices included in the system 10 is a consumer electronics (CE) device, such as an audio video device (AVD) 12, such as, but not limited to, a theater display system (which may be projector-based) or an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). Alternatively, the AVD 12 may also be a computerized Internet-enabled ("smart") phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or headphones such as smart glasses or VR headsets, another wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headset, a computerized Internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to implement the principles of the present invention (e.g., communicate with other CE devices to implement the principles of the present invention, execute the logic described herein, and perform any other functions and / or operations described herein).

[0026] Thus, to implement these principles, an AVD 12 may be built with some or all of the components shown. For example, the AVD 12 may include one or more touch-enabled displays 14, which may be implemented with a high-definition or ultra-high-definition "4K" or higher definition flat panel screen. The touch-enabled display 14 may include, for example, a capacitive or resistive touch sensing layer with an electrode grid for touch sensing consistent with the principles of the present invention.

[0027] AVD 12 may also include: one or more speakers 16, which are used to output audio according to the principles of the present invention; and at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to AVD 12 to control AVD 12. Example AVD 12 may also include one or more network interfaces 20 for communicating through at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. Therefore, interface 20 can be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to implement the principles of the present invention, including controlling other elements described herein of AVD 12, such as controlling display 14 to present images on the display and receiving input from the display. In addition, it is noted that network interface 20 can be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above, etc.

[0028] In addition to the foregoing, the AVD 12 may also include one or more input and / or output ports 26, such as a high-definition multimedia interface (HDMI) port or a universal serial bus (USB) port physically connected to another CE device and / or a headset port for connecting a headset to the AVD 12 to present the audio from the AVD 12 to the user through the headset. For example, the input port 26 may be connected to a cable or satellite source 26a of audio and video content by wire or wirelessly. Therefore, the source 26a may be a separate or integrated set-top box, or a satellite receiver. Alternatively, the source 26a may be a game console or disk player containing content. The source 26a may include some or all of the components described below with respect to the CE device 48 when implemented as a game console.

[0029] AVD 12 may also include one or more computer memories / computer readable storage media 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices. In some cases, the one or more computer memories / computer readable storage media are embodied as independent devices in the housing of the AVD, or as personal video recorders (PVRs) or video disk players for playing back AV programs inside or outside the housing of the AVD, or as removable storage media or servers described below. In addition, in some embodiments, AVD 12 may include a position / location receiver, such as, but not limited to, a cellular phone receiver, a GPS receiver, and / or an altimeter 30, which is configured to receive geographic location information from a satellite or cellular phone base station, and provide the information to the processor 24 and / or determine the altitude of the AVD 12 in conjunction with the processor 24.

[0030] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, IR sensors, event-based sensors, and / or cameras integrated into the AVD 12, and may be controlled by the processor 24 to collect pictures / images and / or videos according to the principles of the present invention. The AVD 12 may also include The transceiver 34 and other near field communication (NFC) elements 36 may be used to communicate with other devices using Bluetooth and / or NFC technology, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0031] In addition, the AVD 12 may include one or more auxiliary sensors 38 that provide input to the processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors that form a layer of the touch-enabled display 14 itself, and may be, but are not limited to, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, and the like. Other sensor examples include pressure sensors, motion sensors (such as accelerometers, gyroscopes, odometers, or magnetic sensors), infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands). Thus, the sensor 38 may be implemented by one or more motion sensors, such as separate accelerometers, gyroscopes, and magnetometers, and / or an inertial measurement unit (IMU), which typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions, or by an event-based sensor, such as an event detection sensor (EDS). An EDS consistent with the present disclosure provides an output indicating a change in light intensity sensed by at least one pixel of a light sensing array. For example, if the light sensed by the pixel is decreasing, the output of the EDS may be -1; if the light sensed by the pixel is increasing, the output of the EDS may be + 1. The absence of a change in light intensity below a certain threshold may be indicated by an output binary signal of 0.

[0032] AVD 12 may also include a wireless TV broadcast port 40 for receiving OTA TV broadcasts to provide input to processor 24. In addition to the foregoing, it should be noted that AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering AVD 12, such as a kinetic energy harvester that may convert kinetic energy into electricity to charge the battery and / or power AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more tactile / vibration generators 47 may be provided for generating tactile signals that may be sensed by a person holding or touching the device. Thus, the tactile generator 47 can vibrate all or part of the AVD 12 using an electric motor that is connected to an eccentric and / or unbalanced counterweight via a rotatable shaft of the motor so that the shaft can rotate under the control of the motor (which in turn can be controlled by a processor such as processor 24) to produce vibrations of various frequencies and / or amplitudes and force simulation in various directions.

[0033] A light source may also be included, such as a projector, such as an infrared (IR) projector.

[0034] In addition to the AVD 12, the system 10 may also include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to send audio and video of a computer game to the AVD 12 via commands sent directly to the AVD 12 and / or through a server as described below, while the second CE device 50 may include

[0035] Components similar to the first CE device 48. In the example shown, the second CE device 50 can be configured as a computer game controller manipulated by the player or a head-mounted display (HMD) worn by the player. The HMD may include a heads-up transparent or opaque display for presenting AR / MR content or VR content (more generally, extended reality (XR) content), respectively. The HMD can be configured as a glasses-type display, or as a larger VR-type display sold by computer game device manufacturers.

[0036] In the example shown, only two CE devices are shown, and it should be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may be combined with some or all of the components shown in the context of AVD 12.

[0037] Referring now to the aforementioned at least one server 52, the at least one server includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 58, which allows communication with other illustrated devices over the network 22 under the control of the server processor 54, and can actually facilitate communication between the server and client devices in accordance with the principles of the present invention. Note that the network interface 58 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as, for example, a wireless telephone transceiver).

[0038] Thus, in some embodiments, the server 52 may be an Internet server or an entire server "farm," and in an example embodiment for, for example, a networked gaming application, the server may include and perform "cloud" functionality such that the devices of the system 10 may access the "cloud" environment via the server 52. Alternatively, the server 52 may be implemented by one or more gaming consoles or other computers in the same room or nearby as the other devices shown.

[0039] The components shown in the following figures may include some or all of the components shown herein.Any user interface (UI) described herein may be merged and / or extended, and UI elements may be mixed and matched between UIs.

[0040] The present invention principle can adopt various machine learning models, including deep learning models. According to the machine learning model of the present invention principle, various algorithms trained in a manner including the following items can be used: supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning and other forms of learning. Examples of such algorithms that can be implemented by computer circuit systems include one or more neural networks, such as convolutional neural networks (CNNs), recursive neural networks (RNNs), and a type of RNN called long short-term memory (LSTM) networks. Support vector machines (SVMs) and Bayesian networks can also be considered as examples of machine learning models. In addition to the above-mentioned network types, the model herein can also be implemented by a classifier.

[0041] As understood herein, performing machine learning may therefore involve accessing training data and then training a model based on the training data to enable the model to process additional data to make inferences. Thus, an artificial neural network / artificial intelligence model trained by machine learning may include an input layer, an output layer, and a plurality of hidden layers therebetween, the plurality of hidden layers being configured and weighted to make inferences about appropriate outputs.

[0042] Reference now Figure 2 A computer simulation (such as a computer game, such as a traditional 2D game) may be sent from a computer game console 200 or a computer game server 202 to a display device 204 (such as a TV) for display on one or more computer simulation controllers 206 (such as, but not limited to, A computer simulation is presented under the control of a controller or other controller).

[0043] One or more haptic generator speakers 208 may play audio from game console 200 or server 202 . Figure 2 One or more components shown in may include one or more cameras 210, including a camera on a head mounted device 212 that may be worn by a player 214 of a virtual reality (VR) or augmented reality (AR) computer simulation, such as a computer game. Figure 2 The elements of the system shown in the above reference can be combined with Figure 1 Some or all of the suitable devices and components described.

[0044] Figure 3 A UI 300 is shown, which may be presented on a display 302 (such as any display herein). Figure 3 The UI 300 in FIG. 1 includes a selector area 304 that includes a plurality of selectors 306 (in Figure 3308, allowing the player to view the selector 306 corresponding to the player's selection. The selection represented by the selector 306 may include, for example, game character actions, non-character selections such as voting, navigation commands, etc.

[0045] Figure 4 Beginning at block 400, using eye tracking on images from any camera herein, for each of a plurality of players, by detecting an eye movement from Figure 3 The player in the game may detect their selection by looking at the selector that they are looking at. In some embodiments, the player's gaze must stay on the selector for at least a certain threshold period of time for the gaze to be counted as a selection.

[0046] Continuing to block 402, data regarding the players' selections as detected by eye tracking at block 400 is received in aggregate, where the selections are tabulated according to the number of players who selected each selection at block 404. If desired only for gaming systems that opt-in to community control, the selection with the highest number of player gaze selections is returned at block 406, and associated operations are implemented at block 408.

[0047] Figure 5 and Figure 6 A specific use case is shown. Figure 5 In block 500, a Figure 4 A user can select a gaze-based group or community and control a character in a simulation (such as a player character in a computer game) based on that selection.

[0048] In contrast, Figure 6 In the example embodiment, a gaze-based group or community selection is received at block 600, and a non-character option is controlled based on the selection at block 602. For example, an upward arrow may have been gazed at by most players, in which case a screen cursor or other object may move upward on the display. Alternatively, a vote may be recorded based on the most popular group or community selection among a set of options that players may vote on.

[0049] Figure 7 A UI 700 is shown that may be presented on a display 702 to present an at-a-glance heat map indicating community gaming activity. Figure 7 , assume that area 704 indicates the gaze area of ​​the first player, and area 708 indicates the gaze area of ​​the second player, it should be understood that more players can be considered.

[0050] Option 706 (in Figure 7The selection is recorded, and the most popular gaze-based selection in the community is thus identified as contributing to the action associated with the most popular location / selection.

[0051] Similarly, region 710 indicates the gaze area of ​​a first player, while region 714 indicates the gaze area of ​​a second player, with the understanding that more players may be considered.

[0052] Option 712 (in Figure 7 In some embodiments, the gaze-based selections (labeled "A" and "D" in FIG. 1 ) are in an area where the gaze areas of two players overlap (for the requisite gaze dwell time in some embodiments) and are therefore determined to be community selections. In this case, the selection is recorded, and the most popular gaze-based selection in the community is thus identified as the action that facilitates the most popular location / selection. Thus, multiple input commands may be associated with a single area where the gazes of most players overlap.

[0053] While particular embodiments have been herein shown and described in detail, it is to be understood that the subject matter encompassed by the present invention is limited only by the claims.

Claims

1. A device comprising: At least one processor configured with instructions executable to: identifying respective gaze directions of a plurality of computer-simulated viewers; and The computer-simulated input signal is generated based at least in part on the gaze direction.

2. The apparatus of claim 1, wherein the input signal is a single input signal determined by the gaze direction.

3. The apparatus of claim 1, wherein the input signal is a selection signal identifying one of a plurality of options presented in the computer simulation.

4. The apparatus of claim 1, wherein a first set of gaze directions are directed toward a first object in a computer, a second set of gaze directions are directed toward a second object in the computer simulation, the input signal corresponds to the first object, and the input signal is generated based on the first set of gaze directions being greater than the second set of gaze directions.

5. The apparatus of claim 1, wherein the input signal is a character control signal for controlling a character in the computer simulation.

6. The apparatus of claim 1, wherein the computer simulation is a 2D computer simulation.

7. The apparatus of claim 1, wherein the input signal comprises a vote.

8. The apparatus of claim 1, wherein gaze directions are counted only in response to gaze being at a location for at least a threshold duration.

9. The device of claim 1, wherein the input signal comprises a navigation command.

10. A method comprising: imaging the eyes of multiple players in a computer simulation; as well as Based on the imaging, a population-based input command is identified for the computer simulation. The method of claim 10 , wherein the input command is a single input command determined by the gaze direction.

12. The method of claim 10, wherein the input command is a selection signal identifying one of a plurality of options presented in the computer simulation.

13. The method of claim 10, wherein the input commands include character control signals for controlling a character in the computer simulation. The method of claim 10 , wherein the input command comprises voting. The method of claim 10 , wherein the input signal comprises a navigation command.

16. An apparatus comprising: at least one computer storage device, the at least one computer storage device being not a transient signal and comprising instructions executable by at least one processor to: Tracking the gaze of multiple users of a computer simulation; and Community-based control of the computer simulation is enabled based on the gaze.

17. The apparatus of claim 16, comprising the at least one processor.

18. The apparatus of claim 16, wherein the instructions are executable to: control at least one character in the computer simulation based on the gaze.

19. The apparatus of claim 16, wherein the instructions are executable to: identify at least one vote based on the gaze.

20. The apparatus of claim 16, wherein the instructions are executable to: input at least one screen navigation command based on the gaze.

Images