Adaptive screen share pointer and reactions during gameplay

Abstract

To enhance the overall execution environment of a game itself, and provide enriched technology-based interactivity features.SOLUTION: Game-specific screen share features can be provided to viewers of a video game livestream. The viewers can control a themed or adaptive cursor presented to the gamer during gameplay, and can also provide themed or adaptive sentiment-based reactions to the gamer during gameplay. The cursor and reactions can then be surfaced to the gamer in a number of ways and can even be tied back into the game execution environment itself.SELECTED DRAWING: None

Description

[Technical field]

[0001] The following disclosure relates generally to adaptive screen sharing pointers and reactions for computer gameplay. [Background technology]

[0002] As will be appreciated, some video gamers stream their gameplay to other users so that others can watch them. As will be recognized, current systems are technically limited in their ability to provide sufficient audience participation, but the present principles recognize that from a technical standpoint, they may provide a richer and more engaging viewing experience. Summary of the Invention [Problem to be solved by the invention]

[0003] The present principles thereby aim to enhance the overall execution environment of the game itself, as well as to provide rich technology-based interactive capabilities. [Means for solving the problem]

[0004] As such, in one aspect, an apparatus includes a processor assembly programmed with instructions for executing a computer game in which a first person plays the computer game, the processor assembly programmed with instructions for receiving input including cursor control commands and / or graphics-based reactions from a second person watching a live stream of the computer game but not controlling a character in the game, the processor assembly programmed with instructions for presenting, based on the input, an output on a display associated with the first person that illustrates the input in a game-specific context.

[0005] In any example implementation, the output may include the cursor being presented with an appearance associated with an ongoing game context of the computer game. Additionally or alternatively, the output may include the cursor being presented with an appearance associated with a current game event.

[0006] As another example, the cursor control command may be a first cursor control command and the processor assembly may be configured to execute the first cursor control command during a first portion of the computer game. According to this example, the processor assembly may be configured to determine that a cutscene is occurring and, based on the determination, to refuse to process additional cursor control commands.

[0007] As yet another example, the cursor control command may also be a first cursor control command, but wherein the processor assembly may be configured to alternate authority to issue cursor control commands among viewers of the computer game based on past viewer input associated with gameplay of the first person of the computer game.

[0008] Also consistent with present principles, in some cases, the processor assembly may be configured to present a selectable graphical object on a display associated with the second person, where the processor assembly may also be configured to receive the cursor control commands that may select the selectable graphical object, and to control an output of the computer game presented on the display associated with the first person based on the cursor control commands, where the output of the computer game may have the effect of altering a current game state of the computer game.

[0009] Further, if desired, the processor assembly may be configured to receive the cursor control command that may direct placement of a graphical object associated with the second person at a permanent location within a field of view of the first person presented on the display, and based on the cursor control command, the processor assembly may be configured to present the graphical object at the permanent location.

[0010] In another implementation, the processor assembly may be configured to receive the cursor control command to select a game object and highlight the selected game object based on the cursor control command, where the highlighting may not include a trace of the cursor control command itself.

[0011] Further, in some cases, the processor assembly may be configured to receive the cursor control commands and, based on the cursor control commands, perform predefined game actions that are reserved for performance by a person not controlling a character during the computer game.

[0012] As another example, the input may include eye movement input associated with the second person, where the processor assembly may be configured to move the cursor on the display in response to the eye movement input, determine that the second person has one eye closed, and modify an appearance of the cursor presented on the display based on the determination.

[0013] Optionally, the processor assembly may be configured to display the cursor on the display with a visual appearance that conveys a feeling about a terrain over which the first person's game character is traversing.

[0014] As yet another example, the processor assembly may be configured to display the cursor on the display with a visual appearance that indicates data regarding current game statistics of a first game character, the first game character being a game character not controlled by the first person.

[0015] With respect to the graphics-based reactions, in some cases, the processor assembly may be configured to receive respective graphics-based reactions from respective people watching the live stream but not controlling a character of the game, and to modify playout of the computer game in response to a predetermined graphics-based reaction threshold being met, the predetermined graphics-based reaction threshold may be associated with multiple graphics-based reactions of the same type being received.

[0016] Additionally or alternatively, the processor assembly may be configured to present haptic feedback to the first person based on the graphics-based response that varies based on a response type associated with the graphics-based response.

[0017] In another aspect, a method includes executing a computer game in which a first person plays the computer game and receiving input from a second person watching a live stream of the computer game but not controlling a character in the game. The input includes cursor control commands and emotion-based reactions. The method further includes presenting game output on a display associated with the first person based on the input.

[0018] In one example, the method may include aggregating emotion-based reactions provided from a plurality of viewers and presenting a reaction summary to the first person based on the aggregation.

[0019] In yet another aspect, a system includes at least one computer storage device that is not a transitory signal. The computer storage device includes instructions executable by at least one processor for receiving input from a second person executing a computer game and viewing a live stream of the computer game in which a first person plays the computer game. The input includes cursor control commands and emotion-based reactions. The instructions are also executable to present an output on a display associated with the first person based on the input.

[0020] In one example, the instructions may be executable to award at least one trophy to the first person based on an amount of emotion-based responses received while playing one or more particular aspects of the computer game, and to reward at least one live stream viewer based on input provided by the viewer, the input may be established by one or more cursor control commands and / or emotion-based responses.

[0021] The details of the structure and operation of the present application can best be understood with reference to the accompanying drawings, in which like reference numbers refer to like parts and in which: [Brief description of the drawings]

[0022] [Figure 1] 1 is a block diagram of an exemplary system consistent with present principles. [Diagram 2] 1 shows a schematic diagram of a streamer playing a video game consistent with the present principles streaming the gameplay to viewers watching the gameplay using different types of devices. [Figure 3A] 14 outlines different types of streamer / viewer interactions that may occur consistent with the present principles. [Figure 3B] 10 outlines different types of streamer / viewer interactions that may occur consistent with the present principles. [Figure 3C]10 outlines different types of streamer / viewer interactions that may occur consistent with the present principles. [Figure 4] A first example of a graphic user interface (GUI) that is overlaid on a streamer's point of view (POV) while the streamer is playing a computer game consistent with the present principles is presented, where a viewer controls a cursor presented to the streamer on the GUI. [Diagram 5] 5 shows an expanded view of the cursor of FIG. 4 in accordance with the present principles. [Figure 6] Consistent with the present principles, we provide an example of a GUI overlaid on the stream's POV while the streamer is playing a computer game, where one viewer provides emotion-based reactions that are presented to the streamer. [Figure 7] FIG. 1 illustrates an example in the context of using a pointer to aid in competitive tactics while playing a sports computer game. [Figure 8] An example consistent with the present principles is provided in the context of using a pointer to aid in competitive tactics while playing a first-person shooter computer game. [Figure 9] We present an example consistent with the present principles in the context of using a pointer to aid in exploration of a virtual / game world. [Figure 10] An example consistent with the present principles is given in the context of using a pointer to assist in looting / searching for hidden objects in a game world. [Figure 11] An example consistent with the present principles is given in the context of using a pointer to assist in how to hit a virtual ball. [Figure 12] An example consistent with the present principles is given in the context of using a pointer to defeat a boss in a computer game. [Figure 13] We present an example consistent with the present principles in which groups of viewers provide a streamer with different reactions that surface in different game contexts. [Figure 14]We present an example consistent with the present principles in which groups of viewers provide a streamer with different reactions that surface in different game contexts. [Figure 15] We present an example consistent with the present principles in which groups of viewers provide a streamer with different reactions that surface in different game contexts. [Figure 16] We present an example consistent with the present principles in which groups of viewers provide a streamer with different reactions that surface in different game contexts. [Figure 17] 13A-13C show different examples of adaptive cursors controlled by the viewer and presented to the streamer consistent with the present principles. [Figure 18] 13A-13C show different examples of adaptive cursors controlled by the viewer and presented to the streamer consistent with the present principles. [Figure 19] 13A-13C show different examples of adaptive cursors controlled by the viewer and presented to the streamer consistent with the present principles. [Figure 20] 13A-13C show different examples of adaptive cursors controlled by the viewer and presented to the streamer consistent with the present principles. [Figure 21] 1 illustrates an example GUI that includes a palette of different selectable objects that may be used by a viewer consistent with the present principles. [Figure 22] 13A-13C show different examples of adaptive cursors controlled by the viewer and presented to the streamer consistent with the present principles. [Diagram 23] 1 shows an example of an emotion-based response questionnaire that may be presented to a viewer consistent with the present principles. [Figure 24] FIG. 23 shows an example of a survey result GUI that is consistent with the present principles. [Figure 25-1] 1 illustrates exemplary logic in the form of an exemplary flowchart that may be executed by a processor assembly consistent with the present principles. [Figure 25-2] 1 illustrates exemplary logic in the form of an exemplary flowchart that may be executed by a processor assembly consistent with the present principles. [Figure 26]1 illustrates an example of a settings GUI that may be presented on a display to configure one or more settings of a system / processor assembly operating consistent with the present principles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The present disclosure generally relates to computer ecosystems including aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks, where the system may include a server component and a client component that may be connected via a network such that data may be exchanged between the client component and the server component. The client components may include one or more computing devices including Sony PlayStation® game consoles, game consoles from Microsoft, Nintendo or other manufacturers, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart televisions, Internet-enabled televisions), portable computers such as laptops and tablet computers, other mobile devices including smartphones, as well as additional examples described below. These client devices may operate in a variety of operating environments. For example, some of the client computers may employ a Linux operating system, an operating system from Microsoft, a Unix operating system, or an operating system from Apple or Google, by way of example only. These operating environments may be used to run one or more browsing programs, such as browsers created by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by Internet servers, as described below. An operating environment consistent with the present principles may also be used to execute one or more computer game programs.

[0024] The server and / or gateway may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the clients and servers may be connected via a local intranet or a virtual private network. The server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, or the like.

[0025] Information may be exchanged between the client and the server via a network. For this purpose and for security, the server and / or the client may include firewalls, load balancers, temporary storage and proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus that implements the method for providing a secure community, such as an online social website, to network members.

[0026] The processor may be a single-chip or multi-chip processor capable of executing logic through various wiring such as address, data and control lines, registers and shift registers. A processor assembly, whether in one device or multiple devices, may include one or more processors that operate independently or in conjunction with each other to execute an algorithm.

[0027] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or illustrated in the drawings may be combined, interchanged, or excluded in other embodiments.

[0028] "A system having at least one of A, B, and C" (and similarly "a system having at least one of A, B, or C", "a system having at least one of A, B, C") includes systems having A only, B only, C only, A and B, A and C, B and C, and / or A, B and C, etc.

[0029] The present principles may employ machine learning models, including deep learning models. Machine learning models use a variety of algorithms trained in ways that include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms that may be implemented by computer circuitry include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs) that are suitable for learning information from a series of images, and a type of RNN known as a long short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models.

[0030] As will be appreciated, performing machine learning involves accessing training data and then training a model on the training data so that the model can process additional data and make predictions. A neural network may include an input layer, an output layer, and multiple hidden layers in between that are configured and weighted to infer an appropriate output.

[0031] Specifically, referring to FIG. 1, an exemplary system 10 is shown, which may include one or more of the exemplary devices described above and further below in accordance with the present principles. A first exemplary device included in the system 10 is a consumer electronics (CE) device, such as an audio-video device (AVD) 12, including, but not limited to, an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). Alternatively, the AVD 12 may be a computerized Internet-enabled ("smart") phone, a tablet computer, a notebook computer, an HMD, a computerized wearable device, a computerized Internet-enabled music player, a computerized Internet-enabled headphones, a computerized Internet-enabled implantable device such as an implantable skin device, and the like. In any case, it should be understood that the AVD 12 is configured to implement the present principles (e.g., to communicate with other CE devices to execute the present principles, execute the logic described herein, and perform other functions and / or operations described herein).

[0032] Accordingly, to realize such principles, the AVD 12 may be comprised of some or all of the components shown in FIG. 1. For example, the AVD 12 may include one or more displays 14, which may be realized by high definition or ultra-high definition "4K" or higher flat screens and may be touch-enabled for receiving user input signals via touch on the display. The AVD 12 may include one or more speakers 16 for outputting audio and at least one additional input device 18, such as a voice receiver / microphone for inputting audible commands to the AVD 12 for controlling the AVD 12, in accordance with the present principles. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, a LAN, etc., under the control of one or more processors 24. Thus, the interface 20 may be a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to implement the present principles, including controlling display 14 to present images thereon and to receive input therefrom, including other elements of AVD 12 as described herein. Additionally, network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or the Wi-Fi transceiver discussed above.

[0033] In addition to the above, the AVD 12 may also include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI) port, a USB port, and / or a headphone port for connecting headphones to the AVD 12 so that audio from the AVD 12 is presented to a user through the headphones. For example, the input port 26 may be connected via wire or wireless to a cable or satellite source 26a of audio-video content. Thus, the source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, the source 26a may be a game console or disc player containing the content. The source 26a when implemented as a game console may include some or all of the components described below in connection with the CE device 48.

[0034] AVD 12 may further include one or more computer memories 28, such as disk-based or solid-state storage that is not a transitory signal, possibly embodied as a personal video recording device (PVR) or video disk player for playing AV programs either within the AVD chassis as a standalone device, or inside or outside the AVD chassis, or as a removable memory medium or server as described below. In some embodiments, AVD 12 may also include a position or location receiver, such as, but not limited to, a cellular receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular tower and provide that information to processor 24 and / or in conjunction with processor 24 to determine the altitude at which AVD 12 is disposed. Component 30 may be implemented by an inertial measurement unit (IMU), typically including a combination of accelerometers, gyroscopes, and magnetometers, or by event-based sensors to determine the location and orientation of AVD 12 in three dimensions.

[0035] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, event-based sensors, and / or cameras integrated into the AVD 12 and controllable by the processor 24 to collect pictures / images and / or videos in accordance with the present principles. The AVD 12 may also include a Bluetooth transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth and / or Near Field Communication (NFC) technologies, respectively. An example of an illustrative NFC element may include a radio frequency identification (RFID) element.

[0036] Additionally, the AVD 12 may include one or more auxiliary sensors 38 (e.g., motion sensors such as accelerometers, gyroscopes, cyclometers or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)) for providing input to the processor 24. The AVD 12 may include an over-the-air (OTA) television broadcast port 40 for receiving an OTA television broadcast for providing input to the processor 24. In addition to the above, it is noted that the AVD 12 may further include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12 and may also be a kinetic energy harvesting device that may convert kinetic energy into power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more haptic generators 47 may be provided to generate haptic signals that can be sensed by a person holding or touching the device.

[0037] Continuing to refer to FIG. 1, in addition to the AVD 12, the system 10 may include one or more other types of CE devices. In one embodiment, the first CE device 48 may be a computer game console that can be used to send audio and video of a computer game to the AVD 12 via commands sent directly to the AVD 12 and / or through a server, as described below, while the second CE device 50 may include components similar to the first CE device 48. In the embodiment shown, the second CE device 50 may be configured as a computer game controller operated by a player or a head mounted display (HMD) worn by a player. In the embodiment shown, only two CE devices are shown, but it should be understood that a fewer or greater number of devices may be used. Here, the devices may implement some or all of the components shown for the AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for the AVD 12.

[0038] Referring now to the aforementioned at least one server 52, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with other devices of Figure 1 via the network 22, and may indeed facilitate communications between the server and client devices in accordance with the present principles. The network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as a wireless telephone transceiver.

[0039] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm," and may include and perform "cloud" functionality such that, for example, in an exemplary embodiment for a network gaming application, devices of system 10 may access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown in FIG.

[0040] The components shown in the following figures may include some or all of the components shown in Figure 1. The user interfaces (UIs) described herein can be integrated or extended, and UI elements can be mixed and matched between UIs.

[0041] With the above in mind, the present Principles recognize that computer / video game screen sharing allows gamers and viewers to share any game moments or content with each other. The following aspects can be combined with game streaming services where the flow of communication is often one-to-many and viewers are unfamiliar viewers, not necessarily friends of the gamer. These aspects, along with others described below, can help to improve the social game playing experience and the technical capabilities of computer-based game systems in particular.

[0042] Thereby, according to the disclosure below, the screen sharing feature allows gamers to easily and directly share their real-time gaming moments with friends and others, fulfilling gamers' desire to enjoy the moments together. Thus, gamers can casually stream their gameplay to friends or small groups in real time while chatting and playing together in parties and other scenarios, creating a sense of togetherness despite the decentralized and sometimes disjunctive nature of particularly computer-based games. The present principles may be employed on mobile applications running on video game consoles, smartphones and other mobile devices, as well as other types of devices, including laptops and desktop computers.

[0043] With the above in mind, reference is now made to FIG. 2. This diagram illustrates a Playstation 5 gamer 200 playing a computer game whose visual content is presented on a local television display 210 at 1080p resolution and a frame rate of 30 frames per second (fps). Once the A / V content stream is routed through one or more transcoding / selection and forwarding unit (SFU) servers 240 and received at a local game console connected to a local television 250 in the viewer's 220 home, the gamer / streamer 200 also streams the gameplay to a remote viewer 220 who is viewing the gameplay audio and video (A / V) content 230 on another television 250. This gameplay is also streamed to a mobile application viewer 260, where the A / V content 230 is viewed on the smartphone once the A / V content stream is routed through the server 240 and received by the game console manufacturer's dedicated mobile application running on the smartphone. Thus, FIG. 2 illustrates different types of devices and systems that may be used to view the gameplay A / V content stream 230.

[0044] Figures 3A-3C further illustrate the present principles: Figure 3A shows that a gamer 200 can stream gameplay to one or more friends 220, 260 via a server 240, allowing those two viewers to watch together and enjoy the game with the gamer 200 and each other.

[0045] FIG. 3B illustrates that the gamer / streamer 200 may stream the gameplay to the friend 300 via the server 240, allowing the friend 300 to assist or coach the gamer 200, and also the friend 300 may assist or coach the viewer 300, if the friend 300 is also playing the same game instance. Thus, it should be understood that the gamer 200's speaking voice, detected by a local microphone near the gamer 200, may be streamed to the viewer 300, and vice versa. This may be accomplished through a separate dedicated out-of-band audio channel, or as part of the interactive game stream itself. Also, according to FIG. 3B, the person 310 shown in FIG. 3B is a visual representation of the coaching that may occur by the gamer 200 (even if the gamer 200 and the viewer 300 are located apart in different geographic locations).

[0046] Turning to FIG. 3C, this figure shows that streaming of the gamer / streamer's 200 gameplay to a friend 350 can be used to allow the gamer 200 and friend 350 to see each other's point of view (POV) (the other player's perspective, rather than the player's own perspective of the game character) while playing a multiplayer game.

[0047] Turning now to FIG. 4, it can be seen that the screen sharing experience can be further enhanced with unique interactive features to encourage viewer engagement and general screen sharing usage. These features include pointer / cursor control and reaction features. FIG. 4 shows an example of the former, where a pointer / cursor 400 is displayed on the gamer / streamer's own display 410 while playing a particular video game, with the cursor 400 being controlled by a remote viewer named Alex. With Alex in control of the cursor 400, a text display 420 is superimposed on the game video as part of the streamer user interface (UI), the display 420 showing Alex's name and indicating that Alex is controlling the cursor 400. As better shown in the enlarged close-up view of the cursor 400 shown in FIG. 5, the cursor 400 also comprises or includes an image 500 associated with Alex (and not other active gamers / viewers). The image 500 can be a graphic or avatar associated with Alex, or a profile picture of Alex himself.

[0048] Returning to FIG. 4, a circle 430 is also shown surrounding the cursor 400. This may serve to highlight graphical objects within the game that the remote viewer wishes to draw the gamer's attention to as potential areas that the gamer may travel or explore with the gamer's virtual character. Thus, the viewer's own device or the intermediary server itself may track the location of the cursor 400 as it moves across the gamer's own POV as it is presented on the display 410. The device / server may then present a larger circular graphical overlay 430 to highlight (e.g., surround) the graphical object of the game that it has determined the cursor 400 is over. In this example, the overlay 430 highlights a building that the viewer has indicated the gamer should explore with the gamer's game character.

[0049] Also note that while positioning / hovering a cursor over a game object can be one form of selection of the game object, other forms of selection consistent with the present principles can include inputting a left-click or right-click on the game object with a mouse after hovering an associated mouse-controlled cursor over the game object, touch-based selection of the game object if the viewer is using a touch-enabled display, etc. Additionally, note that according to the particular example of FIG. 4 , highlighting a game object can include presenting overlay 430 but not presenting a digital trace of the cursor control command itself (e.g., if the cursor control command involves moving cursor 400 over the game object while a left-click selection input is continually provided and rendered on the streamer's screen in response to the cursor movement, this aspect is contemplated in other scenarios consistent with the present principles as well).

[0050] 4 also shows that the gaming video presented on display 410 may also have an additional display 440 overlaid on top of it as part of the UI. Display 440 indicates that screen sharing with a viewer is occurring and that a screen sharing overlay UI is active (in this example, the UI displays elements 400, 420, and 430 according to the description above).

[0051] FIG. 6 continues this example, but now shows the viewer Alex's reaction as a smiling emoji with heart-shaped eyes 600, which is provided by Alex to show that he likes what he sees from the gamer as the gamer plays a computer game. Alex's avatar or profile picture 610 is also displayed so that the gamer can identify who provided the emoji 600. The elements 600, 610 may be presented using the same UI as described in the paragraph above, which is overlaid on the gamer / streamer's POV as the game is played on the display 410. Additionally, other cursor and reaction elements, as described below, may also be displayed using this UI. Thus, according to FIG. 6, it is understood that as another aspect of the present principles, viewers may send reactions to the shared screen / AV feed UI to show support and congratulations to the streaming gamer.

[0052] FIG. 7 further illustrates the present principles in a competitive tactics scenario of a sports game (in this example the sport is soccer). It is imagined that two friends are playing a soccer computer game, with a third-party viewer assisting one of the streaming gamers. The viewer wants to indicate where one of the streamers should pass the ball, so they can use the pointer / cursor 700 to draw on the screen / streamer POV 710 to indicate where on the virtual soccer field the streamer should pass the ball. For example, the cursor 700 can be positioned at a particular POV location, and then a selection command or left-click input (e.g., in the case of a mouse controlling the cursor 700) can be continually provided to draw freehand virtual ink in response to subsequent movements of the cursor 700 while the selection command is still being provided. Then, once the game is over, the viewer can again use the pointer 700 to analyze the particular game situation in detail for the streamer, to better prepare the streamer for the next game.

[0053] FIG. 8 shows another example of competitive tactics, where shooter tactics may be exchanged. For example, it is assumed that two friends are playing a first-person shooter computer game, but the viewer has already been eliminated and is simply watching until the game ends. It is also assumed that the viewer wants to show the streamer (and the other friend who is still playing the game) where his virtual character is located in the virtual world, and which direction he should face to ensure victory in the game. The viewer can achieve this by controlling a cursor 800 that is displayed on the streamer's own screen 810 (and is also mirrored on the viewer's screen). It is also assumed that the viewer then spots a virtual enemy coming from a certain direction and quickly uses the pointer / cursor 800 to indicate the direction from which the attack is coming.

[0054] Turning to FIG. 9, yet another example is shown, this time of exploring a virtual world. Thereby, a virtual map 900 of the game world is presented on the display 910 of the streamer's device. It is then assumed that the streamer is playing a particular game and shares his / her screen with the viewers so that they can explore the world together. The streamer is exploring a large area, but when the viewer notices an area that has not been explored, the viewer uses the pointer / cursor 920 to mark on the map 900 exactly where that area is located. The viewer can do so by simply hovering the cursor 920 over the area or by providing a selection input (e.g., left click of the mouse) when the cursor 920 is over the area, instructing the game to persistently display an "X" mark on the map 900 as "X indicates spot" accordingly.

[0055] Now referring to FIG. 10, another example of virtual world exploration is shown, but with looting. It is assumed that the streamer is playing a particular game while the viewer is just watching and mainly following the storyline. The viewer then realizes that the streamer did not check the potential loot pool. Thus, the viewer can use the pointer / cursor 1000 to specifically indicate to the streamer where on the screen to look (under or beside the virtual dresser 1010), thus making the viewer an active participant in the game.

[0056] Now, two examples of the use of the pointer / cursor for highlighting have been described. With reference to FIG. 11, it is assumed that two friends are playing a certain game and one of the friends (the viewer in this example) is assisting the other friend (the streamer / gamer in this example). The viewer may notice that when the streamer takes a shot (e.g., kicking or hitting the ball towards the goal), he always misses the ball. Thus, the viewer can use the pointer / cursor 1100 to mark which part of the ball 1110 the streamer should aim at in order to make contact with and hit the ball 1110 more accurately. Thus, the viewer is indicated with the "sweet spot" where the ball 1110 should be kicked.

[0057] Another example of a challenge situation is shown in FIG. 12. Assume that a streamer is playing a particular game but is having trouble fighting a tough boss or enemy. The streamer then asks a friend who has already cleared / beaten the boss 1210 to help the streamer by watching the streamer's screen. The friend consults with the streamer about weak points in the streamer's attack plan and uses the pointer / cursor 1200 to highlight the exact location / way to attack the boss 1210.

[0058] Moving on to an example involving reactions rather than the pointer / cursor action itself, we refer to FIG. 13. This diagram illustrates a situation where many viewers are gathered together to watch a remote streamer's gameplay (in some cases they are playing the same game instance). Due to the large number of people, it may be difficult for the streamer to keep track of who is saying what in the audio chat where everyone is talking in the game stream. Instead of trying to pay attention to it, the viewers can provide their own emoji reactions 1300 with animated emotions to show their excitement and appreciation for the great move by the streamer in the game without causing any disruption to the streamer. The streamer may feel even better about that great move because the on-screen reactions 1300 amplify the game experience. Here, the reactions 1300 are mostly, if not all, positive.

[0059] 14 shows another situation where certain viewers are muted from the audio chat or simply cannot speak for some reason. Again, a group of friends may be playing a game together while others are just hanging out watching the gamers' collective gameplay. Even if some viewers do not have access to a microphone or are muted, they can still continue to enjoy themselves by casting some kind of reaction 1400 on the screen 1410.

[0060] Next, Figures 15 and 16 show a small party reaction scenario. In the small party scenario of Figure 15, it is assumed that two friends are playing a certain game and one of the players (the streamer in this example) is struggling to beat a difficult boss and keeps dying over and over again. The viewers can then decide to always react with an emote 1500 as a cheer every time to encourage the streamer to continue. The emote 1500 can be a heart as shown and can be permanently displayed on the streamer's screen 1510 even if it is provided by the viewers at different times.

[0061] 16 illustrates a trolling scenario in which two friends are playing a game together and the streamer (one of the players) is attempting to build / complete a structure as a challenge. The viewers (other players) may find the streamer's inability to do so amusing and may tease the streamer by adding funny reactions every time the streamer nearly gets there. In this example, the reactions include an excrement / poop emoji 1600 that may be persistently displayed on the streamer's screen 1610 until the streamer successfully completes the task, in response to which the emoji 1600 may be removed from display.

[0062] With reference to FIGS. 17-20, it should be appreciated that an adaptive pointer / cursor may be presented consistent with the above description based on the game context (e.g., a cursor that is consistent in appearance and behavior / animation with the ongoing game context). Thus, the game engine may use an artificial neural network (ANN), such as a feed-forward neural network, a deep neural network, or other type of context-aware or pattern-aware neural network, to identify the current game context and adjust the cursor's appearance in response. The ANN may be trained in an unsupervised manner based on one or more sets of training data including the respective context (and / or raw game state data, which may include metadata, video data, and audio data from which the context can be determined). The training data set may also include the respective ground truth pointer appearance to present for each context / game data.

[0063] Then, while the ANN is deployed when a particular game instance is running, the execution device may receive game state data and / or current game context from the game engine itself, or simply receive A / V data from the game itself if the game engine itself is not accessible (such as older legacy games). The device may then determine the context from the game state / AV data if not provided directly from the game engine, and either way provide the received / determined context and / or raw game state data to the ANN as input to the ANN, and generate an inference output in response. That output may be a cursor image to display or a reference identifier for the type of cursor to search for and display. The device may then present an inferred cursor image that corresponds to that output.

[0064] It should be noted that in some examples, the different cursors themselves may all be included in a reference library of cursors accessible to the system. The cursors may each have their own reference ID. Some or all of the cursors in the library may be used as ground truth for the training described above, so that cursors do not need to be dynamically generated during gameplay, and inference output from the ANN may indicate a particular cursor from the library to use. However, in other examples, the system may use the generated image output by the ANN as the cursor image, since the cursor image may actually be indicated in the output even if it is not associated with an existing reference cursor.

[0065] With the above in mind, reference is now made to FIG. 17. This figure illustrates a triangular cursor 1700 with an exclamation mark inside it, which may be presented during a combat sequence in a particular video game. Additionally, cursor 1700 may turn red and pulsate, emphasizing tension. Next, according to FIG. 18, when the game situation changes from combat to a victory celebration, cursor 1700 may morph into cursor 1800 of FIG. 18. As shown in FIG. 18, cursor 1800 may be a green circle with a green check mark inside it to indicate victory, and cursor 1800 may even animate to have virtual fireworks emanating from it.

[0066] 19, in a virtual world exploration context, cursor 1800 may morph into a magnifying glass cursor 1900 to symbolize and encourage the viewer / streamer to explore the virtual world in more detail. Next, FIG. 20 shows another example in which cursor 2000 may be presented as a house burning down in flames based on the streamer's character's health dropping below a threshold amount (e.g., 20 percent) and / or based on the streamer's character losing a boss fight and dying.

[0067]

[00136] Referring now to Figure 21, this figure shows a virtual palette 2100 that may be presented on a viewer's screen while watching a streamer / gamer's gameplay. The palette 2100 may thus be presented as part of the UI described above with reference to Figure 6, for example, but only displayed on the viewer's display and not on the streamer's display. As such, it may be presented semi-transparently as an overlay on the gaming video stream itself, or it may be displayed as an opaque inset in the gaming video, or it may be presented opaquely to one side of the video but still displayed on the same display / UI.

[0068] As shown in FIG. 21, the palette 2100 may include a first section 2110 presenting various emojis, emotes, and / or other viewer-selectable graphic reactions. The palette 2100 may also include a second section 2120 presenting various viewer-selectable sound effects, which when selected may instruct the game engine to insert the corresponding sounds into the audio of the game itself so that the streamer / gamer can hear the sounds as they play the game and enjoy verbal and / or non-verbal audio feedback from the viewer. As shown in section 2120, verbal feedback may include the exclamation "Ouch" (top selector), while non-verbal feedback may include laughter (middle selector) or the ba-dum-tsh sound (bottom selector) that frequently follows jokes in the entertainment industry during broadcasts.

[0069] Section 2130 of FIG. 21 shows various stickers and persistent pointer objects that are also selectable by the viewer. Note that the viewer thereby sends various stickers, emojis, and / or predefined objects from section 2130. Thus, the viewer can drag a selected element onto the game screen, and the element may be affixed to a viewer-specified area in the streamer's POV as a persistent overlay that is not removed except at the user's command or upon the end of the game or game level. If desired, stickers can also be differentiated for each viewer, so the streamer knows which stickers are associated with which viewer. Thus, these elements may remain in place for a set period of time or until removed. Viewers may be creative in resizing, rotating, and positioning the elements on the screen, thereby expressing their individuality and enhancing their visual experience. By way of example, section 2130 shows that the viewer's avatar image may constitute one of these elements (top selector) and the "#1 viewer" sticker may constitute the other one of these elements (bottom selector).

[0070] Thus, more generally, it should be understood that the system may receive a cursor control command instructing the viewer to place a graphical object (the sticker of section 2130) uniquely associated with the viewer at a permanent location within the streamer's field of view / POV to be presented on the streamer's own display. Based on this cursor control command, the system may then present the graphical object (the sticker) at the permanent location.

[0071] 21 also shows that the palette 2100 may include a section 2140 presenting different viewer-selectable cursors that may be used by the viewer and therefore presented to the streamer in accordance with the above description. Note that as examples, a circular pointer (top), an arrow pointer (middle), and an option to instruct the gaming system to use an adaptive pointer in accordance with the above description are shown (bottom).

[0072] Turning to FIG. 22, dynamic terrain analysis may also be used by the game system to present different looking cursors, which may provide a feeling about the terrain and / or real-time terrain analysis. Thus, for example, as the streamer explores different areas of the game world, the cursor may adapt to provide information about the game environment / terrain in which the streamer's game character is currently moving. For example, the cursor may change color or display icons (such as cursor 1700) to indicate danger zones, to indicate hidden treasure, or to indicate safe paths. Alternatively, as shown specifically in FIG. 22, the cursor may provide information about enemy statistics for enemy game characters as non-players of the game. In this example, cursor 2200 indicates the percentage of the boss's health remaining (in relation to the boss being fought by the streamer). Other example statistics may include points gained, inventory fullness, or inventory items, etc. This dynamic guidance may thus enhance the viewer's understanding of the game world through the adaptive cursor, aiding both the streamer's navigation and decision making. The stats cursor may rotate at predefined time intervals (e.g., every 5 seconds) between different stats for the boss and / or the streamer's own character.

[0073] With reference to Figures 23 and 24, the principles also include enabling streamer-generated and viewer-generated surveys. Thus, both streamers and viewers can create surveys and create survey questions and prompts. This can be done by providing voice input to a large-scale language model (LLM) (e.g., a ChatGPT-based application programming interface (API) or a digital assistant running on a game engine or the entire system) and instructing the system to generate a survey according to one or more prompts provided by the person via voice. The LLM or assistant can then generate survey questions and freely select survey answers based on different non-verbal responses that other people may select. Thus, for example, because each emoji or other response in the reference library includes metadata indicating the emotion associated with the respective emoji, the LLM / assistant can select as a potential answer an emoji associated with the emotion correlated by the LLM / assistant to the survey question. Viewers and other poll respondents can select answers using the designated emoji or other responses.

[0074] This is shown in the example of FIG. 23, where Cody, as another person playing against the streamer, provided voice input in which the streamer asked the LLM, "Do you think Cody should be punked?" A poll 2300 is then dynamically generated and presented to viewers of the live stream of the game, and different emojis 2310 are presented and may be selected by the viewer as a poll answer. As shown, a smiley face emoji and a laughing face emoji may be included, which may be selectable to provide an approval (affirmative) answer to the poll question. As shown, a frown emoji may be included, which may be selectable to provide a disapproval (negative) answer to the poll question.

[0075] Once the survey ends after a threshold period, a streamer-specified period, or other trigger, an overlay 2400, as shown in FIG. 24, may be displayed on both the streamer's screen and the viewers' screens. The overlay 2400 may aggregate and display the survey responses in a visually appealing format and show the percentage contribution 2410 of each voting option selected. The streamer may recognize the viewer's preference by selecting a selector 2420 (which may result in the digital assistant / system providing an audio output of a computer-generated voice yelling "It's time to punk!") and use this information to customize content or make in-game decisions.

[0076] 25, exemplary logic that may be executed by a system such as system 10 and / or its respective components (e.g., a console or a remote server) consistent with the present principles is shown. Note that although the logic of FIG. 25 is shown in flow chart form, other suitable logic may be used.

[0077] From block 2500, the system may execute a computer game in which a first person plays the computer game. The logic may then proceed to block 2505, where the system may stream the first person's gameplay to other viewers, for example, via the Internet, a third party streaming website, a dedicated console manufacturer's network, etc. The logic may then proceed to block 2510, where the system may receive input from viewers including at least a second person who is watching the live stream of the computer game but not controlling a character in the game. The input may include cursor control commands to move one of the aforementioned cursors within the first person's viewpoint / POV, as well as graphic-based emotional responses (e.g., emojis) as also described above.

[0078] The logic of FIG. 25 then proceeds to block 2515, where the system may present one or more corresponding outputs on a display associated with the first person based on the input(s) received at block 2510. The output(s) may indicate the input in a game-specific context. As one example of an input indicated in a game-specific context, at block 2520, the system may change the appearance of a cursor based on the game context and / or specific game events, as described above.

[0079] The logic may then proceed from block 2520 to decision diamond 2525. At diamond 2525, the system may determine (e.g., using instructions from the game engine) that a cutscene is occurring, perhaps after executing at least a first cursor control command during a first portion of the computer game. In various examples, a cutscene may be a non-interactive scene in a game where no user input is processed, and the cutscene is initiated when the streamer reaches a particular point in the game (e.g., the end of a level or the streamer's virtual character dies). Thus, a cutscene may interrupt gameplay with a conversation sequence between two game characters, a series of events related to the game's plotline, an awards ceremony where the streamer is provided with one or more rewards, etc.

[0080] Based on / in response to a determination that a cutscene is not occurring, the logic may proceed directly to block 2535, as described below. However, based on / in response to a determination that a cutscene is occurring, the logic may instead proceed first to block 2530, where the system may refuse to process additional cursor control commands and / or reactions during the playout of the cutscene. A multi-viewer UI such as that of FIG. 6 may thereby be contextually controlled (e.g., on and off) during the game to enable automatic detection of cutscenes, scene changes, and / or important moments (e.g., bosses) in the game where viewer input may be too intrusive. When such moments occur, pointer and reaction overlays may be temporarily turned off to ensure that both the streamer and viewer can fully appreciate the story without distraction. When gameplay resumes from the end of the cutscene, scene change, or boss fight, interactions may be re-enabled and replayed.

[0081] The logic may then proceed from block 2530 to block 2535. At block 2535, the system, in some implementations, may alternate authority to issue cursor control commands (and even reactions) among viewers of the computer game. This avoids situations where multiple cursors are floating around on the streamer's screen at the same time, which may distract the streamer and unduly interfere with the viewing of the game video. Thus, in certain implementations, only one cursor may be displayed at a given point during gameplay, in which case the viewer who is allowed to control the cursor may be determined based on past viewer input associated with the first person's gameplay of the computer game.

[0082] The feature may thereby be thought of as passing a virtual baton, with cursor control operating on a turn-by-turn basis. The UI / overlay may track viewer engagement, for example, in terms of volume of reactions. As viewers accumulate points through more interactions (cursor control and reactions), they advance higher in the queue, and the viewer at the front / top of the queue gains control of the pointer. Control may then remain with that viewer until one or more conditions are met, such as the controlling viewer providing a selection command with the cursor, a threshold time expiring (e.g., 30 seconds), or the controlling viewer relinquishing control.

[0083] Additionally or alternatively, the streamer may determine who can use the pointer at any given time by providing a voice command designating a particular user, or simply verbally instructing the system to "take over," which may prompt a handoff of control to the next viewer in the queue. This feature may thus encourage continued, gamified engagement on the part of the viewer themselves.

[0084] The logic may then proceed from block 2535 to block 2540, where the system may continue to control output, game actions, and / or game state based on the viewer's cursor commands and reactions. The logic may then proceed to block 2545, where the system may execute an eye tracking algorithm to track the eyes of one or more viewers, in some implementations using an image of the viewers provided by a camera for imaging the viewers' eyes. The camera may be located, for example, on a headset worn by each viewer or on a television that each viewer is using to watch the game.

[0085] Eye movements from the viewer may constitute input that may be used to move a cursor on the streamer's display according to the eye movement input at block 2550. For example, if the viewer is looking left, the cursor moves a proportional amount to the left, and if the viewer is looking right, the cursor moves a proportional amount to the right. Also at block 2550, in some examples, the system may determine that a tracked viewer has one eye closed, such as by winking or blinking with one eye (but not both eyes simultaneously). Based on a determination that the viewer has one eye closed, the system may change the appearance of the cursor presented on the streamer's display.

[0086] As a specific example, if a viewer is wearing a virtual reality (VR) headset for viewing VR gameplay, an eye-tracking sensor on the headset can be used to establish VR eye-tracking pointer movement and send a (pulsating) pulse to the pointer that is presented on the streamer's display when the viewer blinks / winks with one eye. This can be an easy and intuitive way to draw the streamer's attention to the game object that the cursor moves over and hovers over.

[0087] The logic may then proceed from block 2550 to decision diamond 2555. At diamond 2555, the system may determine whether a predetermined graphics-based reaction threshold has been reached based on receipt of respective graphics-based reactions (such as may occur at block 2540) from respective people watching the live stream but not controlling a character in the game. This may include a predetermined number of all or the same type of reactions provided by viewers within a set time, such as 20 seconds (which eliminates false positives from outside the time window while implicitly indicating that the reactions are all related to the same game event). Alternatively, this may more generally include a predetermined number of all or the same type of reactions provided by viewers during a particular game scene or game level. Here, the same type may relate to specific emojis, each considered a unique type, or it may relate to emojis grouped into different types based on overall emotion (e.g., positive emotion emojis, such as happy face and laughing emojis, are grouped as one type, and negative emotion emojis, such as frowning face and angry face emojis, are grouped as another type).

[0088] A negative determination may cause the logic to proceed directly to block 2565, as briefly described below. However, a positive determination at decision diamond 2555 may instead cause the logic to proceed to block 2560 in response to a predetermined graphics-based reaction threshold being met. At block 2560, the system may alter the playout of the computer game based on the overall or majority / plurality of emotions establishing at least a threshold number. For example, an accumulation of an overall positive emotional reaction may cause the playout of the game to be altered by presenting fireworks in the background of a game scene. For example, an accumulation of an overall negative emotional reaction may cause the playout of the game to be altered by presenting a red flash in the background of a game scene.

[0089] The logic may then proceed from block 2560 to block 2565. In this step, based on the one or more graphics-based responses, the system may present haptic feedback to the first person (the streamer). The haptic feedback may be presented using a vibrator on a game controller the first person is using, a vibrator on a VR headset or other headset type the first person is wearing, a vibrator on the first person's smartphone or other connected device, etc. The haptic feedback may vary based on the response type associated with the graphics-based response that caused the vibration.

[0090] Thus, for example, as a viewer submits a reaction, the streamer may receive subtle haptic feedback corresponding to the type and intensity of the reaction. Thus, if a smiley face emoji is provided, a discrete pulse of vibration of relatively low intensity / amplitude may be provided, whereas if a laughing emoji is provided (designated as a stronger positive reaction), a discrete pulse of vibration of higher intensity / amplitude may be provided. Conversely, if a frown emoji is provided, a single short vibration of relatively low intensity / amplitude may be provided, whereas if an angry face emoji is provided (designated as a stronger negative reaction), a single short vibration of higher intensity / amplitude may be provided.

[0091] The logic may then proceed from block 2565 to block 2570. At block 2570, if desired, the system may aggregate emotion-based reactions provided by multiple viewers. This may be done to present the first person (the streamer) with a reaction summary summarizing the amount of reactions received, also based on the aggregation at block 2570, which reaction summary is categorized and visually displayed by reaction type.

[0092] The logic may then proceed from block 2570 to block 2575. At this step, the system may perform one or more of the following: First, the system may award at least one virtual trophy to the first person (the streamer) based on the amount of emotion-based reactions received while the first person is playing one or more particular aspects of the computer game (such as a combat sequence, a game level, a game scene, etc.). Second, the system may reward at least one viewer of the live stream based on input provided by the viewer and established by one or both of cursor control commands and emotion-based reactions. As the input increases, the reward may increase proportionately.

[0093] Thereby, with regard to trophies and challenges, new or unique types of trophies may be provided to streamers based on the amount of viewer response received, with trophies becoming progressively larger or more important as more viewer input is received. Thus, streamers may earn predefined trophies associated with viewer input, and may earn trophies for sequence, timing, and / or amount of response in a particular challenge, game sequence, game level, etc. This concept can also be applied to games with daily / weekly in-game challenges.

[0094] Similar to the above, with regard to reaction tiers and corresponding rewards to viewers, viewer reactions can be divided into tiers based on the level of engagement (e.g., total amount of reactions provided), such as bronze, silver, and gold levels. As the viewer continually engages in the gameplay with their reactions, the tier progresses, gradually accumulating engagement points and unlocking different rewards with gradually increasing reaction input. A progress bar in a UI overlay visible to the viewer (possibly invisible to the streamer) may indicate the viewer's progress to the next tier, encouraging continued interaction to reach the next tier. This may also be applied on the streamer's side (e.g., the more reactions from the viewer, the more rewards, game points, etc. the streamer can earn).

[0095] Continuing with the detailed description with reference to Figure 26, an exemplary graphical UI 2600 that may be presented on a display of a gaming system to configure one or more settings of the system to implement the present principles is shown. Each of the exemplary options described below may be selected via touch input to the display if it is touch enabled, cursor input (such as mouse or trackpad input), or other input to checkboxes (and indeed selectable items) associated with this example (any of those disclosed herein may be selected by these methods).

[0096] As shown in Fig. 26, the GUI 2600 may include a first option 2610 that may be selectable only once to set / configure the system to perform the above-mentioned functions for multiple future games / game instances. Thus, selecting option 2610 enables viewer-based interaction with the streamer / gamer via cursor control commands and graphic-based non-verbal responses. Thus, for example, the logic of Fig. 25 may be executed for multiple future game instances based on option 2610 being pre-selected.

[0097] The GUI 2600 may also include an option 2620 that may be selected to set up or configure the system so that viewer input can actually control the game itself, rather than simply presenting content to the streamer. Thus, for example, selecting option 2620 may instruct the system to subsequently execute an algorithm that causes the system to present selectable graphical objects on the viewer's display as part of the game itself. The system may then receive cursor control commands that select the selectable graphical objects, and based on the cursor control commands, control the output of the computer game that is displayed on the display associated with the streamer. The output itself may have the effect of altering the current game state of the computer game itself.

[0098] As a specific example, viewers may be allowed to interact with game layer objects. For example, a software development kit (SDK) may allow designated objects in the game world to display markers that the viewer can interact with. When the viewer clicks on one of these markers, the overlay / UI may send commands to the game simulating the viewer's input. For example, if the viewer clicks on a lever in the overlay, the game may receive a command that triggers the action of the lever, changing something within the game itself. This two-way communication can bridge the virtual gap between the viewer, streamer, and game, allowing the viewer to actively participate in the gameplay.

[0099] As another example, consider an SDK for game publishers. This could be a "generic" SDK that allows game publishers to define specific triggers, events, objects, and responses that occur when the viewer interacts through the overlay / UI. For example, developers could integrate unique weapon drops or special character animations that are triggered by the viewer's actions. From a store perspective, viewers could "click" on skins and other digital add-ons to see the purchasable content in the store and even add them to their wishlist.

[0100] Additionally, in some cases, reactive environments are employed, where pointer movements affect the game environment in real time. Viewers can use their pointer to activate switches, create temporary bridges, or manipulate objects in the game. These interactions affect the streamer's gameplay and may lead to secret passages or "Easter eggs" that reveal hidden items in the game.

[0101] Thus, more generally, the system may receive cursor control commands and, based on the cursor control commands, perform predefined game actions intended to be performed only by persons not controlling characters during a computer game (such as viewers not playing a video game / controlling a game character).

[0102] Viewers may mark game layer objects. Thus, for example, a viewer may draw or place a marker on the streamer's screen to highlight a particular game object, enemy, or tactical point. For example, to draw, a user may left-click and hold the left mouse button, then move the mouse to draw on the UI. By placing the cursor over a game object and left-clicking up or down without moving the mouse, the cursor may be controlled to place a visual marker in the streamer's viewpoint (e.g., anchored to a particular virtual location in the game). In some cases, this functionality may also be implemented as part of the SDK.

[0103] Additionally, for viewer-generated tasks, viewers may use the UI / overlay drawing tools (some of which were just mentioned) to sketch tasks, challenges, or proposals directly on the streamer's screen. The streamer can review the tasks and choose to accept or reject them or negotiate variances with the viewer.

[0104] Moving on to reactions, it is further noted that emotion-based visual effects are also included in the present principles. Thus, the system disclosed herein adds the ability to monitor the cumulative emotion of the viewers' reactions by analyzing factors such as type, frequency, and speed of reactions. When a certain reaction threshold is met / exceeded, the UI / overlay may trigger a corresponding visual effect for the streamer and / or viewers. For example, a surge in positive reactions may result in a flurry of fireworks animation as described above. On the other hand, a surge in negative reactions may result in a subtle shadow being cast on the screen.

[0105] Avatar-based reactions may also be used; thus, depending on the game's character or avatar, the appearance of the reaction in a particular game may be altered to match the in-game persona (e.g., the avatar used for a reaction could be an image of one of the game's characters). Additionally, viewers can customize their reactions by choosing from a variety of animations, icons, and even short sound clips.

[0106] Additional visual overlays may also be deployed by type. These additional overlays may include themed filters that change the game's visual palette for different moods or settings, countdown timers to in-game events, sound effects or images that are overlaid on specific sections of the screen, etc. Viewers and streamers can enable or disable these elements, providing a dynamic and customizable visual experience.

[0107] Cursor ghosting may also be used in accordance with the present principles. Thus, the viewer may use the cursor to select and tag a player character, and the cursor may automatically follow the character without any additional input from the viewer. The cursor itself may create a visual trail behind it (such as a comet tail) indicating where the cursor has been.

[0108] Cursor hysteresis is included as well: the cursor may disappear or vanish to signal something, for example it could disappear in flames due to a change of emotion ("I was wrong", meaning the viewer was wrong about the suggestion they provided to the streamer).

[0109] Additionally, if the viewer says "left" with the cursor (e.g., by pointing the cursor to the left or writing "L" on the screen for "left"), the cursor / highlight may remain where the input was commanded. The cursor may then continue to be displayed if the player does not perform an action, or may disappear if the player performs an action, so the viewer knows the player has responded to the request.

[0110] Additionally, the artificial intelligence based model may analyze game objects, so that when a viewer points at an object or circles an object with a cursor, that object may be highlighted in the game with the color of the pointer itself, and the system may then modify the in-game object based on this functionality.

[0111] Although particular embodiments are shown and described in detail herein, it should be understood that the subject matter encompassed by the present invention is limited only by the scope of the claims.

Claims

1. executing the computer game in which a first person plays the computer game; receiving input from a second person viewing a live stream of the computer game but not controlling a character of the game, the input including one or more of cursor control commands and graphics-based responses; Based on the input, presenting an output on a display associated with the first person that illustrates the input in a game-specific context.

23. An apparatus comprising a processor assembly programmed with instructions for:

2. the input includes the cursor control command; 2. The apparatus of claim 1.

3. the output includes the cursor being presented with an appearance associated with an ongoing game context of the computer game.

3. The apparatus of claim 2.

4. the output includes the cursor being presented with an appearance associated with a current game event.

3. The apparatus of claim 2.

5. the cursor control command is a first cursor control command, the processor assembly executing the first cursor control command during a first portion of the computer game; It is determined that a cut scene is occurring. configured to refuse to process additional cursor control commands based on said determination.

3. The apparatus of claim 2.

6. the cursor control command is a first cursor control command, the processor assembly configured to alternate authority to issue cursor control commands among viewers of the computer game based on past viewer input associated with gameplay of the first person of the computer game.

3. The apparatus of claim 2.

7. The processor assembly includes: presenting a selectable graphical object on a display associated with the second person; receiving the cursor control command for selecting the selectable graphical object; configured to control an output of the computer game presented on the display associated with the first person based on the cursor control command and having the effect of modifying a current game state of the computer game.

3. The apparatus of claim 2.

8. The processor assembly includes: receiving the cursor control command directing placement of a graphical object associated with the second person at a permanent location within the field of view of the first person presented on the display; configured to present the graphical object at the persistent position based on the cursor control command.

3. The apparatus of claim 2.

9. The processor assembly includes: receiving said cursor control command for selecting a game object; configured to highlight the selected game object based on the cursor control command, the highlighting not including a trace of the cursor control command itself; 3. The apparatus of claim 2.

10. The processor assembly includes: receiving the cursor control command; configured to perform predefined game actions based on the cursor control commands, the predefined game actions being reserved for execution by a person not controlling a character during the computer game.

3. The apparatus of claim 2.

11. The input includes an eye movement input associated with the second person, and the processor assembly further comprises: moving the cursor on the display in response to the eye movement input; determining that the second person has one eye closed; configured to modify an appearance of the cursor presented on the display based on the determination.

3. The apparatus of claim 2.

12. The processor assembly includes: configured to display the cursor on the display with a visual appearance that conveys a feeling about a terrain over which the first person's game character is traversing.

3. The apparatus of claim 2.

13. The processor assembly includes: configured to display the cursor on the display with a visual appearance indicative of data regarding a current game statistic of a first game character, the first game character being a game character not controlled by the first person; 3. The apparatus of claim 2.

14. the input includes the graphics-based response; 2. The apparatus of claim 1.

15. The processor assembly includes: receiving a respective graphically-based response from each person watching the live stream but not controlling a character of the game; configured to alter playout of the computer game in response to a predetermined graphics-based response threshold associated with a plurality of graphics-based responses of the same type being received being met.

15. The apparatus of claim 14.

16. The processor assembly includes: configured to present, based on the graphics-based response, haptic feedback to the first person that varies based on a response type associated with the graphics-based response.

15. The apparatus of claim 14.

17. executing a computer game in which a first person plays the computer game; receiving input from a second person viewing a live stream of the computer game but not controlling a character of the game, the input including one or more of cursor control commands and emotion-based responses; displaying a game output on a display associated with the first person based on the input; and A method comprising:

18. aggregating emotion-based responses provided by a plurality of viewers; presenting a reaction summary to the first person based on the aggregation; and 20. The method of claim 17, comprising:

19. It is not a temporary signal, executing the computer game in which a first person plays the computer game; receiving input from a second person viewing the live stream of the computer game, the input including one or more of cursor control commands and emotion-based responses; at least one computer storage device including instructions executable by at least one processor for displaying an output on a display associated with the first person based on the input; system.

20. The instruction: awarding at least one trophy to the first person based on an amount of emotion-based responses received while playing one or more particular aspects of the computer game; and executing a reward for at least one viewer of the live stream based on input provided by the viewer and established by one or more cursor control commands and / or emotion-based responses.

20. The system of claim 19.

Images