Adaptive screen sharing pointer and in-game response

Abstract

To enhance the overall execution environment of a game itself, and provide enriched technology-based interactivity features.SOLUTION: Game-specific screen share features can be provided to viewers of a video game livestream. The viewers can control a themed or adaptive cursor presented to the gamer during gameplay, and can also provide themed or adaptive sentiment-based reactions to the gamer during gameplay. The cursor and reactions can then be surfaced to the gamer in a number of ways and can even be tied back into the game execution environment itself.SELECTED DRAWING: None

Description

Technical Field

[0001] The following disclosure generally relates to an adaptive screen sharing pointer and reaction for computer game play.

Background Art

[0002] Here, as will be understood, some video gamers stream their game play to other users so that others can view their game play. As is recognized here, current systems are technically limited in their ability to provide sufficient participation by viewers, but the present principle recognizes that, from a technical perspective, it can provide a more rich and engaging viewing experience.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Thereby, the present principle aims to enhance the overall execution environment of the game itself and to provide a rich technology-based interaction function.

Means for Solving the Problems

[0004] As such, in one aspect, the apparatus comprises a processor assembly programmed with instructions for executing a computer game in which a first person plays the computer game. The processor assembly is programmed with instructions for receiving an input including cursor control commands and / or graphic-based reactions from a second person who is viewing a live stream of the computer game but not controlling the characters of the game. The processor assembly is programmed with instructions for presenting, on a display associated with the first person, an output indicating the input in a game-specific context based on the input.

[0005] In any exemplary implementation, the output may include the cursor being presented in a manner associated with the ongoing game context of the computer game. Furthermore, or alternatively, the output may include the cursor being presented in a manner associated with the current game event.

[0006] As another example, the cursor control command may be a first cursor control command, and the processor assembly may be configured to execute the first cursor control command during a first part of the computer game. In this example, the processor assembly may be configured to determine that a cutscene is occurring and, based on that determination, refuse to process any additional cursor control commands.

[0007] As yet another example, the cursor control command may also be a first cursor control command, in which case the processor assembly may be configured to alternate the authority to issue cursor control commands among viewers of the computer game based on past viewer input related to the gameplay of the first person in the computer game.

[0008] Furthermore, consistent with this principle, the processor assembly may, in some cases, be configured to present selectable graphic objects on a display associated with the second person. Here, the processor assembly may also be configured to receive the cursor control commands that allow the selection of the selectable graphic objects, and to control the output of the computer game presented on the display associated with the first person based on the cursor control commands. The output of the computer game may have the effect of changing the current game state of the computer game.

[0009] Furthermore, the processor assembly may, if necessary, be configured to receive the cursor control command which instructs the system to position a graphic object associated with the second person at a permanent location within the field of view of the first person presented on the display. Based on the cursor control command, the processor assembly may be configured to present the graphic object at the permanent location.

[0010] In another implementation, the processor assembly may be configured to receive the cursor control command for selecting a game object and to highlight the selected game object based on the cursor control command. The highlighting does not necessarily include a trace of the cursor control command itself.

[0011] Furthermore, in some cases, the processor assembly may be configured to receive the cursor control command and, based on the cursor control command, execute a predefined game action reserved for execution by a person who does not control a character during the computer game.

[0012] As another example, the input may include eye movement input associated with the second person, where the processor assembly may be configured to move the cursor on the display in response to the eye movement input, determine that the second person has one eye closed, and, based on the determination, change the appearance of the cursor presented on the display.

[0013] If necessary, the processor assembly may be configured to display the cursor on the display in a visual appearance that conveys emotions about the terrain the first person's game character is moving through.

[0014] As yet another example, the processor assembly may be configured to display the cursor on the display in a visual appearance that shows data relating to the current game statistics of a first game character, which is a game character not controlled by the first person.

[0015] With respect to the graphics-based responses, the processor assembly may, in some cases, be configured to receive graphics-based responses from each person watching the livestream but not controlling the game characters, and to modify the playout of the computer game in response to a predetermined graphics-based response threshold being met. The predetermined graphics-based response threshold may be related to multiple graphics-based responses of the same type that are being received.

[0016] In addition, or alternatively, the processor assembly may be configured to present the first person with haptic feedback that changes based on the type of response associated with the graphics-based response, based on the graphics-based response.

[0017] In another embodiment, the method includes running the computer game in which a first person plays the computer game, and receiving input from a second person who is watching a live stream of the computer game but is not controlling the characters in the game. The input includes cursor control commands and emotion-based responses. The method further includes presenting game output on a display associated with the first person based on the input.

[0018] For example, this method may include aggregating emotion-based responses provided by multiple viewers and presenting a summary of those responses to the first person based on the aggregation.

[0019] In yet another embodiment, the system includes at least one computer memory that is not a transient signal. The computer memory includes instructions executable by at least one processor for receiving input from a second person who is running the computer game in which a first person plays the computer game and watching a live stream of the computer game. The input includes cursor control commands and emotion-based responses. The instructions are also executable to present output on a display associated with the first person based on the input.

[0020] In one example, the command may be executable to award the first person at least one trophy based on the amount of emotion-based responses received while playing one or more specific aspects of the computer game, and to reward at least one viewer of a live stream based on input provided by that viewer. The input may be established by one or more cursor control commands and / or emotion-based responses.

[0021] The details of the structure and operation of this application can be best understood by referring to the attached drawings. In the attached drawings, similar reference numbers refer to similar parts. [Brief explanation of the drawing]

[0022] [Figure 1] This is a block diagram of an exemplary system consistent with this principle. [Figure 2] This diagram illustrates how a streamer playing a video game consistent with this principle streams gameplay to viewers using different types of devices. [Figure 3A] This outlines different types of streamer / viewer interactions that may occur in accordance with this principle. [Figure 3B] This outlines different types of streamer / viewer interactions that may occur in accordance with this principle. [Figure 3C]Shows a schematic of different types of streamer / viewer interactions that may occur in accordance with this principle. [Figure 4] Shows a first example of a graphical user interface (GUI) overlaid on a streamer's point of view (POV) while the streamer is playing a computer game in accordance with this principle, where a viewer is controlling a cursor presented to the streamer on the GUI. [Figure 5] Shows an enlarged view of the cursor of FIG. 4 in accordance with this principle. [Figure 6] Shows an example of a GUI overlaid on the POV of the stream while the streamer is playing a computer game in accordance with this principle, where a viewer provided an emotion-based reaction presented to the streamer. [Figure 7] A diagram showing an example in the situation of using a pointer to assist competitive tactics while playing a sports computer game. [Figure 8] Shows an example in accordance with this principle in the situation of using a pointer to assist competitive tactics while playing a first-person shooting computer game. [Figure 9] Shows an example in accordance with this principle in the situation of using a pointer to assist in exploring a virtual / game world. [Figure 10] Shows an example in accordance with this principle in the situation of using a pointer to assist in looting / searching for hidden objects in a game world. [Figure 11] Shows an example in accordance with this principle in the situation of using a pointer to assist in the method of hitting a virtual ball. [Figure 12] Shows an example in accordance with this principle in the situation of using a pointer to defeat the boss of a computer game. [Figure 13] Shows an example in accordance with this principle where a group of viewers provides different reactions surfaced to the streamer in different game contexts. [Figure 14]This example demonstrates how different groups of viewers can provide a streamer with different reactions that surface in different game contexts, thus illustrating the principle's consistency. [Figure 15] This example demonstrates how different groups of viewers can provide a streamer with different reactions that surface in different game contexts, thus illustrating the principle's consistency. [Figure 16] This example demonstrates how different groups of viewers can provide a streamer with different reactions that surface in different game contexts, thus illustrating the principle's consistency. [Figure 17] This section presents different examples of viewer-controlled, adaptive cursors presented to the streamer that align with this principle. [Figure 18] This section presents different examples of viewer-controlled, adaptive cursors presented to the streamer that align with this principle. [Figure 19] This section presents different examples of viewer-controlled, adaptive cursors presented to the streamer that align with this principle. [Figure 20] This section presents different examples of viewer-controlled, adaptive cursors presented to the streamer that align with this principle. [Figure 21] This example shows a GUI that includes a palette of different selectable objects, which may be used by viewers who agree with this principle. [Figure 22] This section presents different examples of viewer-controlled, adaptive cursors presented to the streamer that align with this principle. [Figure 23] Here is an example of an emotion-based response questionnaire that may be presented to an audience consistent with this principle. [Figure 24] Figure 23 shows an example of a GUI for the survey results that is consistent with this principle. [Figure 25-1] This diagram illustrates exemplary logic in flowchart form that can be executed by a processor assembly consistent with this principle. [Figure 25-2] This diagram illustrates exemplary logic in flowchart form that can be executed by a processor assembly consistent with this principle. [Figure 26]This document provides an example of a configuration GUI that may be presented on a display to configure one or more settings for a system / processor assembly that operates in accordance with this principle. [Modes for carrying out the invention]

[0023] This disclosure generally relates to a computer ecosystem including, but not limited to, a home electrical (CE) device network such as a computer game network. The system may include server and client components that can be connected via a network so that data can be exchanged between client and server components. Client components may include one or more computing devices, including Sony PlayStation® game consoles, game consoles from Microsoft, Nintendo, or other manufacturers, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart TVs, internet-enabled TVs), portable computers such as laptops and tablet computers, other mobile devices including smartphones, and additional embodiments described below. These client devices may operate in a variety of operating environments. For example, some client computers may employ, as an example, the Linux® operating system, an operating system from Microsoft, the Unix® operating system, or an operating system from Apple or Google. These operating environments may be used to run one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by the Internet servers described below. Furthermore, the operating environment relating to this principle can be used to run one or more computer game programs.

[0024] A server and / or gateway may include one or more processors that execute instructions to configure the server to receive and send data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or a virtual private network. The server or controller may be instantiated by a game console such as Sony PlayStation (trademark), a personal computer, etc.

[0025] Information can be exchanged between a client and a server over a network. For this purpose and for security, the server and / or client may include firewalls, load balancers, temporary storage and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device that implements a method for providing a secure community, such as an online social website, to network members.

[0026] A processor can be a single-chip or multi-chip processor capable of executing logic through various wirings such as address lines, data lines, and control lines, as well as registers and shift registers. A processor assembly may contain one or more processors that operate independently or in conjunction with each other to execute algorithms, whether they reside in one device or multiple devices.

[0027] Components included in one embodiment can be used in other embodiments in appropriate combinations. For example, any of the various components described herein and / or shown in the drawings may be combined with, replaced by, or excluded from other embodiments.

[0028] "A system having at least one of A, B, and C" (similarly, "A system having at least one of A, B, or C", "A system having at least one of A, B, and C") includes systems having only A, only B, only C, A and B, A and C, B and C, and / or A, B and C, etc.

[0029] This principle can employ machine learning models, including deep learning models. These machine learning models utilize a variety of algorithms trained in ways that include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms, which can be implemented by computer circuits, include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs) suitable for learning information from a series of images, and types of RNNs known as long short-term memory (LSTM) networks. Support vector machines (SVMs) and Bayesian networks can also be considered examples of machine learning models.

[0030] As can be understood here, performing machine learning involves accessing training data, then training a model on that training data, and enabling that model to process further data and make predictions. A neural network may include an input layer, an output layer, and several hidden layers in between that are configured and weighted to infer appropriate outputs.

[0031] Specifically, referring to Figure 1, an exemplary system 10 is shown, which may include one or more exemplary devices described above and further described below in accordance with the present principle. A first exemplary device included in system 10 is a consumer electrical (CE) device such as an audio-video device (AVD) 12, which includes, but is not limited to, an internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). Alternatively, the AVD 12 may be a computerized internet-enabled ("smart") phone, a tablet computer, a notebook computer, an HMD, a computerized wearable device, a computerized internet-enabled music player, a computerized internet-enabled headphones, or a computerized internet-enabled implantable device such as an implantable skin device. In any case, it should be understood that the AVD 12 is configured to implement the present principle (e.g., to communicate with other CE devices to implement the present principle, to execute the logic described herein, and to perform other functions and / or operations described herein).

[0032] Accordingly, to realize this principle, the AVD12 can be composed of some or all of the components shown in Figure 1. For example, the AVD12 may be realized by a high-definition or ultra-high-definition "4K" or higher flat screen and may include one or more displays 14 that are touch-enabled for receiving user input signals via touch on the display. In accordance with this principle, the AVD12 may include one or more speakers 16 for outputting sound and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to the AVD12 in order to control the AVD12. Also, an exemplary AVD12 may include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, WAN, or LAN, under the control of one or more processors 24. Thus, the interface 20 may be a Wi-Fi transceiver, which is an example of a wireless computer network interface, but is not limited to a mesh network transceiver. It should be understood that the processor 24 controls the AVD12 to realize this principle, including controlling the display 14 to display images on it and receiving input from it, as well as other elements of the AVD12 described herein. Furthermore, the network interface 20 may be a wired or wireless modem or router, or other suitable interface such as a wireless telephone transceiver or the aforementioned Wi-Fi transceiver.

[0033] In addition to the above, the AVD12 may also include one or more input and / or output ports 26, such as a High Definition Multimedia Interface (HDMI®) port or USB port for physically connecting to another CE device, and / or a headphone port for connecting headphones to the AVD12 to present audio from the AVD12 to the user through headphones. For example, an input port 26 may be connected via wired or wireless cable or to a satellite source 26a of audio-video content. Thus, the source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, the source 26a may be a game console or disc player containing content. When implemented as a game console, the source 26a may include some or all of the components described below in relation to the CE device 48.

[0034] AVD12 may further include one or more computer memories 28, such as disk-based or solid-state storage, which are not transient signals, and may be embodied as a standalone device within the AVD chassis, or as a personal video recording device (PVR) or video disc player for playing AV programs, or as removable memory media or a server as described below, either inside or outside the AVD chassis. In some embodiments, AVD12 may also include a location or place receiver, such as a cell phone receiver, GPS receiver and / or altimeter 30, etc., which is configured to receive geographic location information from a satellite or cell phone base station and provide that information to a processor 24, and / or to work in conjunction with the processor 24 to determine the altitude at which AVD12 is located. Component 30 may be implemented by an inertial measurement unit (IMU), which typically includes a combination of an accelerometer, gyroscope and magnetometer, or by an event-based sensor, to determine the location and orientation of AVD12 in three dimensions.

[0035] Continuing the description of AVD12, in some embodiments, AVD12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, event-based sensors, and / or cameras integrated into AVD12 and controllable by a processor 24 to collect photographs / images and / or videos, in accordance with the present principle. AVD12 may also include Bluetooth® transceivers 34 and other NFC elements 36 for communicating with other devices using Bluetooth® and / or Near Field Communication (NFC) technology, respectively. An example of an exemplary NFC element may be a radio frequency identification (RFID) element.

[0036] Furthermore, the AVD12 may include one or more auxiliary sensors 38 for providing input to the processor 24 (e.g., motion sensors such as accelerometers, gyroscopes, cyclometers or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)). The AVD12 may include an OTA television broadcast port 40 for receiving over-the-air (OTA) television broadcasts for providing input to the processor 24. In addition to the above, it should be noted that the AVD12 may further include an infrared (IR) transmitter and / or IR receiver and / or IR transceiver 42, such as an IR Data Association (IRDA) device. A battery (not shown) may be provided to power the AVD12 and may also be a kinetic energy collector capable of converting kinetic energy into power to charge the battery and / or power the AVD12. A graphics processing unit (GPU) 44 and a field-programmable gate array 46 may also be included. One or more tactile generators 47 may be provided to generate tactile signals that can be perceived by a person holding or touching the device.

[0037] Continuing to refer to Figure 1, in addition to the AVD12, the system 10 may include one or more other types of CE devices. In one embodiment, the first CE device 48 may be a computer game console that can be used to transmit audio and video of a computer game to the AVD12 via commands sent directly to the AVD12 and / or through a server described later, while the second CE device 50 may include components similar to the first CE device 48. In the shown embodiment, the second CE device 50 may be configured as a computer game controller operated by a player, or a head-mounted display (HMD) worn by a player. Although only two CE devices are shown in the shown embodiment, it should be understood that fewer or more devices may be used. Here, the devices may implement some or all of the components shown for the AVD12. Any of the components shown in the following figures may incorporate some or all of the components shown for the AVD12.

[0038] Referring here to the aforementioned at least one server 52, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56 such as disk-based storage or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with other devices in Figure 1 via the network 22 and can actually facilitate communication between the server and client devices in accordance with the present principle. The network interface 58 may be, for example, another suitable interface such as a wired or wireless modem or router, a Wi-Fi transceiver, or a wireless telephone transceiver.

[0039] In some embodiments, the server 52 may be an internet server or an entire server "farm," and may include and perform "cloud" functionality, such that, for example, in an exemplary embodiment for a network game application, devices of system 10 can access a "cloud" environment via the server 52. Alternatively, the server 52 may be implemented by one or more game consoles or other computers in the same room or near the other devices shown in Figure 1.

[0040] The components shown in the following diagram may include some or all of the components shown in Figure 1. The user interfaces (UIs) described herein can be integrated and extended, and UI elements can be mixed and matched across UIs.

[0041] With the above in mind, this principle recognizes that computer / video game screen sharing allows gamers and viewers to share any moment or content of the game with each other. The following aspects often involve a one-to-many flow of communication and can be combined with game streaming services where viewers do not necessarily have to be friends of the gamer and may be strangers. These aspects, along with the other aspects described below, can help improve the social gameplay experience and, in particular, the technical capabilities of computer-based game systems.

[0042] As a result, according to the following disclosure, screen sharing functionality can easily and directly share real-time gaming moments with friends and others, fulfilling gamers' desire to enjoy those moments together. Thus, gamers can easily stream gameplay in real time to friends and small groups while chatting and playing together in parties and other scenarios, creating a sense of unity, especially despite the distributed and sometimes isolated nature of computer-based games. This principle can be employed on video game consoles, mobile applications that can run on smartphones and other mobile devices, and other types of devices including laptops and desktop computers.

[0043] With the above in mind, refer to Figure 2. This figure shows a PlayStation 5 gamer 200 playing a computer game, with its visual content presented on a local television display 210 at 1080p resolution and a frame rate of 30 frames per second (fps). The A / V content stream is routed through one or more transcode / selective transfer unit (SFU) servers 240 and received by a local game console connected to a local television 250 at the home of a viewer 220. The gamer / streamer 200 also streams the gameplay to a remote viewer 220 who is watching the audio and video (A / V) content 230 of the gameplay on another television 250. This gameplay is also streamed to a mobile application viewer 260, and the A / V content 230 is viewed on a smartphone when the A / V content stream is routed through the server 240 and received by a dedicated mobile application from the game console manufacturer running on the smartphone. Thus, Figure 2 illustrates different types of devices and systems that may be used to view the A / V content stream 230 of the gameplay.

[0044] Figures 3A to 3C further illustrate this principle. Figure 3A shows that when a gamer 200 streams gameplay via a server 240 to one or more friends 220, 260, those two viewers can watch together and enjoy the game with gamer 200 and with each other.

[0045] Figure 3B illustrates that Gamer / Streamer 200 can stream gameplay to Friend 300 via Server 240, allowing Friend 300 to assist or mentor Gamer 200, and Gamer 200 can also assist or mentor Viewer 300 if Friend 300 is playing the same game instance. Therefore, it should be understood that the voice of Gamer 200, detected by a local microphone near Gamer 200, can be streamed to Viewer 300, and vice versa. This can be achieved through a separate dedicated out-of-band audio channel or as part of the two-way game stream itself. Also, according to Figure 3B, Person 310 shown in Figure 3B is a visual representation of how Gamer 200 might provide coaching that could occur (even if Gamer 200 and Viewer 300 are located in different geographical locations).

[0046] Moving on to Figure 3C, this figure shows that streaming of gamer / streamer 200's gameplay to friend 350 can be used so that gamer 200 and friend 350 can see each other's point of view (POV) (not the player's own point of view relative to the game character, but the other player's point of view) when they are playing a multiplayer game.

[0047] Moving on to Figure 4, we see that the screen sharing experience can be further enhanced by unique interactive features to facilitate viewer engagement and the general use of screen sharing. These features include pointer / cursor control and reaction features. Figure 4 shows an example of the former, where a pointer / cursor 400 is displayed on the gamer / streamer's own display 410 while playing a particular video game, and the cursor 400 is controlled by a remote viewer named Alex. As Alex controls the cursor 400, a text display 420 is overlaid on the game video as part of the streamer user interface (UI), and the display 420 shows Alex's name and indicates that Alex is controlling the cursor 400. As better shown in the enlarged close-up view of the cursor 400 shown in Figure 5, the cursor 400 also has or includes an image 500 associated with Alex (not another active gamer / viewer). The image 500 could be a graphic or avatar associated with Alex, or Alex's own profile picture.

[0048] Returning to Figure 4, a circle 430 surrounding the cursor 400 is also shown. This can be useful for highlighting in-game graphic objects that you want to draw the gamer's attention to as areas that a remote viewer might be able to move around and explore with the gamer's virtual character. Thus, the viewer's own device or the intermediary server itself can track the position of the cursor 400 as it moves across the gamer's own POV when presented on the display 410. The device / server can then highlight (e.g., surround) the game graphic object that the cursor 400 is determined to be located on by presenting a larger circular graphic overlay 430. In this example, the overlay 430 highlights a building that the viewer indicates the gamer should explore with their game character.

[0049] Furthermore, while positioning / hovering a cursor over a game object can be one form of selecting a game object, it should be noted that other forms of selection consistent with this principle include placing a mouse-controlled cursor over the game object and then inputting left-click or right-click commands on the game object with the mouse, or selecting the game object via touch if the viewer is using a touch-enabled display. Additionally, it should be noted that, as shown in the specific example in Figure 4, highlighting a game object may include presenting the overlay 430 but not presenting a digital trace of the cursor control command itself (for example, if the cursor control command involves moving the cursor 400 over the game object while a left-click selection input is continuously provided and drawn on the streamer's screen in response to the cursor's movement, this aspect is also assumed in other scenarios consistent with this principle).

[0050] Figure 4 also shows that the game video presented on display 410 may also have an additional display 440 that is overlaid on top of it as part of the UI. Display 440 indicates that screen sharing with the viewer is occurring and the screen sharing overlay UI is active (in this example, elements 400, 420, and 430 are displayed in the UI as described above).

[0051] Figure 6 is a continuation of this example, where the viewer Alex's reaction is shown with a smiling emoji 600 with heart-shaped eyes. This is provided by Alex to show that he likes what the gamer is seeing while the gamer is playing a computer game. Alex's avatar and profile picture 610 are also displayed so that the gamer can identify who provided the emoji 600. Elements 600, 610 may be presented using the same UI described in the paragraph above, which is overlaid on the gamer / streamer's POV when the game is played on display 410. In addition, other cursor and reaction elements described below may also be displayed using this UI. Thus, according to Figure 6, it is understood that in another aspect of this principle, a viewer can send a reaction to the shared screen / AV feed UI to show support and congratulations to the streaming gamer.

[0052] Figure 7 further illustrates this principle in a competitive tactics scenario for a sports game (in this example, the sport is soccer). It is hypothesized that two friends are playing a soccer computer game, and a third-party viewer is assisting one of the streaming gamers. The viewer wants to indicate where the streamer should pass the ball, and can do so by using a pointer / cursor 700 to draw on the screen / streamer POV 710, indicating where the streamer should pass the ball on the virtual soccer field. For example, the viewer can position the cursor 700 at a specific POV location and then continuously provide selection commands or left-click inputs (e.g., a mouse controlling the cursor 700) to draw virtual ink freehand according to the subsequent movement of the cursor 700 while selection commands continue to be provided. After the game ends, the viewer can again use the pointer 700 to analyze the specific game situation in detail for the streamer, allowing the streamer to better prepare for the next game.

[0053] Figure 8 illustrates another example of competitive tactics, where the shooter's tactics may be swapped. For example, imagine two friends playing a first-person shooter computer game, but the viewer has already been eliminated and is simply watching until the game ends. Also imagine the viewer wants to show the streamer (the other friend still playing the game) where their virtual character is located in the virtual world and which direction they should be facing to ensure victory. The viewer can achieve this by controlling a cursor 800 that is displayed on the streamer's own screen 810 (and mirrored on the viewer's screen). It is also assumed that the viewer then spots a virtual enemy coming from a particular direction and quickly uses the pointer / cursor 800 to indicate the direction from which the attack is coming.

[0054] Moving on to Figure 9, yet another example is shown, this time concerning the exploration of a virtual world. In this example, a virtual map 900 of the game world is presented on the display 910, which is the streamer's device. It is then hypothesized that the streamer is playing a particular game and sharing the screen with viewers, allowing them to explore the world together. The streamer is exploring a large area, and when the viewers notice an area that has not been explored, they use a pointer / cursor 920 to mark the exact location of that area on the map 900. Viewers can do this by simply hovering the cursor 920 over that area, or by providing a selection input (e.g., a left mouse click) when the cursor 920 is over that area, and in response, instructing the game to permanently display an "X" mark on the map 900 as "X indicates a spot".

[0055] Referring to Figure 10, we see an example of exploring another virtual world, but this time it concerns looting. It is hypothesized that the streamer is playing a particular game while the viewers are watching, primarily following the story. The viewers then notice that the streamer has not checked the potential loot pool. Therefore, the viewers become actively involved in the game, as they can use a pointer / cursor 1000 to specifically indicate to the streamer where on the screen (below or to the side of the virtual dresser 1010) they should look.

[0056] Here, we have described two examples of using pointers / cursors for highlighting. Referring to Figure 11, it is hypothesized that two friends are playing a particular game, and one of the friends (in this example, the viewer) is helping the other friend (in this example, the streamer / gamer). The viewer may notice that when the streamer takes a shot (for example, when kicking or hitting the ball towards the goal), he always misses the ball. Therefore, the viewer can use pointer / cursor 1100 to mark which part of the ball 1110 the streamer should aim at in order to make contact with the ball 1110 and hit it more accurately. Thus, the viewer indicates the "sweet spot" where the ball 1110 should be kicked.

[0057] Another example of a challenge scenario is shown in Figure 12. It is hypothesized that a streamer is playing a particular game but is having trouble fighting a tough boss or enemy. The streamer then asks a friend who has already cleared / defeated boss 1210 to help by watching the streamer's screen. The viewer friend consults with the streamer about weaknesses in the streamer's attack plan and uses a pointer / cursor 1200 to highlight the exact location / method to attack boss 1210.

[0058] Moving on to examples involving reactions rather than the pointer / cursor actions themselves, refer to Figure 13. This figure illustrates a situation where many viewers are gathered to watch a streamer's gameplay from a remote location (and in some cases, they are playing the same game instance). Due to the large number of people, the streamer may find it difficult to keep track of who is saying what in the audio chat where everyone is talking within the game stream. Instead of trying to pay attention to that, the viewers can provide individual emoji reactions with animated emotions to show excitement and appreciation for the streamer's great moves in the game without causing confusion for the streamer. The streamer may feel even better about those great moves because the reactions on screen amplify the gaming experience. Here, the reactions are mostly, if not all, positive.

[0059] Figure 14 illustrates another scenario where certain viewers are muted from the voice chat or simply unable to speak for some reason. Here again, a group of friends may be playing the game together while others are simply lurking and watching the gamers' collective gameplay. Even if some viewers don't have access to their microphones or are muted, they can still enjoy themselves by throwing some kind of reaction 1400 at the screen 1410.

[0060] Next, Figures 15 and 16 illustrate a small party reaction scenario. In the small party scenario of Figure 15, it is hypothesized that two friends are playing a particular game, and one of the players (the streamer in this example) is struggling to defeat a difficult boss and keeps dying repeatedly. The viewer can then decide to always react with emote 1500 as encouragement each time to motivate the streamer to continue. Emote 1500 can be a heart, as shown, and can be permanently displayed on the streamer's screen 1510, even if provided by the viewer at different times.

[0061] Figure 16 shows a trolling scenario in which two friends are playing a game together, with the streamer (one of the players) attempting to build / complete a structure as a challenge. The viewers (the other players) may find it amusing when the streamer fails to do so and may tease the streamer by adding funny reactions each time the streamer gets close. In this example, this reaction may include a dung / poop emoji 1600 that may persist on the streamer's screen 1610 until the streamer successfully completes the task, and the emoji 1600 may be removed from display in response to completion.

[0062] Referring to Figures 17-20, it should be understood that adaptive pointers / cursors can be presented in accordance with the above description based on the game context (e.g., a cursor whose appearance and behavior / animation match the ongoing game context). Therefore, a game engine may use artificial neural networks (ANNs), such as feedforward neural networks or deep neural networks, or other types of context-aware or pattern-recognition neural networks, to identify the current game context and adjust the cursor's appearance in response. An ANN can be trained unsupervised on one or more sets of training data, each containing context (and / or raw game state data, which may include metadata, video data, and audio data that can determine the context). The training dataset may also include the appearance of each ground truth pointer to present for each context / game data.

[0063] Next, while the ANN is deployed when a particular game instance is running, the running device may receive game state data and / or the current game context from the game engine itself, or, if it cannot access the game engine itself (e.g., an older legacy game), it may simply receive A / V data from the game itself. The device may then determine the context from the game state data / AV data if it is not provided directly from the game engine, and either way, provide the received / determined context and / or raw game state data to the ANN as input, generating an inference output in response. That output may be a cursor image to display or a reference identifier for the type of cursor to retrieve and display. The device may then present the inferred cursor image corresponding to that output.

[0064] Therefore, it should be noted that in some examples, all different cursors themselves may be included in a system-accessible reference library of cursors. Each cursor may have its own reference ID. Since some or all of the cursors in the library may be used as the training ground truth as described above, the cursors do not need to be dynamically generated during gameplay, and the inference output from the ANN may indicate a specific cursor from the library to be used. However, in other examples, the system may use a generated image output by the ANN as the cursor image, since a cursor image may actually be shown in the output even if it is not associated with an existing reference cursor.

[0065] With the above in mind, refer to Figure 17. This figure shows a triangular cursor 1700 with an exclamation mark inside, which may be presented during a battle sequence in a particular video game. Furthermore, the cursor 1700 may turn red, pulsate, and emphasize the tension. Next, according to Figure 18, when the game situation changes from battle to victory celebration, the cursor 1700 may transform into the cursor 1800 in Figure 18. As shown in Figure 18, the cursor 1800 may be a green circle with a green checkmark inside to indicate victory, and the cursor 1800 may even be animated so that virtual fireworks erupt from it.

[0066] Referring to Figure 19, in a virtual world exploration context, cursor 1800 may transform into a magnifying glass cursor 1900, symbolizing and encouraging the viewer / streamer to examine the details of the virtual world. Next, Figure 20 shows another example where cursor 2000 may be presented as a house burning down in flames, based on the streamer's character's health falling below a threshold amount (e.g., 20 percent) and / or based on the streamer's character losing a boss fight and dying.

[0067] Referring now to Figure 21, this figure shows a virtual palette 2100 that may be presented on the viewer's screen while watching a streamer / gamer's gameplay. Thus, the palette 2100 may be presented as part of the UI described above, for example, referring to Figure 6, but it is displayed only on the viewer's display and not on the streamer's display. Therefore, it may be presented semi-transparently as an overlay of the game video stream itself, as an opaque inset of the game video, or opaquely on one side of the video, yet still appear on the same display / UI.

[0068] As shown in Figure 21, palette 2100 may include a first section 2110 that presents a variety of emojis, emotes, and / or other viewer-selectable graphic reactions. Palette 2100 may also include a second section 2120 that presents a variety of viewer-selectable sound effects, which, when selected, may instruct the game engine to insert the corresponding sounds into the game's own audio so that the streamer / gamer can hear the sounds while playing the game and enjoy verbal and / or non-verbal audio feedback from the viewer. As shown in section 2120, verbal feedback may include the exclamation "ouch" (top selector), while non-verbal feedback may include laughter (middle selector) or a ba-dum-tsh sound (bottom selector) that frequently follows jokes in the entertainment industry during broadcasts.

[0069] Section 2130 of Figure 21 similarly shows various stickers and persistent pointer objects that can be selected by the viewer. Note that the viewer can send various stickers, emojis, and / or predefined objects from Section 2130. Thus, the viewer can drag selected elements onto the game screen, and those elements can be pasted as persistent overlays in areas designated by the viewer within the streamer's POV, and will not be removed except by user commands or at the end of the game or game level. Stickers can also be distinguished for each viewer if desired, so that the streamer knows which sticker is associated with which viewer. Thus, these elements may remain in place for a set period of time or until removed. Viewers can creatively resize, rotate, and position elements on the screen, so they can express their personality and enhance the visual experience. As an example, Section 2130 shows that a viewer's avatar image may constitute one of these elements (top selector) and a "#1 Viewer" sticker may constitute another of these elements (bottom selector).

[0070] Therefore, more generally, it should be understood that this system may receive a cursor control command instructing it to place a graphic object (sticker in Section 2130) uniquely associated with the viewer in a permanent location within the streamer's field of view / POV, which is presented on the streamer's own display. Thus, based on this cursor control command, the system may present the graphic object (sticker) in a permanent location.

[0071] Furthermore, Figure 21 shows that the palette 2100 may include a section 2140 that presents different viewer-selectable cursors, which may be presented to the streamer in accordance with the above description, as they may be used by the viewers. Note that examples are shown, including a circular pointer (top), an arrow pointer (middle), and an option to instruct the game system to use an adaptive pointer in accordance with the above description (bottom).

[0072] Moving on to Figure 22, dynamic terrain analysis can also be used by the game system to present cursors with different appearances, which can provide a sense of the terrain and / or real-time terrain analysis. For example, as a streamer explores different areas of the game world, the cursor can adapt to provide information about the game environment / terrain that the streamer's game character is currently moving through. For example, the cursor (such as cursor 1700) may change color or display icons to indicate danger zones, hidden treasures, or safe paths. Alternatively, as shown particularly in Figure 22, the cursor can provide information about enemy statistics for enemy game characters as non-players in the game. In this example, cursor 2200 shows the remaining percentage of the boss's health (in relation to the boss being fought by the streamer). Other statistical examples may include points earned, inventory sufficiency, or inventory items. Thus, this dynamic guidance can enhance the viewer's understanding of the game world through adaptive cursors and support both the streamer's navigation and decision-making. The statistics cursor may rotate between different statistics for the boss and / or the streamer's own character at predefined time intervals (e.g., every 5 seconds).

[0073] Referring to Figures 23 and 24, this principle also includes enabling both streamer-generated and viewer-generated polls. Therefore, both streamers and viewers can create polls and craft the poll questions and prompts. This can be done by providing voice input to a Large-Scale Language Model (LLM) (e.g., a ChatGPT-based Application Programming Interface (API)) or a digital assistant running on a game engine or entire system, instructing the system to generate polls according to one or more prompts provided by the person via voice. The LLM or assistant can then generate poll questions and freely select poll responses based on different nonverbal responses that others may choose. For example, since each emoji or other response in a reference library contains metadata indicating the sentiment associated with that emoji, the LLM / assistant can select emojis associated with sentiments correlated to the poll questions as potential responses. Viewers and other pollers can then select their responses using the given emojis or other responses.

[0074] This is illustrated in the example in Figure 23, where Cody is another person playing against the streamer, and the streamer provides an audio input asking LLM, "Do you think we should punk Cody?" A poll 2300 is then dynamically generated and presented to viewers of the game's live stream, with different emojis 2310 being presented and selectable by viewers as their poll responses. As illustrated, this may include smiling and laughing emojis, which may be selectable to provide an affirmative response to the poll question. As illustrated, a frowning emoji may be included and may be selectable to provide a negative response to the poll question.

[0075] Once the survey ends after a threshold period, a period specified by the streamer, or other trigger, an overlay 2400, as shown in Figure 24, may appear on both the streamer's and the viewer's screens. The overlay 2400 may aggregate and display the survey responses in a visually appealing format, showing the percentage 2410 for each selected voting option. The streamer can recognize the viewer's preferences by selecting a selector 2420 (which may result in a digital assistant / system providing an audio output such as a computer-generated voice shouting "It's time for punk!"), and use this information to customize content or make decisions within the game.

[0076] Referring now to Figure 25, an exemplary logic that may be executed by a system 10 and / or its respective components (e.g., a console or a remote server) that conforms to this principle is shown. Note that while the logic in Figure 25 is shown in flowchart form, other appropriate logic may also be used.

[0077] Starting from block 2500, the system may run a computer game in which a first person plays the computer game. The logic then proceeds to block 2505, where the system may stream the first person's gameplay to other viewers, for example, via the internet, a third-party streaming website, or a dedicated console manufacturer's network. The logic then proceeds to block 2510, where the system may receive input from viewers, including at least a second person who is watching the live stream of the computer game but is not controlling the game character. The input may include cursor control commands to move one of the aforementioned cursors within the first person's viewpoint / POV, and also graphic-based emotional responses (such as emojis) as described above.

[0078] Next, the logic in Figure 25 proceeds to block 2515, where the system may present one or more corresponding outputs on the display associated with the first character based on the input(s) received in block 2510. The outputs(s) may represent inputs in a game-specific context. As an example of inputs represented in a game-specific context, in block 2520, the system may change the appearance of the cursor based on the game context and / or a specific game event, as described above.

[0079] Next, the logic may proceed from block 2520 to decision diamond 2525. At diamond 2525, the system may determine that a cutscene has occurred (e.g., using instructions from the game engine) after executing at least one first cursor control command, possibly during the first part of the computer game. In various examples, a cutscene may be a non-interactive scene in the game where no user input is processed, and the cutscene begins when the streamer reaches a certain point in the game (e.g., the end of a level or the death of the streamer's virtual character). Thus, a cutscene may interrupt gameplay by a sequence of conversation between two game characters, a series of events related to the game's plotline, or an awards ceremony in which the streamer is awarded one or more rewards.

[0080] Based on / in response to the determination that no cutscene is occurring, the logic may proceed directly to block 2535, as will be explained later. However, based on / in response to the determination that a cutscene is occurring, the logic instead proceeds to block 2530 first, and the system may refuse to process additional cursor control commands and / or reactions during the cutscene playout. This allows the multi-viewer UI, as shown in Figure 6, to be controlled in-game based on context (e.g., on and off) and to automatically detect in-game cutscenes, scene changes, and / or important moments (e.g., bosses) where viewer input would be too intrusive. When such a moment occurs, the pointer and reaction overlays are temporarily turned off to ensure that both the streamer and viewers can fully appreciate the narrative without distractions. When gameplay resumes after the end of a cutscene, scene change, or boss fight, the interaction can be re-enabled and reproduced.

[0081] The logic can then proceed from block 2530 to block 2535. In block 2535, the system may, in some implementations, alternate the authority to issue cursor control commands (and even reactions) among viewers of a computer game. This avoids a situation where multiple cursors are floating around on the streamer's screen simultaneously, potentially distracting the streamer and excessively hindering the viewing of the game video. Thus, in certain implementations, only one cursor may be displayed at a given time during gameplay, in which case the viewer permitted to control the cursor may be determined based on past viewer input related to the gameplay of the first person in the computer game.

[0082] Therefore, this feature can be thought of as a virtual baton being passed around, with cursor control operating on an alternating basis. The UI / overlay may track viewer engagement, for example, in terms of the amount of response. As viewers accumulate points through more interactions (cursor control and responses), they move up the queue, and the viewer at the front / top of the queue gains control of the pointer. Control may then remain with that viewer until one or more conditions are met, such as the viewer providing a selection command with the cursor, a threshold time running out (e.g., 30 seconds), or the viewer relinquishing control.

[0083] In addition, or alternatively, streamers can determine which user can use the pointer at a given time by providing voice commands that specify a particular user, or simply by voice-instructing the system to "pass over." This can facilitate the passing of control to the next viewer in the queue. Thus, this feature can encourage continuous, game-like engagement from the viewers themselves.

[0084] The logic may then proceed from block 2535 to block 2540. In block 2540, the system may continue to control outputs, game actions, and / or game states based on the viewer's cursor commands and responses. The logic may then proceed to block 2545, where the system may, in some implementation forms, run an eye-tracking algorithm that tracks the eyes of one or more viewers using images of the viewers provided by cameras for capturing the viewers' eyes. The cameras may be located, for example, on a headset worn by each viewer or on a television that each viewer is using to watch the game.

[0085] Eye movements from the viewer can constitute inputs that can be used in block 2550 to move a cursor on the streamer's display according to the eye movement input. For example, if the viewer is looking left, the cursor moves proportionally to the left, and if the viewer is looking right, the cursor moves proportionally to the right. Also in block 2550, in some examples, the system may determine that the tracked viewer is closing one eye, such as by winking or blinking (but not both eyes at the same time). Based on the determination that the viewer is closing one eye, the system may change the appearance of the cursor presented on the streamer's display.

[0086] As a concrete example, if a viewer is wearing a VR headset for watching virtual reality (VR) gameplay, the eye-tracking sensor on the headset can be used to establish the movement of a VR eye-tracking pointer, and when the viewer blinks / winks with one eye, a (pulsating) pulse can be sent to the pointer which is then presented on the streamer's display. This could be a simple and intuitive way to draw the streamer's attention to a game object that the cursor is moving and hovering over.

[0087] Next, the logic may proceed from block 2550 to decision diamond 2555. At diamond 2555, the system may determine whether a predetermined graphic-based reaction threshold has been reached, based on the reception of each graphic-based reaction from each person watching the livestream but not controlling a game character (which may occur in block 2540). This may include all or the same type of a predetermined number of reactions provided by viewers within a set time, such as 20 seconds (this implicitly indicates that all reactions are related to the same game event while eliminating false positives from outside the time frame). Or, more generally, this may include all or the same type of reactions provided by viewers during a particular game scene or game level, where the same type may relate to a specific emoji, each considered a distinct type, or to emojis that have been grouped into different types based on an overall emotion (for example, positive emotion emojis such as happy face or laughing emojis may be grouped as one type, while negative emotion emojis such as frowning or angry face emojis may be grouped as another type).

[0088] A negative judgment may cause the logic to proceed directly to block 2565, as briefly described below. However, a positive judgment in decision diamond 2555 may instead cause the logic to proceed to block 2560 in response to the fulfillment of a predetermined graphics-based response threshold. In block 2560, the system may change the playout of the computer game based on overall or majority / multiple emotions that establish at least a threshold number. For example, if positive emotional responses accumulate as an overall, the game playout may be changed by presenting fireworks in the background of the game scene. For example, if negative emotional responses accumulate as an overall, the game playout may be changed by presenting a red flash in the background of the game scene.

[0089] Next, the logic may proceed from block 2560 to block 2565. In this step, the system may present haptic feedback to the first person (streamer) based on one or more graphics-based responses. The haptic feedback may be presented using the vibrators of a game controller used by the first person, the vibrators of a VR headset or other headset type worn by the first person, the vibrators of the first person's smartphone or other connected device, etc. The haptic feedback may vary based on the response type associated with the graphics-based response that caused the vibration.

[0090] Therefore, for example, when a viewer sends a reaction, the streamer may receive subtle haptic feedback corresponding to the type and intensity of the reaction. Thus, if a smiling emoji is provided, a separate pulse of vibration with relatively low intensity / amplitude may be provided, but if a laughing emoji is provided (designated as a stronger positive reaction), a separate pulse of vibration with higher intensity / amplitude may be provided. Conversely, if a frowning emoji is provided, a single short vibration with relatively low intensity / amplitude may be provided, but if an angry face emoji is provided (designated as a stronger negative reaction), a single short vibration with higher intensity / amplitude may be provided.

[0091] Next, the logic may proceed from block 2565 to block 2570. In block 2570, if necessary, the system may aggregate emotion-based responses provided by multiple viewers. This may also be done in block 2570 to present a response summary to the first person (streamer) that summarizes the amount of responses received, categorized by response type and displayed visually.

[0092] Next, the logic may proceed from block 2570 to block 2575. In this step, the system may perform one or more of the following: Firstly, the system may award the first person (streamer) at least one virtual trophy based on the amount of emotion-based responses received while the first person is playing one or more specific aspects of a computer game (e.g., a battle sequence, a game level, a game scene). Secondly, the system may reward at least one viewer of the livestream based on inputs provided by that viewer, established by one or both of cursor control commands and emotion-based responses. As the input increases, the reward may also increase proportionally.

[0093] As a result, with regard to trophies and challenges, new or unique types of trophies may be offered to streamers based on the amount of viewer response received, with trophies gradually becoming larger or more important as more viewer input is received. Thus, streamers may earn predefined trophies associated with viewer input, and trophies may be earned based on the sequence, timing, and / or amount of response in specific challenges, game sequences, game levels, etc. This concept can also be applied to games with daily / weekly in-game challenges.

[0094] Similarly, regarding the hierarchy of responses and corresponding rewards for viewers, viewer responses can be divided into tiers such as Bronze, Silver, and Gold levels based on the level of engagement (e.g., the total amount of responses provided). As viewers continuously engage with gameplay through their responses, they progress through the tiers, gradually accumulating engagement points, and unlocking different rewards as their response input increases. A progress bar in a UI overlay visible to viewers (likely invisible to the streamer) may indicate the viewer's progress to the next tier and encourage continued interaction to reach that tier. This can also apply to the streamer (for example, the more responses they receive from viewers, the more rewards and game points they can earn).

[0095] Continuing the detailed explanation with reference to Figure 26, an exemplary graphic UI 2600 that may be presented on the game system's display to configure one or more settings of the system in order to implement this principle is shown. Each exemplary option described below may be selected via touch input to the display if touch-enabled, cursor input (such as mouse or trackpad input), or other input (actually selectable items) to checkboxes related to this example (those disclosed herein may be selected by these methods).

[0096] As shown in Figure 26, GUI 2600 may include a first option 2610, which may be selectable only once to set / configure the system to perform the functions described above for multiple future games / game instances. Selecting option 2610 enables viewer-based interaction with the streamer / gamer via cursor control commands and graphics-based nonverbal responses. Therefore, for example, the logic in Figure 25 may be executed for multiple future game instances based on the pre-selected option 2610.

[0097] GUI2600 may also include option 2620, which can be selected to set up or configure the system so that viewer input can actually control the game itself, rather than simply presenting content to the streamer. For example, selecting option 2620 may instruct the system to then execute an algorithm that presents selectable graphic objects on the viewer's display as part of the game itself. The system then receives cursor control commands to select these selectable graphic objects and, based on these cursor control commands, can control the output of the computer game displayed on the display associated with the streamer. The output itself may have the effect of changing the current game state of the computer game itself.

[0098] As a specific example, viewers may be allowed to interact with game layer objects. For instance, using a Software Development Kit (SDK), markers that viewers can interact with can be displayed on designated objects in the game world. When a viewer clicks on one of these markers, the overlay / UI may send a command to the game that simulates the viewer's input. For example, if a viewer clicks on a lever in the overlay, the game receives a command that triggers the lever's action, changing something within the game itself. This two-way communication can bridge the virtual gap between the viewer, streamer, and game, allowing viewers to actively participate in the gameplay.

[0099] As another example, consider an SDK for game publishers. This could be a "general" SDK that allows game publishers to define specific triggers, events, objects, and responses that occur when viewers interact with the overlay / UI. For example, developers could integrate custom weapon drops or special character animations that are triggered by viewer actions. From a store perspective, viewers could also "click" on skins and other digital add-ons to see purchasable content in the store or add them to their wishlist.

[0100] Furthermore, in some cases, a reactive environment may be employed where pointer movements affect the game environment in real time. Viewers can use the pointer to activate switches, create temporary bridges, or manipulate objects within the game. These interactions can influence the streamer's gameplay and potentially lead to secret passages or "Easter eggs" that reveal hidden items within the game.

[0101] Therefore, more generally, a system may receive cursor control commands and, based on those commands, execute predefined game actions that are only performed by people who do not control the characters during a computer game (video game players / viewers who do not control the game characters).

[0102] Viewers can mark game layer objects. Therefore, for example, viewers can draw or place markers on the streamer's screen to highlight specific game objects, enemies, or tactical points. For example, to draw, a user might left-click and hold the left mouse button while moving the mouse to draw on the UI. By positioning the cursor over a game object and left-clicking up or down without moving the mouse, the cursor can be controlled to place a visual marker in the streamer's viewpoint (e.g., fixed to a specific virtual location in the game). In some cases, this functionality may also be implemented as part of the SDK.

[0103] Furthermore, for viewer-generated tasks, viewers may use UI / overlay drawing tools (some of which were described earlier) to sketch tasks, issues, or suggestions directly on the streamer's screen. The streamer can review the tasks and choose to approve or reject them, or negotiate with viewers about any changes.

[0104] Moving on to reactions, it should be noted that emotion-based visual effects are also included in this principle. Therefore, the system disclosed here adds the ability to monitor the cumulative emotion of viewer reactions by analyzing elements such as the type, frequency, and speed of the reactions. When a certain reaction threshold is met / exceeded, the UI / overlay may trigger a corresponding visual effect for the streamer and / or viewer. For example, a surge in positive reactions might display an animation of fireworks going off one after another, as described above. Conversely, a surge in negative reactions might display a subtle shadow on the screen.

[0105] Avatar-based reactions can also be used. Therefore, depending on the game character or avatar, the appearance of reactions in a particular game may change to match the in-game persona (for example, the avatar used for a reaction could be an image of one of the game's characters). Furthermore, viewers can customize their reactions by choosing from a variety of animations, icons, and even short sound clips.

[0106] Additional visual overlays may also be deployed on a per-type basis. These additional overlays may include theme filters that change the game's visual palette to suit different moods and settings, countdown timers to in-game events, and sound effects or images that are overlaid on specific sections of the screen. Viewers and streamers can enable or disable these elements, providing a dynamic and customizable visual experience.

[0107] Cursor ghosting can also be used according to this principle. Therefore, if a viewer uses the cursor to select and tag a player's character, the cursor can automatically track the character without any further input from the viewer. The cursor itself can create a visual trail behind it (like a comet's tail) indicating where it went.

[0108] Cursor hysteresis is also included. Cursors may disappear or vanish to signal something. For example, they may disappear in flames due to a change in emotion ("I was wrong," meaning the viewer was wrong about a suggestion given to the streamer).

[0109] Furthermore, if a viewer uses the cursor to say "left" (for example, by pointing the cursor to the left or writing an "L" on the screen to mean "left"), the cursor / highlight may remain at the location indicated by the input. The cursor may then remain visible if the player does not take action, and may disappear if the player takes action. This allows the viewer to know that the player has responded to the request.

[0110] Furthermore, an AI-based model may analyze game objects. Therefore, when a viewer points to an object or draws a cursor around it, that object may be highlighted in the game with the pointer's own color. Thus, the system may modify in-game objects based on this functionality.

[0111] While this specification illustrates and describes specific embodiments in detail, it should be understood that the subject matter covered by the present invention is limited only by the claims.

Claims

1. A computer game played by a first person on a first device, The system receives raw game state data, which includes at least one of metadata, video data, or audio data that determines the ongoing game context of the computer game. A second device, which is watching a live stream of the aforementioned computer game but is not controlling the characters of the aforementioned computer game, receives input including one or more of the following: a cursor control command or a graphics-based response that conveys the emotions of a second person associated with the second device. Based on the input, the output is displayed on the display associated with the first device. A processor assembly programmed with instructions for, The input includes the cursor control command, The apparatus includes an output in which the cursor is presented in a manner that can be transformed based on the raw game state data in accordance with changes in the ongoing game context of the computer game.

2. The processor executes a computer game played by a first person on a first device, The processor receives raw game state data, which includes at least one of metadata, video data, or audio data that determines the ongoing game context of the computer game. The processor receives input from a second device that is watching a live stream of the computer game but is not controlling the computer game characters, including one or more of the following: a cursor control command or a graphics-based response that conveys the emotions of a second person associated with the second device. The processor causes the output to be displayed on the display associated with the first device based on the input, Includes, The input includes the cursor control command, A method comprising the output being presented in such a way that the cursor can be transformed based on the raw game state data in accordance with changes in the ongoing game context of the computer game.

3. A computer game played by a first person is executed on a first device, The system receives raw game state data, which includes at least one of metadata, video data, or audio data that determines the ongoing game context of the computer game. A second device, which is watching a live stream of the aforementioned computer game but is not controlling the characters of the aforementioned computer game, receives input including one or more of the following: a cursor control command or a graphics-based response that conveys the emotions of a second person associated with the second device. The system includes at least one computer storage that includes instructions executable by at least one processor for displaying an output on a display associated with the first device based on the input, The input includes the cursor control command, The system includes the output being presented in such a way that the cursor can be transformed based on the raw game state data in accordance with changes in the ongoing game context of the computer game.

4. The apparatus of claim 1, wherein the raw game state data includes metadata, video data, and audio data that determine the ongoing game context of the computer game.

5. The apparatus of claim 1, wherein the appearance of the cursor, which can be modified based on the raw game state data in accordance with changes in the game context during the computer game, includes a change in the color of the cursor.

6. The apparatus of claim 1, wherein the appearance of the cursor, which can be modified based on the raw game state data in accordance with changes in the game context during the computer game, includes changes in the symbolism of the cursor.

7. The apparatus of claim 1, wherein the appearance of the cursor, which can be deformed based on the raw game state data in accordance with changes in the game context of the computer game in progress, includes the appearance of the cursor that conveys information about the terrain on which the character of the computer game is moving.

8. The apparatus of claim 1, wherein the ongoing game context of the computer game is estimated using an artificial neural network trained at least partially on raw game states from previous gameplay of the computer game.

9. The method of claim 2, wherein the raw game state data includes metadata, video data, and audio data that determine the ongoing game context of the computer game.

10. The method of claim 2, wherein the appearance of the cursor, which can be modified based on the raw game state data in accordance with changes in the game context during the computer game, includes a change in the color of the cursor.

11. The method of claim 2, wherein the appearance of the cursor, which can be modified based on the raw game state data in accordance with changes in the game context of the computer game in progress, includes a change in the symbolism of the cursor.

12. The method of claim 2, wherein the appearance of the cursor, which can be deformed based on the raw game state data in accordance with changes in the game context of the computer game in progress, includes the appearance of the cursor that conveys information about the terrain on which the character of the computer game is moving.

13. The method of claim 2, wherein the ongoing game context of the computer game is estimated using an artificial neural network trained at least partially on raw game states from previous gameplay of the computer game.

14. The system of claim 3, wherein the raw game state data includes metadata, video data, and audio data that determine the ongoing game context of the computer game.

15. The system of claim 3, wherein the appearance of the cursor, which can be modified based on the raw game state data in accordance with changes in the game context during the computer game, includes a change in the color of the cursor.

16. The system of claim 3, wherein the appearance of the cursor, which can be transformed based on the raw game state data in accordance with changes in the game context during the computer game, includes changes in the symbolism of the cursor.

17. The system of claim 3, wherein the appearance of the cursor, which can be deformed based on the raw game state data in accordance with changes in the game context of the computer game in progress, includes the appearance of the cursor that conveys information about the terrain on which the character of the computer game is moving.

18. The system of claim 3, wherein the ongoing game context of the computer game is estimated using an artificial neural network trained at least partially on raw game states from previous gameplay of the computer game.

Images