Enhanced screen sharing for online computer applications

Abstract

The invention relates to enhanced screen sharing for online computer applications. In a method and system for viewer interaction with streaming media, a scene from a media presentation is displayed with a broadcast device including operating system level broadcaster interface overlay. The overlay generates image frames from the scene presented by the media. The image frame is sent to a viewer device. Viewer interaction parameters are received from a viewing device, and viewer interactions are displayed over subsequent scenes of the media presentation with the broadcaster interface overlay. On a viewing device, an image frame of the media presentation is received from the broadcasting device. An operating system level viewer interface overlay is generated on the image frame, and the image frame of the media presentation is displayed. Viewer interaction parameters are generated from viewer interactions with the operating system level viewer interface overlay and sent to the broadcast device.

Description

[0001] This patent application is a divisional application of the following invention patent application:

[0002] Application Number: 202380047844.9

[0003] Application date: May 19, 2023

[0004] Invention Title: Enhanced Screen Sharing for Online Computer Applications Technical Field

[0005] Various aspects of this disclosure relate to streaming video, and more specifically, various aspects of this disclosure relate to adding viewer-broadcast interaction to streaming video. Background Technology

[0006] Streaming media presentations such as video games, movies, and television programs has become popular. Streaming websites such as Twitch and Facebook Games are increasingly popular. A unique aspect of streaming websites lies in the interaction with the broadcaster. Generally, viewers' interaction with broadcasters using streaming websites is limited to applications that work in conjunction with the media presentation on the broadcasting device. Therefore, any interactive features of a streaming application must be compatible with the media presentation on a case-by-case basis. Thus, the most common interaction between viewers and broadcasters is through a chat window. Broadcasters typically have multiple monitors and, in some cases, multiple devices for broadcasting the media presentation and interacting with viewers.

[0007] A broadcaster can have a monitor display a media presentation on a first application and a second monitor display user interactions on a second application. Here, the viewer interaction does not appear on the media presentation itself, but rather on a second application that may include text or voice chat functionality. The second application may also display a streaming window showing image frames captured from the media presentation. The streaming window may also include user interactions within the image frames. Meanwhile, the media presentation itself operates without any user interaction. In cases where the media presentation is a video game or other mentally demanding application, the broadcaster's full attention may be occupied by the media presentation on the first display, and they may be unable to look at the second display or application for user interaction. Furthermore, there is often a significant delay (e.g., one to tens of seconds) between the media presentation and the interaction displayed by the second application. In some cases, this delay is to prevent cheating in multiplayer games, but it is usually due to system or network latency. Therefore, it would be advantageous for user interactions to be displayed directly on the broadcaster's media presentation with low latency. It is in this context that aspects of this disclosure are presented. Attached Figure Description

[0008] The teachings of this disclosure can be readily understood by considering the specific embodiments described below in conjunction with the accompanying drawings, in which:

[0009] Figure 1 It is a diagram of a broadcaster screen with viewer interaction according to one aspect of this disclosure, the viewer interaction including circle, square, arrow and cursor functions in the broadcaster interaction overlay.

[0010] Figure 2 It is a diagram of a broadcaster screen with viewer interaction according to one aspect of this disclosure, the viewer interaction including zoom and freeform line functions in the broadcaster interaction overlay.

[0011] Figure 3 It is a diagram of a broadcaster screen with viewer interaction according to one aspect of this disclosure, the viewer interaction including emoji, emoticons and text functions in the broadcaster interaction overlay.

[0012] Figure 4A It is a diagram depicting a display screen with viewer interaction in a picture-by-picture mode, according to one aspect of this disclosure.

[0013] Figure 4B This is a diagram illustrating a display screen with viewer interaction in a picture-in-picture mode according to one aspect of this disclosure.

[0014] Figure 5 This is a block diagram illustrating a computer system operation control stack including screen sharing according to one aspect of this disclosure.

[0015] Figure 6 This is a flowchart illustrating a method for viewer interaction on a broadcasting device according to one aspect of this disclosure.

[0016] Figure 7 This is a flowchart illustrating a method for viewer interaction on a viewer device according to one aspect of this disclosure.

[0017] Figure 8 This is a timing diagram illustrating the operation of a viewer device and a broadcasting device according to one aspect of this disclosure.

[0018] Figure 9 This is a diagram illustrating how, according to one aspect of this disclosure, user interaction on a viewer interface overlay is transformed into a broadcast device interface overlay on a media presentation.

[0019] Figure 10 It is a system diagram depicting a system for viewer interaction according to one aspect of this disclosure. Detailed Implementation

[0020] Although the following detailed description contains many specific details for illustrative purposes, it will be understood by any person skilled in the art that many variations and modifications to these details are within the scope of the invention. Therefore, the exemplary embodiments of the invention described below are presented without losing the generality of the claimed invention and without imposing limitations on it.

[0021] Viewer interaction is a major factor in attracting a large audience to streaming media. Viewers enjoy chatting with their favorite media broadcasters (also known as “streamers”) via text or voice chat. Sometimes, streamers will ask their viewers for help with media presentations such as video games. Help from their viewers is usually limited to text-based explanations or explanations via voice chat. Some applications have integrated with streaming video platforms, allowing for application-programmed implementation of specific user interactions. Currently, there is no way to enable users to interact with media presentations that are not specifically designed to accommodate these interactions.

[0022] Additionally, streaming media presentations are becoming more common among smaller groups of friends. Often, friends will share streaming media to share new games or get help with gameplay; friends sometimes host movie nights. It can be useful for friends to be able to use on-screen interactions (such as arrows, circles, squares, zoom areas, freeform lines, and on-screen text messages) to help each other.

[0023] This disclosure adds application-agnostic features, lower system latency, and viewer interactivity to the media presentation for broadcasters. Operating system-level media presentation streaming and interactive overlay code can be used to provide application-agnostic interactivity and reduced latency.

[0024] Figure 1This is a diagram of a broadcaster screen with viewer interaction according to one aspect of this disclosure, the viewer interaction including circles, squares, arrows, and cursor functions in a broadcaster interaction overlay. A display screen 109 of a broadcasting device 100 is depicted. Display screen 109 displays a media presentation. In this example, the media presentation includes a first knight 104, a second knight 105, and a castle 106. An operating system-level broadcaster interface overlay generates viewer interactions 101, 103, 108, and 102 based on viewer interaction parameters and displays these viewer interactions on the media presentation. In some implementations, the broadcaster interface overlay may also provide the broadcaster with viewer information 110, such as the number of viewers actively watching the media presentation and their names. By way of example and not limitation, this information may be presented to the broadcaster in a sidebar of the display screen. This broadcaster-specific information is only visible to the broadcaster and is not available in the video layer sent to the viewer, and therefore is not presented on the viewer's device. In this example, the viewer interaction includes placing a large arrow 101 pointing to castle 106, an oval shape 102 surrounding castle 106, and a square 103 highlighting the weakness of the second knight 105. Here, the square 103 or oval 102 may be filled with a semi-transparent or semi-opaque color. This color may be uniquely associated with a specific viewer generating a particular viewer interaction. The outline of the square 103 or oval 102 may be delineated with an opaque, semi-opaque, or semi-transparent color. Additionally, a broadcaster interface overlay may display a viewer cursor 108 on the media presentation. The viewer cursor 108 may be displayed in a different color than the broadcast device cursor 107, or it may be semi-transparent. Here, the media presentation includes, but is not limited to, video games, movies, television programs, streamed video, or images. Here, viewer interaction means information containing interaction parameters sent from a second device that receives image frames (also called streaming) from a media presentation displayed on a first device. Although viewer interaction may be associated with a graphical user interface, aspects of this disclosure are not limited to such implementations. In alternative implementations, viewer interaction can be associated with other elements, such as a touchscreen, game controller, or other peripheral devices (like a customizable control panel). Interaction parameters can include the type of interaction displayed on the broadcaster's interactive overlay (e.g., an arrow, square, or oval) and the location of the interaction. For the sake of illustration rather than limitation, the location of the viewer interaction can be determined based on the viewer's input to the graphical user interface associated with the viewer's device.

[0025] Figure 2This is a diagram of a broadcaster screen with viewer interaction according to one aspect of this disclosure, the viewer interaction including zoom and freeform line functions in a broadcaster interaction overlay. According to various aspects of this disclosure, the broadcaster interaction overlay is displaying zoom function 201 and freeform lines 207, 209 and cursor 208, and viewer interaction.

[0026] Here, the viewer can activate the zoom function 201 by using their cursor 208 to execute the "shrink to zoom" function. A square or other shape can be displayed on the area of ​​the broadcaster's interactive interface where zooming is taking place. The zoom function 201 uses pixel values ​​from the media rendering in the zoom area 203 and applies a scaling algorithm to the pixel values ​​from the media rendering in the zoom area to scale the image proportionally. The scaled image can be overlaid by the broadcaster's interactive interface on the original position 203 or in the second position 201. The scaling algorithm can be, for example, but not limited to, nearest neighbor interpolation, bicubic or bilinear scaling, sinc or lanczos resampling, box sampling, Fourier transform-based interpolation, edge-guided interpolation, HQX, etc. In some implementations, the scaling algorithm can apply machine learning-based methods.

[0027] Additionally, the broadcaster's interface can display other user interactions, such as freeform arrows 207 or freeform lines 205, or viewer cursors 208. Viewer interactions can also be combined with a zoom function. As shown, here, freeform arrow 207 is drawn on the magnified castle tower 206, while freeform line 205 is drawn in the magnified area 201. Here, the zoom function can expose a person on the castle wall, who might be too small for the broadcaster to see without it. Freeform line 205 and freeform arrow 207 can be drawn by the same viewer performing the zoom function, or they can be drawn by different viewers. The zoom function can continue as long as the viewer maintains the shrinking zoom 208, or zooming can be performed on a timer. The length of the zoom timer can be selected by the viewer or the broadcaster.

[0028] Figure 3This is a diagram of a broadcaster screen with viewer interaction according to one aspect of this disclosure, the viewer interaction including emoji, emoticons, and text functionality in a broadcaster interaction overlay. In this implementation, the broadcaster is playing a media presentation of a racing game. The broadcaster's car 301 has won the race, and to express their support and enthusiasm, viewers can use the broadcaster interaction overlay to place emoticons or emojis directly on the media presentation of the broadcasting device. In some implementations, the broadcaster interaction overlay may give the broadcaster the option to disable specific types of emojis or emoticons available to the viewer. Alternatively, the broadcaster interaction overlay may allow the broadcaster to provide custom emoticons for viewer use. The broadcaster may have interactive controls that allow the broadcaster to specify an area 307 of the broadcaster interaction overlay where viewer interactions can be displayed; additionally, the broadcaster can select the types of viewer interactions allowed to be displayed in the broadcaster interaction overlay. This allows the broadcaster to avoid intrusive or unwanted viewer interactions. Here, the broadcaster has designated an area 307 for viewer interaction, and the viewer places smiley faces 302 and party confetti 303 emoticons within the designated area 307. The broadcaster also allows the viewer to create text boxes 304 and arrows 305 outside the designated area 307.

[0029] In some implementations, the broadcaster interface overlay can allow the broadcaster to control other types of unwanted interactions, such as undesirable interactions from malicious viewers. In some implementations, the broadcaster interface overlay can allow the broadcaster to prioritize interactions, for example, by giving higher priority to interactions from VIP viewers, friends, or influencers. In some implementations, interactions can be prioritized based on the popularity of feedback providers so that their feedback appears before other feedback providers. In other implementations, some feedback can be excluded if the feedback volume is high enough. In this case, the system might create an alternative view of the feedback, such as "Top Feedback Only." Furthermore, broadcasters may have the option to selectively disable viewer interactions from some or all viewers, for example, if they believe the feature hinders their viewing of the media presentation, or if they wish to allow only a limited group of friends or subscribers on the platform to use the feature.

[0030] Figure 4A This is a diagram depicting a display screen with viewer interaction in a picture-by-picture mode according to one aspect of this disclosure. As shown, media presentation 401 may be playing on viewer device 400, and frames 402 of the media presentation may come from a broadcasting device. The viewer's media presentation 401 is displayed side-by-side with frames 402 of the media presentation from the broadcasting device on viewer device 400. Here, the viewer's media presentation 401 may be displayed with the same window size or resolution as the frames from broadcasting device 402. Figure 4A In this illustration, the viewer's media presentation 401 is shown to have a window size or resolution equal to the frame from the broadcasting device 402. However, aspects of this disclosure are not limited thereto, and the viewer's media presentation of any size or resolution can be displayed alongside frames from the broadcasting device of any size or resolution. This display configuration allows viewers to quickly interact with the broadcaster's stream by moving their cursor through the image frame window from the broadcasting device, as will be discussed in later sections.

[0031] Figure 4B This is a diagram illustrating a display screen with viewer interaction in a picture-in-picture mode according to one aspect of this disclosure. Here, a viewer's media presentation 403 is shown on a viewer device 400. An image frame 404 from a broadcasting device is displayed within the viewer's media presentation 403. The image frame 404 from the broadcasting device is in a smaller window or at a lower resolution than the viewer's media presentation 403. The display of the image frame 404 from the broadcasting device within the viewer's media presentation 403 allows for more display space for the viewer's media presentation while still allowing the viewer to view and interact with the image frame 404 from the broadcasting device. While implementations of displaying image frames from the broadcasting device within the viewer's media presentation have been discussed, aspects of this disclosure are not limited thereto. In alternative implementations, the viewer's media presentation may be displayed within image frames from the broadcasting device.

[0032] Figure 5This is a block diagram illustrating a computer system operation control stack including screen sharing according to one aspect of this disclosure. Herein lies the software and firmware stack levels in computer system 500. The base layer 501 is hardware configured to form a general-purpose computing device, including circuitry, resistors, capacitors, transistors, transformers, etc. Above the hardware level is the hardware abstraction layer 502. The hardware abstraction layer 502 interacts with the base hardware and is further configured to operate alongside higher-level software during the operation of the general-purpose computer. The hardware abstraction layer may include a bootloader that loads the operating system into hardware memory, a board support package that enables higher-level software to perform generalized operations on hardware functions, and device drivers that enable hardware components and devices that can be plugged into the hardware to operate. Hardware components and devices may include a keyboard, mouse, joystick, graphics card, monitor, etc. The next level is the operating system (OS) level 503. Here, the operating system includes screen sharing features 506, as described herein, that facilitate interaction between the viewer and the broadcaster. The operating system 503 communicates with the hardware abstraction layer 502 to allow operation of higher-level software. The OS includes automation of supporting operations in the form of a hardware abstraction layer. These supporting operations may include, but are not limited to, scheduling, file synchronization, and networking. Here, screen sharing plus function 506 operates at the OS level, thereby allowing for generalized support throughout the system architecture without the need for specialized application integration.

[0033] The next layer is the system services layer 507, which provides the service interface to the operating system 503 for the application layer 508. The system services layer may include: a file system, which indexes files stored in the hardware by the operating system; a graphical user interface (GUI), which graphically represents the services provided by OS 503 and allows users to more easily operate the general-purpose computer; and task management, which manages the services provided by the OS. Finally, the application layer 508 uses the OS services provided by the system services layer to generate additional computer functionality separate from the OS. Applications run in a lower privilege state than the operating system and typically require an application programming interface (API) to interact with the OS. Generally, specially programmed applications do not interact with other applications except through the OS layer. The OS layer's native screen sharing specifications allow for general screen sharing purposes without the need for specialized applications or custom applications for screen sharing.

[0034] Figure 6This is a flowchart illustrating a method for viewer interaction on a broadcasting device according to one aspect of this disclosure. By way of example and not limitation, this method can be implemented wholly or partially by operating system features such as screen sharing plus feature 506 discussed above. First, at 601, the broadcasting device can begin displaying a media presentation to the broadcaster. The media presentation can be, for example, but not limited to, video games, movies, television programs, streamed video, or images. The broadcaster can initiate streaming of the media presentation. As part of streaming the media presentation, the broadcasting device generates image frames from the media presentation, as indicated at 602. At 603, the image frames can be generated before or after the viewer connects to the broadcasting device. The connection with the viewer can be facilitated by using a streaming server that can receive frames from the broadcasting device and pass those frames to the connected viewing device. For the purposes of this document, this can be considered as the viewing device connecting to the broadcasting device. In addition to passing frames from the broadcaster to the viewer, the streaming server can also transcode alternative resolutions that the broadcaster may not provide.

[0035] At 604, after establishing a connection with the viewing device, the broadcaster can enable viewer-controlled events. If viewer-controlled events are not enabled, the broadcasting device may not display viewer interactions even if viewer interaction parameters are received. A message can also be sent to the viewing device notifying it that viewer-controlled events are enabled on the broadcasting device. The broadcasting device may include enableable restrictions on the type of viewer interaction and the position of the viewer interaction on the broadcasting device's display. At 605, the broadcasting device may have user-controlled settings that control the position and type of viewer interaction. At 606, after establishing a connection with the viewing device, the broadcasting device can send image frames to the viewing device. In some implementations, the image frames can be sent to a streaming server, which then sends the image frames to the viewing device.

[0036] After an image frame is sent to a viewing device, the viewing device can generate viewer interaction parameters based on the viewer's interaction with the image frame. At 607, these viewer interaction parameters can be sent back to the broadcasting device, whereby the viewer interaction parameters are received by the broadcasting device. In an alternative implementation, the viewing device can send the viewer interaction parameters to a streaming server, and the streaming server sends the viewer interaction parameters to the broadcasting device. At 608, the broadcaster interface overlay can take the viewer interaction parameters and overlay the interaction onto the media presentation. Here, user interaction parameters can include, for example, but not limited to, interaction type and / or interaction position. Interaction type can describe, for example, but not limited to, the displayed interaction, such as zoom, arrow, freeform line, cursor, etc. Interaction position can describe the location of the interaction in the media presentation, such as the pixel position or coordinate grid position (such as height, width, center, radius, diameter, edge position, etc.) of the interaction or part of the interaction. By way of example and not limitation, the interaction type and the position of the viewer interaction can be determined based on the viewer's input to the graphical user interface software and / or hardware associated with the viewer's device.

[0037] Viewer interaction can be displayed back to the viewer. This allows the viewer to more easily understand the broadcaster's live narration or actions. The broadcasting device can generate subsequent image frames, including viewer interaction, from the broadcaster's interactive interface overlay, as indicated at 609. As long as the frame index keeps moving forward, subsequent image frames can be generated from subsequent scenes in the media presentation or frames currently displayed on the broadcasting device to avoid sending frames more than once. At 610, these subsequent image frames can then be sent to the viewing device. The viewer can then interact with the subsequent image frames and send interaction parameters to the broadcasting device, repeating the process at 611, returning to step 607, and performing the remaining method steps 608 through 610. Alternatively, at 612, the broadcaster can terminate the streaming of the media presentation.

[0038] Figure 7 This is a flowchart illustrating a method 700 for viewer interaction on a viewer device according to one aspect of this disclosure. By way of example and not limitation, this method can be implemented wholly or partially by operating system features such as screen sharing plus feature 506 discussed above. At 701, the viewer device may initially connect to a broadcasting device. Alternatively, the viewer device may connect to the broadcasting device via a streaming server, which receives communications from the viewer device and sends the communications to the broadcasting device, for example via a cloud-based streaming server. The broadcasting device may send image frames from media presentation to the viewer device. Subsequently, at 702, the viewer device may receive the image frames. Additionally, the viewer device may receive information from the broadcasting device indicating that interaction is enabled or the location where viewer interaction is enabled, or the type of viewer interaction allowed.

[0039] At 703, an image frame from the media presentation on the broadcast device is displayed on the viewing device. Additionally, at 704, a viewer interaction overlay is generated on the image frame; this can occur simultaneously with, before, or after the display of the image frame. The viewer interaction overlay may be invisible to the user, or it may display the types of interactions the user can place within the image frame. In any case, at 705, the viewer interaction overlay captures the viewer's interactions with the image frame and translates those interactions into interaction parameters. By way of example and not limitation, the interaction overlay may include graphical user interface software and / or hardware associated with the viewer device, which allows the viewer to determine interaction parameters, such as interaction type and interaction location.

[0040] Then, at 706, the interaction parameters are sent to the broadcasting device. As described above, the broadcasting device can include user interaction in subsequent image frames. Then, at 707, the subsequent image frame including the viewer's interaction is transmitted to and received by the viewer device. Then, at 703, the subsequent image frame can be displayed, and steps 704, 705, 706, and 707 are repeated at 708 until the viewer or broadcaster ends the streaming or stops allowing viewer interaction at 709.

[0041] You can refer to Figure 8 The timing diagram depicted is used to understand the interaction between the viewer and broadcaster according to various aspects of this disclosure. The timing diagram illustrates communication 800 between broadcast device 802 and viewing device 801 via network 803. Network 803 may include a web server and / or a streaming server that transmits communication from broadcast device 802 to viewing device 801 and / or transmits communication from viewing device to broadcast device. Additionally, in some implementations, one or more web servers or streaming servers may provide other services such as viewer authentication, secure communication, payment processing, and similar services. As described above, at 804, broadcast device 802 may first begin displaying media presentation. At 806, broadcast device 802 may generate image frames from media presentation for streaming. At 805, viewing device 801 may request a connection to broadcast device. At 807, in response to receiving a connection request from viewing device, broadcast device 802 may send a connection response. The connection response may include, for example, but not limited to, information such as connection address and port number. In an alternative implementation, the connection request may be facilitated by one or more web servers, which may route the connection request or allow the viewer device to form an independent connection. At 808, once a connection response is received, the viewer device can use the information contained in the connection response to send an acknowledgment message.

[0042] At 809, once communication is established with viewing device 801, broadcasting device 802 sends image frames to viewing device. At 810, viewing device displays the image frames from broadcasting device. Viewing device can provide viewer interaction overlays on the displayed image frames. At 811, using viewer interaction overlays, viewing device can capture viewer interactions and translate them into interaction parameters. Then, at 812, interaction parameters are sent to broadcasting device. At 813, broadcasting device 802 can generate and display subsequent scenes from media presentation. In the subsequent scenes, at 814, viewer interactions can be displayed and overlaid using viewer interaction parameters. Then, at 815, broadcasting device 802 can generate image frames for streaming to viewing device, the image frames including the viewer's overlay interactions. Including viewer interactions in subsequent image frames allows viewers to understand the context of the broadcaster's actions and also to know that their interactions are received and displayed on the broadcasting device. Broadcaster-specific information, such as the number and names of viewers, can be omitted from the images generated from the broadcast overlays. At point 816, subsequent image frames, including viewer interaction, are sent to the viewer device. At point 817, the viewer device can display image frames including viewer interaction, but omit any broadcaster-specific information.

[0043] Figure 9This diagram illustrates how, according to one aspect of this disclosure, user interactions on a viewer interface overlay are transformed into a broadcast device interface overlay on a media presentation. In this implementation, a viewing device 910 displays a media presentation 902 for the viewer and an image frame 901 from a broadcast device. The viewer interface overlay can capture the viewer's interaction with the image frame. Here, viewer interaction can be captured using a grid with X grid lines 904 and Y grid lines 903. Alternatively, pixel positions can be used to capture viewer interaction. The viewer interaction shown here is, for example, a square 905 drawn in the image frame around the area of ​​interest to the viewer using graphical interface hardware and / or software. Furthermore, the position of the viewer cursor 906 can be captured as user interaction. The position and type of user interaction (referred to as viewer interaction parameters) can be captured using X coordinates 904 and Y coordinates 903 or pixel positions determined from the viewer interaction overlay. The viewer interaction parameters can be transmitted to the broadcast device 911 via a network 907. Since image frames from a broadcast device are displayed in a smaller window or at a reduced resolution compared to the media presentation of the broadcast device, a transformation can be applied to change the viewer interaction parameters from the viewer device 910 to the broadcast device 911. The transformed viewer interaction parameters can then be used to display the user interaction 910. Transformations can include, but are not limited to, grid or pixel coordinate offsets, coordinate multipliers, dividers, etc. Transformations can be used to handle problems such as different resolutions or different viewing devices between the broadcaster and the viewer. Furthermore, transformations can address the offset requirements caused by off-grid clicks due to different aspect ratios at the broadcaster and the viewing device. As shown, the X 909 and Y 908 grid or pixel coordinates have been transformed to fit the broadcaster interaction overlays 908 and 909 of the broadcast device 911. Additionally, the viewer cursor 906 is displayed through the broadcaster interface overlay.

[0044] In some implementations, the location of user interaction may follow the location of objects within the image frame. For example, in Figure 9 In the image displayed on both the broadcaster and viewer devices, a square 905 can move along with a ladder following a hole as the knight moves. To achieve this, certain information can be captured from the viewer side to take certain actions and transmitted to the broadcaster to maintain accuracy, since the position of the selected object is precisely located on the broadcaster side. For example, the system can capture the position of the viewer cursor 906 and transmit the pixel coordinates and color information (for a small pixel grid) obtained on the viewer side. Once the broadcaster device 911 receives the cursor information, it attempts to match the color and pixel coordinates within a threshold area to improve accuracy. When a prediction attempt to match the coordinate radius and color cannot be satisfied, the broadcaster device 911 can render using only the pixel coordinates as a fallback.

[0045] Figure 10This is a block diagram depicting a system for viewer interaction according to one aspect of this disclosure. By way of example and not limitation, according to various aspects of this disclosure, system 1000 can be an embedded system, a mobile phone, a personal computer, a tablet computer, a portable gaming device, a workstation, a game console, etc.

[0046] System 1000 typically includes a central processing unit (CPU) 1003 and memory 1004. System 1000 may also include well-known support functions 1006, which may communicate with other components of the system, for example, via a data bus 1005. Such support functions may include, but are not limited to, input / output (I / O) elements 1007, power supplies (P / S) 1011, clocks (CLK) 1012, and caches 1013.

[0047] System 1000 may include a mass storage device 1015, such as a disk drive, CD-ROM drive, flash memory, solid-state drive (SSD), tape drive, etc., to provide non-volatile storage for programs and / or data. System 1000 may also optionally include a user interface unit 1016 to facilitate interaction between system 1000 and the user. User interface 1016 may include a keyboard, mouse, joystick, light pen, or other devices that can be used in conjunction with a graphical user interface (GUI). System 1000 may also include a network interface 1014 to enable the device to communicate with other devices via network 1020. Network 1020 may be, for example, a local area network (LAN), a wide area network (such as the Internet), a personal area network (such as a Bluetooth network), or other types of networks. These components may be implemented in hardware, software, or firmware, or some combination of both or more of these.

[0048] CPU 1003 may include one or more processor cores, such as a single core, two cores, four cores, eight cores, or more cores. In some implementations, CPU 1003 may include a GPU core or multiple cores of the same Accelerated Processing Unit (APU). Memory 1004 may be in the form of an integrated circuit providing addressable memory (e.g., random access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.). Main memory 1004 may include applications 1021 used by processor 1003 to generate applications such as chat programs, internet browsers, video games, movies, etc. Main memory 1004 may also include an operating system (OS) 1022, which can run on a hardware abstraction layer, such as... Figure 5 As depicted. (e.g.) Figure 5The operating system may include an interaction interface 1009 that uses and / or generates interaction parameters 1010 as components in screen sharing. Interaction parameters 1010 may be stored in memory 1004 or cache 1013. Additionally, memory 1004 may include media presentation 1008, such as, but not limited to, video games, movies, television programs, streamed video, or images. The OS-layer interface and interaction parameters provide viewer interaction that is agnostic to both the application and the media presentation, and reduce waiting time compared to current application-layer user interaction programs, because the interaction operates at a system layer lower than the media presentation or application.

[0049] According to various aspects of this disclosure, the processor 1003 can implement connectivity and viewer interaction, for example, as described above regarding Figure 6 , 7 As described in 8, these methods can be loaded into memory 1004 as part of OS 1022. Processor 1003 can generate one or more interaction parameters 1010, which can be sent to a broadcasting device via network 1020 using network interface 1014, and receive image frames from media presentation 1008 from the broadcasting device. Network interface 1014 can also receive interaction parameters 1010 from a viewing device, and can stream image frames from media presentation 1008 to the viewing device.

[0050] Mass storage device 1015 may contain an application or program 1017, which is loaded into main memory 1004 when processing begins on application 1023. Additionally, mass storage device 1015 may contain data 1018 used by the processor to load OS 1009 during startup.

[0051] As used herein and as commonly understood by those skilled in the art, an application-specific integrated circuit (ASIC) is tailored for a specific purpose rather than intended for general use.

[0052] As used herein and as commonly understood by those skilled in the art, a Field Programmable Gate Array (FPGA) is an integrated circuit designed to be configured by a customer or designer after manufacturing (hence the term "field-programmable"). FPGA configuration is typically specified using an HDL, similar to a Hardware Description Language (HDL) used for ASICs.

[0053] As used herein and as commonly understood by those skilled in the art, a system-on-a-chip (SoC or SOC) is an integrated circuit (IC) that integrates all components of a computer or other electronic system onto a single chip. It can contain digital, analog, mixed-signal, and often radio frequency functions—all on a single chip substrate. Typical applications are in the field of embedded systems.

[0054] A typical SoC includes the following hardware components:

[0055] One or more processor cores (e.g., microcontroller, microprocessor, or digital signal processor (DSP) cores).

[0056] Memory blocks, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), and flash memory.

[0057] Timing sources, such as oscillators or phase-locked loops.

[0058] Peripheral devices, such as counters-timers, real-time timers, or power-on reset generators.

[0059] External interfaces, such as industry standards, include Universal Serial Bus (USB), FireWire, Ethernet, Universal Asynchronous Receiver / Transmitter (USART), and Serial Peripheral Interface (SPI) bus.

[0060] Analog interfaces, which include analog-to-digital converters (ADCs) and digital-to-analog converters (DACs).

[0061] Voltage regulator and power management circuit.

[0062] These components are connected via proprietary or industry-standard buses. The Direct Memory Access (DMA) controller routes data directly between the external interface and memory, bypassing the processor core and thereby increasing the data throughput of the SoC.

[0063] A typical SoC includes both the hardware components described above and the executable instructions (e.g., software or firmware) that control the processor core, peripheral devices, and interfaces.

[0064] The various aspects of this disclosure allow for enhanced remote interaction between video game players. The implementations discussed herein are not limited to allowing players to share content on their respective screens. Furthermore, such implementations allow viewer interactions to be displayed directly on the broadcaster's media presentation with low latency.

[0065] While the foregoing is a complete description of preferred embodiments of the invention, various alternatives, modifications, and equivalents may be used. Therefore, the scope of the invention should not be determined by reference to the foregoing description, but rather by reference instead to the appended claims and their equivalents throughout. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, The indefinite article "a" or "a kind" "This refers to the quantity of one or more items following the article, unless otherwise expressly stated therein. The appended claims should not be construed as including means plus functional limitations, unless such limitations are expressly stated in a given claim using the phrase "component for..."

Claims

1. A method for viewer interaction with streaming media on a broadcasting device, comprising: Displaying a scene from media presentation, wherein the broadcasting device includes an operating system-level broadcaster interface overlay; Generate image frames from the scene presented by the media; The image frames from the scene presented by the media are sent to the viewing device; Receive viewer interaction parameters; as well as Using the viewer interaction parameters, the viewer interaction is displayed over the subsequent scene from the media presentation using the operating system-level broadcaster interface, wherein the viewer interaction parameters include the interaction position; The viewer interaction parameters include a transformation for the interaction position, which is used to change the viewer interaction position from the resolution at the viewer device to the resolution at the broadcast device.

2. The method for viewer interaction according to claim 1, wherein the viewer interaction parameters include the interaction type.

3. The method for viewer interaction according to claim 1, wherein the viewer interaction parameters include data for displaying emoticons or emojis in the operating system-level broadcaster interface overlay.

4. The method for viewer interaction according to claim 1, wherein the viewer interaction parameters include data for displaying freeform lines, rectangles, circles, or ovals in the operating system-level broadcaster interface overlay.

5. The method for viewer interaction according to claim 1, wherein the viewer interaction parameters include data for displaying a highlighted area, cursor, arrow, or text annotation in the operating system-level broadcaster interface overlay.

6. The method for viewer interaction according to claim 1, wherein the viewer interaction parameters include data for magnifying a portion of the scene presented by the media in the operating system-level broadcaster interface overlay.

7. The method for viewer interaction according to claim 1, further comprising establishing viewer interaction restrictions using the operating system-level broadcaster interface overlay.

8. The method of claim 1, wherein the operating system-level broadcaster interface overlay includes information not included in the image sent to the viewing device.

9. A system for viewer interaction with streaming media, comprising: processor; A memory coupled to the processor; Non-transitory instructions embodied in the memory, when executed by the processor, cause the processor to implement a method for interaction between a viewer and streaming media on a broadcasting device, the method comprising: Displaying a scene from media presentation, wherein the broadcasting device includes an operating system-level broadcaster interface overlay; Generate image frames from the scene presented by the media; The image frames from the scene presented by the media are sent to the viewing device; Receive viewer interaction parameters; Using the viewer interaction parameters, the viewer interaction is displayed over the subsequent scene from the media presentation using the operating system-level broadcaster interface, wherein the viewer interaction parameters include the interaction position; The viewer interaction parameters include a transformation for the interaction position, which is used to change the viewer interaction position from the resolution at the viewer device to the resolution at the broadcast device.

10. A method for viewer interaction with streaming media on a viewing device, comprising: Requesting a connection to the broadcasting device; Receive image frames from media presentation from the broadcasting device; Generate an operating system-level viewer interface overlay on the image frame; Display the image frames presented by the media; Viewer interaction parameters are generated from viewer interactions with the operating system-level viewer interface overlay. The viewer interaction parameters are sent to the broadcasting device, wherein the viewer interaction parameters include the interaction position; The viewer interaction parameters include a transformation for the interaction position, the transformation being used to change the position of the viewer interaction from the resolution at the viewing device to the resolution at the broadcasting device.

11. The method for viewer interaction according to claim 10, wherein the viewer interaction parameters include an interaction type.

12. The method for interaction according to claim 10, wherein the viewer interaction parameters include data for displaying emoticons or emojis in the operating system-level viewer interface overlay.

13. The method for interaction according to claim 10, wherein the viewer interaction parameters include data for displaying freeform lines, rectangles, circles, or ovals in the operating system-level viewer interface overlay.

14. The method for interaction according to claim 10, wherein the viewer interaction parameters include data for displaying a highlighted area, cursor, arrow, or text annotation in the operating system-level viewer interface overlay.

15. The method for interaction according to claim 10, wherein the viewer interaction parameters include data for magnifying a portion of the scene presented by the media in the operating system-level viewer interface overlay.

16. The method for interaction according to claim 10, further comprising receiving viewer interaction restrictions from the broadcasting device.

17. The method for interaction according to claim 10, wherein generating viewer interaction parameters includes capturing an interaction type and the interaction location from a region of the display screen of the viewing device.

18. A system for viewer interaction with streaming media, comprising: processor; A memory coupled to the processor; Non-transitory instructions embodied in the memory, when executed by the processor, cause the processor to implement a method for interacting with streaming media on a viewing device, the method comprising: Requesting a connection to the broadcasting device; Receive image frames presented by the media from the broadcasting device; Generate an operating system-level viewer interface overlay on the image frames from the media presentation; Display the image frames presented by the media; Viewer interaction parameters are generated from viewer interactions with the operating system-level viewer interface overlay. The viewer interaction parameters are sent to the broadcasting device, wherein the viewer interaction parameters include the interaction position; The viewer interaction parameters include a transformation for the interaction position, the transformation being used to change the position of the viewer interaction from the resolution at the viewing device to the resolution at the broadcasting device.

Images