Enhanced screen sharing for online computer applications

Abstract

In a method and system for viewer interaction with streaming media, a scene from a media presentation is displayed with a broadcast device including operating system level broadcaster interface overlay. The overlay generates image frames from the scene presented by the media. The image frame is sent to a viewer device. Viewer interaction parameters are received from a viewing device, and viewer interactions are displayed over subsequent scenes of the media presentation with the broadcaster interface overlay. On a viewing device, an image frame of the media presentation is received from the broadcasting device. An operating system level viewer interface overlay is generated on the image frame, and the image frame of the media presentation is displayed. Viewer interaction parameters are generated from viewer interactions with the operating system level viewer interface overlay and sent to the broadcast device.

Description

Technical Field

[0001] Aspects of the present disclosure relate to streaming video, and more particularly, aspects of the present disclosure relate to adding viewer and broadcaster interaction to streaming video. Background Art

[0002] Streaming of media presentations such as video games, movies, and television shows has become popular. Streaming websites such as Twitch and Facebook games are becoming increasingly popular. A unique aspect of streaming websites is the interaction with broadcasters. Generally speaking, viewers' interaction with broadcasters using streaming websites is limited to applications that operate in conjunction with the media presentation on the broadcasting device. Therefore, any interactive features of streaming applications must be compatible with the media presentation on a case-by-case basis. Therefore, the most common interaction between viewers and broadcasters is through a chat window. Broadcasters typically have multiple monitors and, in some cases, multiple devices for broadcasting media presentations and interacting with viewers.

[0003] The broadcaster can have one monitor display the media presentation on the first application, and have the second monitor display the user interaction on the second application. Here, the viewer interaction does not appear on the media presentation itself, but on the second application that can include text or voice chat function. The second application can also display a streaming window, which displays the image frame captured from the media presentation. The streaming window can also include the user interaction in the image frame. At the same time, the media presentation itself can run without any interaction from the user. In the case where the media presentation is a video game or other application that requires mental work, the broadcaster's full attention may be occupied by the media presentation on the first display, and may not be able to watch the second display or application for user interaction. In addition, there is usually a large delay (e.g., one second to tens of seconds) between the media presentation and the interaction displayed by the second application. In some cases, this delay is to prevent cheating in multi-player games, but it is usually due to system or network latency. Therefore, it would be advantageous for user interactions to be directly displayed on the broadcaster's media presentation with low latency. It is in this context that various aspects of the present disclosure are proposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The teachings of the present disclosure may be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

[0005] Figure 1 is a diagram of a broadcaster screen with viewer interaction including circle, box, arrow, and cursor functionality in a broadcaster interaction overlay according to one aspect of the present disclosure.

[0006] Figure 2 is a diagram of a broadcaster screen with viewer interaction including zoom and freeform line functionality in a broadcaster interaction overlay according to one aspect of the present disclosure.

[0007] Figure 3 is a diagram of a broadcaster screen with viewer interaction including emoji, emoticon, and text functionality in a broadcaster interaction overlay according to one aspect of the present disclosure.

[0008] Figure 4A is a diagram depicting a display screen with viewer interaction in a picture-by-picture mode according to one aspect of the present disclosure.

[0009] Figure 4B is a diagram illustrating a display screen with viewer interaction in a picture-in-picture mode according to one aspect of the present disclosure.

[0010] Figure 5 is a block diagram illustrating a computer system operation control stack including screen sharing according to one aspect of the present disclosure.

[0011] Figure 6 is a flow chart illustrating a method for viewer interaction on a broadcaster according to one aspect of the present disclosure.

[0012] Figure 7 is a flow chart illustrating a method for viewer interaction on a viewer device according to one aspect of the present disclosure.

[0013] Figure 8 is a timing diagram illustrating the operation of a viewer device and a broadcasting device according to one aspect of the present disclosure.

[0014] Fig. 9 is a diagram illustrating transformation of user interactions on a viewer interface overlay to a broadcaster interface overlay on a media presentation according to one aspect of the present disclosure.

[0015] Fig.10 is a system diagram depicting a system for viewer interaction according to one aspect of the present disclosure. DETAILED DESCRIPTION

[0016] Although the following detailed description contains many specific details for illustrative purposes, it will be appreciated by any person skilled in the art that many variations and modifications to the following details are within the scope of the present invention. Therefore, the exemplary embodiments of the present invention described below are set forth without losing the generality of the claimed invention and without imposing limitations on the claimed invention.

[0017] Viewer interaction is a major factor that attracts many streaming media viewers. Viewers like to chat with their favorite media broadcasters (also called "streamers") in text chat or voice chat. Sometimes, streamers ask their viewers for help on a media presentation (such as a video game). The help from their viewers is usually limited to text-based explanations or explanations through voice chat. Some applications have been integrated with streaming video platforms, allowing specific user interactions to be implemented through application programming. Currently, there is no way for users to interact with media presentations that are not specifically designed to accommodate these interactions.

[0018] Additionally, streaming media presentations is becoming common among smaller groups of friends. Often, friends will share streaming presentations with each other to share a new game or get help with a game; friends sometimes host movie nights. It would be useful for friends to be able to help other friends using on-screen interactions such as arrows, circles, boxes, magnified areas, free-form lines, and on-screen text messages.

[0019] Aspects of the present disclosure add application-agnostic, low system latency, viewer interaction to a broadcaster's media presentation. Operating system level media presentation streaming and interaction overlay code can be used to provide application-agnostic interaction and reduced latency.

[0020] Figure 1is a diagram of a broadcaster screen with viewer interactions including circle, box, arrow, and cursor functionality in a broadcaster interaction overlay according to one aspect of the present disclosure. Depicted is a display screen 109 of a broadcast device 100. The display screen 109 displays a media presentation. In this example, the media presentation includes a first knight 104 and a second knight 105 and a castle 106. The operating system level broadcaster interface overlay generates viewer interactions 101, 103, 108, 102 based on viewer interaction parameters and displays the viewer interactions on the media presentation. In some implementations, the broadcaster interface overlay may also provide viewer information 110 to the broadcaster, such as the number of viewers actively viewing the media presentation and the viewer names. By way of example and not limitation, this information may be presented to the broadcaster on a sidebar of the display screen. This broadcaster-unique information is visible only to the broadcaster and is not available in the video layer sent to the viewer and therefore is not presented on the viewer device. In this example, the viewer interaction includes placing a large arrow 101 pointing to the castle 106, an oval 102 surrounding the castle 106, and a square 103 highlighting the weakness of the second knight 105. Here, the square 103 or oval 102 can be filled with a semi-transparent or semi-opaque color. Such a color may be uniquely associated with a specific viewer generating a specific viewer interaction. The outline of the square 103 or oval 102 can be outlined with an opaque color, a semi-opaque color, or a semi-transparent color. In addition, the broadcaster interface overlay can display a viewer cursor 108 on the media presentation. The viewer cursor 108 can be displayed in a different color from the broadcaster cursor 107, or can be semi-transparent. Here, the media presentation includes, but is not limited to, video games, movies, television programs, streamed videos or pictures. Here, viewer interaction means information containing interaction parameters sent from a second device, which receives image frames (also known as streaming) from the media presentation displayed on the first device. Although viewer interaction can be associated with a graphical user interface, aspects of the present disclosure are not limited to such implementations. In alternative implementations, viewer interactions may be associated with something else, such as a touch screen, game controller, or other peripheral device (such as a customizable control panel). Interaction parameters may include the type of interaction (e.g., arrow, square, or oval) and the location of the interaction for display on the broadcaster interaction interface overlay. By way of example and not limitation, the location of the viewer interaction may be determined based on viewer input to a graphical user interface associated with the viewer device.

[0021] Figure 2is a diagram of a broadcaster screen with viewer interactions including zoom and freeform line functionality in a broadcaster interaction overlay according to one aspect of the disclosure. According to aspects of the disclosure, the broadcaster interaction overlay is displaying a zoom 201 functionality and freeform lines 207, 209 and cursor 208, viewer interactions.

[0022] Here, the viewer may activate the zoom function 201 by performing a "pinch to zoom" function with their cursor 208. A square or other shape may be displayed on the area of ​​the broadcaster interaction interface being zoomed. The zoom 201 function may use pixel values ​​from the media presentation in the zoom area 203 and apply a scaling up algorithm to the pixel values ​​from the media presentation in the zoom area to scale up the image. The scaled up image may be displayed overlaid by the broadcaster interaction interface on the original position 203 or in the second position 201. The scaling up algorithm may be, for example, but not limited to, nearest neighbor interpolation, bicubic or bilinear scaling, sinc or lanczos resampling, box sampling, Fourier transform based interpolation, edge guided interpolation, hqx, etc. In some implementations, the scaling algorithm may apply a machine learning based approach.

[0023] In addition, the broadcaster interaction interface can also display other user interactions, such as free form arrows 207 or free form lines 205 or viewer cursors 208. Viewer interactions can also be combined with zoom functions. As shown, here, free form arrows 207 are drawn on the enlarged castle tower 206, and free form lines 205 are drawn in the enlarged area 201. Here, the zoom function can expose a person on the wall, who may be too small to be seen by the broadcaster without the function. The free form line 205 and free form arrows 207 can be drawn by the same viewer who performs the zoom function, or can be drawn by different viewers. As long as the viewer keeps the contraction zoom 208, the zoom function can continue, or the zoom can be performed on a timer. The length of the zoom timer can be selected by the viewer or by the broadcaster.

[0024] Figure 3is a diagram of a broadcaster screen with viewer interactions according to one aspect of the present disclosure, the viewer interactions including emojis, emoticons, and text functionality in a broadcaster interaction overlay. In this implementation, the broadcaster is playing a media presentation of a racing game. The broadcaster's car 301 wins the race, and to show their support and enthusiasm, viewers can use the broadcaster interaction interface overlay to place emoticons or emojis directly on the media presentation of the broadcaster. In some implementations, the broadcaster interaction interface overlay can give the broadcaster the option to disable specific types of emojis or emoticons available to viewers. Alternatively, the broadcaster interaction interface overlay can allow the broadcaster to provide custom emoticons for viewers to use. The broadcaster can have an interaction control that allows the broadcaster to specify an area 307 of the broadcaster interaction interface overlay where viewer interactions can be displayed, and in addition, the broadcaster can select the types of viewer interactions that are allowed to be displayed in the broadcaster interaction interface overlay. This allows the broadcaster to avoid disruptive or undesired viewer interactions. Here, the broadcaster has designated an area 307 for viewer interaction, and the viewer places smiley face 302 and party popper 303 emoticons in the designated area 307. Here, the broadcaster also allows the viewer to create a text box 304 and an arrow 305 outside the designated area 307.

[0025] In some implementations, the broadcaster interaction interface overlay may allow the broadcaster to control other types of unwanted interactions, such as unwanted interactions from bad actor viewers. In some implementations, the broadcaster interaction interface overlay may allow the broadcaster to prioritize interactions, for example, by giving higher priority interactions to VIP viewers or friends or influencers. In some implementations, interactions may be prioritized based on the popularity of the feedback provider so that their feedback appears before other feedback providers. In other implementations, some feedback may be excluded if the feedback volume is high enough. In this case, the system may create an alternative view of the feedback, such as "top feedback only." In addition, broadcasters may have the option to selectively disable viewer interactions for some or all viewers, for example, if they believe that the feature is hindering their viewing of the media presentation, or if they wish to allow only a limited set of friends or subscribers on the platform to use the feature.

[0026] Figure 4A is a diagram depicting a display screen with viewer interaction in a picture-by-picture mode according to one aspect of the present disclosure. As shown, a media presentation 401 may be playing on a viewer device 400, and a frame 402 of the media presentation may be from a broadcaster. The viewer's media presentation 401 is displayed side-by-side with the frame 402 of the media presentation from the broadcaster on the viewer device 400. Here, the viewer's media presentation 401 may be displayed in the same window size or resolution as the frame from the broadcaster 402. Figure 4A In the example, the window size or resolution of the viewer's media presentation 401 is equal to the frame from the broadcaster 402, but aspects of the present disclosure are not limited thereto, and the viewer's media presentation of any size or resolution can be displayed next to the frame from the broadcaster of any size or resolution. This display configuration can allow viewers to quickly interact with the broadcaster's stream by moving their cursor into the image frame window from the broadcaster, as will be discussed in a later section.

[0027] Figure 4B is a diagram illustrating a display screen with viewer interaction in a picture-in-picture mode according to one aspect of the present disclosure. Here is a viewer's media presentation 403 on a viewer device 400. An image frame 404 from a broadcaster is displayed within the viewer's media presentation 403. The image frame 404 from the broadcaster is in a smaller window or lower resolution than the viewer's media presentation 403. The display of the image frame 404 from the broadcaster within the viewer's media presentation 403 allows for more display screen space to be provided for the viewer's media presentation while still allowing the viewer to view and interact with the image frame 404 from the broadcaster. Although implementations of displaying the image frame from the broadcaster within the viewer's media presentation are discussed, aspects of the present disclosure are not limited thereto. In alternative implementations, the viewer's media presentation may be displayed within the image frame from the broadcaster.

[0028] Figure 55 is a block diagram illustrating a computer system operation control stack including screen sharing according to one aspect of the present disclosure. Here are the software and firmware stack levels in the computer system 500. The base layer 501 is hardware configured to form a general-purpose computing device with circuit systems, resistors, capacitors, transistors, transformers, etc. Above the hardware level is the hardware abstraction layer 502. The hardware abstraction layer 502 level interacts with the basic hardware and is further configured to operate with higher-level software during the operation of the general-purpose computer. The hardware abstraction layer may include a boot loader that loads the operating system into the hardware memory, a board support package that enables higher-level software to generalize the hardware functions, and a device driver that enables the hardware components and devices that can be inserted into the hardware to operate. The hardware components and devices may include a keyboard, a mouse, a joystick, a graphics card, a display, etc. The next level is the operating system (OS) level 503. Here, the operating system includes a screen sharing plus feature 506 that facilitates interaction between viewers and broadcasters, as described herein. The operating system 503 communicates with the hardware abstraction layer 502 to allow the operation of higher-level software. The OS includes automation of support operations in the hardware abstraction layer in the form of services. These support operations may include, for example, but not limited to, scheduling, file synchronization, and networking. Here, the screen sharing plus function 506 runs at the OS level, allowing generalized support to be provided throughout the system architecture without the need for specialized application integration.

[0029] The next layer is the system service layer 507 that provides a service interface to the operating system 503 for the application layer 508. The system service layer may include: a file system that indexes files stored in hardware by the operating system; a graphical user interface (GUI) that graphically represents the services provided by the OS 503 and allows the user to more easily operate a general-purpose computer; and task management that manages the services provided by the OS. Finally, the application layer 508 uses the services of the OS provided by the system service layer to generate additional computer functionality separate from the OS. Applications run in a lower privilege state than the operating system and typically require an application programming interface API to interact with the OS. In general, a specially programmed application will not interact with other applications unless through the OS layer. The native screen sharing provisions of the OS layer allow general screen sharing purposes without the need to use specialized applications or customize applications for screen sharing.

[0030] Figure 6is a flow chart illustrating a method for viewer interaction on a broadcast device according to one aspect of the present disclosure. By way of example and not limitation, such a method may be implemented in whole or in part by an operating system feature such as the screen sharing plus feature 506 discussed above. First, at 601, a broadcast device may begin a media presentation displayed to a broadcaster. The media presentation may be, for example, but not limited to, a video game, a movie, a television program, a streamed video, or a picture. The broadcaster may initiate streaming of the media presentation. As part of streaming the media presentation, the broadcast device generates image frames from the media presentation, as indicated at 602. At 603, the image frames may be generated before or after the viewer connects to the broadcast device. Connection to the viewer may be facilitated by using a streaming server that may receive frames from the broadcast device and pass those frames to a connected viewing device. For the purposes of this document, this may be considered as a viewing device connecting to the broadcast device. In addition to passing frames from the broadcaster to the viewer, the streaming server may also transcode alternative resolutions that may not be provided by the broadcaster.

[0031] At 604, after a connection has been established with the viewing device, the broadcaster may enable viewer controlled events. If viewer controlled events are not enabled, the broadcaster may not display the viewer interactions even if viewer interaction parameters are received. A message may also be sent to the viewing device notifying it that viewer controlled events have been enabled on the broadcaster. The broadcaster may include enabled capability restrictions on the type of viewer interactions and the location of viewer interactions on the broadcaster display. At 605, the broadcaster may have user controlled settings that control viewer interaction locations and user interaction types. At 606, after a connection has been established with the viewing device, the broadcaster may send image frames to the viewing device. In some implementations, the image frames may be sent to a streaming server, which then sends the image frames to the viewing device.

[0032] After the image frame is sent to the viewing device, the viewing device may generate viewer interaction parameters based on the viewer's interaction with the image frame. At 607, these viewer interaction parameters may be sent back to the broadcast device, where the viewer interaction parameters are received by the broadcast device. In an alternative implementation, the viewing device may send the viewer interaction parameters to a streaming server, and the streaming server sends the viewer interaction parameters to the broadcast device. At 608, the broadcaster interface overlay may adopt the viewer interaction parameters and overlay the interaction on the media presentation. Here, the user interaction parameters may include, for example, but not limited to, the interaction type and / or the interaction position. The interaction type may, for example, but not limited to, describe the displayed interaction, such as zoom, arrow, free form line, cursor, etc. The interaction position may describe the location of the interaction in the media presentation, such as the pixel position or coordinate grid position (such as height, width, center, radius, diameter, edge position, etc.) of the interaction or part of the interaction. By way of example and not limitation, the interaction type and the location of the viewer interaction may be determined based on the viewer's input to the graphical user interface software and / or hardware associated with the viewer device.

[0033] The viewer interactions may be displayed back to the viewer. This may allow the viewer to more easily understand the broadcaster's live commentary or actions. The broadcaster may overlay the generation of subsequent image frames including the viewer interactions from the broadcaster interaction interface, as indicated at 609. As long as the frame index is advancing, subsequent image frames may be generated from subsequent scenes of the media presentation or frames currently displayed on the broadcaster to avoid sending frames more than once. At 610, these subsequent image frames may then be sent to the viewing device. The viewer may then interact with the subsequent image frames and send the interaction parameters to the broadcaster, which repeats returning to step 607 at 611 and performing the remaining method steps 608 to 610. Alternatively, at 612, the broadcaster may end streaming of the media presentation.

[0034] Figure 7 is a flow chart illustrating a method 700 for viewer interaction on a viewer device according to one aspect of the present disclosure. By way of example and not limitation, such a method may be implemented in whole or in part by an operating system feature such as the screen share plus feature 506 discussed above. At 701, a viewing device may initially be connected to a broadcaster. Alternatively, the viewing device may be connected to the broadcaster through a streaming server that receives communications from the viewing device and sends the communications to the broadcaster, for example, via a cloud-based streaming server. The broadcaster may send image frames from a media presentation to the viewing device. Subsequently, at 702, the viewing device may receive the image frames. In addition, the viewing device may receive information from the broadcaster indicating that interaction has been enabled or where viewer interaction is enabled or the type of viewer interaction that is allowed.

[0035] At 703, an image frame from a media presentation on a broadcast device is displayed on a viewing device. Additionally, at 704, a viewer interaction interface overlay is generated on the image frame, which may occur simultaneously with the display of the image frame, before the image frame is displayed, or after the image frame is displayed. The viewer interaction interface overlay may not be visible to the user, or may display the type of interaction that the user may place within the image frame. In any case, at 705, the viewer interaction overlay captures the viewer's interactions with the image frame and translates those interactions into interaction parameters. By way of example and not limitation, the interaction overlay may include graphical user interface software and / or hardware associated with a viewer device that allows the viewer to determine interaction parameters, such as interaction type and interaction location.

[0036] Then, at 706, the interaction parameters are sent to the broadcaster. As described above, the broadcaster may include the user interaction in a subsequent image frame. Then, at 707, the subsequent image frame including the viewer interaction is transmitted to and received by the viewer device. Then, at 703, the subsequent image frame may be displayed, and steps 704, 705, 706, 707 are repeated at 708 until the viewer or broadcaster ends streaming or stops allowing viewer interaction 709.

[0037] You can refer to Figure 8 801 through a network 803. The network 803 may include a network server and / or a streaming server that passes communications from the broadcast device 802 and passes them to the viewing device 801 and / or passes communications from the viewing device to the broadcast device. In addition, in some implementations, one or more network servers or streaming servers may provide other services, such as viewer authentication, secure communications, payment processing, and similar services. As described above, at 804, the broadcast device 802 may first begin displaying the media presentation. At 806, the broadcast device may generate image frames from the media presentation for streaming. At 805, the viewing device 801 may request to connect to the broadcast device. At 807, in response to receiving a connection request from the viewing device, the broadcast device 802 may send a connection response. The connection response may include information such as, but not limited to, a connection address and a port number. In alternative implementations, the connection request may be facilitated by one or more network servers, which may route the connection request or the viewer device may form an independent connection. At 808, upon receiving the connection response, the viewer device may send a confirmation message using the information contained within the connection response.

[0038] At 809, once communication is established with viewing device 801, broadcast device 802 sends the image frame to the viewing device. At 810, the viewing device displays the image frame from the broadcast device. The viewing device may provide a viewer interaction overlay on the displayed image frame. At 811, using the viewer interaction overlay, the viewing device may capture viewer interactions and translate the interactions into interaction parameters. Then, at 812, the interaction parameters are sent to the broadcast device. At 813, broadcast device 802 may generate a subsequent scene from the media presentation and display the subsequent scene. In the subsequent scene, at 814, the viewer interaction parameters may be used to display and overlay the viewer interaction. Then, at 815, broadcast device 802 may generate an image frame for streaming to the viewing device, the image frame including the overlay interaction of the viewer. Including viewer interactions in subsequent image frames may allow viewers to understand the context of the broadcaster's actions, and may also understand that their interactions are received and displayed on the broadcaster. Information unique to the broadcaster, such as the number and name of viewers, may be omitted from the image generated by the broadcast overlay. At 816, subsequent image frames including the viewer interaction are sent to the viewer device. At 817, the viewer device may display the image frames including the viewer interaction, but omitting any broadcaster specific information.

[0039] Fig. 9is a diagram illustrating the transformation of user interactions on a viewer interface overlay into a broadcaster interface overlay on a media presentation according to one aspect of the present disclosure. In this implementation, a viewing device 910 displays a media presentation 902 of a viewer and an image frame 901 from a broadcaster. The viewer interface overlay can capture the viewer's interaction with the image frame. Here, the viewer interaction can be captured by a grid having X grid lines 904 and Y grid lines 903. Alternatively, the viewer interaction can be captured using pixel positions. The viewer interaction shown here is a square 905 drawn in the image frame around an area of ​​interest to the viewer using, for example, graphical interface hardware and / or software. In addition, the position of the viewer cursor 906 can be captured as a user interaction. The location and type of user interaction (referred to as viewer interaction parameters) can be captured using X coordinate 904 and Y coordinate 903 coordinates or pixel positions determined from the viewer interaction overlay. The viewer interaction parameters can be sent to the broadcaster 911 via a network 907. Since the image frames from the broadcaster are displayed in a smaller window or at a reduced resolution compared to the media presentation of the broadcaster, a transformation can be applied to change the viewer interaction parameters from the viewer device 910 to the broadcaster 911. The transformed viewer interaction parameters can then be used to display the user interaction 910. The transformation can include, for example, but not limited to, a grid or pixel coordinate offset, a coordinate multiplier, a divider, and the like. The transformation can be used to handle issues such as different resolutions or different viewing devices between the broadcaster and the viewer. In addition, the transformation can also address the need for offsets caused by off-grid clicks due to different aspect ratios at the broadcaster and viewing device. As shown, the X 909 and Y 908 grid or pixel coordinates have been transformed to fit the broadcaster interaction overlays 908, 909 of the broadcaster 911. In addition, the viewer cursor 906 is displayed through the broadcaster interface overlay.

[0040] In some implementations, the location of the user interaction may follow the location of the object within the image frame. Fig. 9 , the square 905 in the image on the broadcaster device and the viewer device can move with the ladder following the hole as the knight moves. To achieve this, certain information can be captured from the viewer side to take certain actions and passed to the broadcaster to maintain accuracy because the location of the selected object is precisely located on the broadcaster side. For example, the system can capture the location of the viewer cursor 906 and pass the pixel coordinates and color information (for the small pixel grid) obtained on the viewer side. Once the broadcaster 911 receives the cursor information, it will try to match the color and pixel coordinates within the threshold area to improve accuracy. When the predicted attempt of the coordinate radius color match cannot be met, the broadcaster device 911 can only use the pixel coordinates as a fallback for rendering.

[0041] Fig.101 is a block diagram depicting a system for viewer interaction according to one aspect of the present disclosure. By way of example and not limitation, according to aspects of the present disclosure, system 1000 may be an embedded system, a mobile phone, a personal computer, a tablet computer, a portable gaming device, a workstation, a gaming console, and the like.

[0042] The system 1000 typically includes a central processing unit (CPU) 1003 and a memory 1004. The system 1000 may also include well-known support functions 1006, which may communicate with other components of the system, for example, via a data bus 1005. Such support functions may include, but are not limited to, input / output (I / O) elements 1007, a power supply (P / S) 1011, a clock (CLK) 1012, and a cache 1013.

[0043] The system 1000 may include a mass storage device 1015, such as a disk drive, a CD-ROM drive, a flash memory, a solid state drive (SSD), a tape drive, etc., to provide non-volatile storage for programs and / or data. The system 1000 may also optionally include a user interface unit 1016 for facilitating interaction between the system 1000 and a user. The user interface 1016 may include a keyboard, a mouse, a joystick, a light pen, or other devices that can be used in conjunction with a graphical user interface (GUI). The system 1000 may also include a network interface 1014 for enabling the device to communicate with other devices via a network 1020. The network 1020 may be, for example, a local area network (LAN), a wide area network (such as the Internet), a personal area network (such as a Bluetooth network), or other types of networks. These components may be implemented in hardware, software, or firmware, or some combination of two or more of these.

[0044] The CPU 1003 may include one or more processor cores, for example, a single core, two cores, four cores, eight cores, or more cores. In some implementations, the CPU 1003 may include a GPU core or multiple cores of the same accelerated processing unit (APU). The memory 1004 may be in the form of an integrated circuit that provides addressable memory (e.g., random access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.). The main memory 1004 may include application programs 1021 that the processor 1003 uses to generate, for example, chat programs, Internet browsers, video games, movies, etc. The main memory 1004 may also include an operating system (OS) 1022, which may run on a hardware abstraction layer, such as Figure 5 As depicted. Figure 5As described, the operating system may include an interaction interface 1009 that uses and / or generates interaction parameters 1010 as a component in screen sharing. The interaction parameters 1010 may be stored in the memory 1004 or cache 1013. In addition, the memory 1004 may include a media presentation 1008, such as but not limited to a video game, a movie, a television program, a streamed video or a picture. The OS layer interaction interface and interaction parameters provide a viewer interaction that is agnostic to the application and the media presentation, as well as an interaction that reduces latency than current application layer user interaction procedures, because the interaction operates at a lower system layer than the media presentation or application.

[0045] According to aspects of the present disclosure, the processor 1003 can implement connection and viewer interaction, for example, as described above with respect to Figure 6 , 7 8, these methods may be loaded into memory 1004 as part of OS 1022. Processor 1003 may generate one or more interaction parameters 1010, which may be sent to a broadcaster over network 1020 using network interface 1014, and receive image frames of media presentation 1008 from the broadcaster. Network interface 1014 may also receive interaction parameters 1010 from a viewing device, and may stream image frames from media presentation 1008 to the viewing device.

[0046] The mass storage device 1015 may contain an application or program 1017 that is loaded into the main memory 1004 when processing begins on the application 1023. In addition, the mass storage device 1015 may contain data 1018 that the processor uses to load the OS 1009 during startup.

[0047] As used herein and generally understood by those skilled in the art, an application specific integrated circuit (ASIC) is customized for a specific use and is not intended for general use.

[0048] As used herein and generally understood by those skilled in the art, a field programmable gate array (FPGA) is an integrated circuit designed to be configured by a customer or designer after manufacture (hence "field programmable"). FPGA configuration is typically specified using a hardware description language (HDL) similar to that used for ASICs.

[0049] As used herein and generally understood by those skilled in the art, a system on a chip (SoC or SOC) is an integrated circuit (IC) that integrates all components of a computer or other electronic system into a single chip. It can contain digital, analog, mixed signal, and often radio frequency functions - all on a single chip substrate. Typical applications are in the field of embedded systems.

[0050] A typical SoC includes the following hardware components:

[0051] One or more processor cores (e.g., microcontroller, microprocessor, or digital signal processor (DSP) cores.

[0052] Memory blocks such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), and flash memory.

[0053] A timing source, such as an oscillator or a phase-locked loop.

[0054] Peripherals such as counter-timers, real-time timers, or power-on reset generators.

[0055] External interfaces, for example, industry standards such as Universal Serial Bus (USB), FireWire, Ethernet, Universal Asynchronous Receiver / Transmitter (USART), Serial Peripheral Interface (SPI) bus.

[0056] Analog interfaces include analog-to-digital converters (ADCs) and digital-to-analog converters (DACs).

[0057] Voltage regulators and power management circuits.

[0058] These components are connected by proprietary or industry standard buses. A direct memory access (DMA) controller routes data directly between the external interface and the memory, bypassing the processor core and thereby increasing the data throughput of the SoC.

[0059] A typical SoC includes both the hardware components described above and executable instructions (eg, software or firmware) that control the processor core, peripherals, and interfaces.

[0060] Various aspects of the present disclosure allow for enhanced remote interaction between video game players. The various implementations discussed herein are not limited to allowing players to share content on their respective screens. In addition, such implementations allow viewer interactions to be displayed directly on the broadcaster's media presentation with low latency.

[0061] Although the above is a complete description of the preferred embodiment of the present invention, various alternatives, modifications and equivalents may be used. Therefore, the scope of the present invention should not be determined with reference to the above description, but should instead be determined with reference to the appended claims and their full range of equivalents. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, Indefinite article "a" or "a kind" It refers to the quantity of one or more of the item following the article unless explicitly stated otherwise. The appended claims should not be read as including means-plus-function limitations unless such a limitation is explicitly recited in a given claim using the phrase "means for..."

Claims

1. A method for viewer interaction with streaming media on a broadcast device, comprising: a) displaying a scene from a media presentation, wherein the broadcaster includes an operating system level broadcaster interface overlay; b) generating an image frame from said scene from said media presentation; c) sending said image frames of said scene from said media presentation to a viewing device; d) receiving viewer interaction parameters; e) displaying the viewer interaction on a subsequent scene from the media presentation using the viewer interaction parameter with the operating system level broadcaster interface overlay.

2. The method for viewer interaction according to claim 1, further comprising: f) generating subsequent image frames from subsequent scenes from said media presentation, said media presentation including said viewer interaction in said operating system level broadcaster interface overlay; as well as g) sending said subsequent image frames including said viewer interaction to said viewing device.

3. The method for viewer interaction according to claim 1, wherein the viewer interaction parameters include interaction type and interaction location.

4. The method for viewer interaction of claim 3, wherein the viewer interaction parameters include a transformation for the interaction location, the transformation being used to transform the location of the viewer interaction from a resolution at a viewer device to a resolution at the broadcast device.

5. The method for viewer interaction of claim 1, wherein the viewer interaction parameters include data for displaying emoticons or emojis in the operating system level broadcaster interface overlay.

6. The method for viewer interaction of claim 1, wherein the viewer interaction parameters include data for displaying a free form line, rectangle, circle, or oval in the operating system level broadcaster interface overlay.

7. The method for viewer interaction of claim 1, wherein the viewer interaction parameters include data for displaying a highlight area, cursor, arrow, or text annotation in the operating system level broadcaster interface overlay.

8. The method for viewer interaction of claim 1, wherein the viewer interaction parameters include data for magnifying a portion of the scene of the media presentation in the operating system level broadcaster interface overlay.

9. The method for viewer interaction of claim 1, further comprising establishing viewer interaction limitations utilizing the operating system level broadcaster interface overlay.

10. The method for viewer interaction of claim 1, wherein the operating system level broadcaster interface overlay includes information not included in the image sent to the viewing device.

11. A system for viewer interaction with streaming media, comprising: processor; a memory coupled to the processor; non-transitory instructions embodied in the memory, which, when executed by the processor, cause the processor to implement a method for viewer interaction with streaming media on a broadcast device, the method comprising: a) displaying a scene from a media presentation, wherein the broadcaster includes an operating system level broadcaster interface overlay; b) generating an image frame from said scene from said media presentation; c) sending said image frames of said scene from said media presentation to a viewing device; d) receiving viewer interaction parameters; e) displaying the viewer interaction on a subsequent scene from the media presentation using the viewer interaction parameter with the operating system level broadcaster interface overlay.

12. A method for interacting with streaming media on a viewing device, comprising: a) Request to connect to the broadcasting device; b) receiving image frames from a media presentation from said broadcasting device; c) generating an operating system level viewer interface overlay on the image frame; d) displaying the image frame of the media presentation; e) generating viewer interaction parameters from viewer interactions with said operating system level viewer interface overlay; f) sending viewer interaction parameters to said broadcaster.

13. The method for interacting according to claim 12, further comprising: g) receiving a subsequent image frame including the viewer interaction from the broadcaster; and h) displaying the subsequent image frame with the viewer interaction.

14. The method for interacting according to claim 12, wherein the viewer interaction parameters include interaction type and interaction location.

15. A method for interacting according to claim 14, wherein the viewer interaction parameters include a transformation for the interaction location, the transformation being used to transform the location of the viewer interaction from a resolution at the viewing device to the resolution at the broadcast device.

16. The method for interacting according to claim 12, wherein the viewer interaction parameters include data for displaying emoticons or emojis in a broadcaster interface overlay.

17. The method for interacting according to claim 12, wherein the user interaction parameters include data for displaying a free form line, rectangle, circle or oval in a broadcaster interface overlay.

18. The method for interacting of claim 12, wherein the user interaction parameters include data for displaying a highlight area, cursor, arrow, or text annotation in a broadcaster interface overlay.

19. The method for interacting of claim 12, wherein the user interaction parameters include data for zooming in on a portion of the scene of the media presentation in a broadcaster interface overlay.

20. The method for interacting of claim 12, further comprising receiving viewer interaction restrictions from a broadcaster.

21. The method for interacting of claim 12, wherein generating viewer interaction parameters comprises capturing an interaction type and an interaction location from the location of the viewer interaction from an area of ​​a display screen of the viewing device.

22. A system for viewer interaction with streaming media, comprising: processor; a memory coupled to the processor; non-transitory instructions embodied in the memory, which, when executed by the processor, cause the processor to implement a method for interacting with streaming media on a broadcast device, the method comprising: a) Request to connect to the broadcasting device; b) receiving image frames of a media presentation from said broadcasting device; c) generating an operating system level viewer interface overlay on the image frame; d) displaying the image frame of the media presentation; e) generating viewer interaction parameters from viewer interactions with said operating system level viewer interface overlay; f) sending viewer interaction parameters to said broadcaster.

Images