Distributed video and graphics rendering system

Abstract

A receiving device receives a command stream from a base unit via a network, the command stream including drawing commands for generating a graphics frame of a user interface. The receiving device is connected to a display. The receiving device parses the command stream to determine the drawing commands and their arguments. The receiving device executes the drawing commands in a graphics engine of the receiving device to generate a graphics frame of the user interface. Execution includes executing a rendering function and interpolating coordinates for animation. The receiving device then outputs the graphics frame of the user interface for display.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of media devices; and more particularly, to distributed video and graphics rendering systems. Background Art

[0002] The modern living room media consumption experience, which mainly consists of media viewing and gaming, is achieved through game consoles, streaming devices, set-top boxes connected to TVs, and smart TVs. Inexpensive "casting sticks" that stream media or gaming content can be used to expand media playback and gaming devices. Casting sticks that stream media typically do so by receiving a media URL and streaming the media content from that URL. Casting sticks are used as an inexpensive means of providing services for another screen without incurring the cost of a full-featured entertainment device.

[0003] Cast sticks have audio / video processing and output capabilities, but they are typically designed with simple user interfaces. The inexpensive hardware of cast sticks is often not capable of running the technology stacks commonly used to implement user interfaces (e.g., Java, HTML5). As a result, the user experience of using a cast stick is often inferior to that of using a full-featured primary media consumption device. Summary of the Invention

[0004] A distributed video and graphics rendering system is described, comprising a base unit and one or more receiving devices. The base unit receives one or more events associated with the receiving devices, the receiving devices being communicatively coupled to the base unit via a network. The base unit determines how the one or more events inform or direct the output of content currently being rendered on the receiving devices. The base unit generates an instruction stream comprising drawing commands for generating a graphics frame for a user interface of the receiving device corresponding to the one or more events. Generating the instruction stream does not include executing rendering functions in a graphics engine and generating interpolated coordinates for animation. The base unit transmits the generated instruction stream to the receiving devices via the network.

[0005] A receiving device receives a command stream from a base unit via a network, the command stream including drawing commands for generating a graphics frame of a user interface. The receiving device is connected to a display. The receiving device parses the command stream to determine the drawing commands and their arguments. The receiving device executes the drawing commands in a graphics engine of the receiving device to generate a graphics frame of the user interface. This execution includes executing a rendering function and interpolating coordinates for animation. The receiving device then outputs the graphics frame of the user interface for display.

[0006] In an embodiment, the instruction stream is vector graphics-based and low-latency, and allows for the generation and interaction of a user interface that is indistinguishable from a user interface generated locally on the base unit. That is, even though the main application may be running on the base unit, it may appear to the user that each of the receiving devices is running the main application at its full capabilities.

[0007] In an embodiment, the base unit is relatively more powerful than the receiving device. Each receiving device is relatively cheap and relatively less powerful than the base unit. However, when used together as described herein, a user experience on the receiving device that matches the user experience on the base unit is provided. Most of the application logic and other computing overhead reside on the base unit, while a complex and fully functional user interface (UI) is represented by the receiving device. In addition, streaming different types of media, including video, music, and games, is provided. Therefore, a replica of the user experience can be provided to multiple displays while enjoying the cost advantage provided by the relatively cheap receiving device hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention may best be understood by referring to the following description and accompanying drawings which illustrate embodiments of the invention. In the drawings:

[0009] Figure 1 An example of a distributed video and graphics rendering system according to an embodiment is shown.

[0010] Figure 2 An example of a distributed video and graphics rendering system according to an embodiment is shown.

[0011] Figure 3 An example of a distributed video and graphics rendering system responsive to events according to an embodiment is shown.

[0012] Figure 4 An example of the architecture of an instruction stream dispatcher of a basic unit according to an embodiment is shown.

[0013] Figure 5 An example of the architecture of a distributed video and graphics rendering system according to an embodiment is shown, where a sender can operate in a dual mode including generating an instruction stream for a receiving device and performing direct rendering for output to a locally connected display.

[0014] Figure 6 An example architecture of an instruction stream processor of a receiving device according to an embodiment is shown.

[0015] Figure 7 is a flow chart illustrating exemplary operation of a distributed video and graphics rendering system according to an embodiment.

[0016] Figure 8An example of a distributed video and graphics rendering system according to an embodiment is shown, where a video stream is transmitted from a sender to a receiver.

[0017] Figure 9 is a flow chart illustrating exemplary operations for receiving and processing a video stream according to an embodiment.

[0018] Figure 10 is a flow chart illustrating exemplary operation of a distributed video and graphics rendering system driven by user input events received at a receiving device, according to an embodiment.

[0019] Figure 11 is a flowchart illustrating exemplary operation of a receiving device receiving and processing two concurrent streams according to an embodiment.

[0020] Figure 12 is a block diagram illustrating an exemplary architecture of a receiving device according to an embodiment.

[0021] Figure 13 is a block diagram illustrating an exemplary architecture of a basic unit according to an embodiment. DETAILED DESCRIPTION

[0022] A distributed video and graphics rendering system is described. The system includes a base unit and a collection of one or more receiving devices. The base unit sends a presentation stream to the receiving device, which in turn interprets and outputs graphics frames to a display. The base unit and / or the receiving device can receive user input (e.g., via a remote control, game controller, keyboard, mouse, gesture interpretation, or voice). The receiving device can send the user input to the base unit for processing.

[0023] In an embodiment, the presentation stream is an instruction stream that includes drawing commands, which the receiving device parses and executes to generate graphics frames. For example, the instruction stream includes drawing commands for a user interface and interactions with the user interface displayed by the receiving device. In such an embodiment, the receiving device includes an application to interpret the instruction stream and convert it into graphics frames for display. In such an embodiment, the base unit does not execute rendering functions through the graphics engine and does not generate interpolated coordinates for animation when preparing the instruction stream. This reduces the computational load on the base unit.

[0024] In an embodiment, the presentation stream is a video stream encoded in a standard video format, which the receiving device decodes and outputs to a display. For example, the video stream can be encoded using H.264, H.265, AV1, VP9, etc., and sent using Real-time Transport Protocol (RTP), Real-time Messaging Protocol (RTMP), etc. In addition to receiving the video stream from the base unit, or as an alternative, the receiving device can receive the video stream from a remote WAN server.

[0025] In an embodiment, a receiving device can receive at least two concurrent streams. For example, a receiving device can receive two instruction streams simultaneously, receive an instruction stream and a video stream simultaneously, and / or receive two video streams simultaneously. The receiving device can output each stream in the concurrent stream to the same display at the same time. For example, the receiving device can mix the concurrent streams together, or display them side by side or picture-in-picture. For example, a video stream can be received, and an instruction stream for navigating a user interface (UI) can be overlaid by the receiving device. As another example, a video stream representing a game can be received and displayed, and a video stream of another video can be displayed as a picture-in-picture. As another example, an instruction stream for a user interface can be received, and it can be shown side by side with a video of a video stream received from a remote server.

[0026] In an embodiment, the instruction stream is vector graphics-based and low-latency, and allows for the generation and interaction of a user interface that is indistinguishable from a user interface generated locally on the base unit. That is, even though the main application may be running on the base unit, it may appear to the user that each of the receiving devices is running the main application at its full capabilities.

[0027] In an embodiment, the distributed video and graphics rendering system is event-driven. In such an embodiment, the base unit and / or the receiving device can receive user input (e.g., controller input, voice input, image input) and process the user input accordingly. If the receiving device receives the user input, the user input is transmitted to the base unit, which in turn processes the user input to inform or guide the output of the content currently being rendered by the receiving device.

[0028] In an embodiment, the base unit is relatively more powerful than the receiving device. Each receiving device is relatively inexpensive and less powerful than the base unit. However, when used together as described herein, a user experience on the receiving device that matches the user experience on the base unit is provided. The application logic and other computing overhead largely reside on the base unit, while a complex and fully functional UI is presented by the receiving device. In addition, streaming of different types of media, including videos, music, and games, is provided. Therefore, a replica of the user experience can be provided to multiple displays while enjoying the cost advantages provided by the relatively inexpensive receiving device hardware.

[0029] Figure 1An example of a distributed video and graphics rendering system according to an embodiment is shown. The base unit 110 is a computing device, which may be a set-top box (STB), a streaming device, a game console, a smart TV, a smart speaker, or other computing device. In an embodiment, the base unit 110 drives the receiving device 120, for example, to generate commands representing a user interface to be displayed by the receiving device 120. In addition to driving the receiving device 120, the base unit 110 may also have independent functions (e.g., act as an STB, a streaming device, a game console, a smart TV, a smart speaker, etc.). The base unit 110 may perform its own independent functions while driving the receiving device 120. Although Figure 1 A single sink device is shown, but the base unit 110 can drive multiple sink devices simultaneously.

[0030] Base unit 110 includes a transmitter application 115 that generates and transmits a presentation stream 170 to receiving device 120. Presentation stream 170 may include an instruction stream comprising drawing commands and their arguments, which receiving device 120 parses and executes to generate graphics frames 180 to be displayed on display 130. The instruction stream may include commands for generating and interacting with a user interface displayed by receiving device 120. If the instruction stream is transmitted to receiving device 120, transmitter application 115 does not perform rendering functions via a graphics engine and does not generate interpolated coordinates for animation. Instead, receiving device 120 performs rendering functions and interpolated coordinates for animation. This reduces the computational load on transmitter application 115 on base unit 110.

[0031] Presentation stream 170 may include a video stream, wherein receiving device 120 decodes the video stream to generate graphic frames 180 to be displayed on display 130. The video stream may be encoded using H.264, H.265, AV1, VP9, etc., and transmitted using Real-time Transport Protocol (RTP), Real-time Messaging Protocol (RTMP), etc. If receiving and processing the video stream, receiving device 120 includes an application for decoding the video stream and outputting the decoded video stream. In an embodiment, presentation stream 170 may include both an instruction stream and a video stream.

[0032] The presentation stream 170 can be sent via a wireless network such as Wi-Fi, Bluetooth, or other wireless networks, or via a wired network such as Ethernet. The presentation stream 170 can be sent using the WebSocket protocol or other low-level network protocols. Typically, the base unit 110 and the receiving device 120 are located in the same physical structure, such as in the same home unit.

[0033] The receiving device 120 is connected to the display 130 and may be relatively small (e.g., it is typically smaller than the display 130 and the base unit 110). The receiving device 120 may take the form of a cast bar. The receiving device 120 may be plugged into the HDMI port of the display 130. The receiving device 120 is relatively inexpensive compared to the base unit 110. For example, the receiving device 120 may be implemented as a system on a chip (SoC). The receiving device 120 includes a receiver application 125 that receives the presentation stream 170, processes the presentation stream 170, and outputs the graphics frames 18 to be displayed on the display 130.

[0034] In an embodiment, in addition to or instead of receiving presentation stream 170 from base unit 110, receiving device 120 receives and processes presentation stream 172 received from remote WAN server 140. In such an embodiment, WAN server 140 includes a sender application 145 to generate and send presentation stream 172.

[0035] Figure 2 An example of a distributed video and graphics rendering system according to an embodiment is shown. Figure 2 As shown, the transmitter application 115 of the base unit 110 includes an instruction stream transmitter 210 and a video stream transmitter 215, and the receiver application 135 of the receiving device 120 includes an instruction stream processor 220 and a video stream processor 225. The instruction stream transmitter 210 generates an instruction stream 240 processed by the instruction stream processor 220 of the receiver application 135 and transmits it to the receiving device 120. The video stream transmitter 215 generates and transmits a video stream 245 processed by the video stream processor 225 of the receiver application 135. The transmitter application 145 of the WAN server 140 may include a similar instruction stream transmitter for generating and transmitting an instruction stream 250 and / or a similar video stream transmitter for generating and transmitting a video stream 255.

[0036] If the instruction stream is sent to the receiving device 120, the instruction stream sender 210 does not perform rendering functions via the graphics engine and does not generate interpolated coordinates for animation. Instead, the instruction stream processor 220 of the receiver application 135 of the receiving device 120 performs rendering functions and interpolated coordinates for animation. This reduces the computational load on the instruction stream sender 210 on the base unit 110. In an embodiment, the instruction stream 240 is sent via a protocol that enables lossless, keyframe-based interpolated remote rendering by the instruction stream processor 220.

[0037] In an embodiment, the instruction stream transmitter 210 is invisible to the main operating system of the base unit 110. For example, a hypervisor with two different operating systems can be used on the base unit 110, one operating system for the main operation of the base unit 110 and one operating system dedicated to driving the receiving device.

[0038] The instruction stream processor 220 receives and parses the instruction stream 240. The parsed instruction stream 240 includes drawing commands and their arguments. The instruction stream processor 220 executes the drawing commands and arguments to generate the graphics frame 180. The drawing commands are mapped to rendering functions in the graphics engine of the instruction stream processor 220. In one embodiment, the instruction stream processor 220 generates interpolated coordinates for the rendering functions.

[0039] The video stream processor 225 receives and decodes the video stream 245. The decoded video stream 245 is represented by the graphics frame 180.

[0040] In an embodiment, the distributed video and graphics rendering system is responsive to events received at a base unit and / or a receiving device. Figure 3 An example of a distributed video and graphics rendering system responsive to events according to an embodiment is shown. The base unit 110 can be connected to a user input device 310 and receive user input 315. The user input device 310 can be, for example, a controller, keyboard, mouse, external camera, or other input device that is wirelessly (e.g., Bluetooth, infrared, etc.) or wired to the base unit 110. In addition or alternatively, the base unit 110 can receive other user input 315 (e.g., voice input, gesture interpretation input, etc.). The user input 315 can inform or guide the output of the content currently being rendered by the receiving device 120.

[0041] Additionally or alternatively, the receiving device 120 may be connected to a user input device 320 and receive user input 325. The user input device 320 may be, for example, a controller, keyboard, mouse, external camera, or other input device connected wirelessly (e.g., Bluetooth, infrared, etc.) or wired to the receiving device 120. Additionally or alternatively, the receiving device 120 may receive other user input 325 (e.g., voice input, gesture interpretation input, etc.). In an embodiment, if the receiving device 120 receives the user input 325, the receiving device 120 sends a corresponding event 330 to the base unit 110 for further processing.

[0042] For example, if the input is audio input, a sound file can be recorded by the receiving device 120 and provided to the base unit 110. The base unit 110 can then process the audio input accordingly. As another example, if the input is a video captured by the receiving device 120, a video file can be recorded by the receiving device 120 and provided to the base unit 110. The base unit 110 can then process the video and act accordingly. For example, the base unit 110 can include capabilities for facial login and process the video to log the user into the application if the user is recognized. The receiving device 120 can capture audio and send the audio to the base unit 110 for processing, thereby allowing far-field voice control. The receiving device 120 can capture video and send the video to the base unit 110 for processing, allowing video features such as facial login.

[0043] The base unit 110 processes events (received from the receiving device 120 and / or directly from user input 315) to inform or direct the output of content currently being rendered on the receiving device 120. For example, events are processed by the sender application 115, and their effects are reflected in the presentation stream 170. Events are specific to the applications executing on the base unit 110 and / or receiving device 120. For example, if the application executing on the base unit 110 is a media application (e.g., a television application, a streaming video application, etc.), events are specific to the media application, such as events for the user interface of the media application (e.g., navigation controls, media asset selections, etc.).

[0044] In an embodiment, the user input device 310 is permanently paired with the base unit 110. If the computer vision system of the base unit 110 sees the user input device 310 in its field of view, the base unit 110 assumes that input events from the user input device 310 are intended for the base unit 110 and processes the input events accordingly. However, if the computer vision system of the base unit 110 does not see the user input device 310 in its field of view, the base unit 110 assumes that the user input device 310 is physically located near the receiving device 120 and assumes that input events received from the user input device 310 are intended for the receiving device 120 and processes the input events accordingly (e.g., processes the events to drive the presentation model of the receiving device 120).

[0045] Figure 4An example of the architecture of the instruction stream transmitter 210 according to an embodiment is shown. The base unit 110 executes the instruction stream transmitter 210. The WAN server 140 may also execute a similar instruction stream transmitter. The instruction stream transmitter 210 is logically divided into an input section, a main / rendering section, and an output section. The input section processes input such as user input 410. User input 410 may be received from the user input device 310 or other user input (e.g., voice input, gesture interpretation input, etc.).

[0046] The application logic 412 processes the user input 410 to determine one or more events 414 corresponding to the user input 410. The events 414 may inform or direct the output of content currently being rendered on the receiving device 120. For example, the events 414 may be for game controller input (button presses, directional movements, etc.), navigation input (e.g., navigating a menu), power input, volume input, etc. The application logic 412 and events 414 are specific to the application executing on the base unit 110 and / or receiving device 120. For example, if the application executing on the base unit 110 and / or receiving device 120 is a media application (e.g., a television application, a streaming video application, etc.), the application logic 412 and events 414 are specific to the media application, such as the media application's user interface and the navigation controls of the user interface.

[0047] Events 414 are received by UI views 421A-421N. Each view 421 is a graphics plane. Multiple planes form a composite "window" or "overall view" presented to the user. Views 421A-421N are individually positioned, animated, and ordered, and form a containment hierarchy. Key handling function 416 handles events that may force an update or re-rendering of the graphical user interface by a rendering function. Rendering function 422 is called from a drawable 420 to determine the position of the graphics primitives. Since interpolation will not be performed on the sender application if an instruction stream is sent (interpolation will be performed on the receiving application (e.g., in the receiving device)), relatively few renderings are performed per second (e.g., 3 renderings per second instead of 60 renderings per second).

[0048] The rendering functionality 422 determines the location of graphics primitives using a plurality of rendering functions 440. Because graphics primitives are not drawn by the instruction stream sender (they are drawn by the receiver), the results of the rendering functions 440 are not processed by the graphics engine of the base unit 110, but are packaged as commands by the instruction stream module 460 and sent in the instruction stream 240.

[0049] The rendering function may include one or more of the following functions: draw_cache function, render_rect function, clear function, set_background function, initialize function, set_geometry function, set_scroll_offset function, animate function, set_image function, load_image function, render_text function, stroke_rect function, frame_done function, and destroy function.

[0050] The draw_cache function sets the size of the graphics plane. It takes as parameters a reference to the cache object, the position and dimensions in screen space (e.g., int x, y, w, h), and a boolean indicating whether the object is animatable.

[0051] The render_rect function renders a filled rectangle at a certain position and size using a given style. The parameters of the render_rect function can include the position and dimensions in screen space (e.g., int x, y, w, h), a string style, and a Boolean value indicating whether the object is animatable.

[0052] The clear function clears the graphics plane (e.g., to transparent black). It can take a position and dimensions in screen space (e.g., int x, y, w, h) as parameters.

[0053] The set_background function sets the size of the background graphics plane. It takes as a parameter a string specifying the background, such as a solid color, gradient, or particle system specifier.

[0054] The initialization function initializes a new graphics frame and passes the cache size to synchronize the client and server's understanding of the least recently used (LRU) cache. It takes as parameters a string version identifier indicating the requested renderer version, a boolean indicating whether planes should be rendered exclusively, an integer for the size of the image cache, and an integer for the size of the text cache.

[0055] The set_geometry function sets the size of the graphics plane. It takes a position and dimensions in screen space (e.g., int x, y, w, h) as parameters.

[0056] The set_scroll_offset function sets the new scroll offset of an animatable object at the last keyframe. It takes as parameters the position (e.g., int x, y) of the new scroll offset to be notified to the animatable object.

[0057] The animate function triggers the animation of an animatable object in the current frame. This command specifies the current frame as a keyframe. It takes as parameters a position (e.g., int x, y) that anchors the animatable object's current scroll offset and a string specifying a timing function. The timing function quantizes the acceleration and velocity of the three phases of the animation (acceleration, cruising, and deceleration).

[0058] The set_image function sets the image from the given binary blob and caches it in the cache object for later use by the draw_cache function. Its parameters include a reference to the cache object, the width and height of the image representing it (e.g., int w, h), and the blob with the data for the image to be set.

[0059] The load_image function loads an image from a given URL and caches it in a cache object for later use by the draw_cache function. Its parameters include a reference to the cache object and a string URL of the image to be loaded.

[0060] The render_text function renders a text string at a certain position and size using a given style. Its parameters may include the position and size in screen space (e.g., int x, y, w, h), the string text, and the string style.

[0061] The stroke_rect function renders an outlined rectangle at a certain position and size using a given style. Its parameters include the position and dimensions in screen space (e.g., int x, y, w, h), a string style, and a boolean indicating whether the object is animatable.

[0062] The frame_done function declares the end of the instruction flow for the current frame. The frame_done function does not have any parameters.

[0063] The destroy function destroys / releases the graphics plane and all associated caches. The destroy function does not have any parameters.

[0064] As described above, the results of the rendering function 440 are processed and packaged into commands by the instruction stream module 460. In one embodiment, the instruction stream 240 is sent using the network socket protocol using a network socket 470. In such an embodiment, the instruction stream module 460 includes a network socket stream module 462, which includes a send function 464 and a send binary function 466. The send function 464 is used to send at least some of the drawing commands as text strings. The send binary function 466 is used to send at least some of the drawing commands as binary.

[0065] Figure 5 An example of the architecture of a distributed video and graphics rendering system on the sender side (eg, executing on base unit 110 ) is shown. Figure 5 The architecture in illustrates the ability to operate in a dual mode, in which the base unit 110 can generate an instruction stream for the sink device and the base unit 110 can perform direct rendering for output to the locally connected display, depending in part on whether the output is to the sink device or a locally connected display. Figure 5 The illustrated architecture can be used by the base unit 110. The instruction stream sender 210 is logically divided into an input section, a main / rendering section, and an output section. The input section processes input such as user input 510. The user input 510 can be received from the user input device 310 or other user input (e.g., voice input, gesture interpretation input, etc.).

[0066] Application logic 512 processes user input 510 to determine one or more events 514 corresponding to user input 510. Events 514 may inform or direct the output of content currently being rendered on receiving device 120. For example, events 514 may be for game controller input (button presses, directional movements, etc.), navigation input (e.g., navigating a menu), power input, volume input, etc., similar to events 414. Application logic 512 and events 514 are specific to applications executing on base unit 110 and / or receiving device 120, similar to application logic 412 and events 414.

[0067] Events 514 are received by UI views 521A-521N. Each view 521 is a graphics plane. Multiple planes form a composite "window" or "overall view" presented to the user. Views 521A-521N are individually positioned, animated, and ordered, and form a containment hierarchy. Key handling function 516 handles events that may force an update or re-rendering of the graphical user interface by the rendering function, including determining whether to generate 518 an instruction stream or perform direct rendering. The state about the object can be accessed to determine whether to generate an instruction stream. If an instruction stream is to be generated, the rendering function 522 is called from the drawable object 520. Since interpolation will not be performed on the sender application if an instruction stream is sent (interpolation will be performed on the receiving application (e.g., in the receiving device)), relatively few renderings are performed per second (e.g., 3 renderings per second instead of 60 renderings per second).

[0068] If no instruction stream is generated (e.g., direct rendering is to be performed), animation can be performed by a timer-driven physics module that calls a rendering function with interpolated drawing coordinates. Thus, if direct rendering is to be performed, an animation generator 530 using a physics module 532 is called to generate a plurality of intermediate graphics commands using a timer 534 on a predefined schedule and provide them to a rendering function 522 called from a drawable object 520. For example, 60 frames per second can be rendered.

[0069] The renderer 585 includes a plurality of rendering functions 540. If an instruction stream is to be generated, the results of the functions do not go to the graphics engine 555, but rather they are packaged as commands by the instruction stream module 560 and sent in the instruction stream 240. If an instruction stream is not to be generated (e.g., direct rendering is performed instead), the results of the rendering functions 540 are processed by the graphics engine 555 and output to the display 580. The rendering functions 540 may be the same as the rendering functions 440.

[0070] As described above, if the event is related to an instruction stream, the result of the rendering function 540 is processed by the instruction stream module 560 and packaged as a command. In one embodiment, the instruction stream 240 is sent using the network socket protocol using a network socket 570. In such an embodiment, the instruction stream module 560 includes a network socket stream module 562, which includes a send function 564 and a binary send function 566. The send function 564 is used to send at least some of the drawing commands as text strings. The binary send function 566 is used to send at least some of the drawing commands as binary.

[0071] Figure 6 An example architecture of the instruction stream processor 220 according to an embodiment is shown. The receiving device 120 executes the instruction stream processor 220. The instruction stream processor 220 is logically divided into an input section, a main / rendering section, and a synchronization section. The input section processes input, such as the instruction stream 240 received at the instruction stream receiver 610 via the network socket 608.

[0072] The instruction stream processor 220 parses the instruction stream 240 to determine the commands and their arguments included in the instruction stream 240. The parser 615 parses each received message 618. Each plane, acting as a graphics view, has a receive buffer A 620 and an execute buffer B 622. The parser 615 parses the instruction stream 240 and writes the commands and arguments to receive buffer A 620. When the end of a frame is received (e.g., a frame complete command is received), the plane is delimited and receive buffer A 620 is swapped with execute buffer B 622, which is arbitrated by the physics module 626. The drawing function 619 is then triggered (or periodically with the display refresh cycle), and the drawing function 619 executes 630 the commands in execute buffer B 622. Most of these commands are directly mapped to the rendering function 640 in the graphics engine 672. Some commands that change state or call functions other than the renderer 685 are not mapped to the rendering function 640, such as the frame complete command and the set scroll offset command, which are consumed by the current graphics plane.

[0073] Some commands can be processed out-of-band; they can be processed immediately rather than being written to receive buffer A 620 for delayed execution. Such commands may include an animate command for animating graphics on a graphics plane, a load image command for loading an image from a URL into a low-level graphics representation (e.g., a texture), and a set image command for setting image data into a texture.

[0074] Animation commands 628 from animation plane 624 trigger physics module 626, which injects itself between command execution 630 and rendering function 640 and generates interpolated coordinates for rendering function 640 (using lambdas 632) for the animation duration specified in the animation command, while respecting the timing function specified in the animation command. The use of lambdas 632 (e.g., dx / dy) modulates the coordinates in execution buffer B 622 to rendering function 640 (which operates in absolute coordinates) to indicate to renderer 685 what to draw and where. Animation commands effectively synthesize animations in renderer 685 that would otherwise be executed on base unit 110. While active, physics module 626 prevents receive buffer A 620 from being swapped with execution buffer B 622 if an animation is currently in progress. However, if a new animation command arrives out of band, this arbitration is relaxed, potentially causing the internal animation state to extend by the combined duration of the current and incoming animations, leading to swapping in new buffers while an animation is active. In this case, the timing functions of the current animation and the old animation may also be merged to form a new single uninterrupted animation.

[0075] The renderer 685 is used to generate graphics frames for display on the display 130. The renderer 685 includes a set image function 662 and a load image function 664, a rendering function 640, and a graphics engine 672. The renderer 685 can be implemented on native graphics, threading, and decoding APIs. The graphics engine 672 includes a decode image function 676, among other functions. The decode image function 676 is called by the load image function 664 and generates textures from data formatted according to the image format. The graphics engine 672 includes a synchronization function 678 (e.g., VSYNC), which calls a notification function 682 to signal the master / drawing segment to check for a state change 616 that requires redrawing the graphics plane (e.g., an unrendered frame or a frame that needs to be rerendered). If so, a draw function 619 is triggered, which executes 630 commands to execute buffer B 622.

[0076] In an embodiment, the receiving device 120 performs an extension function that extends the animation to produce a smooth animation that appears continuous if an animation command is received while another animation is currently executing and the new animation is in the same direction as the currently executing animation. The extension function merges the timing function of the next animation (specified in the most recently received animation command) with the timing function of the current animation (specified in the animation command currently being processed). The timing function quantifies the acceleration and speed of the three phases of the animation: acceleration, cruising, and deceleration. The merging effectively extends the animation envelope and provides seamless animation execution. If the current animation is in the cruising phase, the merged animation continues cruising (skipping the declaration and acceleration phases of the current and next animations). If the cruising speeds of the current and next animations are different, the cruising speed is adjusted according to the acceleration curve of the next animation if the speed is higher, or according to the deceleration curve of the current animation if the speed of the next animation is lower. If the current animation is in the acceleration phase, the cruising phases are linked together as described above. If the current animation is in the deceleration phase, the merged animation accelerates back to cruising speed according to the corresponding acceleration and cruising speed curves of the next animation. Any number of extensions can be performed, effectively allowing infinite animations (animations with no predefined end).

[0077] Figure 7is a flow chart illustrating exemplary operations of a distributed video and graphics rendering system according to an embodiment. In operation 710, a receiving device 120 receives an instruction stream from a base unit 110, the instruction stream comprising drawing commands for generating a graphics frame for a user interface. The receiving device 120 is connected to a display 130. The user interface may be for a media application, such as a television application, a streaming video application, a video-on-demand application, a gaming application, a music application, a photo application; a social networking application; a smart home or smart office application; or any other application that utilizes a user interface. The instruction stream is received across a network (which may be a wireless network, such as Wi-Fi or Bluetooth), or via a wired network (such as Ethernet). The instruction stream may be received at the receiving device 120 using a network socket protocol or other low-level network protocol. At least some of the drawing commands may be received as text strings. Some drawing commands may be received as binary packets. The instruction stream may be lossless. That is, the receiving device 120 is able to use the information in the instruction stream to perfectly reconstruct the original data representing the user interface.

[0078] Next, the receiving device 120 parses the instruction stream to determine the drawing command and its arguments in operation 715. In an embodiment, the receiving device 120 writes the parsed command and its arguments into a receiving buffer.

[0079] As examples, drawing commands may include one or more of the following:

[0080] - draw cache command that sets the size of the graphics plane using arguments including a reference to a cache object, the position and dimensions of the graphics plane in screen space (e.g., int x, y, w, h values), and a Boolean indicating whether the object is animatable (this command corresponds to the draw_cache function described previously);

[0081] - Render rectangle command, which renders a filled rectangle at a certain position and size using a given style, where the arguments indicate the position and dimensions of the filled rectangle in screen space (e.g., int x, y, w, h values), a string style, and a boolean indicating whether the object is animatable (this command corresponds to the render_rect function described previously);

[0082] - a clear command for clearing the graphics plane (e.g., to transparent black), where the arguments indicate the position and dimensions of the clear in screen space (e.g., int x, y, w, h) (this command corresponds to the clear function described previously);

[0083] - Set Background command, used to set the size of the background's graphics plane, where the arguments specify the background, such as a solid color, gradient, and particle system specifier (this command corresponds to the set_background function described previously);

[0084] - an initialization command for initializing a new graphics frame and transmitting a cache size to synchronize the understanding of the LRU cache between the receiving device 120 and the base unit 110, wherein the arguments include a string version identifier indicating the requested renderer version, a Boolean value indicating whether a plane should be rendered exclusively, an integer indicating the size of the image cache, and an integer indicating the size of the text cache (this command corresponds to an initialization function);

[0085] - set_geometry command, used to set the size of the graphics plane, where the arguments include the position and dimensions in screen space (e.g., int x, y, w, h values) (this command corresponds to the set_geometry function described previously);

[0086] - Set scroll offset command, used to set the new scroll offset of the animatable object in the last keyframe, where the arguments include the position of the animatable object (e.g., int x, y values) with the new scroll offset information (this command corresponds to the set_scroll_offset function described previously);

[0087] - an animate command for triggering animation of an animatable object in the current frame, where the arguments include a location (e.g., int x, y values) anchoring the current scroll offset of the animatable object and a string specifying a timing function (this command corresponds to the animate function described previously);

[0088] - set image command, used to set an image from a given binary blob and cache it in a cache object for later call by the draw_cache function, where the arguments include a reference to the cache object, the width and height of the image representing it (e.g., int w, h values), and the blob with the data of the image to be set (this command corresponds to the set_image function described previously);

[0089] - load_image command, used to load an image from a given URL and cache it in a cache object for later call by the draw_cache function, where the arguments include a reference to the cache object and a string URL of the image to be loaded (this command corresponds to the load_image function described previously);

[0090] - render_text command, which renders a text string at a certain position and size using a given style, where the arguments include the position and dimensions of the text in screen space (e.g., int x, y, w, h), the string text, and the string style (this command corresponds to the render_text function described previously);

[0091] - Render Outlined Rectangle command, which renders an outlined rectangle at a certain position and size using a given style, where the arguments include the position and dimensions in screen space (e.g., int x, y, w, h values), a string style, and a boolean indicating whether the object is animatable (this command corresponds to the stroke_rect function described previously);

[0092] - a frame done command, used to declare the end of the instruction stream for the current frame (this command may not have any arguments, and corresponds to the frame_done function described previously); and

[0093] - A destroy command, used to destroy / release the graphics plane and all associated caches (this command may have no arguments and corresponds to the destroy function described previously).

[0094] Next, at operation 720, receiving device 120 executes the drawing commands in its graphics engine to generate a graphics frame for the user interface, including executing a rendering function and interpolating coordinates for animation. In one embodiment, execution includes swapping commands and arguments from a receive buffer to an execution buffer upon receiving the end of a frame (frame_done command), and then triggering a drawing function (or periodically with the refresh cycle of display 130) to execute the commands in the execution buffer. These commands are directly mapped to a rendering function in the graphics engine, which generates a graphics frame for output to display 130. Alternatively, or in addition, executing the drawing commands includes executing an animation command, which triggers a physics module that injects itself between the command execution and the rendering function to generate interpolated coordinates for the rendering function within the animation duration specified in the animation command, while adhering to the timing function specified in the animation command. Executing the animation command on receiving device 120 synthesizes the animation on receiving device 120, rather than having the base unit 110 perform the animation. The receiving device 120 may use native graphics, threading, and decoding APIs to execute the drawing commands.

[0095] Next, at operation 725, the sink device 120 outputs a graphic frame of the user interface for display on the display 130. The graphic frame may be output via an HDMI connection or other video connection connecting the sink device 120 and the display 130.

[0096] Figure 7 1 and 2. Operations performed at receiving device 120 when receiving and processing an instruction stream are shown. Instead of, or in addition to, receiving and processing an instruction stream, receiving device 120 may receive and process a video stream. Figure 8 An example of a distributed video and graphics rendering system according to an embodiment is shown, in which a video stream is transmitted from a transmitter to a receiver. The video stream transmitter 215 of the base unit 110 includes a video encoder 810. The video stream processor 225 of the receiving device 120 includes a video decoder 815. The video encoder 810 encodes a video stream 245, which can be a standard video format, such as H.264, H.265, AV1, VP9, etc. The video stream transmitter 215 sends the video stream 245 to the receiving device 120. The transmission can use RTP, RTMP or other protocols. The video stream processor 225 receives the video stream 245 and uses the video decoder 815 to decode the encoded video. The decoded video is output to the display 130 as a graphics frame. Although Figure 8 The receiving device 120 is shown receiving the video stream from the base unit 110, but alternatively or in addition, the receiving device 120 may receive the video stream from a remote server such as the WAN server 140. The WAN server 140 may include a similar video encoder, or may send already encoded video. Figure 8 The base unit 110 is shown encoding the video stream, but in some cases the base unit 110 sends an encoded video file to the receiving device 120 without encoding the video.

[0097] Figure 9 9 is a flow chart illustrating exemplary operations for receiving and processing a video stream according to an embodiment. In operation 910, the receiving device 120 receives an encoded video stream. The encoded video stream may be received from the base unit 110 or from a remote WAN server 140. The encoded video stream may be encoded in a standard video format (e.g., H.264, H.265, AV1, VP9, etc.) and sent using RTP, RTMP, or other protocols. Next, in operation 915, the receiving device 120 decodes the encoded video stream. Next, in operation 920, the receiving device 120 outputs the decoded video stream to the display 130.

[0098] As mentioned previously, the distributed video and graphics rendering system can be driven by user input events. Figure 10 is a flow chart illustrating exemplary operation of a distributed video and graphics rendering system driven by user input events received at a receiving device, according to an embodiment.

[0099] At operation 1010, the receiving device 120 receives user input. The user input may be received from a user input device 320 connected to the receiving device 120. Alternatively, or in addition, the user input may be received via voice input and / or gesture interpretation. Example user inputs include navigation input (e.g., navigating a menu displayed on the display 130 via the receiving device 120), media asset selection, game controller input (button presses, directional movements, etc.), and the like.

[0100] Next, in operation 1015, the receiving device 120 transmits one or more events representing the user input to the base unit 110, and in operation 1020, the base unit 110 receives the one or more events from the receiving device 120. Figure 10 Base unit 110 is shown receiving events from receiving device 120, but alternatively or in addition, base unit 110 receives events from user input at base unit 110 (e.g., from user input device 310 or through other user input such as voice input or gesture input received at base unit 110).

[0101] Next, at operation 1025, the base unit 110 determines how the one or more events inform or direct the output of the content currently being rendered on the receiving device 120. For example, the base unit 110 determines how to indicate to the receiving device 120 a change / update to the content being displayed by the receiving device 120. The event is specific to the application executing on the receiving device 120. For example, if the application is a media application and the event is scrolling through a list of media assets playable by the media application, the base unit 110 determines what content should be shown on the media application as a result of the scrolling.

[0102] Next, in operation 1030, base unit 110 generates an instruction stream including drawing commands, and drawing commands are used to generate a graphics frame reflecting the one or more events. When generating the instruction stream, base unit 110 does not execute the rendering function in the graphics engine and does not generate interpolation coordinates for animation. On the contrary, the rendering function and any interpolation coordinates for animation will be executed at the receiving device 120. Base unit 110 executes one or more functions to generate one or more commands included in the instruction stream to be executed by the receiving device 120 to reflect the one or more events. The one or more functions (previously described) can include one or more of the following: draw_cache function, render_rect function, clear function, begin_frame instruction, set_background function, initialization function, set_geometry function, set_scroll_offset function, animation function, set_image function, load_image function, render_text function, stroke_rect function, frame_done function, and destruction function. The command can be configured to be sent as a text string and / or binary command.

[0103] Next, at operation 1035, the base unit 110 sends the instruction stream to the receiving device 120. The instruction stream can be lossless. That is, the receiving device 120 can use the information in the instruction stream to perfectly reconstruct the original data representing the changes to the user interface. The instruction stream can be sent using a network socket protocol. The receiving device 120 receives the instruction stream at operation 1040.

[0104] Despite Figure 10 Although not shown in the figure, the base unit 110 may include at least two operating systems, one of which performs event processing and instruction stream generation to drive the receiving device 120, and another operating system that performs tasks unrelated to processing events and user interface commands for the receiving device 120. For example, the other operating system may be the main operating system of the base unit 110. The separation of operating systems can be achieved through the use of a hypervisor. In another embodiment, the sender application 115 on the base unit 110 runs invisibly as a service in the operating system of the base unit 110 or in a virtualized headless operating system.

[0105] In operation 1040, the receiving device 120 receives an instruction stream from the base unit 110, the instruction stream including drawing commands for generating a graphics frame of the user interface to reflect the user input. Next, in operation 1045, the receiving device 120 parses the instruction stream to determine the drawing commands and their arguments. Operation 1045 is similar to operation 715 described previously and includes examples of drawing commands that may be included in the instruction stream. Next, in operation 1050, the receiving device 120 executes the drawing commands in its graphics engine to generate the graphics frame of the user interface, including executing rendering functions and interpolating coordinates for animation. Operation 1050 is similar to operation 720 described previously. Next, in operation 1055, the receiving device 120 outputs the graphics frame for display on the display 130. Operation 1055 is similar to operation 725 described previously.

[0106] In an embodiment, the receiving device 120 can receive at least two concurrent streams. For example, the receiving device can receive two instruction streams at the same time, receive an instruction stream and a video stream at the same time, and / or receive two video streams at the same time. The receiving device can output each of the concurrent streams to the same display at the same time. For example, the receiving device can mix the concurrent streams together, or display them side by side or picture-in-picture. For example, a video stream can be received, and an instruction stream for navigating a UI can be overlaid by the receiving device. As another example, a video stream representing a game can be received and displayed, and a video stream of another video can be displayed as a picture-in-picture. As another example, an instruction stream for a user interface can be received and shown side by side with the video of a video stream received from a remote server.

[0107] Figure 11 1 is a flow chart illustrating exemplary operations of a receiving device receiving and processing two concurrent streams according to an embodiment. In operation 1110, receiving device 120 receives a first presentation stream and a second presentation stream. The first presentation stream and the second presentation stream may both be instruction streams, both be video streams, or one instruction stream and one video stream. The first presentation stream and the second presentation stream may both be received from base unit 110 and from WAN server 140, or one may be received from base unit 110 and the other from WAN server 140.

[0108] Next, in operation 1115, the receiving device 120 processes the first presentation stream and the second presentation stream. The receiving device 120 processes each stream in the presentation stream differently depending on whether the presentation stream is a video stream or an instruction stream. For example, if the presentation stream is an instruction stream, the receiving device 120 parses the instruction stream to determine drawing commands and arguments, and executes these commands to generate graphics frames, as previously described herein. If the presentation stream is a video stream, the receiving device 120 decodes the video stream and outputs the decoded video, as previously described herein.

[0109] Next, in operation 1120, the receiving device 120 outputs the first presentation stream and the second presentation stream to the display 130. The receiving device 120 can mix the concurrent streams or display them side by side or picture-in-picture. For example, a video stream can be received, and an instruction stream for navigating a UI can be overlaid by the receiving device 120 to be displayed on the display 130. As another example, a video stream representing a game can be received and displayed, and a video stream of another video can be displayed as a picture-in-picture. As another example, an instruction stream for a user interface can be received and shown side by side with the video of a video stream received from a remote server.

[0110] Although the sink device has been described as being connected to an external display for display, the embodiments described herein also apply if the sink device has its own internal display. That is, if the sink device has its own internal display, the graphics frames can be displayed on the sink device itself rather than outputting the graphics frames to be displayed on the external display.

[0111] Figure 12 1 is a block diagram illustrating an exemplary architecture of a receiving device 120 according to an embodiment. The receiving device 120 includes one or more microphones 1210 for voice processing, one or more cameras 1215 for image processing, a display connection 1220 (e.g., an HDMI connection, a DisplayPort connection, etc.), one or more wireless transceivers 1225 (e.g., a Bluetooth transceiver, a Wi-Fi transceiver, etc.), a processing system 1230, one or more USB inputs 1235, non-volatile memory 1240 (e.g., a hard drive, flash memory, phase change memory (PCM), etc.), and volatile memory 1245 (e.g., random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM)). The processing system 1230 includes a central processing unit (CPU) and a graphics processing unit (GPU) (integrated graphics unit or a separate GPU). The processing system 1230 can retrieve instructions from the volatile memory 1245 and / or the non-volatile memory 1240 and execute the instructions to perform the operations described herein, including executing commands to generate graphics frames as described herein. Figure 12 The components of the receiving device 120 may be coupled together using one or more buses. One or more of these components may be optional. For example, in some embodiments, the receiving device does not include a microphone and / or a camera. In addition, the receiving device 120 may include other components. For example, in an embodiment, the receiving device 120 may include a wired transceiver (e.g., an Ethernet transceiver) for wired communication. The receiving device 120 may be implemented on a SoC.

[0112] Figure 13is a block diagram illustrating an exemplary architecture of a base unit 110 according to an embodiment. The base unit 110 includes one or more microphones 1310 for voice processing, one or more cameras 1315 for image processing, a display connection 1320 (e.g., an HDMI connection, a DisplayPort connection, etc.), one or more wireless transceivers 1325 (e.g., a Bluetooth transceiver, a Wi-Fi transceiver, etc.), a processing system 1330, one or more USB inputs 1335, non-volatile memory 1340 (e.g., a hard drive, flash memory, phase change memory (PCM), etc.), and volatile memory 1345 (e.g., random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM)). The processing system 1330 includes a central processing unit (CPU) and a graphics processing unit (GPU) (an integrated graphics unit or a separate GPU). The processing system 1330 may retrieve instructions from the volatile memory 1345 and / or the non-volatile memory 1340 and execute the instructions to perform the operations described herein, including receiving events and generating presentation streams, as part of the distributed video and graphics rendering system described herein. Figure 13 The components of the base unit 110 may be coupled together using one or more buses. One or more of these components may be optional. For example, in some embodiments, the base unit 110 does not include a microphone and / or a camera. In addition, the base unit 110 may include additional components. For example, in an embodiment, the base unit 110 may include a wired transceiver (e.g., an Ethernet transceiver) for wired communication. When compared to the receiving device 120, the base unit 110 is relatively more powerful (e.g., a more powerful processing system, larger and faster volatile memory and / or non-volatile memory).

[0113] The techniques shown in the accompanying figures can be implemented using code and data stored and executed on one or more computing devices (e.g., base units, receiving devices, WAN servers, etc.). Such computing devices use machine-readable media (e.g., non-transitory machine-readable storage media (e.g., magnetic disks, optical disks, random access memory, read-only memory, flash memory devices, phase-change memory)) and machine-readable communication media (e.g., electrical, optical, acoustic, or other forms of propagated signals, such as carrier waves, infrared signals, digital signals, etc.) to store and communicate (internally and / or to other computing devices over a network). In addition, such computing devices typically include a collection of one or more processors coupled to one or more other components, such as one or more storage devices, user input / output devices (e.g., keyboards, touch screens, and / or displays), and network connections. The processors and other components are typically coupled via one or more buses and bridges (also known as bus controllers). The storage devices and the signals carrying network traffic represent one or more machine-readable storage media and machine-readable communication media, respectively. Thus, the storage devices of a given computing device typically store code and / or data for execution on the collection of one or more processors of that computing device. Of course, one or more parts of the embodiments may be implemented using different combinations of software, firmware and / or hardware.

[0114] The operations in the flowcharts are described with reference to the exemplary embodiments of other figures. However, it should be understood that the operations of the flowcharts may be performed by embodiments other than those discussed with reference to the other figures, and that the embodiments discussed with reference to these other figures may perform operations different from those discussed with reference to the flowcharts.

[0115] Although the flowcharts in the figures illustrate a particular order of operations performed by particular embodiments, such order is exemplary (eg, alternative embodiments may perform operations in a different order, combine particular operations, overlap particular operations, etc.).

[0116] Generally, unless clearly given and / or different meanings are implied from the context in which it is used, all terms used herein will be interpreted according to their ordinary meanings in the relevant technical field. Unless clearly stated otherwise, all references to element, device, assembly, means, step, etc. will be openly interpreted as referring to at least one instance of this element, device, assembly, means, step, etc. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless steps are clearly described as being after or before another step and / or wherein implicit steps must be after or before another step. Where appropriate, any feature of any embodiment disclosed herein may be applied to any other embodiment. Equally, any advantage of any embodiment may be applied to any other embodiment, and vice versa. From the foregoing description, other purposes, features and advantages of the attached embodiments will be apparent.

[0117] Although the present invention has been described in terms of several embodiments, those skilled in the art will recognize that the present invention is not limited to the embodiments described and that modifications and variations can be made to practice the present invention within the spirit and scope of the appended claims. Therefore, this description should be considered illustrative rather than restrictive.

Claims

1. A method performed at a receiving device (120) of a distributed video and graphics rendering system, comprising: receiving ( 710 ) a first portion of an instruction stream from a base unit ( 110 ) and across a network, the first portion of the instruction stream comprising a first plurality of drawing commands for generating a first plurality of graphics frames of a user interface, wherein the receiving device is connected to a display; parsing ( 715 ) the first portion of the received instruction stream to determine the first plurality of drawing commands and arguments for the first plurality of drawing commands; writing a subset of the first plurality of drawing commands and arguments of the subset of the first plurality of drawing commands into a receive buffer; preventing swapping of the subset of the first plurality of drawing commands with arguments of the subset of the first plurality of drawing commands from the receive buffer to the execute buffer while there are animation commands being executed; swapping the subset of the first plurality of drawing commands and arguments of the subset of the first plurality of drawing commands from the receive buffer to the execute buffer when there are no animation commands being executed; executing ( 720 ) the first plurality of drawing commands in a graphics engine of the receiving device to generate the first plurality of graphics frames of the user interface, the executing comprising executing a rendering function and interpolating coordinates for animation, wherein the executing comprises executing the subset of the first plurality of drawing commands of the execution buffer; as well as The first plurality of graphical frames of the user interface are output (725) for display on the display.

2. The method according to claim 1, further comprising: receiving user input at the receiving device; sending one or more events representing the user input to the base unit; receiving a second portion of the instruction stream from the base unit, the second portion of the instruction stream comprising a second plurality of drawing commands for generating a second plurality of graphics frames, the second plurality of graphics frames reflecting a result of the received user input; parsing the second portion of the received instruction stream to determine the second plurality of drawing commands and arguments of the second plurality of drawing commands; executing the second plurality of drawing commands in the graphics engine to generate the second plurality of graphics frames; as well as The second plurality of graphics frames are output for display on the display.

3. The method according to claim 1, further comprising: receiving, at the receiving device, an encoded video stream from the base unit and across the network; Decoding the encoded video stream; as well as The decoded video stream is output for display on the display. The method of claim 1 , wherein the instruction stream is lossless.

5. The method according to claim 1, further comprising: receiving, at the receiving device, an encoded video stream from a remote server; Decoding the encoded video stream; as well as The decoded video stream is output for display on the display.

6. The method of claim 5, wherein outputting the first plurality of graphics frames of the user interface for display on the display and outputting the decoded video stream for display on the display occur simultaneously.

7. A non-transitory machine-readable storage medium (1240) providing instructions that, when executed by a processor (1230) of a receiving device (120), cause the processor to perform operations comprising: receiving ( 710 ) a first portion of an instruction stream from a base unit ( 110 ) and across a network at the receiving device, the first portion of the instruction stream comprising a first plurality of drawing commands for generating a first plurality of graphics frames of a user interface, wherein the receiving device is to be connected to a display; parsing ( 715 ) the first portion of the received instruction stream to determine the first plurality of drawing commands and arguments for the first plurality of drawing commands; writing a subset of the first plurality of drawing commands and arguments of the subset of the first plurality of drawing commands into a receive buffer; preventing swapping of the subset of the first plurality of drawing commands with arguments of the subset of the first plurality of drawing commands from the receive buffer to the execute buffer while there are animation commands being executed; swapping the subset of the first plurality of drawing commands and arguments of the subset of the first plurality of drawing commands from the receive buffer to the execute buffer when there are no animation commands being executed; executing ( 720 ) the first plurality of drawing commands in a graphics engine of the receiving device to generate the first plurality of graphics frames of the user interface, wherein the executing includes executing a rendering function and interpolating coordinates for animation, wherein the executing includes executing the subset of the first plurality of drawing commands of the execution buffer; and The first plurality of graphical frames of the user interface are output (725) for display on the display.

8. The non-transitory machine-readable storage medium of claim 7, wherein the operations further comprise: receiving user input at the receiving device; sending one or more events representing the user input to the base unit; receiving a second portion of the instruction stream from the base unit, the second portion of the instruction stream comprising a second plurality of drawing commands for generating a second plurality of graphics frames, the second plurality of graphics frames reflecting a result of the received user input; parsing the second portion of the received instruction stream to determine the second plurality of drawing commands and arguments of the second plurality of drawing commands; executing the second plurality of drawing commands in the graphics engine to generate the second plurality of graphics frames; as well as The second plurality of graphics frames are output for display on the display.

9. The non-transitory machine-readable storage medium of claim 7, wherein the operations further comprise: receiving, at the receiving device, an encoded video stream from the base unit and across the network; Decoding the encoded video stream; as well as The decoded video stream is output for display on the display.

10. The non-transitory machine-readable storage medium of claim 7, wherein the instruction stream is lossless.

11. The non-transitory machine-readable storage medium of claim 7, wherein the operations further comprise: receiving, at the receiving device, an encoded video stream from a remote server; Decoding the encoded video stream; as well as The decoded video stream is output for display on the display.

12. The non-transitory machine-readable storage medium of claim 11, wherein outputting the first plurality of graphics frames of the user interface for display on the display and outputting the decoded video stream for display on the display occur simultaneously.

Images