virtual machine top

Abstract

Techniques for providing multimedia content in a virtual machine top system are described. In some embodiments, a method for UI rendering stream switching is performed at a device (e.g., a server) having one or more processors and a non-transitory memory. The server distributes a first stream comprising a plurality of frames. The server receives a request for a user interface (UI) from a client. In response to receiving the request, the server generates a second stream by locating an encoder to encode the plurality of frames into the second stream based on the first stream, instructing the encoder to accelerate encoding of the plurality of frames, and facilitating blending of the UI with the plurality of frames into the second stream. The server then switches the client from the first stream to the second stream.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to multimedia content delivery, and more specifically to virtualized set tops for multimedia content delivery. BACKGROUND

[0002] Moving client-side applications such as generating a graphical interface (UI) on a set-top-box (STB) to a virtualized 2-way environment is complex and expensive. To deliver high quality and low latency streams to each client, previously existing systems attempted to provide each client its own dedicated stream. Such solutions are expensive and inefficient, as most viewing sessions involve watching a common video feed, and the same UI (e.g., overlays) are generated from common data in the cloud over and over again. BRIEF DESCRIPTION OF DRAWINGS

[0003] Accordingly, as will be appreciated by those skilled in the art, the present disclosure can be embodied in a variety of ways, some of which are illustrated in the figures and described below. It is to be understood that any feature described herein that is entirely or partially under the control of the software and / or firmware can be implemented using one or more computer systems, which can include one or more computers or servers.

[0004] FIG. 1 is a block diagram of an exemplary virtual set top system for media content delivery according to some embodiments;

[0005] FIG. 2A and FIG. 2B is a block diagram illustrating an exemplary UI engine for user interface (UI) element caching and sharing according to some embodiments;

[0006] FIG. 3 is a flow diagram illustrating a method of sharing UI objects in an exemplary virtual set top system according to some embodiments;

[0007] FIG. 4 is a block diagram illustrating use of shared UI objects for UI rendering in an exemplary virtual set top system according to some embodiments;

[0008] FIG. 5 is a flow diagram illustrating a hybrid UI rendering method according to some embodiments;

[0009] FIG. 6A and FIG. 6B is a block diagram illustrating an embodiment of hybrid UI rendering for display on a client device according to some embodiments;

[0010] FIG. 7A - FIG. 7C is a block diagram illustrating transitioning between streams in an exemplary virtual set top system according to some embodiments; is a block diagram illustrating transitioning between streams in an exemplary virtual set top system according to some embodiments;

[0011] FIG. 8A - FIG. 8C FIG. 3 is a flow diagram illustrating a method of stream switching in an example virtual set-top system in response to a request for a UI according to some embodiments;

[0012] FIG. 9 FIG. 4 is a flow diagram illustrating a method of stream switching in an example virtual set-top system according to some embodiments;

[0013] FIG. 10 FIG. 5 is a block diagram of a computing device for an example virtual set-top system according to some embodiments; and

[0014] FIG. 11 FIG. 6 is a block diagram of another computing device for an example virtual set-top system according to some embodiments.

[0015] According to convention, the various features illustrated in the drawings can not be drawn to scale. Accordingly, the dimensions of the various features can be arbitrarily expanded or reduced for the clarity of presentation. In addition, some of the drawings can not depict all of the components of a given system, method or device. Finally, like reference numerals can be used to denote like features throughout the specification and figures. DETAILED DESCRIPTION

[0016] Many details are described to provide a thorough understanding of the example embodiments illustrated in the present disclosure. However, the disclosure can be practiced without some or all of these details. In other instances, well-known systems, methods, components, devices and circuits have not been described in detail to avoid obscuring aspects of the example embodiments described herein. In addition, the one or more example embodiments can be implemented by various means, such as in hardware, software, firmware, or a combination thereof. In one embodiment, one or more example embodiments are implemented by computer readable instructions on a non-transitory computer readable medium, such as a computer data storage device or a computer data signal. The computer readable instructions can be executed by one or more processors to produce a machine, such as a general purpose computer or a special purpose computer.

[0017] SUMMARY

[0018] According to some embodiments, the virtual top system described herein merges clients into a common video segment while the clients are watching a video and seamlessly switches to a client-specific segment (e.g., a personal segment) when a particular client loads a user interface (UI) element related to display. Thus, the system provides a personalized UI for each client without interrupting the video stream. In some embodiments, to improve UI rendering efficiency, the system stores metadata and rendered UI objects in a shared cache for multiple clients. Further, in some embodiments, some portions of the UI can be downloaded as a native application and the client device can utilize the native application to render the portions locally (e.g., a grid with fast navigation). Thus, the seamless switching between the common stream and the unique stream, combined with the use of rendered UI objects in the shared cache and having portions of the UI rendered at the client and portions of the UI rendered in the cloud, enables the virtual top system described herein to deliver high quality multimedia content efficiently at low cost.

[0019] According to various embodiments, a method of stream switching in response to a request for a UI is performed at a device (e.g., a server) having one or more processors and a non-transitory memory. The method includes distributing a first stream including a plurality of frames. The method also includes receiving a request for a user interface (UI) from a client. The method additionally includes generating a second stream in response to receiving the request, where the second stream is generated by locating an encoder to encode the plurality of frames into the second stream based on the first stream, instructing the encoder to accelerate encoding of the plurality of frames, and facilitating mixing the UI with the plurality of frames into the second stream. The method further includes switching the client from the first stream to the second stream.

[0020] SUMMARY

[0021] The virtual set-top system described herein addresses the performance and cost issues associated with virtualized set-top boxes (STBs) described above. In some embodiments, a single virtualization process on the server side (e.g., in the cloud) manages multiple client devices on the server side and utilizes a cache of rendered user interface (UI) objects for improved efficiency. When multiple client devices are watching a video, the virtual set-top system provides a common video segment to the multiple client devices in a common stream. When a particular client device sends a request for a UI while the video is playing, the virtual set-top system seamlessly switches the particular client device from the common stream to a dedicated personal stream for high quality and low latency delivery of media content, which can include rendered UI, video, and / or rendered UI mixed with video. Additionally, in some embodiments, the virtual set-top system employs hybrid UI rendering for enhanced user experience. Hybrid UI rendering allows portions of a UI to be rendered in the cloud while other portions of the UI are rendered locally on the client device. In this way, the virtual set-top system creates a technically and cost-feasible solution for high quality multimedia content delivery through the various optimizations described herein.

[0022] Reference is now made to the following drawings, in which FIG. 1 , FIG. 1is a block diagram of an exemplary virtual machine top system 100 for media content delivery according to some embodiments. Media content (hereinafter also referred to as "multimedia content", "media content item", or "content") can include any multimedia data, such as visual data, audio data, and / or text, etc. In some embodiments, the media content delivery system 100 includes a server 110 (e.g., a headend and / or edge device) that obtains media content from various sources, such as recorded content 101a and / or live broadcast 101b. In some embodiments, the recorded content 101a and / or live broadcast 101b is raw (e.g., unencoded). Alternatively, the recorded content 101a and / or live broadcast 101b is encoded and obtained by the server 110. The server 110 encodes and / or re-encodes the media content and delivers the encoded media content to a plurality of client devices 180 (hereinafter also referred to as clients 180), e.g., client device 1 180-1, client device 2 180-2,..., client device N 180-N, via a network 170. In some embodiments, the network 170 further includes a plurality of content delivery networks 175 (CDNs), e.g., CDN 1 175-1, CDN 2 175-2,..., CDN M 175-M. The plurality of CDNs 175 cache the media content and increase content availability. As such, in some embodiments, the server 110 provides the encoded media content to a plurality of non-colocated client devices 180 via the plurality of CDNs 175, e.g., the server 110 is different from and remote to the plurality of client devices 180.

[0023] In FIG. 1In some embodiments, the exemplary client device 1180-1 includes a decoder 182 for decoding received media content. Upon receiving encoded media content, the client device 1180-1 decodes the media content for display. In some embodiments, the client device 180 also includes an input device (not shown) for receiving input from a user, e.g., buttons, voice recorder, sensors, etc., and / or a network interface (not shown) for connecting to the network 170. Upon receiving input from a user, where the input corresponds to one or more requests for one or more user interfaces (UIs), the respective client device 180 sends the requests to the server 110 and receives from the server 110 one or more rendered UIs in a stream along with the media content. In some embodiments, the respective client device 180 also includes a manifest manipulator 184 for transitioning between streams. In this way, the manifest selector 160 on the server side, the manifest manipulator 184 on the client side, or a combination of the two, facilitate delivery of mixed streams from the server 110 to the respective client device 180.

[0024] In some embodiments, the server 110 includes an encoder / re-encoder 140 for encoding raw content (e.g., content that has not been compressed and / or encoded) from the recorded content 101a and / or the live broadcast 101b. In some embodiments, the encoder / re-encoder 140 includes at least one decoder and at least one encoder, such that the server 110 obtains an encoded stream, decodes the stream, mixes a UI with the decoded stream, and re-encodes the mixed stream. As illustrated by the dashed arrows in FIG. 1 As illustrated by the dashed arrows in

[0025] It should be noted that although a single server 110 is exemplified in FIG. 1 In some embodiments, the server 110 includes an encoder / re-encoder 140 for encoding raw content (e.g., content that has not been compressed and / or encoded) from the recorded content 101a and / or the live broadcast 101b. In some embodiments, the encoder / re-encoder 140 includes at least one decoder and at least one encoder, such that the server 110 obtains an encoded stream, decodes the stream, mixes a UI with the decoded stream, and re-encodes the mixed stream. As illustrated by the dashed arrows in FIG. 1The encoder / re-encoder 140 is illustrated as part of the server 110, but the encoder / re-encoder 140 can be distributed and / or separate from the server 110, e.g., not co-located on the same device. As such, the encoder or re-encoder 140 can be on a different device, in a different system, and / or at a different location to pre-encode media content so that the server 110 obtains the pre-encoded stream for further processing. In particular, in some embodiments, the encoding portion of the re-encoder can be separate from the decoding portion, e.g., a single decoder provides multiple UI engines 130 and / or supports encoding for multiple re-encoders. For simplicity, the subject matter will be described below with reference to a single server 110 to a great extent, but any portion of the single server 110, e.g., the UI object cache 120, the UI engines 130, the encoder 140, the re-encoder 140, the mixer 150, and / or the manifest selector 160, etc., can be separate from each other and / or from the server 110.

[0026] In some embodiments, the server 110 also includes multiple UI engines 130 for rendering UIs for multiple client devices 180. In some embodiments, the server 110 also includes a shared UI object cache 120 for storing rendered UI objects (e.g., rendered images) that are shared by multiple virtualized STBs. According to some embodiments, when rendered UI objects (e.g., posters and banners) are shared between users and / or sessions, the server 110 runs multiple virtualized STBs in one application so that the multiple virtualized STBs share a common layer of rendered images. This allows UI objects generated for one user and / or session to be reused for another user and / or session without having to recompute to re-render the same images. As used herein, an application refers to an executable program, or a list of instructions for execution, that defines a UI for display on a display device. The UI engine 130 is configured to include, e.g., a web browser, a script interpreter, an operating system, or other computer-based environment that is accessed during the running of the application.

[0027] In some embodiments, the server 110 also includes a mixer 150 to mix rendered UI objects with video. As will be described below with reference to FIG. 2, the mixer 150 can be configured to mix rendered UI objects with video in real-time, e.g., as the video is being streamed to the client device 180. In some embodiments, the mixer 150 is configured to mix rendered UI objects with video that is stored on the server 110, e.g., for later playback. FIG. 6A and FIG. 6BUI objects, described in further detail below, include not only graphical elements and UI artifacts rendered by the UI engine 130, but also native applications generated by the UI engine 130 or data and metadata associated with native applications. For example, a UI can include a video component, such as a live stream of a native channel, and a UI element, such as a banner with native channel information. In response to a request for such a UI, the UI engine 130 renders the banner on the server side, the mixer 150 mixes the rendered banner with the live video of the native channel into one stream. The server 110 then sends the mixed stream to the client device 180 for display. Upon receiving the mixed stream, the client device 180 displays only the mixed stream, in which the banner is positioned by the mixer 150 on top of the streaming native channel, e.g., as an overlay on top of the video. In some embodiments, when at least a portion of the UI is rendered by a native application on the client device 180, the mixer 150 facilitates mixing of relevant data, indicators, and / or metadata with the video component, and the server 110 sends the mixed stream to the client device 180. Upon receiving the mixed stream, the client device 180 utilizes the data, indicators, and / or metadata for native UI rendering.

[0028] In some embodiments, the server 110 provisions encoders / re-encoders 140 (e.g., for UI mixing in a provisioned attached mode) and maintains encoders / re-encoders 140 (e.g., activates or deactivates). In some embodiments, the server 110 includes an encoder controller (not shown) for provisioning and maintaining encoders / re-encoders 140 and for selecting a particular encoder / re-encoder 140 to generate a personalized stream in response to a request for a UI. According to some embodiments, the encoder controller further instructs the encoder to accelerate video encoding upon selection of an encoder / re-encoder 140 to ensure seamless transition from one stream to another.

[0029] In some embodiments, the server 110 additionally includes a manifest selector 160 for providing links to streams. In some embodiments, the links are provided in a manifest file to a plurality of client devices 180 for retrieving media content. As will be shown and described in further detail below in FIG. 7A - FIG. 7C According to some embodiments, the manifest selector 160 facilitates a transition between a public stream and a corresponding private stream. The transition can be triggered by or in response to actions on the client side and / or the server side.

[0030] For example, a client-side user action (such as a button press, gesture, voice input) can be associated with a request for a UI, and such a user action can trigger an update to the links in the manifest. In another embodiment, an event from the cloud (such as a published banner, notification, message, or advertisement) can be associated with a request for a UI, and such an event can trigger an update to the links in the manifest. In the example system 100, the client device 1 180-1 links the manifest on the client device to the server-side manifest selector 160 through the manifest manipulator 184. In such an implementation, the client device 1 180-1 plays the same stream from the server 110, while the server-side manifest selector 160 links to a common video stream (e.g., common video without a UI) or to a unique personal segment (e.g., a unique stream with video and a requested UI).

[0031] It should be noted that the virtual top system 100 can include more, fewer, and / or different elements than shown in FIG. 1 as will be described in further detail below with respect to FIG. 2A - FIG. 2B and FIG. 6A - FIG. 6B Each of the elements in the virtual top system 100 can include more, fewer, and / or different sub-elements than shown in FIG. 1 as will be described in further detail below with respect to

[0032] FIG. 2A and FIG. 2B are block diagrams 200A and 200B illustrating example UI engines 130A and 130-B for UI element caching and sharing in accordance with some embodiments. As explained above with respect to FIG. 1 The virtual top system described herein allows multiple virtualized STBs to share a common layer of rendered graphics. The sharing can be between multiple application instances within one UI engine as shown in FIG. 2A and / or between multiple UI engines as shown in FIG. 2B .

[0033] In FIG. 2A , the UI engine 130-A (e.g., UI engine 130, FIG. 1) includes a pool of engines, such as engine 1 210-A1 and engine 2 210-A2, hereinafter collectively referred to as engines 210-A. Each of the engines 210-A includes a shared UI object cache (e.g., as part of the UI object cache 120, FIG. 1 ). For example, engine 1 210-A1 includes a UI object cache 1 120-A1 that stores rendered UI objects 205-A1 that are shared among multiple application instances 220-A1. In another embodiment, engine 2 210-A2 includes a UI object cache 2 120-A2 that stores rendered UI objects 205-A2 that are shared among multiple application instances 220-A1. Each of the engines 210-A provides, for example, one application in the example virtual machine top system. Further, according to some embodiments, each of the application instances 220-A1 and 220-A2 corresponds to one virtualized STB and is responsible for rendering the UI associated with the application for one respective client device.

[0034] In FIG. 2B , the UI engine 180-B includes a cache server 230 external to engine 1 210-B1 and engine 2 210-B2. The cache server 230 allows for a central cache of rendered UI objects 205-B stored in the UI object cache 120-B. The UI objects 205-B are shared by both engine 1 210-B1 and engine 2 210-B2, for example, also by application instance 220-B1 running through engine 1 210-B1 and application instance 220-B2 running through engine 2 210-B2. For example, the application instance 220-B1 can look up the UI object cache 120-B in response to receiving a request for a UI. In the event that no rendered UI object corresponding to a UI element of the UI in the UI object cache 120-B is found, the UI element is rendered and the rendered UI object is uploaded to the UI object cache 120-B. In some embodiments, upon detecting a new rendered UI object in the UI object cache 120-B, the cache server 230 can determine whether to allow sharing of the UI object with other application instances and / or engines, for example, with the application instance 220-B2 on engine 2 210-B2. In the event that sharing of the UI object with the application instance 220-B2 is allowed, the UI object can be retrieved from the UI object cache 120-B and reused by one of the application instances 220-B2 upon request.

[0035] It is noted that although FIG. 2AUI object cache 1 120-A1 and UI object cache 2 120-A2 are part of engine 1 210-A1 and engine 2 210-A2, respectively, but UI object cache 1 120-A1 can be separate and distinct from engine 1 210-A1 and UI object cache 2 120-A2 can be separate and distinct from engine 2 210-A2. Further, UI object cache 1 120-A1 and UI object cache 2 120-A2 can be separate and distinct from UI engine 130-A. Likewise, FIG. 2B Cache server 230 shown in FIG. 2B can be part of UI engine 130-B or on a separate virtual machine, instance, and / or device. Further, UI engines 130-A and 130-B can include more, less, and / or different elements for UI rendering than shown in block diagrams 200A and 200B. For example, a UI engine controller (not shown) can be used by the server for selecting a UI engine and / or application instance for UI rendering. Those skilled in the art will appreciate from the present disclosure that various other features and configurations are not illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the implementations disclosed herein.

[0036] FIG. 3 is a flow diagram illustrating a method 300 of sharing UI objects in a virtual machine top system, in accordance with some embodiments. In some embodiments, method 300 is performed on server 110 FIG. 1 ), for example, by UI engine 130 FIG. 1 Because many graphical elements are shared between users and / or viewing sessions (e.g., a poster for an event on a channel), server 110 described above with reference to FIG. 1 provides a common layer of rendered UI objects shared by multiple virtualized STBs (e.g., application instances 220-A1 and 220-A2 in FIG. 2A or application instances 220-B1 and 220-B2 in FIG. 2B ) in one application and by multiple virtualized STBs. For example, UI object cache 120 is shared by UI engine 130 in FIG. 1 In another embodiment, UI object cache 1 120-A1 is shared by application instance 220-A1 and UI object cache 2 120-A2 is shared by application instance 220-A2 in FIG. 2A In yet another embodiment, UI object cache 120-B is shared by application instances 220-B1 and 220-B2 in FIG. 2B

[0037] ​In some embodiments, when the corresponding virtualized STB receives a request for a UI from a client (e.g., one of the client devices 180, FIG. 1 ) as represented by block 310, the corresponding virtualized STB identifies the UI elements within the UI and checks whether a rendered UI object corresponding to the UI element of the UI exists in the UI object cache as represented by decision block 320. In the case where a rendered UI object corresponding to the UI element is found in the UI object cache (from the "Yes" -branch of block 320), the corresponding virtualized STB foregoes rendering the UI element. Instead, as represented by block 330, the corresponding virtualized STB copies the UI object from the UI object cache 120( FIG. 1 ). Further, as represented by block 340, the corresponding virtualized STB builds the UI using the rendered UI object from the cache, e.g., UI composition using the copied UI object, thereby improving the efficiency of UI rendering.

[0038] On the other hand, in the case where a UI object corresponding to the UI element of the UI is not found in the UI object cache 120( FIG. 1 ) (from the "No" -branch of block 320), the corresponding virtualized STB renders the UI for the client, including rendering the UI element of the UI as represented by block 350. Once rendered, the corresponding virtual machine top further stores the rendered UI object in the UI object cache 120( FIG. 1 ) as represented by block 360. In addition, the corresponding virtual machine top indicates that the stored UI object can be shared with other virtualized STBs to reduce duplication and improve rendering efficiency as represented by block 370. In some embodiments, the stored UI object is shared with multiple instances of the UI engine (as shown in FIG. 2A ) or across multiple UI engines (as shown in FIG. 2B ).

[0039] FIG. 4is a diagram illustrating an example virtual machine top environment 400 for UI object sharing in accordance with some embodiments. In the example virtual machine top environment 400, the UI engine 130 runs multiple application instances 410 to provide virtualized STBs for multiple client devices 180, e.g., application instance 1 410-1 as a virtualized STB for client device 1 180-1, application instance 2 410-2 as a virtualized STB for client device 2 180-2, and application instance 3 410-3 as a virtualized STB for client device 3 180-3. The multiple application instances 410 render personalized UIs for the client devices 180, e.g., application instance 1 410-1 renders UI A 420-A for client device 1 180-1, application instance 2 410-2 renders UI B 420-B for client device 2 180-2, and application instance 3 410-3 renders UI C 420-C for client device C 180-C.

[0040] In the example environment 400, UI A 420-A requested by client device 1 180-1 includes UI element 1 430-A1, UI element 2 430-A2, UI element 3 430-A3, and UI element x 430-Ax, collectively referred to hereinafter as UI elements 430-A. UI B 420-B requested by client device 2 180-2 includes UI element 1 430-B1 and UI element y 430-By, collectively referred to hereinafter as UI elements 430-B. UI C 420-C requested by client device 3 180-3 includes UI element 2 430-C2 and UI element z 430-Cz, collectively referred to hereinafter as UI elements 430-C. In some embodiments, each of the application instances 410 is unique for an active viewing session, e.g., each active profile, active user, and / or active device, etc. Further, in some embodiments, each of the application instances 410 has its own state, e.g., rendering the beginning of a movie or the middle of an advertisement, etc. Thus, each of the application instances 410 as virtualized STBs renders personalized UIs for the corresponding client devices 180.

[0041] The example sharing environment 400 includes the UI object cache 120, as described above with reference to FIG. 1 and FIG. 2A - FIG. 2BUI objects 405 are described, for example, UI object 1 405-1 and UI object 2 405-2, are stored. The rendered UI objects 405 are shared by multiple application instances 410, for example, by application instance 1 410-1, application instance 2 410-2, and application instance 3 410-3. Among the UI elements 430-A, 430-B, and 430-C, UI element 1 430-B1 in UI B 420-B is the same as UI element 1 430-A1 in UI A 420-A, for example, the same poster for the same event on the same channel. Likewise, UI element 2 430-B2 in UI B 420-B is the same as UI element 2 430-A2 in UI A 420-A.

[0042] According to the UI object sharing method 300 described above, once application instance 1 410-1 or application 2 410-2 renders UI element 1 430-A1 or UI element 1 430-B1 and stores the rendered UI element as UI object 1 405-1 in the UI object cache 120, any of the application instances 410 can use the rendered UI object 1 405-1 without having to re-render the same graphical element. In another embodiment, once UI element 2 430-A2 or UI element 2 430-C2 is rendered and stored as UI object 2 405-2 in the UI object cache 120, any of the application instances 410 can use the rendered UI object 2 405-2 without having to re-render the same graphical element. In this way, by not having each application instance 410 render the same UI element that has already been rendered by another application instance 410 for a different viewing session, the UI engine 130 saves computational resources and improves cloud rendering efficiency.

[0043] FIG. 5 is a flowchart illustrating a hybrid UI rendering method 500 in a virtual machine top system (e.g., virtual machine top system 100, FIG. 1 ) in accordance with some embodiments. In some embodiments, the method 500 is performed by one of the UI engines 130 FIG. 1 ) on the server 110 FIG. 1 ) in accordance with some embodiments. In particular, in some embodiments, the method 500 is performed at a virtualized STB, such as one of the application instances (e.g., FIG. 2A application instances 220-A1 and 220-A2 in the virtualized STB 100-A, FIG. 2B application instances 220-B1 and 220-B2 in the virtualized STB 100-B, or FIG. 4one of the application instances 410 in the server 110).

[0044] The method 500 begins by a respective UI engine receiving a request for a UI from a client device, as represented by block 510. The method 500 continues with the respective UI engine analyzing the request and / or the requested UI to determine whether the UI can be rendered, at least in part, on the client device, as represented by block 520, e.g., based on UI complexity, user interactions associated with the UI, an amount of personalization of the UI, and / or processing and networking capabilities of the client device, etc. In cases where the client device is allowed to render at least part of the UI (from the "yes" -branch of block 520), the server prepares a local application for the client device, as represented by block 530, and instructs the client device to locally run the application for rendering at least part of the UI on the client device, as represented by block 540. For example, the server can generate a local application for the client device to download, where the local application can be loaded to the client device at any time, e.g., at startup, during a pre-scheduled update, and / or upon sending a UI event. In some embodiments, the server prepares the local application for the client device by triggering the local application to be executed, e.g., instructing the client device to load the local application and / or relevant data for application to the local application. On the other hand, in cases where the client device is not allowed to render at least part of the UI (from the "no" -branch of block 520), the server renders the UI for the client device, e.g., by a virtualized STB, as represented by block 550, and sends the rendered UI to the client device for display, as represented by block 560.

[0045] For example, FIG. 6A and FIG. 6B are block diagrams 600A and 600B illustrating a number of embodiments for hybrid UI rendering for display on a client device. In FIG. 6A In the server 110-A (e.g., the server 110 in the FIG. 1 includes a UI engine 130-A (e.g., the UI engine 130 in the FIG. 1 includes an encoder / re-encoder 140-A (e.g., the encoder / re-encoder 140 in the FIG. 1 includes a mixer 150-A (e.g., the mixer 150 in the FIG. 1 The server 110-A delivers the mixed stream to a client device 180-A (e.g., the client device 180 in the FIG. 1the client device 180) for display on the display 610-A. Thus, for simple UIs that do not require a persistently present response, it is more cost effective for the client device 180-A to have the UI rendered on the server 110-A by the UI engine 130-A and mixed by the mixer 150-A with the video from the encoder / re-encoder 140-A.

[0046] In FIG. 6B , similar to FIG. 6A , the server 110-B (e.g., the server 110 in FIG. 1 ) includes a UI engine 130-B (e.g., the UI engine 130 in FIG. 1 ) for rendering the UI, an encoder / re-encoder 140-B (e.g., the encoder / re-encoder 140 in FIG. 1 ) for encoding the video, and a mixer 150-B (e.g., the mixer 150 in FIG. 1 ) for mixing the UI with the video. Unlike the UI engine 130-A that generates rendered graphical images, in some embodiments, the UI engine 130-B includes a local application generator 620 that generates one or more local applications for the client 180-B (e.g., the client device 180 in FIG. 1 ) as part of the UI or provides data, metadata, and / or indicators associated with the one or more local applications.

[0047] As explained above with reference to FIG. 5 , the one or more local applications can be downloaded and / or loaded at any time. As such, in some embodiments, the one or more local applications are downloaded separately from the video (e.g., delivered out-of-band, not mixed with the video, and / or using a separate delivery path), while in some other embodiments, the one or more local applications are then mixed with the video from the encoder / re-encoder 140-B by the mixer 150-B. In some embodiments, the mixer 150-B mixes the data, metadata, and / or indicators with the video. On the client side, the client device 180-B includes a local application engine 630 for detecting and / or receiving the one or more local applications, data, metadata, and / or indicators mixed with the video. The local application engine 630 then downloads and / or executes the one or more local applications according to the data, metadata, and / or indicators. In some embodiments, the one or more local applications locally render UI elements over the video on the client device 180-B for display on the display 610-B.

[0048] As FIG. 6BThe hybrid UI rendering shown improves the user experience for rendering UIs such as TV guides involving fast navigation and / or deep personalization, for example, UIs with personalization levels exceeding a threshold and / or UI user interactions exceeding a threshold. UI engine 130-B utilizes native application generator 620 to prepare one or more native applications associated with the TV guide for client device 180-B, such as a native application for a channel within the TV guide. Upon receiving the one or more native applications, data, metadata, and / or indicators associated with the one or more native applications, native application engine 630 downloads and / or runs the one or more native applications when the user selects one or more tiles on the TV guide. Based on the data, metadata, and / or indicators associated with the one or more native applications, client device 180-B may render the TV guide as an overlay on the video, render the TV guide in the absence of video in the background, and / or render the TV guide while pausing the video until further instructions for starting are received. In some embodiments, UI elements are rendered on transparent or semi-transparent surfaces, ordered higher than the video, without blurring the video. Thus, for UIs such as TV guides, the native application engine 630 facilitates client-side UI rendering, improving the user experience and reducing rendering latency.

[0049] FIG. 7A - FIG. 7C This illustrates an exemplary virtual machine top system (e.g., virtual machine top system 100) according to some implementation schemes. FIG. 1 Block diagrams 700A-700C illustrate the transitions between flows within the block diagram. FIG. 7A In the context of multiple client devices 180 requesting videos without a UI, the client devices 180 obtain links to a common stream (e.g., a video stream), for example, by using CDN 175 to return the links to network 170. FIG. 1 ) list selector 160 ( FIG. 1 ) and / or via list manipulator 184 ( FIG. 1 Thus, according to some implementation schemes, video encoded by a single video encoder is acquired from a public stream and distributed to multiple client devices 180. Compared to previous solutions that allocated a complete video encoder to each viewing session, the virtual machine top system described herein uses a publicly encoded video and merges clients into a public video segment when the user is watching a video without a UI, thereby reducing redundancy and achieving cost savings.

[0050] exist FIG. 7B In this context, a link to a dedicated stream is obtained by the client device 180 in response to a UI request. For example, the manifest selector 160 (FIG. 1 ) can return a link to stream 1 for client device 1 180-1 in response to an action from client device 1 180-1 for one UI, return a link to stream 2 for client device 2 180-2 in response to an action from client device 2 180-2 for another UI, and return a link to stream N for client device N 180-N in response to an action for yet another UI, etc. In FIG. 7C , once UI visibility ends, e.g., the requested UI is no longer displayed or visible on client device 1 180-1, manifest selector 160( FIG. 1 ) can return a link to a common stream for both client device 1 180-1 and client device 2 180-2, thereby merging client device 1 180-1 and client device 2 180-2 back to a common video segment.

[0051] As shown in FIG. 7A and FIG. 7B , in response to a request for a UI, e.g., triggered by an action from the client side and / or the server side, the virtual top box system described herein transitions from a common stream to a unique stream (with video and UI) for high quality and low latency delivery of a dedicated and / or personal segment. As shown in FIG. 7C , when the visibility of the UI ends, e.g., when the UI element is no longer displayed on client device 1 180-1 and client device 2 180-2, the virtual top box system described herein merges the client devices 180 back to a common stream for cost savings. In this way, the virtual top box system described herein improves efficiency, reduces cost, and delivers high quality media content relative to previously existing solutions.

[0052] FIG. 8A - FIG. 8C is a diagram 800A-800C illustrating encoder selection and stream switching in response to a request for a UI in an example virtual top box system, e.g., virtual top box system 100 of FIG. 1 . As shown in the encoder selection and stream switching of FIG. 8A - FIG. 8C is performed on the server side, e.g., by server 110 of FIG. 1 . In some embodiments, the server prepares and maintains a pool of encoders and possibly a pool of re-encoders, e.g., as described in FIG. 1encoder / re-encoder 140. Thus, in some embodiments, the server includes an encoder controller for maintaining the pool of encoders / re-encoders and for selecting an encoder / re-encoder from the pool of encoders / re-encoders upon request. As used herein, the term "encode" also applies to "re-encode," and "encoder" also corresponds to the re-encoding portion of a "re-encoder." For example, in some cases, when the server prepares a re-encoder, the server instructs the re-encoder to prepare to re-encode the stream after decoding the stream.

[0053] In FIG. 8A The common stream 810 includes a plurality of frames, for example, encoded by encoder 1. The server prepares a pool of encoders, for example, encoder 1, encoder 2, encoder 3, and encoder 4, by instructing the pool of encoders to encode the plurality of frames with varying delays relative to the common stream 810. For example, streams 2a and 3-4 820a and 830-840 are encoded by encoders 2-4 with varying amounts of delay relative to the common stream 810. Specifically, encoder 2 encodes the plurality of frames into stream 2a 820-a with slightly more delay than stream 3 830 encoded by encoder 3, but less delay than stream 4 840 encoded by encoder 4.

[0054] In some embodiments, the client devices report their decoded positions and offsets of the video being played to the server. In such embodiments, the server sets encoders 2-4 to prepare the attach mode by emulating the client's decoding offset. In this way, encoders 2-4 encode the same input stream as the common stream 810, for example, the same plurality of frames, and encode near the position the client devices are decoding, for example, within a threshold delay.

[0055] For example, in FIG. 8AIn this case, the client device reports that due to processing and / or network latency, the client device is decoding frames 80 in the playing stream 805, while encoder 1 has encoded frames beyond frame 80 in the common stream 810. Based on the client reported offset, the server prepares encoders 2-4 so that the latency of encoding by encoders 2-4 is within a threshold of the offset from the position of the decoding by the client device (e.g., within a number of frames and / or sub-seconds, etc.). As a result, at the time of the request for the UI, encoder 2 is encoding frame 78 in stream 2a 820a, encoder 3 is encoding frame 90 in stream 3 830, and encoder 4 is encoding frame 70 in stream 4 840. Thus, relative to the position of the decoding by the client device at frame 80, stream 2a 820a encoded by encoder 2 is closest to the stream 805 being played by the client. In other words, the server identifies for each encoder in the pool of encoders a respective latency of the offset from the position of the decoding, e.g., relative to the position of the decoding 80 in the stream 805 being played by the client device, the server identifies that encoder 2 will encode stream 2a 820a at frame 78 delayed by 2 frames, stream 3 830 encoded by encoder 3 is ahead by 10 frames at frame 90, and encoder 4 will encode stream 4 840 at frame 70 delayed by 10 frames. Thus, the server selects encoder 2 from the pool of encoders 2-4 based on frame 78 being closest to the offset of the position of the decoding at frame 80.

[0056] In FIG. 8B which encoder 2 has been selected to provide the personalized segment to the client, the server (e.g., with an encoder controller) instructs encoder 2 to generate variant stream 2b 820b by quickly encoding / re-encoding frames 79, 80, and 81 to close the gap. Further, in some embodiments, the server (e.g., with a mixer 150, FIG. 1 ) starts encoding video from frame 82 onwards combined with the UI. The server then starts sending variant stream 2b 820b to the client from variant frame 82b 817b instead of the common stream 810, and then, e.g., by way of the FIG. 1 manifest selector 160 and / or manifest manipulator 184 in the server updates the manifest. On the client side, because the decoded stream in the common stream 810 (e.g., frames 78-81) prior to frame 82b 817b is the same as in stream 2b 820b, e.g., the same frames 78-81, the switch from the common stream 810 to stream 2b 820b is seamless. In some embodiments, once the UI ends, e.g., the visibility of the UI ends, the server switches the client back from stream 2b 820b to the common stream 810.

[0057] It is noted that in FIG. 8A and FIG. 8BIn the embodiment shown in FIG. 8B, the switch point, e.g., frame 82, can be a reference frame or a non-reference frame. In other words, the switch point can be at the beginning of a segment or in the middle of a segment. For example, FIG. 8B Frame 82b in FIG. 8B can be its own I-frame. Alternatively, the encoder 2 used to generate variant stream 820b can be able to use frames from encoder 1 (or decode frames from common stream 810) as reference frames for encoding frame 82b. In this way, a client can download and display up to frame 81 from common stream 810 and seamlessly switch to downloading from stream 2b 820b starting at frame 82b, whether frame 82b is at the beginning of a segment (e.g., an I-frame) or in the middle of a segment (e.g., frame 82b is a reference frame to frame 81).

[0058] In some embodiments, instead of maintaining a pool of busy encoders, the server selects any encoder that is available and can be activated to service a request for a UI by a fast enough encoder and a frequent enough key frame. In FIG. 8C In FIG. 8C, encoder 1 encodes a plurality of frames into common stream 810C, where the plurality of frames includes a key frame 815, e.g., an I-frame 815. When the server receives a request for a UI from a client, the server determines that encoder 1 is available and can be activated, e.g., has sufficient processing power and can encode at a rate faster than a threshold bits per second. Further, once the server determines that common stream 810C has a key frame interval less than a threshold, e.g., has frequent enough key frames, the server activates encoder 2 and provides frames from the last key frame in common stream 810C to encoder 2, e.g., provides frames from key frame 815 to encoder 2. According to some embodiments, the server further instructs encoder 2 to generate variant stream 2 820C by quickly encoding frames from key frame 815. In some embodiments, the server also overlays the UI forward from frame 817c. The server then starts sending variant stream 2 820C to the client at variant frame 817c. Thus, the client can switch to stream 2 820C without incurring a significant delay.

[0059] FIG. 9 is a flowchart of a method 900 of stream switching in response to a request for a UI in an example virtual head system (e.g., virtual head system 100 of FIG. 1), FIG. 1 According to some embodiments. In some embodiments, method 900 is performed on a server, e.g., server 110 in virtual head system 100, where the server includes one or more processors and a non-transitory memory. FIG. 1

[0060] ​Method 900 begins with a server distributing a first stream comprising a plurality of frames, as represented by block 910. For example, the first stream can be a video stream encoded by one of the encoders / re-encoders in FIG. 1. In some embodiments, the server obtains a first stream that is pre-encoded (e.g., encoded by an encoder that is different from and separate from the server). Upon request, the server distributes a common video stream to a plurality of client devices, e.g., via a link to the common stream provided through a manifest. FIG. 1

[0061] Method 900 continues with the server (e.g., UI engine 130, FIG. 1 ) receiving a request for a UI from a client, as represented by block 920. In response to receiving the request, the server generates a second stream, as represented by block 930. In some embodiments, the generation of the second stream is: locating, by the server (e.g., encoder controller), an encoder to encode a plurality of frames into the second stream based on the first stream, as represented by block 932, and instructing, by the server (e.g., encoder controller), the encoder to accelerate encoding of the plurality of frames, as represented by block 934. In some embodiments, the second stream is further generated by facilitating mixing of the UI with the plurality of frames into the second stream, as represented by block 936. The server then switches the client from the first stream to the second stream, as represented by block 940.

[0062] In some embodiments, the UI is generated by an application instance, and the application instance has access to a shared cache. Further, in some embodiments, the application instance obtains rendered UI objects of UI elements of the UI corresponding to the UI from the shared cache, and composes the UI for the client using the rendered UI objects. In some other embodiments, the application instance renders the UI for the client, e.g., by generating rendered UI objects of UI elements of the UI, and stores the rendered UI objects to the shared cache.

[0063] For example, in accordance with the UI object sharing method 300( FIG. 3 ), FIG. 4 ​Application instance 1410-1 determines whether a rendered UI object corresponding to UI element 1 430-A1 of UI A 420-A exists in shared UI object cache 120. In the case where a rendered UI object 1 405-1 corresponding to UI element 1 430-A1 in shared UI object cache 120 is located, application instance 1410-1 saves computational resources by not re-rendering UI element 1 430-A1 and uses UI object 1 405-1 for composing UI A 420-A. On the other hand, in the case where a rendered UI object corresponding to UI element x 430-Ax is not located, application instance 1410-1 renders UI element x 430-Ax and stores the rendered UI object in shared UI object cache 120.

[0064] In some embodiments, the UI also includes a client-downloadable native application for rendering a portion of the UI on the client, e.g., a native application for rendering tiles in a TV guide UI. In such embodiments, the server identifies the portion of the UI to be rendered on the client, e.g., based on interactivity of the portion of the UI and / or capabilities of the client device. The server further prepares a native application corresponding to the portion of the UI for the client, e.g., by embedding the native application in a plurality of frames or instructing the client device to load the native application and / or related data for application to the native application.

[0065] For example, in FIG. 6B In some embodiments, the server 110-B prepares one or more native applications for the client device 180-B. The one or more native applications, data associated with the one or more native applications, metadata, and / or indicators are embedded with the video from the encoder 140-B by the mixer 150-B and delivered to the client device 180-B or downloaded separately from the video. In some embodiments, the client device 180-B utilizes a native application engine 630 to download the one or more native applications and / or obtain related data for the one or more native applications for locally rendering one or more elements on the client device 180-B to improve user experience.

[0066] In some embodiments, the server utilizes a pool of encoders to facilitate generation of the second stream. To prepare the pool of encoders, in some embodiments, the server receives, from the client device, an offset of a decoded position relative to the first stream, and instructs the pool of encoders to encode a plurality of frames with varying delays relative to the first stream (including instructing the pool of re-encoders to decode and re-encode), and the varying delays are within a threshold of the offset from the decoded position. For example, in FIG. 8A In some embodiments, the server prepares the pool of encoders by instructing encoders 2-4 to prepare for attach mode by mimicking the decoding delay of the client, e.g., relative to the offset of the decoded position frame 78 in stream 805, instructing encoder 2 to encode stream 2a 820a with a delay of 2 frames, instructing encoder 3 to encode stream 3 830 with a lead of 10 frames, and instructing encoder 4 to encode stream 4 840 with a delay of 10 frames.

[0067] After the pool of encoders has been prepared, the server selects one encoder from the pool of encoders to service the request for the UI. In some embodiments, locating an encoder that encodes a plurality of frames based on the first stream includes identifying, for each encoder in the pool of encoders, a respective delay relative to encoding the plurality of frames into the first stream, and selecting an encoder from the pool of encoders based on the respective delay associated with the encoder that is closest to the offset of the decoded position. For example, in FIG. 8A In some embodiments, the server selects encoder 2 because the difference between the delay of encoded stream 2a 820a and the offset of the decoded position in stream 805 being played by the client is the smallest among streams 2a 820a, 3 830, and 4 840, indicating that the encoding of stream 2a 820a by encoder 2 is closest to the decoding of stream 2a 820 on the client.

[0068] In some embodiments, locating an encoder that encodes a plurality of frames based on the first stream includes identifying a switch point to overlay the UI on the plurality of frames. In such embodiments, in accordance with some embodiments, instructing the encoder to accelerate encoding of the plurality of frames includes instructing the encoder to accelerate encoding of the plurality of frames up to the switch point. Further, in such embodiments, in accordance with some embodiments, facilitating mixing of the UI with the plurality of frames into the second stream includes overlaying the UI on the plurality of frames in the second stream starting at the switch point. Additionally, in such embodiments, in accordance with some embodiments, switching the client from the first stream to the second stream includes switching the client from the first stream to the second stream at the switch point.

[0069] For example, in FIG. 8BIn this context, the server identifies frame 82 as a switching point based on the scenario—for example, the client's decoding rate, the encoder's encoding rate, the timing of UI requests, network conditions, the current frame being displayed or decoded, the current frame being encoded, and / or the characteristics of the UI to be overlaid. Furthermore, the server instructs encoder 2 to accelerate encoding, such as... FIG. 8B As shown, to reduce the gap and begin covering the UI from frame 82 forward, variant frames 82b and 817b are generated, for example, by covering the UI on frame 82. The server then switches the client device from public stream 810 to stream 2b and 820b.

[0070] In some implementations, instead of maintaining a pool of encoders, the server locates any available encoders that can be activated to serve requests for the UI. In such implementations, the server identifies keyframes in a first stream encoded before receiving a request for the UI, and the switching point from those keyframes. Thus, in some implementations, instructing an encoder to accelerate the encoding of the plurality of frames includes providing the encoder with frames starting from the keyframe and instructing the encoder to accelerate the encoding of those frames. Furthermore, the server facilitates blending the UI with the plurality of frames into a second stream by incorporating the UI overlay on the plurality of frames at the switching point in the second stream. Additionally, the server switches the client from the first stream to the second stream at the switching point.

[0071] For example, in FIG. 8C In the process, the server identifies the last keyframe 815 encoded before receiving a request for the UI and the subsequent switch point 817c. The server then feeds the frame starting from the last keyframe 815 to encoder 2 and accelerates the encoding between keyframe 815 and switch point 817c. The server then begins mixing the UI with frame 817c at switch point 817c and forward, and switches the client device from common stream 810C to stream 2 820C at switch point 817c.

[0072] In some implementations, to facilitate stream switching, the server provides a link to the client—for example, via a manifest selector 160. FIG. 1 The link used to switch the client from the first stream to the second stream involves a link in the manifest pointing to the first segment in the first stream before the switch, and in response to a request for the UI, replacing the link to the first segment in the manifest with a link to the second segment in the second stream. Furthermore, in some implementations, the server switches the client back from the second stream to the first stream when the UI display ends. For example, in... FIG. 7A In the process, before the handover, the link provided to client device 180 points to the public stream.FIG. 7B In particular embodiments, in response to a request by client 1 180-1 for the UI, the link provided to client 1 180-1 is updated so that it points to stream 1. In FIG. 7C In particular embodiments, at the end of the UI, e.g., the client device stops displaying the UI, the server again updates the link to point to the common stream.

[0073] FIG. 10 is a block diagram of a computing device 1000 in an example virtual machine top system in accordance with some embodiments. In some embodiments, the computing device 1000 corresponds to the server 110 in FIG. 1 While certain specific features are illustrated, one of ordinary skill in the art will appreciate from the present disclosure that implementation of the embodiments disclosed herein that are illustrated are not all-inclusive and other features can be implemented as part of the embodiments, as one of ordinary skill in the art will appreciate from the present disclosure. As one non-limiting example, in some embodiments, the computing device 1000 includes one or more processing units (CPUs) 1002 (e.g., processors), one or more input / output interfaces 1003 (e.g., input devices, network interfaces, displays, etc.), memory 1006, a programming interface 1008, and one or more communication buses 1004 for interconnecting these and various other components.

[0074] In some embodiments, the communication buses 1004 include circuitry that interconnects and controls communications between system components. The memory 1006 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and, in some embodiments, includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory 1006 optionally includes one or more storage devices remotely located from the CPU(s) 1002. The memory 1006 comprises a non-transitory computer readable storage medium. Moreover, in some embodiments, the memory 1006, or the non-transitory computer readable storage medium of memory 1006, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 1030, a storage module 1035, an encoder controller 1040, a UI engine controller 1050, a mixing module 1060, and a manifest selector 1070. In some embodiments, one or more instructions are included in a combination of logic and non-transitory memory. The operating system 1030 includes procedures for handling various basic system services and for performing hardware dependent tasks.

[0075] In some embodiments, the storage module 1035 is configured to store rendered UI objects in a UI object cache 1037 (e.g.,FIG. 1 UI object cache 120) in the content store 1020. To this end, the storage module 1035 includes a set of instructions 1039a and heuristics and metadata 1039b.

[0076] In some embodiments, the encoder controller 1040 is configured to maintain a pool of encoders 1042 (e.g., encoders / re-encoders 140 in the content store 1020) and / or to select one of the encoders 1042 for use in encoding a frame into a stream. To this end, the encoder controller 1040 includes a set of instructions 1043a and heuristics and metadata 1043b. FIG. 1

[0077] In some embodiments, the UI engine controller 1050 is configured to maintain a pool of UI engines 1052 (e.g., UI engines 130, FIG. 1 ) for UI rendering. In some embodiments, the UI engines 1052 include a local application generator 1054 (e.g., local application generator 620, FIG. 6B ) for generating a local application for a client device. To this end, the UI engine controller 1050 includes a set of instructions 1055a and heuristics and metadata 1055b.

[0078] In some embodiments, a mixing module 1060 (e.g., mixer 150, FIG. 1 ) is configured to mix a UI, a local application for rendering at least a portion of the UI, and / or data, indicators, and / or metadata for positioning the application with a video. To this end, the mixing module 1060 includes a set of instructions 1061a and heuristics and metadata 1061b.

[0079] In some embodiments, a manifest selector 1070 (e.g., manifest selector 160, FIG. 1 ) is configured to provide a link to a segment in a stream to a client device. To this end, the manifest selector 1070 includes a set of instructions 1071a and heuristics and metadata 1071b.

[0080] While the storage module 1035, the encoder controller 1040, the UI engine controller 1050, the mixing module 1060, and the manifest selector 1070 are illustrated as residing on a single computing device 1000, it should be understood that, in other embodiments, any combination of the storage module 1035, the encoder controller 1040, the UI engine controller 1050, the mixing module 1060, and the manifest selector 1070 can reside in separate computing devices. For example, in some embodiments, each of the storage module 1035, the encoder controller 1040, the UI engine controller 1050, the mixing module 1060, and the manifest selector 1070 resides on a separate computing device.​

[0081] Furthermore, contrary to the structural diagram of the implementation scheme described herein, FIG. 10 This is intended more as a functional description of the various features present in a particular embodiment. As those skilled in the art will recognize, items shown individually may be combined and some items may be separated. For example, in multiple embodiments, FIG. 10 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated among them, will vary from one implementation to another and may depend in part on the specific combination of hardware, software, and / or firmware chosen for a particular implementation.

[0082] FIG. 11 This is a block diagram of a computing device 1100 in an exemplary virtual machine top-down system according to some embodiments. In some embodiments, computing device 1100 corresponds to FIG. 1 One of the client devices 180 in the above description performs one or more of the functions described with respect to client device 180. While certain specific features have been illustrated, those skilled in the art will understand from this disclosure that various other features have not been illustrated for the sake of brevity and to avoid obscuring more relevant aspects of the embodiments disclosed herein. Therefore, as a non-limiting embodiment, in some embodiments, computing device 1100 includes one or more processing units (CPUs) 1102 (e.g., processors), one or more input / output interfaces 1103 (e.g., input devices, sensors, network interfaces, displays, etc.), memory 1106, programming interface 1108, and one or more communication buses 1104 for interconnecting these and various other components.

[0083] In some embodiments, communication bus 1104 includes circuitry that interconnects and controls communications between system components. Memory 1106 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and, in some embodiments, includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 1106 optionally includes one or more storage devices remotely located from CPU 1102. Memory 1106 comprises a non-transitory computer readable storage medium. Moreover, in some embodiments, memory 1106, or the non-transitory computer readable storage medium of memory 1106, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 1130, a storage module 1135, a decoder 1140, a manifest manipulator 1150, and a local application engine 1160. In some embodiments, one or more instructions are included in a combination of logic and non-transitory memory. Operating system 1130 includes procedures for handling various basic system services and for performing hardware dependent tasks.

[0084] In some embodiments, storage module 1135 is configured to provide storage for decoding, rendering, and / or displaying media content. To this end, storage module 1135 includes instruction sets 1137a and heuristics and metadata 1137b.

[0085] In some embodiments, decoder 1140 (e.g., decoder 182, FIG. 1 ) is configured to decode streams from a server. To this end, decoder 1140 includes instruction sets 1141a and heuristics and metadata 1141b.

[0086] In some embodiments, manifest manipulator 1150 (e.g., manifest manipulator 184, FIG. 1 ) is configured to manipulate links to media streams (e.g., to public streams, to unique personal segments) and / or links to server-side manifest selectors. To this end, manifest manipulator 1150 includes instruction sets 1151a and heuristics and metadata 1151b.

[0087] In some embodiments, local application engine 1160 (e.g., local application engine 630, FIG. 6B ) is configured to download one or more local applications and use the applications to locally render UIs. To this end, local application engine 1160 includes instruction sets 1161a and heuristics and metadata 1161b.

[0088] Although the storage module 1135, the decoder 1140, the manifest manipulator 1150, and the local application engine 1160 are illustrated as residing on a single computing device 1100, it should be understood that, in other embodiments, any combination of the storage module 1135, the decoder 1140, the manifest manipulator 1150, and the local application engine 1160 can reside in separate computing devices. For example, in some embodiments, each of the storage module 1135, the decoder 1140, the manifest manipulator 1150, and the local application engine 1160 reside on a separate computing device.

[0089] Furthermore, contrary to the structural schematics of the embodiments described herein, FIG. 11 are meant to be taken only as examples of the functionality of various features present in one particular implementation. As those skilled in the art will appreciate, items shown separately could be combined and items shown separately could be separated. For example, in various embodiments, FIG. 11 Some of the functional modules illustrated separately in the various figures can be implemented in a single module and various functions of a single functional block can be implemented by one or more functional blocks. The actual number of modules and the division of particular functions between them, as well as the allocation of features between them, will vary from one implementation to another and can depend in part on the particular combination of hardware, software, and / or firmware chosen to implement a particular implementation.

[0090] While various aspects of implementations within the scope of the appended claims are described above, it should be understood that the various features described herein can be embodied in various forms, and that the described examples are to be considered only illustrative. In the light of the disclosure provided herein, those skilled in the art will appreciate that one aspect described herein can be implemented independently of any other aspects and that two or more of these aspects can be combined in various ways. For instance, an apparatus can be implemented and / or a method can be practiced using any number of the aspects set forth herein. In addition, such an apparatus can be implemented and / or such a method can be practiced using other structure and / or functionality in addition to or other than one or more of the aspects set forth herein.

[0091] It should also be understood that, although the terms “first,” “second,” etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first device could be termed a second device, and, similarly, a second device could be termed a first device, which changes the meaning of the description, so long as all occurrences of the “first device” are renamed consistently and all occurrences of the “second device” are renamed consistently. The first device and the second device are both devices, but they are not the same device.

[0092] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0093] As used herein, the term "if' can be construed to mean "when" or "in response to a determination" or "in response to a detection" that a stated condition precedent has been met, according to context. Similarly, the phrase "if it is determined" or "if [a stated condition precedent] is true" or "when [a stated condition precedent] is true" can be construed to mean "in response to a determination" or "in response to a detection" that the stated condition precedent has been met, according to context.

Claims

1. A method comprising: at one or more servers hosting a user interface (UI) engine for rendering a UI and a plurality of encoders for encoding or re-encoding a rendered UI, wherein the one or more servers comprise one or more processors and a non-transitory memory: distributing a first stream comprising a plurality of frames, wherein the first stream is encoded using a common encoder; receiving a request for a UI from a client; in response to receiving the request, generating a second stream comprising: instructing the plurality of encoders to take the plurality of frames in the first stream and re-encode the plurality of frames with varying delays relative to the first stream, locating an encoder from the plurality of encoders that is different from the common encoder and instructing the encoder to accelerate encoding of the plurality of frames up to a switch point, and facilitating mixing of the UI with the plurality of frames into the second stream starting from the switch point; and switching the client from the first stream to the second stream at the switch point.

2. The method of claim 1, wherein the UI is rendered by an application instance and the application instance has access to a shared cache.

3. The method of claim 2, further comprising: obtaining, by the application instance from the shared cache, rendered UI objects corresponding to UI elements of the UI; and composing, by the application instance, the UI for the client using the rendered UI objects.

4. The method of claim 2, further comprising: rendering, by the application instance, the UI for the client, including generating rendered UI objects corresponding to UI elements of the UI; and storing, by the application instance, the rendered UI objects to the shared cache.

5. The method of claim 1, wherein the UI comprises a portion of a native application downloadable by the client for rendering the UI on the client, and the method further comprises: identifying the portion of the UI to be rendered on the client; and preparing the native application corresponding to the portion of the UI for the client.

6. The method of claim 1, further comprising: obtaining an offset of a decoded position relative to the first stream from the client, wherein the varying delays are within a threshold of the offset of the decoded position.

7. The method of claim 6, wherein locating an encoder from the plurality of encoders that is different from the common encoder comprises: for each encoder of the plurality of encoders, identifying a respective delay relative to encoding the plurality of frames into the first stream; and selecting the encoder from the plurality of encoders based on the respective delay associated with the encoder that is closest to the offset of the decoded position.

8. The method of claim 1, further comprising: identifying the switch point to overlay the UI on the plurality of frames, wherein: the first stream is encoded using a common encoder; Facilitating mixing of the UI with the plurality of frames into the second stream starting at the switch point includes overlaying the UI over the plurality of frames in the second stream starting at the switch point.

9. The method of claim 1, further comprising: identifying a key frame in the first stream encoded prior to receiving the request for the UI and the switch point that follows the key frame, wherein: indicating the encoder to accelerate encoding of the plurality of frames up to a switch point includes: providing frames to the encoder starting at the key frame, and indicating the encoder to accelerate encoding of the frames starting at the key frame; and facilitating mixing of the UI with the plurality of frames into the second stream starting at the switch point includes overlaying the UI over the plurality of frames in the second stream starting at the switch point.

10. The method of claim 1, wherein switching the client from the first stream to the second stream at the switch point includes: providing the client a link in a manifest pointing to a first set of segments in the first stream prior to the switch; and replacing the link pointing to the first set of segments with a link pointing to a second set of segments in the second stream in the manifest in response to receiving the request for the UI.

11. The method of claim 1, further comprising: switching the client from the second stream to the first stream upon ceasing display of the UI on the client.

12. A system comprising: one or more processors; non-transitory memory; and one or more programs stored in the non-transitory memory, which, when executed, cause the one or more processors to perform operations comprising: distributing a first stream comprising a plurality of frames, wherein the first stream is encoded using a common encoder; receiving a request for a user interface (UI) from a client; generating a second stream in response to receiving the request, including: indicating a plurality of encoders to take the plurality of frames in the first stream and re-encode the plurality of frames with varying delays relative to the first stream, locating an encoder from the plurality of encoders that is different from the common encoder and indicating the encoder to accelerate encoding of the plurality of frames up to a switch point, and facilitating mixing of the UI with the plurality of frames into the second stream starting at the switch point; and switching the client from the first stream to the second stream at the switch point.

13. The system of claim 12, wherein the UI is rendered by an application instance and the application instance has access to a shared cache.

14. The system of claim 13, wherein the one or more programs further cause the one or more processors to perform operations comprising: obtaining, by the application instance from the shared cache, a rendered UI object of a UI element corresponding to the UI; and composing, by the application instance, the UI for the client using the rendered UI object.

15. The system of claim 14, wherein the one or more programs further cause the one or more processors to perform operations comprising: receiving, by the application instance from the client, a request for the UI; and in response to receiving the request for the UI, obtaining, by the application instance from the shared cache, a rendered UI object of a UI element corresponding to the UI.

16. The system of claim 15, wherein the one or more programs further cause the one or more processors to perform operations comprising: composing, by the application instance, the UI for the client using the rendered UI object.

17. The system of claim 16, wherein the one or more programs further cause the one or more processors to perform operations comprising: receiving, by the application instance from the client, a request to cease display of the UI; and in response to receiving the request to cease display of the UI, obtaining, by the application instance from the shared cache, a rendered UI object of a UI element corresponding to the UI.

18. The system of claim 17, wherein the one or more programs further cause the one or more processors to perform operations comprising: composing, by the application instance, the UI for the client using the rendered UI object.

15. The system of claim 13, wherein the one or more programs further cause the one or more processors to perform operations comprising: rendering, by the application instance, the UI for the client, including generating rendered UI objects corresponding to UI elements of the UI; and storing, by the application instance, the rendered UI objects to the shared cache.

16. The system of claim 12, wherein the UI includes a portion of the UI that is downloadable by the client for rendering on the client, and the one or more programs further cause the one or more processors to perform operations comprising: identifying the portion of the UI to be rendered on the client; and preparing, for the client, the local application corresponding to the portion of the UI.

17. The system of claim 12, wherein the one or more programs further cause the one or more processors to perform operations comprising: obtaining, from the client, an offset of a decoded position relative to the first stream, wherein the varying latency is within a threshold of the offset from the decoded position.

18. The system of claim 17, wherein locating, from the plurality of encoders, an encoder different from the common encoder comprises: identifying, for each encoder of the plurality of encoders, a respective latency relative to encoding the plurality of frames into the first stream; and selecting, from the plurality of encoders, the encoder based on the respective latency associated with the encoder that is closest to the offset of the decoded position.

19. The system of claim 12, wherein the one or more programs further cause the one or more processors to perform operations comprising: identifying the switch point to overlay the UI on the plurality of frames, wherein: facilitating mixing, from the switch point, the UI with the plurality of frames into the second stream comprises overlaying the UI on the plurality of frames in the second stream starting at the switch point.

20. The system of claim 12, wherein the one or more programs further cause the one or more processors to perform operations comprising: identifying a key frame in the first stream that is encoded prior to receiving the request for the UI and the switch point that follows the key frame, wherein: instructing the encoder to accelerate encoding of the plurality of frames up to the switch point comprises: providing, to the encoder, frames starting from the key frame, and instructing the encoder to accelerate encoding of the frames starting from the key frame; and facilitating mixing, from the switch point, the UI with the plurality of frames into the second stream comprises overlaying the UI on the plurality of frames in the second stream starting at the switch point. ​

Images