Media server proxy for switching streaming media protocols

Abstract

A media server proxy switches a streaming media protocol ("SMP") during streaming of media segments. The media server proxy receives a request from a playback tool to provide information about outgoing media segments of the media sequence in accordance with the first SMP. The media server agent generates information about outgoing media segments and sends the information to a playback tool. The media server proxy also retrieves incoming media content of the media sequence from the remote server according to a second SMP different from the first SMP. The media server agent assembles outgoing media segments based at least in part on the incoming media content. The media server proxy streams outgoing media segments to the playback tool according to the first SMP. In this manner, the media server proxy may deliver media segments with very low latency even when the first SMP typically has a much higher latency.

Description

BACKGROUND

[0001] A streaming protocol ("SMP") defines a way to transmit media content, such as video content and audio content, over a network. A player requests media content from a media server, which streams the media content over the network. The player can be, for example, a web application that plays media content in an html / video element of a web page, and the media server can be a web server that streams the requested media content to the web application over the Internet.

[0002] Two common SMPs are HTTP Live Streaming ("HLS") and Dynamic Adaptive Streaming over HTTP ("DASH"). Typically, a player that supports HLS or DASH retrieves a playlist (sometimes called a manifest) that lists media segments of a media sequence. A media segment of a media sequence is a portion of media that represents a period of time. The player can request a media segment identified in the playlist from a media server using a URL that identifies a file of the corresponding media segment. When the player requests a media segment from the media server, the media server streams the requested media segment to the player. The player receives, decodes, and plays the requested media segment.

[0003] According to the HLS protocol or the DASH protocol, a media server that streams media segments can provide alternative versions of a media segment at different bit rates associated with different quality levels. When requesting encoded data of a media segment for playback, a player can adaptively switch bit rates between segments by changing the version of the requested media segment according to available network bandwidth or device capabilities of the player.

[0004] Streaming according to the HLS protocol or the DASH protocol works well in many usage scenarios, such as on-demand streaming over a reliable network, but can suffer from performance issues in other scenarios. For example, in scenarios that seek to provide playback with very low latency (e.g., sub-second latency), streaming according to the HLS protocol or the DASH protocol typically suffers from poor performance. In particular, in implementations where a web application manages the playback buffer size according to network conditions, or the web application retrieves a playlist too frequently, or the media segments are too long, the playback latency when using HLS or DASH can be too long. A variant of HLS called low-latency HLS ("LL-HLS") achieves lower latency in some cases. However, due to the characteristics of the protocol itself, the best-case latency using LL-HLS can still be too long, and in practice, this amount of best-case latency can not even be achievable on real networks. As another example, in lossy networks, rate control according to HLS or DASH can fail due to ineffective client-side decision processes. Furthermore, the operation of the transport protocol for HLS or DASH can actively "fallback" from timely delivery of network packets, which causes playback to be delayed or stopped in order to avoid packet loss. As another example, for media delivered using HLS or DASH, client-side operations that compose media segments from multiple sequences for playback in a single media sequence are not easily achievable. SUMMARY

[0005] In summary, the detailed description presents innovations in the operation of a media server proxy that switches streaming media protocols ("SMPs"). These innovations can enable a media server proxy that is located between a remote server and a playback tool to provide various functionalities. For example, the innovations can enable a media server proxy to deliver media segments from a remote server to a playback tool with very low latency (e.g., sub-second latency), even when the playback tool uses a conventional SMP that typically has much higher latency. As another example, the innovations can enable a remote server and a media server proxy to adapt the bit rate of media segments delivered over a lossy network, even when the playback tool uses a conventional SMP that only supports client-side rate control processes. As another example, the innovations can enable a media server proxy to compose media segments from multiple sequences for playback in a single media sequence, which can then be streamed to a playback tool using a conventional SMP. The innovations include the features encompassed by the claims.

[0006] According to a first aspect of the techniques and tools described herein, a media server agent resides between a remote server and a playback device. The media server agent receives from the playback device a request for information regarding outgoing media segments of a media sequence, based on a first SMP. The media server agent generates information about the outgoing media segments and sends this information to the playback device. The media server agent also retrieves incoming media content of the media sequence from the remote server based on a second SMP, different from the first SMP. For example, the media server agent sends a request to the remote server for a given portion of the incoming media content. The media server agent receives the given portion of the incoming media content from the remote server and buffers it. (As used herein, the terms "incoming" and "outgoing" indicate the direction from the media server agent's perspective. Incoming media content is received by the media server agent. Outgoing media segments are sent from the media server agent.) The playback tool supports the first SMP but does not need to directly support the second SMP. In some example implementations, the first SMP is HLS, and the second SMP is a low-latency SMP with lightweight congestion control or flow control. More generally, the first SMP can differ from the second SMP in various aspects. For example, the first SMP can have a higher expected minimum latency due to longer buffering by the playback tool before playback begins or a longer time window for receiving network packets. Alternatively, the first SMP can have ineffective bitrate control in a lossy network between the remote server and the playback tool due to limitations of client-side bitrate control, more aggressive congestion control, and / or more aggressive flow control. Or, the first SMP can have features that hinder client-side synthesis of media sequences. However, by supporting the second SMP, the media server agent can effectively overcome the shortcomings of the first SMP.

[0007] The media server agent assembles outgoing media segments based at least in part on incoming media content retrieved from a remote server. This may involve repackaging the incoming media content and / or transcoding the incoming media content. The media server agent stores the outgoing media segments in an outgoing media buffer. The media server agent then streams the outgoing media segments to a playback device according to a first SMP. For example, the media server agent receives a request from the playback device for providing a given outgoing media segment. The media server agent retrieves the given outgoing media segment from the outgoing media buffer and sends the given outgoing media segment to the playback device.

[0008] In some example implementations, the media server proxy and the playback tool are hosted on the same client device. In other example implementations, the media server proxy is hosted on a gateway server, and the playback tool is hosted on a client device connected to the gateway server via a local network. Typically, for the playback tool, the media server proxy can be a remote server streaming media segments according to a first SMP protocol. On the other hand, for the remote server, the media server proxy can be a local client retrieving media content according to a second SMP protocol.

[0009] According to a second aspect of the techniques and tools described herein, a media server proxy located between a remote server and a playback tool includes a request processor, a playlist generator, a server communication module, and a segment assembler. The request processor is configured to perform various request processing operations. For example, request processing operations include: receiving a request from the playback tool, based on a first SMP, for providing information about outgoing media segments of a media sequence; sending information about outgoing media segments to the playback tool; and streaming the outgoing media segments to the playback tool according to the first SMP. The playlist generator is configured to generate information about outgoing media segments. The server communication module is configured to perform various retrieval operations. For example, retrieval operations include: retrieving incoming media content of a media sequence from a remote server according to a second SMP different from the first SMP. The segment assembler is configured to perform various assembly operations. For example, assembly operations include assembling outgoing media segments at least in part based on the incoming media content. Assembly operations may also include operations such as repackaging media segments, transcoding between container formats, transcoding between codec formats, and / or transcoding between in-place bitrate / quality levels.

[0010] According to a third aspect of the techniques and tools described herein, a computer system receives software for a media server proxy. Using this software, the computer system is configured to perform operations as a media server proxy. For example, these operations are those described above with reference to the first or second aspect.

[0011] The innovations described herein can be implemented as part of a method, as part of a computer system (physical or virtual, as described below) configured to perform the method, or as part of a tangible computer-readable medium storing computer-executable instructions for causing one or more processors to perform the method when programmed. Various innovations can be used in combination or individually. The innovations described herein include those covered by the claims. This summary is provided to introduce, in a simplified form, a selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The foregoing and other objects, features, and advantages of the invention will become more apparent from the following detailed description, which continues with reference to the accompanying drawings and illustrates several examples. Examples may also have other and different applications, and some details may be modified in various aspects without departing from the spirit and scope of the disclosed innovations. Attached Figure Description

[0012] Figure 1a and Figure 1b This is a diagram illustrating an example media streaming environment in which some of the embodiments described can be implemented.

[0013] Figure 2 This is a diagram illustrating an example media server proxy in which some of the described embodiments can be implemented.

[0014] Figures 3a-3c This is a diagram illustrating the operation of a media server agent that switches between streaming media protocols (“SMP”).

[0015] Figure 4a This is a flowchart illustrating an example technique for streaming media segments using a media server agent that switches to SMP. Figure 4b and Figure 4c It shows the basis respectively. Figure 4a The flowcharts for example retrieval and streaming operations of the example technology are shown.

[0016] Figure 5 This is a diagram illustrating an example computer system in which some of the described embodiments can be implemented. Detailed Implementation

[0017] This section details an innovation in the operation of a media server agent that switches streaming media protocols (“SMP”). The media server agent sits between the remote server and the playback device. Even when the playback device only supports traditional SMP such as HLS or DASH, the innovation enables the media server agent to provide a variety of functionalities. For example, even when traditional SMP used by the playback device typically has much higher latency, the innovation allows the media server agent to deliver media segments from the remote server to the playback device with very low latency (e.g., sub-second latency). As another example, even when traditional SMP used by the playback device only supports client-side bitrate control processes, the innovation allows the remote server and the media server agent to adapt the bitrate of media segments delivered over lossy networks. As yet another example, the innovation allows the media server agent to combine media segments from multiple sequences for playback in a single media sequence, which can then be streamed to the playback device using traditional SMP.

[0018] When streaming media segments from a remote server over the internet, some playback tools only support traditional SMP such as HLS or DASH. Such traditional SMP may not be suitable for certain use cases. For example, buffer settings or congestion control operations in traditional SMP can prevent playback tools from achieving ultra-low latency delivery of encoded data, even if the network would normally allow for such delivery. As another example, flow control operations in traditional SMP can interfere with the timely delivery of encoded data over lossy networks.

[0019] To address these shortcomings, a media server agent can manage the delivery of media segments between a remote server and a playback tool. For interaction with the playback tool, the media server agent uses a traditional SMP such as HLS. On the other hand, for interaction with the remote server, the media server agent uses a different SMP that does not suffer from the shortcomings of a traditional SMP. During the streaming of media segments, the media server agent can perform operations to manage conversions or transitions between the two SMPs.

[0020] When a playback tool requests media segments of a media sequence from a remote server, a media server proxy intercepts the request. For example, the media segments of the media sequence may be advertised as being "published" to a predefined Uniform Resource Locator ("URL"), a pseudo-URL, or another address. In some example implementations, a service worker implements the media server proxy and manages the delivery of media segments within that service worker's scope. More generally, a media server proxy may intercept URLs referencing specific servers registered with that media server proxy.

[0021] In response to a request for a media segment from a remote server (received from a playback device), a media server agent dynamically generates information about the media segment. This information can be organized as a playlist (also called a catalog) according to traditional SMP principles. In some example implementations, the media server agent generates information about long sequences (e.g., 300 milliseconds per segment) of very short media segments. The media server agent can generate such information without retrieving media segments from the remote server. Specifically, the media server agent can generate information about the media segment based on a static pattern or convention of the media segment to be retrieved from the remote server. For example, the media server lists files such as 001.mp4, 002.mp4, 003.mp4, etc. In any case, the media server agent sends the information about the media segment to the playback device that sent the request. The media server agent can update the information about the media segment from time to time and resend it to the playback device.

[0022] The media server agent retrieves media segments from a remote server using a different SMP that does not suffer from the drawbacks of traditional SMPs. For example, the different SMP could be a low-latency transport protocol with lightweight flow control. More generally, when retrieving media segments from a remote server, both the media server agent and the remote server can use a custom or proprietary SMP. In some example implementations, the media server agent uses a long-lived HTTP connection to retrieve the encoded data of the media segment, delivering packets as a one-way byte stream with latency on the order of tens of milliseconds.

[0023] A media segment is a portion of media representing a period of time. For video, a media segment can be a group of pictures (“GOP”) or multiple GOPs. The encoded data of a media segment is stored in a media container. The media container can store encoded data for video content, audio content, closed-circuit captioning content, augmented reality content, virtual reality content, and / or other media content. In some example implementations, a portion of incoming media content retrieved by a media server agent from a remote server is simply used as an outgoing media segment to be streamed by the media server agent to a playback device. Alternatively, the media server agent can perform operations to repackage the incoming media content into an outgoing media segment (e.g., repackage multiple portions of the incoming media content into a single outgoing media segment, or repackage a single portion of the incoming media content into multiple outgoing media segments). In some example implementations, the media server agent does not change the media container format, media codec format, or bitrate between the incoming media content and the outgoing media segment. Alternatively, the media server agent can perform transcoding operations to switch the container format and / or media codec format or bitrate / quality level.

[0024] As used herein, the terms "incoming" and "outgoing" indicate the direction from the media server agent's perspective. Incoming media content is received by the media server agent. Outgoing media segments based on the incoming media content are sent from the media server agent.

[0025] In response to a request from a playback device for a media segment, the media server agent streams the requested media segment to the playback device according to conventional SMP principles. The media server agent and the playback device can be hosted on the same client device, in which case the delivery of media segments from the media server agent to the playback device has extremely low latency. Alternatively, the media server agent can be hosted on a gateway server, which is separated from the client device hosting the playback device by a very fast network; in this case, the delivery of media segments from the media server agent to the playback device still has very low latency. Where applicable, buffering for preventing interruptions during playback, and conventional SMP features for congestion control, flow control, or bitrate control can be disabled, as they are not needed to manage the delivery of media segments from the media server agent to the playback device.

[0026] Media segments listed in the playlist are not available for streaming to the playback tool until the media server agent has retrieved the encoded data for the media segment from the remote server. If the playback tool is unaware that a media segment is unavailable, it may request the media server agent to provide it. In this case, the media server agent delays the request until the media segment becomes available. The playback tool treats this as a normal networking issue and waits for the requested segment. Once the playback tool has received the media segment, it begins playback. Simultaneously, the media server agent retrieves and caches the next media segment. When the playback tool requests the next media segment from the media server agent, the agent can provide it immediately. In this way, playback is stabilized, allowing the playback of the current media segment to overlap with the retrieval of the next. In general, in some example implementations, achievable latency can be as low as a few hundred milliseconds.

[0027] A media server proxy and / or remote server can adjust the bitrate of media segments retrieved from a remote server and streamed to a playback tool by the media server proxy. In many cases, the remote server or media server proxy is better suited than the playback tool to determine available bandwidth and adjust the bitrate of media segments. In this scenario, the media server proxy updates information about the media segments and sends the updated information to the playback tool. The playback tool can then request the media segments listed in the updated information.

[0028] In some use cases, a media server agent distributes media clips from a remote server to playback tools or multiple playback tools as part of real-time presentation. For example, a media server agent manages the delivery of media clips from one source to n participants, where n is typically a large number such as 100, 500, 1000, or more participants. In real-time presentation, most traffic flows from a single source (at the remote server) to the playback tool, and very low latency is a primary consideration. Leveraging the fast network connection between the remote server and the media server agent, very low latency can be achieved even when the playback tool only supports traditional SMP (such as HLS) that doesn't allow for very low latency. Alternatively, a media server agent can be used in other use cases, such as peer-to-peer communication or delivering media clips over lossy networks.

[0029] In some examples, a media server agent works with a single remote server to retrieve segments of a media sequence. Alternatively, a media server agent may work with multiple remote servers to retrieve segments of a given media sequence (where different servers offer different versions of the same content) or to retrieve segments of multiple media sequences.

[0030] In some examples, the media server agent retrieves segments of a single media sequence. Alternatively, the media server agent retrieves segments of multiple media sequences and combines the media segments into a single composite sequence.

[0031] Example streaming protocol.

[0032] Typically, for interaction with playback devices, media server agents use a first streaming media protocol (“SMP”). On the other hand, for interaction with remote servers, media server agents use a second SMP, which differs from the first SMP. In one or more aspects, the first SMP may have capabilities different from the second SMP. For example, compared to the second SMP, the first SMP has a higher expected minimum latency due to longer buffering by the playback device before playback begins or a longer time window for receiving network packets. Or, as another example, compared to the second SMP, the first SMP has ineffective bitrate control in lossy networks between the remote server and the playback device due to limitations in client-side bitrate control, more aggressive congestion control, and / or more aggressive flow control. Or, as another example, compared to the second SMP, the first SMP has features that hinder client-side synthesis of media sequences.

[0033] In some example implementations, the first SMP is HTTP Live Streaming (“HLS”). In many cases, the playback tool supports HLS but not any other SMP. HLS uses a playlist that includes information about media segments that can be requested. HLS uses Hypertext Transfer Protocol (“HTTP”) as the application layer protocol, Transmission Control Protocol (“TCP”) as the transport layer protocol, and Internet Protocol (“IP”) as the network layer protocol. Encoded data for media segments is transmitted between the media server agent and the playback tool using TCP / IP, where the media server agent acts as the HTTP server and the playback tool acts as the HTTP client. Where possible, the playback tool can disable HLS latency compensation mechanisms when interacting with the media server agent. For example, the playback tool can operate in video-on-demand mode, which relies on the media server agent to manage overall latency.

[0034] Alternatively, the first SMP is a different SMP (such as HTTP Dynamic Adaptive Streaming (“DASH”) or Microsoft Smooth Streaming (“MSS”)).

[0035] In some example implementations, the second SMP is a low-latency proprietary SMP. The second SMP uses a transport protocol with lightweight flow control, along with IP, to deliver packets over the network. This transport protocol may be "lossy" because it tends to quickly designate packets as "lost" without retransmitting them, rather than actively limiting the bitrate or increasing the packet delivery window. The second SMP does not use playlists that list information about media segments for media server agents to choose from. Furthermore, the second SMP does not use media segments in the same way as the first SMP. A portion of the incoming media content according to the second SMP may simply be a media file used to encode data containers. This portion of the incoming media content may or may not overlap with the outgoing media segments according to the first SMP (conextensive).

[0036] In some configurations, the second SMP uses a dedicated network connection between the remote server and the media server agent. Media clips from the remote server are "fan-out" to two layers of network servers, which deliver the media clips as a one-way byte stream to the client device in a predictable manner. Different playback tools can share the buffer at the intermediate server to reduce bandwidth usage.

[0037] Alternatively, the second SMP is a different SMP (e.g., Secure and Reliable Transmission (“SRT”) or Faster-than-Light Transmission (“FTL”)).

[0038] II. Example network environment.

[0039] Figure 1aAn example media streaming environment (101) in which some of the embodiments described herein may be implemented is shown. The example media streaming environment (101) includes one or more remote servers (110…11x), a network (120), and a client device (150). The client device (150) includes a media server agent (152) and a playback tool (154). The media server agent 152 interacts with the playback tool 154 using a first SMP, and the media server agent 152 interacts with the remote server 110 according to a second SMP.

[0040] A remote server (110…11x) stores and streams encoded data of media segments of a media sequence. Typically, the remote server (110…11x) provides encoded data of media segments of a media sequence to a media server agent (152) on a client device (150) according to a second SMP. The remote server (110…11x) retrieves encoded data of media segments from a memory or storage device accessible via a storage area network (not shown) or otherwise. In response to a request, the remote server sends encoded data to the media server agent (152). The remote server (110…11x) itself may receive encoded data from containers from another server. Alternatively, the remote server (110…11x) may include a media decoder, a media encoder, a container format sink, and / or a container format source. The media encoder receives segments of the media sequence and encodes the segments to produce encoded data conforming to a standardized format (e.g., AV1, H.265 / HEVC, or H.266 / VVC for video) or a proprietary format. The media encoder can be a video encoder, an audio encoder, or other types of media encoder. The container format receiver receives encoded data and organizes it into a media container according to a container format (e.g., ISO / IEC 14496-12 (Basic Media File Format), ISO / IEC 14496-14 (MP4), ASF, WebM, or other container formats). For transcoding between container formats, the container format source extracts encoded data from segments of a media sequence organized in a media container according to a container format such as ISO / IEC 14496-12 (Basic Media File Format), ISO / IEC 14496-14 (MP4), ASF, WebM, or other container formats.

[0041] The network (120) can be the Internet or a private network connection.

[0042] The operation of the media server agent (152) is detailed elsewhere. Typically, the media server agent (152) receives a request from the playback device (154) according to a first SMP for information about media segments of a media sequence, generates such information, and provides it to the playback device (154). The media server agent (152) requests encoded data of the media segments from a remote server (110…11x) according to a second SMP. Subsequently, in response to the request from the playback device (154), the media server agent (152) streams the encoded data of the media segments to the playback device (154) according to the first SMP. The media server agent (152) may include references... Figure 2 The components described may have another architecture. For transcoding operations, the media server agent (152) may include media decoders and media encoders, and / or container format receivers and container format sources, which operate as described in the reference remote server (110…11x).

[0043] The playback tool (154) at the client device (150) requests information about a media segment based on a first SMP request, and then requests the encoded data of the media segment based on the first SMP request. The playback tool (154) receives the encoded data of the media segment from the media server agent (152). The playback tool (154) includes a container format source and a media decoder. The container format source receives and processes the media container, removing the encoded data of the media segment from the media sequence, which is organized in the media container according to a container format such as ISO / IEC 14496-12 (Basic Media File Format), ISO / IEC 14496-14 (MP4), ASF, WebM, or another container format. The media decoder receives encoded data conforming to a standardized format (e.g., AV1, H.265 / HEVC, or H.266 / VVC for video) or a proprietary format, and decodes the encoded data to produce a reconstructed media segment. The media decoder can be a video decoder, an audio decoder, or other types of media decoders.

[0044] Figure 1b An alternative media streaming environment (102) in which some of the embodiments described can be implemented is shown. The media streaming environment (102) includes one or more remote servers (110…11x), a network (120), a gateway server (130), another network (140), and one or more client devices (150…15x). The gateway server (130) includes a media server agent (132) that interacts with a playback tool (154) using a first SMP and with the remote servers (110…11x) according to a second SMP. The client devices (150…15x) include the playback tool (154).

[0045] Remote servers (110…11x) store and stream encoded data of media segments within a media sequence. Typically, remote servers (110…11x) are as described in the reference... Figure 1a Operate as described. The network (120) can be the Internet or a private network connection.

[0046] The gateway server (130) can connect to multiple client devices (150…15x) via a network (140), which is typically a very fast local network. The media server agent (132) potentially supports connections to playback devices (154) at the multiple client devices (150…15x) according to a first SMP. In this way, encoded data of media segments can be shared among the playback devices (154) at the multiple client devices (150…15x) and delivered according to the first SMP, rather than retrieving encoded data separately for each playback device.

[0047] The operation of the media server agent (132) is detailed elsewhere. Typically, the media server agent (132) receives a request from the playback device (154) according to a first SMP for information about media segments of a media sequence, generates such information (or reuses previously generated information), and provides this information to the playback device (154). The media server agent (132) requests encoded data of the media segments from a remote server (110…11x) via a network (120) according to a second SMP. Subsequently, in response to the request from the playback device (154), the media server agent (132) streams the encoded data of the media segments to the playback device (154) via another network (140) according to the first SMP. The media server agent (132) may include references Figure 2 The components described may have another architecture. For transcoding operations, the media server agent (132) may include media decoders and media encoders, and / or container format receivers and container format sources, which operate as described in the reference remote server (110…11x).

[0048] The playback tool (154) at the client device (150…15x) sends requests and receives encoded data over the network (140) instead of interacting directly with the media server proxy without an intermediary network. Otherwise, the playback tool (154) at the client device (150) would refer to Figure 1a Operate as described.

[0049] In some example implementations, reference Figure 1a or Figure 1bThe described playback tool (154) is a web application that runs in a browser environment. When the playback tool interacts with the media server agent using HLS, the web application is configured for video-on-demand mode as if the media clips were part of a static movie, playing from beginning to end.

[0050] III. Example Media Server Proxy.

[0051] Figure 2 An example media server agent (200) in which some of the described embodiments can be implemented is shown. The media server agent (200) includes a request processor (210), a playlist generator (220), a playlist buffer (222), a server communication module (230), a clip assembler (240), and an outgoing media buffer (242).

[0052] The request processor (210) is configured to perform various request processing operations. For example, the request processing operations include: receiving a request from the playback device, according to a first SMP, for providing information about outgoing media segments of a media sequence, and sending information about the outgoing media segments to the playback device. In subsequent streaming, the request processing operations may include: streaming the outgoing media segments to the playback device according to the first SMP.

[0053] The playlist generator (220) is configured to generate information about outgoing media segments. The organization of this information depends on the implementation. In some example implementations, the information about the outgoing media segments is organized as a playlist. For a given outgoing media segment, the information about the outgoing media segment may include: the location of the given outgoing media segment, the filename of the given outgoing media segment, the bitrate of the given outgoing media segment, the resolution of the given outgoing media segment, the timestamp of the given outgoing media segment, the duration of the given outgoing media segment, and / or other information about the given outgoing media segment. Alternatively, the information about the outgoing media segments may be organized in some other way. In any case, the playlist buffer (222) is configured to store information about the outgoing media segments for delivery to the playback tool.

[0054] The server communication module (230) is configured to perform various retrieval operations. For example, the retrieval operation includes retrieving incoming media content of a media sequence from a remote server according to a second SMP different from the first SMP. The incoming media content may be temporarily stored in the incoming media buffer (not shown) of the media server agent (200).

[0055] The segment assembler (240) is configured to perform various assembly operations. For example, assembly operations include assembling outgoing media segments based at least in part on incoming media content. Assembly operations may also include: repackaging media segments, transcoding between container formats, transcoding between codec formats, and / or transcoding between in-place bitrate / quality levels. For assembly operations, the segment assembler (240), or more generally, the media server agent (200), may include a media decoder, a media encoder, a container format receiver, and / or a container format source. The media encoder receives segments of a media sequence and encodes the segments to produce encoded data conforming to a standardized format (e.g., AV1, H.265 / HEVC, or H.266 / VVC for video) or a proprietary format. The media encoder may be a video encoder, an audio encoder, or other types of media encoder. For transcoding between codec formats or bitrates, the media decoder receives encoded data conforming to a standardized format (e.g., AV1, H.265 / HEVC, or H.266 / VVC for video) or a proprietary format and decodes the encoded data to produce a reconstructed segment. The media decoder can be a video decoder, an audio decoder, or another type of media decoder. The container format receiver receives the encoded data and organizes it in a media container according to a container format such as ISO / IEC 14496-12 (ISO Basic Media File Format), ISO / IEC 14496-14 (MP4), ASF, WebM, or another container format. For transcoding between container formats, the container format source removes the encoded data from segments of a media sequence that are organized in media containers according to a container format such as ISO / IEC 14496-12 (ISO Basic Media File Format), ISO / IEC 14496-14 (MP4), ASF, WebM, or another container format. The outgoing media buffer (242) stores the encoded data in the containers of the media segments.

[0056] In some example implementations, reference Figure 2 The described media server agent (200) is implemented using a service worker. At the client device or gateway server, the service worker sits between the playback device and the network connecting the service worker to a remote server. The service worker can intercept requests from the playback device and take appropriate actions. Typically, the service worker is an event-driven worker that can register against a source and path. The service worker can control the remote server associated with it, intercepting and modifying requests for media segments.

[0057] Service workers run within a worker context, on a different thread than other applications. Compared to other applications, service workers can run asynchronously in a non-blocking manner.

[0058] Service workers can be implemented using JavaScript files. For service workers executing on the client device, the service worker's code can be retrieved and registered from calls on the web page. When the web page is retrieved and rendered, the service worker's code is also retrieved from the remote server, and the service worker is registered on the client device. The service worker has an associated media segment scope (more generally, an asset or resource scope) accessible through it. The service worker is installed on the client device, and a buffer for the media segment is established on the client device. The service worker is then activated. The service worker retrieves (fetches) the media segment from the remote server using a second SMP. Using a first SMP, the service worker streams the media segment to the playback tool in response to a request from the playback tool (where the request has a URL within the service worker's scope).

[0059] Some client devices support HLS but not other SMPs. Such client devices may also require the complete download of media files (for media segments) before playback begins. Even for such devices, service workers can implement a media server proxy to overcome the limitations of HLS.

[0060] Alternatively, a media server proxy can be implemented using the Media Source Extension (“MSE”) application programming interface (“API”). In this case, a JavaScript file for the MSE API is used to implement the media server proxy.

[0061] IV. Example media server proxy operation.

[0062] Figures 3a-3c The various operations (300, 310, 320) for switching the media server proxy in SMP are illustrated. Reference Figure 1a , 1b The media server agent described in option 2 or other media server agents can perform operations (300, 310, 320). The media server agent can perform different operations concurrently and asynchronously. For example, the media server agent can perform an operation to retrieve a media segment from a remote server while simultaneously performing an operation to stream the (earlier) media segment to a playback tool.

[0063] Figure 3a The operation (300) of generating and updating information about media segments of a media sequence based on a first SMP is illustrated. The media server agent receives a request for information about outgoing media segments of the media sequence. The media server agent generates information about the outgoing media segments and sends this information to the playback device.

[0064] In some example implementations, the media server agent generates information about outgoing media segments based on a defined segment organization pattern, without checking with the remote server about the incoming media content available there. Alternatively, as... Figure 3a As shown, a media server proxy can send a request to a remote server to provide information about the incoming media content. In this case, the remote server generates information about the incoming media content and sends it to the media server proxy. Upon receiving the information about the incoming media content, the media server proxy can use this information when generating information about the outgoing media segment.

[0065] The media server agent may periodically and / or, upon request from the playback tool or a remote server, update information about outgoing media segments and resend it to the playback tool. The updated information may list new outgoing media segments for long sequences. Alternatively, the updated information may reflect bitrate control decisions made by the media server agent and / or the remote server to adjust the bitrate (and quality level) of the encoded data for the media segments.

[0066] For example, such as Figure 3a As shown, a media server agent can receive requests for (updated) information about outgoing media segments. The media server agent updates the information about the outgoing media segments and sends the updated information to the playback tool. In this process, the media server agent can update the information about the outgoing media segments based on a defined segment organization pattern without checking the remote server, or the media server agent can coordinate with the remote server. For example, as... Figure 3a As shown, a media server proxy can send a request to a remote server to provide updated information about the incoming media content. In this case, the remote server updates the information about the incoming media content and sends the updated information to the media server proxy. Upon receiving the updated information about the incoming media content, the media server proxy can use the updated information about the incoming media content when updating the information about the outgoing media segments.

[0067] Figure 3bThe operation (310) of retrieving media segments from a remote server according to a second SMP different from the first SMP is illustrated. A media server agent sends a request to the remote server for providing one or more portions of incoming media content. The remote server retrieves the requested portion of the incoming media content. For example, the remote server retrieves the requested portion of the incoming media content from a memory or storage device accessible to the remote server. The remote server sends the requested portion of the incoming media content to the media server agent. Upon receipt, the media server agent may buffer the portion of the incoming media content in an incoming media buffer. The incoming media buffer may be the same as the outgoing media buffer used to store outgoing media segments, or it may be a different buffer.

[0068] The media server agent assembles one or more outgoing media segments, at least in part, based on portions of incoming media content retrieved from a remote server. This assembly may include re-encapsulation and / or transcoding operations, or it may simply include a pass-through portion of the incoming media content for use as an outgoing media segment. The outgoing media segment is then stored in an outgoing media buffer for subsequent streaming.

[0069] In some examples, portions of incoming media content retrieved from a remote server are streamed as outgoing media segments to the playback tool in a "direct delivery" operation. In other words, for the assembly operation, the media server agent sets up outgoing media segments to stream from portions of the incoming media content. An outgoing media segment is a portion of the retrieved and buffered incoming media content, where each portion of the incoming media content provides a corresponding one in the outgoing media segments.

[0070] In other examples, for the assembly operation, the media server agent repackages portions of the incoming media content already retrieved from the remote server to stream as one or more outgoing media segments to the playback device. For example, a single portion of the retrieved incoming media content can be split into multiple outgoing media segments for streaming. Alternatively, multiple portions of the retrieved incoming media content can be combined into a single outgoing media segment for streaming. More generally, a set of m portions of the incoming media content retrieved from the remote server is repackaged into a set of n outgoing media segments to be streamed to the playback device, where m and n are natural numbers, and m is not equal to n.

[0071] In other examples, for the assembly operation, the media server agent transcodes portions of incoming media content retrieved from a remote server between container formats, and / or between media codec formats or bitrate / quality levels, to produce one or more outgoing media segments streamed to a playback tool. For example, for a given portion of incoming media content retrieved from a remote server, the media server agent transcodes the given portion from a source container format to a target container format different from the source container format. This produces a version of the given portion in the target container format. The version of the given portion in the target container format provides the corresponding outgoing media segment. Alternatively, as another example, for a given portion of incoming media content retrieved from a remote server, the media server agent transcodes the given portion from a source media codec format to a target media codec format different from the source media codec format. This produces a version of the given portion in the target media codec format. The version of the given portion in the target media codec format provides the corresponding outgoing media segment in the outgoing media segment.

[0072] In other examples, as described above, the media server agent performs transcoding operations and also repackages a set of m parts of incoming media content already retrieved from a remote server into a set of n outgoing media segments to be streamed to the playback tool.

[0073] like Figure 3b As shown, the media server agent iteratively performs operation (310) to retrieve portions of incoming media content from the remote server according to the second SMP, and assembles outgoing media segments from the retrieved portions of the incoming media content. For example, the media server agent sends a request to the remote server for providing one or more additional portions of the incoming media content. The remote server retrieves the requested additional portions of the incoming media content and sends them to the media server agent. Upon receipt, the media server agent may buffer the additional portions of the incoming media content. The media server agent assembles one or more additional outgoing media segments based at least in part on the additional portions of the incoming media content retrieved from the remote server. The additional outgoing media segments are then stored in the outgoing media buffer for subsequent streaming.

[0074] Figure 3c The operation (320) of streaming a media segment to a playback device according to a first SMP is illustrated. The media server agent receives a request from the playback device for providing one or more outgoing media segments and retrieves the requested outgoing media segment from a memory or storage device. If the requested outgoing media segment is not yet available to the media server agent, the media server agent can request the encoded data of the media segment from a remote server according to a second SMP, as shown in the reference. Figure 3bThe media server proxy sends the requested outgoing media segment to the playback tool. Upon receiving it, the playback tool buffers the requested outgoing media segment for subsequent decoding and playback.

[0075] like Figure 3c As shown, the media server agent iteratively performs operations (320) to stream media segments to the playback device according to the first SMP. For example, the media server agent receives a request from the playback device for providing one or more additional outgoing media segments and retrieves the requested additional outgoing media segments from memory or storage device (or retrieves their encoded data from a remote server, such as reference 1) Figure 3b (As described above). The media server agent sends the requested additional outgoing media segment to the playback tool. Upon receiving it, the playback tool buffers the requested additional outgoing media segment for subsequent decoding and playback.

[0076] V. Example technologies for switching streaming media protocols.

[0077] Figure 4a An example technique (400) for switching streaming media protocols (“SMP”) using a media server agent is shown. Figure 4b and Figure 4c They respectively showed according to Figure 4a The example technology (400) is illustrated with example retrieval operations (401) and streaming operations (402). Media server agents and playback tools can be hosted on the same client device, as shown in the reference. Figure 1a As described. Alternatively, a media server proxy can be hosted in the reference. Figure 1b On the described gateway server, in this case, one or more playback tools are hosted on corresponding client devices connected to the gateway server via a local network. More generally, as referenced above... Figure 1a , 1b The media server proxy or another media server proxy described in 2 can perform the technology (400-402).

[0078] Typically, for a playback tool, a media server proxy acts as a remote server streaming media segments according to a first SMP. For the remote server, the media server proxy acts as a local client retrieving media content according to a second SMP different from the first SMP. For example, the first SMP is HTTP Live Streaming (“HLS”), and the second SMP is a low-latency proprietary media streaming protocol. Alternatively, the first SMP may be a different SMP (e.g., HTTP Dynamic Adaptive Streaming (“DASH”) or Microsoft Smooth Streaming (“MSS”)) and / or the second SMP may be a different SMP (e.g., Secure Reliable Transport (“SRT”) or Faster Than Light Transport (“FTL”)).

[0079] refer to Figure 4a Initially, the media server agent receives (410) a request from the playback device based on the first SMP for providing information about outgoing (to-stream) media segments of the media sequence. The media sequence includes video content, audio content, closed-circuit captioning content, augmented reality content, virtual reality content, and / or another type of media content.

[0080] Depending on the implementation, a media server proxy may use any of several mechanisms to receive requests for information about outgoing media segments. For example, the media server proxy intercepts requests for information about outgoing media segments. Based at least in part on this request, the media server proxy identifies the remote server in a list of servers in which it acts as an intermediary. Alternatively, as another example, the request for information about outgoing media segments may be part of a pseudo-URL identifying the media server proxy. Specifically, in some example implementations, the playback tool is a web application that can run in a browser environment, and the media server proxy is a service worker. The pseudo-URL is a call to a registered service worker. The outgoing media segment is then a resource within the scope of that service worker. Alternatively, the media server proxy may receive requests for information about outgoing media segments in some other way.

[0081] refer to Figure 4a As part of a processing loop, the media server agent selectively performs the following operations: generating or updating information about outgoing media segments of a media sequence, retrieving incoming media content from a remote server, and streaming outgoing media segments to a playback device. The media server agent can perform some operations concurrently and asynchronously with other operations. For example, the media server agent can retrieve incoming media content from a remote server while simultaneously streaming outgoing media segments (earlier segments in the media sequence) to a playback device. The media server agent can perform operations iteratively (e.g., iteratively retrieving portions of incoming media content from a remote server and iteratively streaming outgoing media segments to a playback device).

[0082] refer to Figure 4aThe media server agent determines (420) whether to generate or update information about outgoing media segments of the media sequence. If so, for example, the media server agent generates (430) information about the outgoing media segments and sends (440) the information about the outgoing media segments to the playback device. In some example implementations, the media server agent generates information about the outgoing media segments based on a defined segment organization pattern without checking with a remote server about the incoming media content available at the remote server. Alternatively, the media server agent may send a request to the remote server for information about the incoming media content and receive information about the incoming media content. In this case, the media server agent uses the information about the incoming media content when generating information about the outgoing media segments of the media sequence.

[0083] refer to Figure 4a When the media server agent does not generate or update information about outgoing media segments, the media server agent determines (450) whether to retrieve the incoming media content of the media sequence from the remote server. If so, the media server agent performs an operation to retrieve (460) the incoming media content of the media sequence from the remote server according to the second SMP.

[0084] The operation performed by the media server agent to retrieve the incoming media content of the media sequence according to the second SMP (460) depends on the implementation method. For example, the media server agent according to Figure 4b The operation (401) shown is performed iteratively. The media server agent sends (462) a request to the remote server for a given portion of the incoming media content. The media server agent later receives (464) the given portion of the incoming media content from the remote server and buffers (466) the given portion of the incoming media content in the incoming media buffer. The incoming media buffer may be the same as the outgoing media buffer used to store outgoing media segments, or it may be a different buffer. Alternatively, the media server agent performs other operations to retrieve (460) the incoming media content of the media sequence according to the second SMP.

[0085] The media server agent assembles (468) outgoing media segments based at least in part on incoming media content retrieved from a remote server. This assembly (468) may include re-encapsulation and / or transcoding operations, as described in reference [reference needed]. Figure 3b The described, or the assembly (468), may simply include portions that directly transmit incoming media content for use as outgoing media segments, as referenced. Figure 3bAs described. After assembly (468), the given outgoing media segment is typically a file of encoded media data within a container. The encoded media data is organized according to a media codec format, and the container is organized according to a container format. The outgoing media segment is then stored in an outgoing media buffer for subsequent streaming.

[0086] refer to Figure 4a When the media server agent does not generate or update information about outgoing media segments and does not retrieve incoming media content from a remote server, the media server agent determines (470) whether to stream the outgoing media segments to the playback device. If so, the media server agent performs an operation to stream (480) the outgoing media segments of the media sequence to the playback device according to the first SMP.

[0087] The operations performed by the media server agent to stream (480) outgoing media segments of media content according to the first SMP streaming method depend on the implementation. For example, the media server agent iteratively performs streaming operations according to the operation (402) shown in FIG1c. The media server agent receives (482) a request from the playback device for providing a given outgoing media segment from the outgoing media segments, retrieves (484) the given outgoing media segment from the outgoing media buffer, and sends (486) the given outgoing media segment to the playback device. Alternatively, the media server agent performs other operations to stream (480) outgoing media segments of media content according to the first SMP streaming method.

[0088] The media server agent checks whether (490) continues operation as part of its processing loop. If so, the media server agent selectively performs the following operations: generates or updates information about outgoing media segments of the media sequence, retrieves incoming media content of the media sequence from a remote server, and streams outgoing media segments to a playback tool.

[0089] During streaming, the media server agent can update information about outgoing media segments of the media sequence. For example, the media server agent updates information about outgoing media segments to include new outgoing media segments, and then sends the updated information about the outgoing media segments to the playback device. The media server agent can also implement bitrate control according to a second SMP. For example, the media server agent determines an adjustment to the bitrate of the media sequence. For server-side bitrate control, the media server agent can receive an instruction to adjust the bitrate of the media sequence from a remote server and determine the adjustment accordingly. Alternatively, for client-side (here, agent-side) bitrate control, the media server agent can measure one or more factors (based on packet loss or latency) and determine the adjustment accordingly. In any case, based at least in part on the adjustment of the bitrate of the media content, the media server agent updates information about outgoing media segments to include new outgoing media segments. The media server agent then sends the updated information about the outgoing media segments to the playback device. The media server agent then retrieves the incoming media content of the media sequence from the remote server at the new bitrate and streams the outgoing media segments to the playback device at the new bitrate.

[0090] In many of the aforementioned examples, the media server agent retrieves the incoming media content of a media sequence from a single remote server. In practice, the media server agent can retrieve incoming media content from one or more other remote servers. Therefore, the media server agent retrieves incoming media content from multiple remote servers for different portions of the media sequence, and the retrieval of portions of the media sequence is interleaved.

[0091] A media server agent can also combine retrieved portions of multiple media sequences into a single composite media sequence. The media server agent can combine incoming media content from multiple remote servers or a single remote server into a composite sequence, such that the composite sequence includes multiple picture windows or overlapping media content. For example, in addition to retrieving a given portion of incoming media content from a remote server, the media server agent also identifies one or more other portions of another media sequence. The media server agent combines the given portion of the incoming media content and the identified other portions into a given outgoing media segment within the outgoing media segment.

[0092] VI. Example of technical advantages.

[0093] When streaming media segments from a remote server over the internet, some playback tools only support traditional SMP, such as HLS or DASH. Such traditional SMP may not be suitable for certain use cases. For example, buffer settings or congestion control operations according to traditional SMP may prevent the playback tool from achieving ultra-low latency in the delivery of encoded data, even if the network would otherwise allow for ultra-low latency delivery. As another example, flow control operations according to traditional SMP may interfere with the timely delivery of encoded data over lossy networks. As yet another example, using traditional SMP may not support client-side operations for synthesized streaming. The methods described in this paper provide technical solutions to these and other technical problems, thus offering various technical advantages.

[0094] For example, using the methods described herein, a media server agent can manage the delivery of media segments from a remote server to the playback tool with ultra-low latency, even when the playback tool only supports traditional SMP. In some example implementations, the latency is on the order of 500 milliseconds or less.

[0095] As another example, using the method described in this paper, a media server agent can manage the delivery of media segments over lossy networks, even when the playback tool only supports traditional SMP. By using a different SMP over the lossy network, the media server agent can avoid the flow control operations required by traditional SMP, which would otherwise interfere with the timely delivery of encoded data.

[0096] As another example, using the method described in this paper, a media server agent can synthesize media segments from multiple sequences for playback in a single media sequence, which can then be streamed to a playback tool using conventional SMP.

[0097] As another example, using the method described in this paper, the media server agent decouples the SMP used for interacting with the remote server from the SMP used for interacting with the playback tool. This allows the media server agent to use a traditional SMP supported by the playback tool without reimplementing the playback tool, which may be provided by a third party. Simultaneously, the media server agent can use a different SMP for “wired” exchange with the remote server. This enables innovations targeting different SMPs to address the shortcomings of traditional SMPs, such as long latency and performance issues on lossy networks.

[0098] VII. Example Computer System.

[0099] Figure 5A generalized example of a suitable computer system (500) in which some of the described innovations can be implemented is shown. The innovations described herein relate to switching SMP media server agents. The computer system (500) is not intended to imply any limitations on its scope of use or functionality, as these innovations can be implemented in a variety of computer systems, including dedicated computer systems.

[0100] refer to Figure 5 The computer system (500) includes a central processing unit (“CPU”) or one or more processing cores (510) of CPUs and local memory (518). For example, the processing cores (510…51x) are processing cores on a single chip and execute computer-executable instructions. The number of processing cores depends on the implementation and can be, for example, 4 or 8. The local memory (518) can be volatile memory (e.g., registers, cache, random access memory (“RAM”), non-volatile memory (e.g., read-only memory (“ROM”), electrically erasable programmable ROM (“EEPROM”), flash memory) or some combination of both, which can be accessed by the respective processing cores (510). Alternatively, the processing cores (510…51x) can be part of a system-on-a-chip (“SoC”), application-specific integrated circuit (“ASIC”), or other integrated circuits.

[0101] Local memory (518) can store software (580) in the form of computer-executable instructions for operations executed by corresponding processing cores (510…51x), which implements innovative aspects of switching SMP media server proxies. Figure 5 In this context, the local memory (518) is an on-chip memory such as one or more caches, for which access operations, transfer operations, etc., using the processing core (510) are fast.

[0102] The computer system (500) also includes a graphics processing unit (“GPU”) or processing cores (530…53x) of multiple GPUs and local memory (538). The number of processing cores (530…53x) of the GPU depends on the implementation. The processing cores (530…53x) are, for example, part of a single instruction multiple data (“SIMD”) unit of the GPU. The SIMD width n, depending on the implementation, indicates the number of elements (sometimes called channels) of the SIMD unit. For example, for an ultra-wide SIMD architecture, the number of elements (channels) of the SIMD unit can be 16, 32, 64, or 128. The GPU memory (538) can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory), or some combination of both, which can be accessed by the respective processing cores (530…53x). The GPU memory (538) can store software (580) in the form of computer-executable instructions such as shader code for operations performed by the corresponding processing cores (530…53x), which implement innovative aspects of switching SMP media server proxies.

[0103] The computer system (500) includes a main memory (520), which may be volatile memory (e.g., RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory), or some combination of both, accessible by the processing core (510). The main memory (520) stores software (580) in the form of computer-executable instructions that implement innovative aspects of switching SMP media server agents. Figure 5 In this chip, the main memory (520) is an off-chip memory, and its access and transfer operations with the processing cores (510…51x, 530…53x) are relatively slow.

[0104] More generally, the term "processor" refers to any device capable of processing computer-executable instructions and can include microprocessors, microcontrollers, programmable logic devices, digital signal processors, and / or other computing devices. A processor can be a CPU, other general-purpose unit, or the processing core of a GPU. A processor can also be a special-purpose processor implemented using, for example, an ASIC or a field-programmable gate array ("FPGA"). A "processor system" is a group of one or more processors that may be located together or distributed across a network. Operations performed by a processor system can be performed by one processor of the processor system, where different processors perform different operations, or multiple processors of the processor system can perform a given operation. (Depending on the implementation, each processor of the processor system may or may not be able to perform all operations of a given subset of operations.)

[0105] The term "control logic" refers to a controller, or more generally, one or more processors, operable to process computer-executable instructions, determine results, and generate output. Depending on the implementation, control logic may be implemented by software executable on a CPU, by software controlling dedicated hardware (e.g., a GPU or other graphics hardware), or by dedicated hardware (e.g., in an ASIC).

[0106] The computer system (500) includes one or more network interface devices (540). The network interface device (540) enables communication with another computing entity (e.g., a server, other computer system) over a network. The network interface device (540) may support wired and / or wireless connections for wide area networks, local area networks, personal area networks, or other networks. For example, the network interface device may include one or more Wi-Fi® transceivers, Ethernet® ports, cellular transceivers, and / or other types of network interface devices, along with associated drivers, software, etc. The network interface device (540) transmits information over the network connection, such as computer-executable instructions, audio or video inputs or outputs, or other data in modulated data signals. A modulated data signal is a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. As an example, the network connection may use electrical, optical, RF, or other carrier waves.

[0107] The computer system (500) optionally includes a motion sensor / tracker input (542) for the motion sensor / tracker, which can track the movement of the user and objects around the user. For example, the motion sensor / tracker allows a user (e.g., a player in a game) to interact with the computer system (500) via a natural user interface using gestures and verbal commands. The motion sensor / tracker may combine gesture recognition, facial recognition, and / or voice recognition.

[0108] The computer system (500) optionally includes a game controller input (544) that receives control signals from one or more game controllers via a wired or wireless connection. The control signals may indicate user input from one or more directional pads, buttons, triggers, and / or one or more joysticks of the game controller. The control signals may also indicate user input from a touchpad or touchscreen, gyroscope, accelerometer, angular velocity sensor, magnetometer, and / or other controls or instruments of the game controller.

[0109] The computer system (500) optionally includes a media player (546) and a video source (548). The media player (546) can play DVDs, Blu-rays, etc. TMDisk, other disk media, and / or other formats of media. The video source (548) may be a camera input that accepts video input in analog or digital form from a camera that captures natural video. Alternatively, the video source (548) may be a screen capture module (e.g., a driver for an operating system, or software coupled to an operating system) that provides screen capture content as input. Alternatively, the video source (548) may be a graphics engine that provides texture data for graphics in a computer-represented environment. Alternatively, the video source (548) may be a video card, a TV tuner card, or other video input that accepts input video in analog or digital form (e.g., from a cable input, a High Definition Multimedia Interface (“HDMI”) input, or other input).

[0110] An optional audio source (550) accepts audio input in analog or digital form from a microphone or other audio input that captures audio.

[0111] The computer system (500) optionally includes a video output (860) that provides video output to a display device. The video output (560) may be an HDMI output or other type of output. An optional audio output (560) provides audio output to one or more speakers.

[0112] The storage device (570) may be removable or non-removable and includes magnetic media (such as disks, magnetic tapes, or tape cartridges), optical disc media, and / or any other media that can be used to store information and is accessible within a computer system (500). The memory (570) stores instructions for software (580) used to implement innovative aspects of the media server agent for switching SMP.

[0113] The computer system (500) may have additional features. For example, the computer system (500) includes one or more other input devices and / or one or more other output devices. Other input devices may be touch input devices such as a keyboard, mouse, pen or trackball, scanning devices, or other devices that provide input to the computer system (500). Other output devices may be printers, CD writers, or other devices that provide output from the computer system (500).

[0114] Interconnection mechanisms such as buses, controllers, or networks (not shown) interconnect with components of the computer system (500). Typically, operating system software (not shown) provides an operating environment for other software executing in the computer system (500) and coordinates the activities of the components of the computer system (500).

[0115] Figure 5 The computer system (500) is a physical computer system. Virtual machines may include, for example, Figure 5 The components of the organization shown.

[0116] The terms "application" or "program" refer to software such as any user-mode instructions that provide functionality. The software of an application (or program) may also include instructions for operating systems and / or device drivers. The software may be stored in associated memory. The software may be, for example, firmware. While a properly programmed general-purpose computer or computing device is expected to be used to execute such software, hardwired circuitry or custom hardware (e.g., ASICs) is also expected to be used in place of or in combination with software instructions. Therefore, the examples described herein are not limited to any particular combination of hardware and software.

[0117] The term "computer-readable medium" refers to any medium that contributes to providing data (e.g., instructions) that can be read by a processor and accessed within a computing environment. Computer-readable media can take many forms, including both non-volatile and volatile media. Non-volatile media include, for example, optical discs or magnetic disks and other persistent storage. Volatile media include dynamic random access memory ("DRAM"). Common forms of computer-readable media include, for example, solid-state drives, flash drives, hard disks, any other magnetic media, CD-ROMs, DVDs, any other optical media, RAM, programmable read-only memory ("PROM"), erasable programmable read-only memory ("EPROM"), USB flash drives, any other memory chips or cartridges, or any other media from which a computer can read. The term "non-transitory computer-readable medium" specifically excludes transiently propagating signals, carrier waves and waveforms, or other intangible or transient media that are still readable by a computer. The term "carrier wave" can refer to an electromagnetic wave modulated in amplitude or frequency to transmit a signal.

[0118] The innovations are described within the general context of computer-executable instructions that can execute on a computer system running on a target real or virtual processor. Computer-executable instructions may include instructions executable on the processing core of a general-purpose processor to provide the functionality described herein, instructions executable to control a GPU or dedicated hardware to provide the functionality described herein, instructions executable on the processing core of a GPU to provide the functionality described herein, and / or instructions executable on the processing core of a dedicated processor to provide the functionality described herein. In some embodiments, computer-executable instructions may be organized into program modules. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split among program modules as needed. The computer-executable instructions for program modules may execute within a local or distributed computer system.

[0119] The computer-executable instructions that implement the innovations described herein can be received and installed at a computer system. Specifically, the computer system receives software comprising the computer-executable instructions. Using this software, the computer system is configured to perform operations according to the computer-executable instructions.

[0120] The terms “system” and “device” are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computer system or device. Generally, a computer system or device can be local or distributed, and can include dedicated hardware and / or any combination of hardware and software that implements the functions described herein.

[0121] Numerous examples are described in this disclosure, and these examples are presented for illustrative purposes only. The described examples are not and are not intended to be limiting in any sense. As will be apparent from this disclosure, the innovations currently disclosed are broadly applicable to many contexts. Those skilled in the art will recognize that the disclosed innovations can be practiced with various modifications and changes, such as structural, logical, software, and electrical modifications. Although specific features of the disclosed innovations may be described with reference to one or more particular examples, it should be understood that, unless expressly stated otherwise, such features are not limited to their use in the one or more particular examples described with reference to them. This disclosure is neither a literal description of all examples nor a list of features of the invention that must be present in all examples.

[0122] When ordinal numbers (such as "first," "second," "third," etc.) are used before a term as an adjective, the ordinal number is used only to indicate a specific characteristic (unless explicitly stated otherwise), for example, to distinguish that specific characteristic from another characteristic described by the same or similar terms. The use of ordinal numbers alone, such as "first," "second," "third," etc., does not indicate any physical order or location, any temporal ranking, or any ranking in terms of importance, quality, or other aspects. Furthermore, the use of ordinal numbers alone does not define any numerical limitations on the characteristics identified by the ordinal numbers.

[0123] When describing elements, the articles “a,” “an,” “the,” and “the” are intended to indicate the presence of one or more elements. The terms “including,” “comprising,” and “having” are intended to be inclusive and mean that additional elements may exist in addition to those listed.

[0124] When describing a single device, component, module, or structure, multiple devices, components, modules, or structures (whether or not they collaborate) may be used instead. A function described as being owned by a single device may be owned by multiple devices, regardless of whether they collaborate. Similarly, where multiple devices, components, modules, or structures are described herein, a single device, component, module, or structure may be used instead of multiple devices, components, modules, or structures, regardless of whether they collaborate. A function described as being owned by multiple devices may be owned by a single device.

[0125] The techniques and tools described herein can be used independently and separately from the other techniques and tools described herein.

[0126] Unless otherwise expressly stated, devices, components, modules, or structures that communicate with each other do not need to communicate continuously. Rather, such devices, components, modules, or structures only need to send data to each other as needed or expected, and they can avoid exchanging data most of the time. For example, a device communicating with another device via the Internet may not continuously send data to that other device for several weeks at a time. Furthermore, devices, components, modules, or structures that communicate with each other can communicate directly or indirectly through one or more intermediaries.

[0127] As used herein, the term "transmit" refers to any manner of transferring information from one device, component, module, or structure to another. The term "receive" refers to any manner of obtaining information from one device, component, module, or structure at another. Devices, components, modules, or structures may be part of the same computer system or different computer systems. Information may be passed by value (e.g., as a parameter of a message or function call) or by reference (e.g., in a buffer). Depending on the context, information may be transmitted directly or through one or more intermediate devices, components, modules, or structures. As used herein, the term "connection" refers to an operational communication link between devices, components, modules, or structures, which may be part of the same computer system or different computer systems. An operational communication link may be a wired or wireless network connection, which may be direct or transmitted through one or more media (e.g., network).

[0128] As used herein, the term "set" as a noun refers to a non-empty group of elements, unless the context explicitly states otherwise. That is, unless the context explicitly states otherwise, a "set" has one or more elements.

[0129] The description of instances with several features does not imply that all or even any of these features are required. Rather, a variety of optional features are described to illustrate various possible examples of the innovations described herein. Unless otherwise explicitly stated, no features are required.

[0130] Furthermore, while process steps and stages can be described in sequential order, such processes can be configured to operate in different orders. A description of a particular order or sequence does not necessarily indicate a requirement that steps or stages be performed in that order. Steps or stages can be performed in any actual order. Moreover, although described or implied to occur non-simultaneously, some steps or stages can be performed concurrently. Describing a process as including multiple steps or stages does not imply that all or even any of the steps or stages are necessary or required. Various other examples may omit some or all of the described steps or stages. Unless explicitly stated otherwise, no step or stage is necessary or required. Similarly, although a product can be described as including multiple aspects, qualities, or characteristics, this does not imply that all of them are necessary or required. Various other examples may omit some or all of the aspects, qualities, or characteristics.

[0131] Unless otherwise expressly stated, the enumerated list of items does not imply that any or all items are mutually exclusive. Similarly, unless otherwise expressly stated, the enumerated list of items does not imply that any or all items are a combination of any category.

[0132] For the sake of presentation, detailed descriptions use terms such as "determine" and "select" to describe computer operations in a computer system. These terms indicate operations performed by one or more processors or other components in the computer system and should not be confused with human actions. The actual computer operations corresponding to these terms vary depending on the implementation.

[0133] In the examples described herein, the same reference numerals in different figures indicate the same components, modules, or operations. More generally, various alternatives to the examples described herein are possible. For example, some methods described herein can be modified by changing the order of the described method actions, by splitting, repeating, or omitting certain method actions, etc. Various aspects of the disclosed techniques can be used in combination or individually. Some innovations described herein solve one or more problems mentioned in the background art. Typically, a given technique or tool does not solve all of these problems. It should be understood that other examples can be utilized, and structural, logical, software, hardware, and electrical changes can be made without departing from the scope of this disclosure.

[0134] Given the many possible embodiments in which the principles of the disclosed invention can be applied, it should be recognized that the illustrated embodiments are merely preferred examples of the invention and should not be considered as limiting the scope of the invention. Rather, the scope of the invention is defined by the following claims. Therefore, we claim protection for the invention falling within the scope and spirit of these claims.

Claims

1. One or more computer-readable media having computer-executable instructions stored thereon, the computer-executable instructions being configured to cause a processor system, when programmed, to perform operations including: At the media server agent, a request for information about outgoing media segments of the media sequence is received from the playback device in accordance with the first streaming media protocol ("SMP"); At the media server proxy, (430) information about the outgoing media segment is generated; Information about the outgoing media segment is sent (440) from the media server agent to the playback tool; At the media server proxy, the incoming media content of the media sequence is retrieved from the remote server according to a second SMP different from the first SMP; At the media server proxy, the outgoing media segment is assembled (468) at least in part based on the incoming media content; as well as The outgoing media segment is streamed from the media server proxy (480) to the playback tool according to the first SMP.

2. The one or more computer-readable media according to claim 1, wherein, The outgoing media segment is stored in the outgoing media buffer, and wherein: Retrieving the incoming media content includes iteratively performing the following operations according to the second SMP: Send a request to the remote server for providing a given portion of the incoming media content; Receive the given portion of the incoming media content from the remote server; and The given portion of the incoming media content is cached in the incoming media buffer; and Streaming the outgoing media segment includes iteratively performing the following operations based on the first SMP: Receive a request from the playback tool to provide a given outgoing media segment from the outgoing media segments; Retrieve the given outgoing media segment from the outgoing media buffer; and Send the given outgoing media segment to the playback tool.

3. The one or more computer-readable media according to claim 1, wherein, Receiving a request to provide information about the outgoing media segment includes: Intercept requests for information about outgoing media segments; and The remote server is identified, at least in part, from the list of servers that the media server proxy acts as an intermediary for, based on the request.

4. One or more computer-readable media according to claim 1, wherein, The request used to provide information about the outgoing media segment is part of a pseudo-URL that identifies the media server proxy.

5. One or more computer-readable media according to claim 1, wherein, The playback tool is a web application that can run in a browser environment, wherein the media server proxy is a service worker, wherein the pseudo URL is a call used to register the service worker, and wherein the outgoing media segment is within the scope of the service worker.

6. One or more computer-readable media according to claim 1, wherein, Information about the outgoing media segments is organized into a playlist, and wherein, for a given outgoing media segment, the information about the outgoing media segment includes: The location of the given outgoing media segment; The filename of the given outgoing media segment; The bit rate of the given outgoing media segment; The resolution of the given outgoing media segment; The timestamp of the given outgoing media segment; and / or The duration of the given outgoing media segment.

7. One or more computer-readable media according to claim 1, wherein, The first SMP is selected from the group consisting of: HTTP Live Streaming ("HLS"), HTTP Dynamic Adaptive Streaming ("DASH"), and Microsoft Smooth Streaming ("MSS"), and the second SMP is selected from the group consisting of: Low Latency Proprietary Media Streaming Protocol, Secure and Reliable Transport ("SRT"), and Faster Than Light Transport ("FTL").

8. One or more computer-readable media according to claim 1, wherein, Compared to the second SMP, the first SMP has: Higher expected minimum latency due to longer buffering by the playback tool before playback begins or a longer window for receiving network packets; Ineffective bitrate control in the lossy network between the remote server and the playback tool due to limitations imposed by client-side bitrate control, more aggressive congestion control, and / or more aggressive flow control; or Features that hinder client-side synthesis of media sequences.

9. One or more computer-readable media according to claim 1, wherein, Assembling the transmitted media segment includes: The outgoing media segment is set from the incoming media content, and each part of the incoming media content provides a corresponding outgoing media segment in the outgoing media segment; The incoming media content is repackaged, wherein the set of m parts of the incoming media content is repackaged into a set of n outgoing media segments, where m and n are natural numbers, and m is not equal to n; or Transcode the incoming media content between container formats and / or between media codec formats.

10. One or more computer-readable media according to claim 1, wherein, The operation also includes at the media server proxy: Identify one or more other portions of another media sequence; as well as A given portion of the incoming media content and one or more other identified portions are combined to form a given outgoing media segment in the outgoing media segment.

11. One or more computer-readable media according to claim 1, wherein, The operation also includes at the media server proxy: Update the information regarding the outgoing media segment to include new outgoing media segments; and Updated information about the transmitted media segment is sent to the playback device.

12. One or more computer-readable media according to claim 1, wherein, The operation also includes at the media server proxy: Determine the adjustment of the bit rate for the media sequence; At least in part based on the adjustment of the bitrate of the media sequence, update the information about the outgoing media segment to include the new outgoing media segment; and Updated information about the transmitted media segment is sent to the playback device.

13. One or more computer-readable media according to claim 1, wherein, The media server proxy: For the playback tool, this manifests as the remote server that streams the outgoing media segment according to the first SMP; and For the remote server, this manifests as a local client retrieving the incoming media content based on the second SMP.

14. One or more computer-readable media according to claim 1, wherein: The media server proxy and the playback tool are hosted on the client device; or The media server proxy is hosted on a gateway server, and the playback tool is hosted on a client device connected to the gateway server via a local network.

15. A computer system comprising one or more processing units and memory, wherein, The computer system implements a media server proxy, which includes: Request processor (210), configured to perform request processing operations, the request processing operations including: According to the first streaming media protocol ("SMP"), a request is received from the playback tool to provide information about the outgoing media segments of the media sequence; Send information about the outgoing media segment to the playback device; and The outgoing media segment is streamed to the playback tool according to the first SMP. A playlist generator (220) is configured to generate information about the outgoing media segments; A server communication module (230) is configured to perform a retrieval operation, the retrieval operation including retrieving incoming media content of the media sequence from a remote server according to a second SMP different from the first SMP; and A fragment assembler (240) is configured to perform an assembly operation, which includes assembling the outgoing media fragments based at least in part on the incoming media content.

Images