Data channel management in interactive real-time streaming network

Abstract

A computer system is provided in which one or more data channels can be established in the same connection as any streaming channel using the same setup protocol. The data channel enabled protocol layer can establish any number and type of channels, whether only one or more channels are one or more data channels. An application using a data channel enabled protocol layer may choose to establish any number and type of channels. Integration of streaming media and related data into multiple different lanes within a single connection provides significant advantages over maintaining a separate connection and supports multiple different use cases.

Description

Data Channel Management in Interactive Real-Time Streaming Networks Background Technology

[0001] Real-time streaming of interactive video, audio, and related data to a large number of participants presents unique challenges that traditional broadcast-oriented streaming protocols have yet to address.

[0002] For example, a commonly used streaming media protocol is the Web Real-Time Communication (WEBRTC) protocol, which was originally designed to facilitate data transfer between peer-to-peer connections. When using WEBRTC, there is a protocol that uses the Hypertext Transfer Protocol (HTTP) to ingest streaming data from the source device into the WEBRTC channel. This protocol is called WebRTC-HTTP Ingestion Protocol (WHIP). There is also a protocol for outputting streaming data from the WEBRTC channel to the client device. This protocol is called WebRTC-HTTP Egress Protocol (WHEP). Neither WHIP nor WHEP protocols support the transmission of metadata or other data related to streaming media within the data channel of the WEBRTC protocol. The draft specifications for WHIP and WHEP protocols recommend using long polling over HTTP or the WebSocket protocol to handle "server send events."

[0003] Current systems supporting real-time interaction typically separate the management of the video stream from the management of any messaging layer carrying the associated data. Managing data transmission across multiple protocols introduces complexity. For example, a sports application might utilize the WebRTC or HTTP Live Streaming (HLS) protocol for video delivery, while employing a separate WebSocket system to transmit data.

[0004] Another technique already in use is to add metadata to the video stream using a technique called "insertable streaming." With this technique, metadata is appended to the video frame after including a known keyword. Standard video players that do not have a program to process that keyword will ignore the data following the keyword. Custom video players, such as web browsers that implement insertable streaming functionality, can read the metadata included after the video data and after the keyword. With insertable streaming, metadata can include information in binary or encoded binary format. With insertable streaming, metadata is stored along with its corresponding video data in any recording file. Furthermore, the metadata is inherently synchronized with the video by being included with the video data.

[0005] However, using pluggable streams has several drawbacks. First, the video must be played for the metadata to be updated in the player. If the video is paused or switched, for example, to a different video, the metadata transmission will stop. Second, if used in a multi-view scenario, the stream carrying the metadata must always be processed by the player, even if that stream is not the video being displayed. This processing doubles bandwidth consumption and can lead to a poor user experience for various reasons. Finally, pluggable streams require a custom video player to extract metadata from the video frames.

[0006] As is well known, managing persistent Transmission Control Protocol (TCP) connections to support long polling in HTTP or WebSocket protocols is challenging, especially when attempting to scale. Each TCP connection involves a three-way handshake to initiate the connection. Afterward, additional messages are sent to ensure the connection remains open. In contrast, User Datagram Protocol (UDP), which underlies the WebRTC protocol, is connectionless. Summary of the Invention

[0007] This invention provides a simplified overview of some concepts, which will be further described in the detailed embodiments below. This summary does not identify key or essential features, nor does it limit the scope of the claimed subject matter.

[0008] A computer system is provided in which one or more data channels can be established within the same connection as any streaming media channel using the same setup protocol. The setup protocol layer enabled by the data channels is capable of establishing any number and type of channels, whether it is only one or more channels, or one or more data channels, including but not limited to: only one data channel, only one or more data channels, only one audio channel, only one or more audio channels, only one video channel, only one or more video channels, or only data and audio channels, or only data and video channels, or only audio and video channels, or only data, audio, and video channels. Regardless of the number and type of connections that the WHIP / WHEP layer with data channels enabled can establish, applications using the WHIP / WHEP layer with data channels enabled can choose to establish fewer connections and, depending on the capability limitations of the WHIP / WHEP layer with data channels enabled, can request to establish any number of one or more data channels, whether it is only one channel or multiple channels, including but not limited to: only one data channel, only one or more data channels, only one audio channel, only one or more audio channels, only one video channel, only one or more video channels, or only data and audio channels, or only data and video channels, or only audio and video channels, or data, audio, and video channels.

[0009] Integrating streaming media and related data into multiple distinct channels within a single connection offers significant advantages over maintaining separate connections and supports multiple different use cases. For example, complexity is reduced by using only one communication protocol. Furthermore, all relevant data can be transmitted through a single WebRTC connection. Additionally, using the WebRTC protocol, data on a data channel can be more easily synchronized with any associated audio or video data. Moreover, each data channel transmits its data using a UDP session within the WebRTC connection, instead of a TCP connection. Establishing data channels using the WHIP / WHEP protocol and transmitting data over a WebRTC connection offers significant performance and scalability advantages compared to using other technologies to transmit streaming media-related data. By using WebRTC, the advantages of User Datagram Protocol (UDP) sessions between each node in the cluster can be leveraged to reduce the load associated with managing multiple persistent Transmission Control Protocol (TCP) connections. Compared to TCP-based solutions, the stateless UDP sessions used in WebRTC offer lower overhead and improved performance and scalability.

[0010] Utilizing bidirectional data channels associated with audio or video, or audio and video, which facilitate communication between multiple publishers and subscribers—including one-to-many, many-to-one, and many-to-many communication formats—a wide variety of real-time, interactive applications and live media delivery can be supported. Furthermore, there are advantages to using WebRTC connections solely for data channels, without audio or video, or only for data and audio. WebRTC connections can be established using only one or more data channels without any audio or video channels. Because data channels are bidirectional, low-latency, and real-time, they allow for a wide range of communication applications exchanging data, such as text-based chat, data from sports applications, telemetry data from vehicles or aircraft, and other applications. These use cases demonstrate the versatility and practicality supported by individual or additional data channels.

[0011] Therefore, in one aspect, a computer system includes a processing system comprising one or more processing devices and one or more computer storage devices, the processing system processing computer program instructions to configure the processing system to process. The configured processing system includes an application for processing data, a setup protocol client, and a real-time streaming client. The setup protocol client communicates with the application and, in response to the application, requests to establish a real-time streaming connection with another device using a real-time streaming protocol. This request includes a request to establish one or more bidirectional data channels within the real-time streaming connection. The real-time streaming client communicates with the application to receive data from the application and to transmit data to another device via the bidirectional data channels of the real-time streaming connection.

[0012] In one aspect, the computer-implemented process involves an application sending a request to a setup protocol client, which includes a request to establish one or more bidirectional data channels with another device within a real-time streaming connection using a real-time streaming protocol. The setup protocol client, in response to the request from the application, establishes a real-time streaming connection with the other device having the requested one or more bidirectional channels. The application initiates data transmission to the other device via the real-time streaming client on the requested one or more bidirectional channels. The real-time streaming client receives data from the application and transmits the data to the other device via the bidirectional data channels of the real-time streaming connection.

[0013] In one aspect, the computer system includes means for receiving a request from an application to establish one or more bidirectional data channels within a real-time streaming connection with another device using a real-time streaming protocol. The computer system also includes means for establishing a real-time streaming connection with the other device having the requested one or more bidirectional channels in response to the request from the application. The computer system further includes means for receiving data from the application to transmit data to the other device via the bidirectional data channels of the real-time streaming connection.

[0014] In one aspect, the computer program product includes a computer storage device storing computer program instructions that, when processed by a processing unit, configure the processing unit to include a setup protocol client. The setup protocol client, in response to a request from an application, establishes a real-time streaming connection with another device using a real-time streaming protocol. This request includes a request to establish one or more bidirectional data channels within the real-time streaming connection.

[0015] In one aspect, a computer system includes a processing system comprising one or more processing devices and one or more computer storage devices. The processing system processes computer program instructions to configure its processing capabilities. The configured processing system includes an application for processing data, a setup protocol client, and a real-time streaming client. The setup protocol client communicates with the application and, in response to the application, requests to establish a real-time streaming connection with another device using a real-time streaming protocol. This request includes a request to establish one or more bidirectional data channels within the real-time streaming connection. The real-time streaming client communicates with the other device to receive data through the bidirectional data channels of the real-time streaming connection and to transmit the received data to the application.

[0016] In one aspect, the computer-implemented process involves an application sending a request to a setup protocol client, which includes a request to establish one or more bidirectional data channels with another device within a real-time streaming connection using a real-time streaming protocol. The setup protocol client, in response to the request from the application, establishes a real-time streaming connection with the other device having the requested one or more bidirectional channels. The real-time streaming client receives data from the other device through the bidirectional data channels of the real-time streaming connection. The real-time streaming application provides the received data to the application.

[0017] In one aspect, the computer system includes means for receiving a request from an application to establish one or more bidirectional data channels within a real-time streaming connection with another device using a real-time streaming protocol. The computer system also includes means for establishing a real-time streaming connection with the other device having the requested one or more bidirectional channels in response to the request from the application. The computer system further includes means for receiving data from the other device via the bidirectional data channels of the real-time streaming connection, and means for transmitting the received data to the application.

[0018] In one aspect, a computer system includes a processing system comprising one or more processing devices and one or more computer storage devices, the processing system processing computer program instructions to configure the processing system to process. The processing system is configured to implement a real-time streaming server instance and a setup protocol server instance. The real-time streaming server instance supports establishing real-time streaming connections with one or more bidirectional data channels to transmit data through the one or more bidirectional data channels of the real-time streaming connection. The setup protocol server instance responds to requests from other devices to establish a real-time streaming connection with the real-time streaming server instance. Such requests include requests to establish one or more bidirectional data channels within the requested real-time streaming connection. In response to a request from a device, the setup protocol server instance sends a request to the real-time streaming server instance to establish the requested real-time streaming connection with the requested one or more bidirectional data channels. The setup protocol server instance responds to the device by returning connection information so that the device completes the requested real-time streaming connection.

[0019] In one aspect, the computer-implemented process includes a setup protocol server instance receiving a request from a device to establish a real-time streaming connection with a real-time streaming server instance. Such requests include requests to establish one or more bidirectional data channels within the requested real-time streaming connection. In response to the request, the setup protocol server instance communicates with the real-time streaming server instance to establish a real-time streaming connection with the requested bidirectional data channels. The setup protocol server instance then responds to the device by returning connection information so that the device can complete the requested real-time streaming connection.

[0020] In one aspect, a computer system includes means for receiving a request from a device to establish a real-time streaming connection with a real-time streaming server instance. Such requests include requests to establish one or more bidirectional data channels within the requested real-time streaming connection. The computer system includes means for communicating with the real-time streaming server instance in response to the request to establish a real-time streaming connection having the requested bidirectional data channels. The computer system also includes means for transmitting connection information to the device in response to the establishment of the real-time streaming connection, causing the device to complete the requested real-time streaming connection.

[0021] In one aspect, the computer program product includes a computer storage device storing computer program instructions that, when processed by a processing unit, configure the processing unit to include a setup protocol server instance. The setup protocol server instance responds to a request from a client device to establish a real-time streaming connection with a real-time streaming server instance using a real-time streaming protocol. This request includes a request to establish one or more bidirectional data channels within the real-time streaming connection. In response to this request, the setup protocol server instance communicates with the real-time streaming server instance to establish a real-time streaming connection with the requested bidirectional data channels. In response to the establishment of the real-time streaming connection, the setup protocol server instance transmits connection information to the device so that the device completes the requested real-time streaming connection.

[0022] In one aspect, a cluster of server computers includes one or more source servers and one or more edge servers, wherein each source server is configured to receive streaming data from one or more publisher devices; and wherein each edge server is configured to transmit streaming data to one or more subscriber devices; wherein each source server is configured to have a corresponding Real-Time Streaming Protocol (RTLP) server instance implementing the RLP; and each edge server is configured to have a corresponding RLP server instance implementing the RLP; and wherein the server computer cluster includes at least one setup protocol server instance that, in response to a request from a publisher device or subscriber device, establishes a real-time streaming connection with one of the RLP server instances in the server computer cluster using the RLP; wherein the request includes a request to establish one or more bidirectional data channels within the real-time streaming connection; and wherein, in response to the request, the setup protocol server instance communicates with one of the real-time streaming server instances to establish a real-time streaming connection having the requested bidirectional data channels; and wherein, in response to the establishment of the real-time streaming connection, the setup protocol server instance transmits connection information to the publisher device or subscriber device for the publisher device or subscriber device to complete the requested real-time streaming connection.

[0023] In any of the foregoing aspects, in some embodiments, the application sends data to another device via a data channel. In some embodiments, the application receives data from another device via a data channel. In some embodiments, the application both sends data to and receives data from another device via a data channel.

[0024] In any of the foregoing aspects, in some embodiments, the application further utilizes the data to process streaming media data; and wherein the setup protocol client further responds to the application to request the establishment of one or more streaming media channels on the live streaming connection; and wherein the live streaming client also communicates with the application to further receive streaming media data and data from the application, and transmits streaming media data through one or more streaming media channels of the live streaming connection, while simultaneously engaging in bidirectional data communication with the application through a bidirectional data channel of the live streaming connection. In some embodiments, the data on the bidirectional data channel includes metadata to be synchronized with the live streaming media data.

[0025] In any of the foregoing aspects, in some embodiments, the setup protocol client includes a WebRTC (WHIP) client for the HTTP Ingest protocol, and the real-time streaming client includes a WebRTC client that supports the WebRTC protocol. In some embodiments, the setup protocol client includes a WebRTC (WHEP) client for the HTTP Ingest protocol, and the real-time streaming client includes a WebRTC client that supports the WebRTC protocol. In some embodiments, the computer system includes a WebRTC streaming server with WebRTC server instances and WHIP / WHEP server instances deployed. In a WebRTC implementation, the WebRTC server instance may also be referred to as a media server.

[0026] In any of the foregoing aspects, in some embodiments, the computer system includes a flow manager for a plurality of server computers, including a plurality of WebRTC server computers, each WebRTC server computer including a corresponding WebRTC server instance. The flow manager includes a WHIP / WHEP server instance. The flow manager is configured to, in response to a request, select a WebRTC server computer for providing a requested WebRTC connection and communicate with the WebRTC server instance of the selected WebRTC server computer. The WHIP / WHEP server instance of the flow manager responds to a device requesting a WebRTC channel.

[0027] In any of the foregoing aspects, in some embodiments, the computer system includes a load balancer for a plurality of server computers, the plurality of server computers including a plurality of WebRTC server computers, each WebRTC server computer including a corresponding WebRTC server instance and an associated WHIP / WHEP server instance. In response to a request, the load balancer selects a WebRTC server computer for providing the requested WebRTC connection and communicates with the WHIP / WHEP server instance of the selected WebRTC server computer. The WHIP / WHEP server instance of the selected WebRTC server computer responds to a device requesting a WebRTC channel.

[0028] In one respect, when a user connects to a single server or when a user connects across a cluster, the computer system can be configured to establish data channels, configure transmission parameters, and facilitate simultaneous data transmission in multiple directions among the participants in the connection.

[0029] Any of the foregoing aspects may include one or more of the following features: The application provides data in a real-time streaming protocol format. The application converts data in the real-time streaming protocol format into metadata for transmission over a bidirectional data channel. The application converts metadata received from the bidirectional data channel into data in the real-time streaming protocol format. Data transmitted over the bidirectional data channel is compressed or encrypted. Compressed or encrypted data received over the bidirectional data channel is decompressed or decrypted.

[0030] Any of the foregoing aspects may include one or more of the following features: Data on the bidirectional data channel includes metadata to be synchronized with live streaming data. The metadata includes metadata timestamps corresponding to the media data timestamps of the live streaming data, and the system also uses these timestamps to ensure accurate and timely transmission of the metadata. Received metadata is cached. The metadata includes event-related data.

[0031] Any of the foregoing aspects may include one or more of the following features. Data transmitted over a bidirectional data channel includes Institute of Cable Telecommunications Engineers (SCTE) 35 data. Data transmitted over a bidirectional data channel includes binary data. Data transmitted over a bidirectional data channel includes one or more of the following data types: Key-Length-Value (KLV), JavaScript Object Notation (JSON), and Remote Procedure Call (RPC).

[0032] Any of the foregoing aspects may include one or more of the following features: Data transmitted over a bidirectional data channel includes statistics related to available bandwidth. The receiver of the live streaming media sends the statistics to the sender of the live streaming media. The sender is configured to adjust the characteristics of the transmitted stream based on the statistics. The sender uses a trained computational model to automatically adjust cluster and stream settings in real time. The statistics include information received based on the receiver's estimated maximum bit rate (REMB) or other Real-Time Transmission Control Protocol (RTCP) messages to adjust the optimal stream without exceeding the available bandwidth. One or more transcoders are used to implement adaptive bit rate. The statistics indicate the available bandwidth. The system is also configured to estimate bandwidth based on the statistics. The system is also configured to allocate network resources based on the statistics.

[0033] Any of the foregoing aspects may include one or more of the following features. Data transmitted over the data channel includes shared objects passed through the data channel to enable efficient and seamless data sharing among participants. The application includes one or more of a chat application, a whiteboard application, or a collaboration application. The application enables video conferencing. Data transmitted over the bidirectional data channel includes Digital Rights Management (DRM) metadata, which is associated with media data transmitted over one or more media data channels.

[0034] Any of the foregoing aspects may include one or more of the following features: Data transmitted over the bidirectional data channel includes Remote Procedure Call (RPC) data. RPC data includes request messages indicating an operation to be performed. RPC data includes response messages indicating the output produced by the operation to be performed. RPC data includes a browser, thereby enabling RPC execution across browsers via the data channel.

[0035] Any of the foregoing aspects may include one or more of the following features. Data transmitted via the data channel includes data and status information of nodes within a distributed computer system. The system maintains synchronization between nodes in the cluster through message passing based on data and status information from the data channel.

[0036] Any of the foregoing aspects may also include an error correction module that processes data received through the data channel.

[0037] Any of the foregoing aspects may be implemented as a computer system, any individual component of such a computer system, a process executed by such a computer system or any individual component of such a computer system, or an article of manufacture including computer memory, in which computer program code is stored, and when the computer program code is processed by the processing systems of one or more computers, the computer program code configures the processing systems of one or more computers to provide such a computer system or individual components of such a computer system.

[0038] The following detailed description refers to the accompanying drawings, which form part of this application and illustrate specific exemplary embodiments by way of illustration. Other embodiments may be implemented without departing from the scope of this disclosure. Attached Figure Description

[0039] Figure 1A is a schematic diagram of the software stack, which uses the traditional WHIP / WHEP protocol only for audio and video channels, while using the WebSocket protocol for other data.

[0040] Figure 1B is a schematic diagram of a software stack using the technology described herein, employing the WHIP or WHEP protocol to support data channels within a WebRTC connection.

[0041] Figure 1C is a block diagram of the interaction between the client and a server computer that supports WebRTC and WHIP or WHEP protocols, as shown in Figure 1B.

[0042] Figure 2 is a sequence diagram illustrating how the data channel is integrated into the example system.

[0043] Figure 3 is a sequence diagram depicting an example session with a single server.

[0044] Figures 4A and 4B are sequence diagrams illustrating an example session using a signaling server and a server cluster.

[0045] Figures 5A and 5B are sequence diagrams describing the first example session with a load-balanced cluster.

[0046] Figures 5C and 5D are sequence diagrams describing a second example session with a load-balanced cluster.

[0047] Figure 6 is a sequence diagram illustrating an example session using a data channel that does not have streaming media capabilities.

[0048] Figure 7 is a block diagram illustrating how such a system can be used to support adaptive bit rate applications, such as transmitting data for bandwidth calculations.

[0049] Figure 8 is a block diagram illustrating how such a system can be used for ad insertion.

[0050] Figure 9 is a block diagram illustrating how such a system can be used for chat.

[0051] Figure 10 is a block diagram illustrating how such a system can be used for file transfer.

[0052] Figure 11 is a block diagram illustrating how such a system can be used for screen sharing control.

[0053] Figure 12 is a block diagram illustrating how such a system can be used to transmit metadata related to real-time events (such as sporting events) to client devices.

[0054] Figure 13 is a block diagram illustrating how such a system can be used to transmit operations related to real-time events from client devices.

[0055] Figure 14 is a block diagram illustrating how such systems propagate data channel messages.

[0056] Figures 15A and 15B are sequence diagrams depicting a session that uses a data channel to transmit digital rights management (DRM) information.

[0057] Figure 16 is a block diagram of an example general-purpose computer. Detailed Implementation

[0058] Figure 1A is a schematic diagram of a software stack using the traditional WHIP / WHEP protocol and the WebSocket protocol. In Figure 1A, application 100 running on client computer 101 is used to communicate with another application on another computer (not shown), and the communication content combines streaming media data and related metadata. The client computer also has instances of WHIP / WHEP client layer 102, WebRTC client layer 104, and WebSocket client layer 106 running on the client computer. Application 100 establishes two communication connections to communicate with the other computer. Using the WHIP or WHEP protocol, signaling is sent to the WebRTC server (not shown) (as shown in 112), and application 100 configures WebRTC layer 104 to perform bidirectional communication transmission of any audio or video streaming data 114 with the WebRTC server over audio and video channel 108 via the WebRTC connection. To communicate the transmission of any other data 116, application 100 configures WebSocket layer 106 to establish a data channel 110 via a TCP connection to transmit this additional data. Alternatively, long polling over an HTTP connection can be used to transmit other data 116.

[0059] At least one of the documents titled "WebRTC-HTTP Inception Protocol (WHIP)" by S. Murillo and A. Gouaillard, published on October 19, 2022, and "WebRTC-HTTP Egress Protocol (WHEP)" by S. Murillo and C. Chen, published on March 29, 2023, describes conventional implementations of the WHIP and WHEP protocols. The contents of these documents are incorporated herein by reference and can be accessed using the following URLs with the HTTPS protocol ("https: / / ", domain name "datatracker.ietf.org / "): doc / draft-ietf-wish-whip / 05 / ; doc / draft-murillo-whep / 02 / ; doc / draft-ietf-wish-whep / 01 / .

[0060] As used herein, a “data channel enabled WHIP / WHEP” component, instance, or layer represents a software component that implements the WHIP protocol or the WHEP protocol, or both, and is modified as described herein to support the establishment of one or more data channels via a WebRTC connection. Which specific implementation is referred to should be obvious from the context. A client machine that sends data using the WebRTC protocol, also referred to herein as a publisher device, establishes a channel using the data channel enabled WHIP protocol and communicates with a corresponding data channel enabled WHIP instance on a server computer. A client machine that receives data using the WebRTC protocol (also referred to herein as a subscriber device) establishes a channel using the data channel enabled WHEP protocol and communicates with a corresponding data channel enabled WHEP instance on a server computer.

[0061] Figure 1B is a schematic diagram of the software stack on client computer 151, which uses a data channel enabled setup protocol, such as a modified WHIP / WHEP protocol. The WebRTC protocol, implemented by the WebRTC client layer 158, supports the establishment of one or more bidirectional real-time data channels for data transmission, one or more real-time audio data channels, or one or more video data channels, or any combination thereof. In the system of Figure 1B, the data channel enabled WHIP / WHEP client layer 150 allows application 152 on client computer 151 to configure one or more data channels for bidirectional data communication via WebRTC connection 156.

[0062] The data channel-enabled WHIP / WHEP client layer 150 is invoked by application 152 using request 155 to perform signaling (as shown in 154) with another computer (not shown), thereby establishing one or more audio channels, one or more video channels, one or more data channels, or any combination thereof, through WebRTC connection 156. The data channel-enabled WHIP / WHEP client layer 150 includes code similar to WHIP / WHEP client layer 102 (Figure 1A) to configure the audio and video channels within WebRTC connection 156 in response to request 155 from application 152. Furthermore, the data channel-enabled WHIP / WHEP client layer 150 also includes code to configure one or more data channels within WebRTC connection 156 in response to request 155 from application 152. Similar modifications are made to the WHIP / WHEP instance on the WebRTC server to respond to data channel establishment signaling. Therefore, by using the WHIP or WHEP protocol enabled by the data channel, application 152 configures the WebRTC client layer 158 to perform bidirectional communication transmission of any audio or video stream data and any other data (as shown in 157) with the WebRTC server over a single WebRTC connection 156.

[0063] The data channel-enabled WHIP / WHEP layer 150 can be programmed to establish any number or type of data channels, whether single or multiple, including but not limited to: single data channel, single or multiple data channels, single audio channel, single or multiple audio channels, single video channel, single or multiple video channels, or data and audio channels only, or data and video channels only, or audio and video channels only, or data, audio, and video channels only. Regardless of the number and type of connections that the data channel-enabled WHIP / WHEP layer can establish, applications using the data channel-enabled WHIP / WHEP layer can choose to establish fewer connections and can request to establish any number or type of data channels, whether single or multiple, including but not limited to: single data channel, single or multiple data channels, single audio channel, single or multiple audio channels, single video channel, single or multiple video channels, or data and audio channels only, or data and video channels only, or audio and video channels only, or data, audio, and video channels only.

[0064] Figure 1C is a block diagram illustrating how a data channel-enabled WHIP / WHEP layer 174 on client computer 170 connects to a corresponding data channel-enabled WHIP / WHEP server instance 172 on server computer 173 via computer network 171 to establish a WebRTC channel 178. The interaction process for establishing the channel is explained in more detail below with reference to Figures 2 through 6 and Figures 15A through 15B. In Figure 1C, application 182 sends a request 184 to the data channel-enabled WHIP / WHEP layer 174 to establish one or more channels. The data channel-enabled WHIP / WHEP client layer 174 communicates with the data channel-enabled WHIP / WHEP server instance 172 on server computer 173 using signaling 180. The data channel-enabled WHIP / WHEP server instance 172 coordinates with the WebRTC server instance 176, typically referred to as the media server, to establish the requested channel within the WebRTC connection, whether it is a data channel, an audio channel, a video channel, or any combination of one or more such channels. Although Figure 1C shows the WebRTC server instance 176 and the WHIP / WHEP server instance 172 on the same server computer 173, the signaling server component (WHIP / WHEP) and the media server component (WebRTC) can be on different machines or different server computers. The data channel-enabled WHIP / WHEP server instance 172 uses signaling 180 to return information about the WebRTC connection and the WebRTC channel to the client application 182 via the data channel-enabled WHIP / WHIP client layer 174. Then, application 182 can begin using the WebRTC layer 188 to communicate and transmit audio data, video data, or other data, or any combination of such data (as shown in 186), to and / or from server application 189 via the requested channel 178 through the WebRTC connection.

[0065] Integrating data channels with any audio or video data channels on the same WebRTC connection offers numerous advantages. For example, complexity is reduced by using only one communication protocol. Furthermore, only one WebRTC connection is managed to transmit all relevant data. Additionally, data on the data channels can be more easily synchronized with any associated audio or video data. Moreover, each data channel transmits its data using a UDP session within the WebRTC connection, rather than a TCP connection. Furthermore, leveraging bidirectional data channels associated with audio or video, or both, which communicate between multiple publishers and subscribers—including one-to-many, many-to-one, and many-to-many communication formats—enables a wide variety of real-time, interactive applications and live media delivery.

[0066] While the example implementations described herein are based on the WHIP / WHEP and WebRTC protocols, the invention is not limited to these technologies. Any real-time streaming protocol can be used that supports the real-time streaming of data from one device to another via a real-time streaming network connection. These devices can be connected in a peer-to-peer relationship or within a cluster of multiple computers, examples of which are described herein. A real-time streaming connection includes one or more bidirectional data channels, one or more streaming audio channels, or one or more streaming video channels, or any combination thereof. Such protocols are typically implemented using real-time streaming client and real-time streaming server instances. Utilizing such real-time streaming protocols, the protocol client and server instance initially communicate in response to requests from applications to handle signaling responsible for establishing the real-time streaming connection and one or more channels within that connection. Given a real-time streaming connection established with one or more bidirectional data channels, a wide variety of applications can be supported.

[0067] Given a WHIP / WHEP layer with data channel enabled as shown in Figures 1B and 1C, a variety of interesting applications can be implemented to provide live, real-time, interactive communication between multiple data producers and consumers in the system. For example, a real-time streaming system can use the WHIP and WHEP protocols to establish channels for real-time streaming media (e.g., video or audio, or both) and related data from source devices to client devices using the WebRTC protocol. This real-time media streaming can be implemented in a cluster architecture that supports a large number of clients while providing a real-time video streaming experience with end-to-end latency of less than 500 milliseconds (ms). In some embodiments, this cluster architecture can be implemented using a set of compute instances on a cloud-based infrastructure, such as server clusters provided by service providers like Amazon Web Services (AWS), Microsoft Azure, or any other cloud provider. These compute instances include stream manager devices, source devices, relay devices, and edge devices deployed on the cloud infrastructure. This cluster architecture can be implemented using technologies other than cloud-based clusters. Decentralized networks, such as those described in the patent applications mentioned below, or other collections of server computers accessible via computer networks, can be used. For example, in cloud infrastructure examples, such deployments may be controlled by the content provider or offered as a service to the content provider. In some implementations, the cluster architecture includes a set of clusters deployed in different geographic regions to serve traffic from anywhere while providing low-latency delivery of video streams. Example implementations of such cluster architectures are described in U.S. Patents 8,019,878, 8,024,469, 8,171,145, 8,166,181, 8,019,867, 11,425,113, 11,438,638, and 11,778,011, and U.S. Publications 2019 / 0320004, 2019 / 0320014, 2019 / 0028,465, 2022 / 0094,729, and 2022 / 0321,945, all of which are incorporated herein by reference.

[0068] Implementations of such live streaming systems may include one or more subscriber devices and one or more publisher devices. In some implementations, a stream manager is used to manage resources used for transmitting data streams. In some implementations, a load balancer is used to manage resources used for transmitting data streams. The following flowchart describes an implementation of a setup process implemented in a WHIP / WHEP layer with data channel enabled to configure such a live streaming system.

[0069] Figure 2 is a sequence diagram illustrating the operation of integrating data channel-enabled WHIP / WHEP client and server instances into the system. In Figure 2, a separate server 200 exists, comprising a server computer configured with a data channel-enabled WHIP / WHEP server instance and a media server (e.g., a WebRTC server). The data channel-enabled WHIP / WHEP server instance has a published URL to which other devices can publish messages, for example, using the HTTP POST command or other HTTP commands. Publisher device or subscriber device 202 also has a data channel-enabled WHIP / WHEP client layer.

[0070] The application on publisher or subscriber device 202 invokes the data channel-enabled WHIP / WHEP client layer on publisher or subscriber device 202, indicating the type of channel the application will use. The data channel-enabled WHIP / WHEP layer of the publisher or subscriber device sends an HTTP POST message to the URL of the data channel-enabled WHIP / WHEP instance on server 200 (step 204). This message includes a proposal with a session description in Session Description Protocol (SDP) format. The data channel-enabled WHIP / WHEP instance on server 200 interacts with the media server instance on server 200 to establish a WebRTC connection with the requested data, audio, and video channels and receives information returned regarding the connection. The WHIP / WHEP server instance returns a response in SDP format to the data channel-enabled WHIP / WHEP client layer in a POST response (step 206). This message includes an SDP-formatted response containing information about the WebRTC connection. In this implementation, details specifying any requested data channel are included in both the proposal and the response. The response also includes information about interactive connection establishment (ICE) candidates for the WebRTC connection. In some implementations, the publisher or subscriber device may send an HTTP Patch request to the URL of a WHIP / WHEP server instance with data channel enabled and receive an HTTP Patch response (step 205).

[0071] Using the provided ICE candidate, the publisher or subscriber device follows the standard steps, as shown in step 208, to complete the WebRTC connection by attempting to establish a connection with the ICE, facilitating the network address translation through STUN (a NAT session traversal utility) and / or TURN (a traversal that bypasses NAT using a relay) servers 201. Once the connection is established, the publisher can transmit audio and video data unidirectionally, but the data channel supports bidirectional communication.

[0072] Before the advent of systems implementing the WHIP / WHEP protocol, these steps to establish a WebRTC connection for audio and video data required sending multiple messages in both directions, increasing the latency of connection establishment.

[0073] In the example of Figure 2, after establishing a data channel on a WebRTC connection with the media server, the publisher device 202 can transmit audio or video streams through any established audio or video channel (step 212), or the subscriber device 202 can receive audio or video streams through any established audio or video channel (step 214). Furthermore, both devices can send and receive other data through any established bidirectional data channel.

[0074] Figures 3 through 6 are sequence diagrams illustrating how to incorporate communication sessions with different types of servers into a WebRTC connection setup using the techniques described in Figure 2.

[0075] Figure 3 is a sequence diagram illustrating a session between publisher device 302 and a single WebRTC server 300. This sequence diagram illustrates the end-to-end process of establishing a data channel connection in WHIP and WHEP protocols, sending and receiving data through the channel, and closing the connection.

[0076] The application on publisher device 302 invokes the data channel-enabled WHIP / WHEP client layer, indicating the type of channel the application will use. The data channel-enabled WHIP / WHEP client layer sends an HTTP POST message to the URL of the data channel-enabled WHIP / WHEP server instance on WebRTC server 300 (step 304). This message includes a proposal with an SDP-formatted session description, indicating the data channel (DC) to be created and any other channels. The data channel-enabled WHIP / WHEP server instance interacts with the WebRTC server instance on WebRTC server 300 to establish a WebRTC connection with the requested data channel and any audio and video channels, and receives information about the channels and the WebRTC connection. The WHIP / WHEP server instance returns a response in SDP format to the data channel-enabled WHIP / WHEP client layer in a POST response (step 306). This message includes an SDP-formatted response containing information about the data channel (DC) and other channels, as well as the WebRTC connection. In this implementation, details specifying any requested data channel are included in the proposal and response. In some implementations, the publisher device can communicate with a WHIP / WHEP server instance that has a data channel enabled to transmit HTTP Patch requests and HTTP Patch responses (step 305).

[0077] The response received in step 306 also includes information about the Interactive Connection Establishment (ICE) candidate for the WebRTC connection. Using the provided ICE candidate, publisher device 302 follows the standard steps, as shown in step 308, to complete the WebRTC connection by initiating ICE negotiation and configuration with STUN / TURN server 301 and WebRTC server 300. In the example of Figure 3, after establishing a data channel on the WebRTC connection, publisher device 302 can use Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to send and receive any data over the data channel, and transmit any audio or video stream over the audio or video channel (step 310). For example, publisher device can use the data channel to send statistics, other information, and chat messages, and to receive statistics, other information, and chat replies. In some implementations, WebRTC server 300 may not receive data, and WebRTC server 300 sends a “SACK” message to the WebRTC layer on publisher device 302, and publisher device 302 retransmits the data in response, as shown in step 312.

[0078] When the publisher device 302 and the application on the WebRTC server 300 stop communicating, the publisher device sends an HTTP Delete message to the URL of the WHIP / WHEP server instance with the data channel enabled (step 314). The WebRTC server 300 terminates the WebRTC connection and sends an HTTP Delete response (step 316).

[0079] Figures 4A and 4B are sequence diagrams illustrating a session with a stream manager cluster. Using the stream manager cluster, publisher device 402 transmits a stream to source server 403, which acts as a client receiver. Similarly, edge server 403 transmits a stream to subscriber device 402. Data is transmitted between the source server and edge server within the stream manager cluster using another low-latency protocol, such as the one described in the patent document cited above.

[0080] In Figure 4A, the initial proposal message is sent from publisher or subscriber device 400 to stream manager server 400 (step 404), which acts as a broker to redirect the message to the source or edge server 403 assigned to the stream. In some embodiments, stream manager server 400 is a signaling server that manages connections between publisher and subscriber devices and the media server. In some embodiments, in addition to performing signaling server functions, stream manager server 400 performs other functions to manage the server cluster, such as monitoring, cluster management, statistics collection, server allocation, and server startup and shutdown. Responses from source or edge server 403 are sent to stream manager server 400, which is then brokered to the corresponding publisher or subscriber device 402, as shown in step 406. In some embodiments, publisher or subscriber devices may communicate with stream manager server 400 via HTTP Patch requests and responses (step 405), which in turn brokers such requests and responses between source or edge servers.

[0081] The response received at step 406 also includes information about interactive connection establishment (ICE) candidates for the WebRTC connection. Using the provided ICE candidates, as shown in step 408, publisher device 402 establishes a WebRTC connection with the assigned source server by initiating ICE negotiation and configuration with STUN / TURN server 401 and source server 403. Similarly, subscriber device 402 establishes a WebRTC connection with the assigned edge server by initiating ICE negotiation and configuration with STUN / TURN server 401 and edge server 403.

[0082] In the example of Figure 4B, after a data channel is established over the WebRTC connection, publisher device 402 can use Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to directly send and receive any data with the assigned source server 403 via the data channel, and transmit any audio or video stream via the audio or video channel (step 410). Similarly, subscriber device 402 can use Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to directly send and receive any data with the assigned edge server 403 via the data channel, and receive any audio or video stream via the audio or video channel (step 410).

[0083] Similar to Figure 3, the data can be resent as shown in step 412. Furthermore, when publisher device 402 or subscriber device 402 and the application on the source server or edge server 403 cease communication, device 402 sends an HTTP Delete message to the URL of the WHIP / WHEP server instance enabled by the data channel of stream manager 400 (step 414). Stream manager 400 coordinates the termination of the WebRTC connection with server 403 and sends an HTTP Delete response to device 402 (step 416).

[0084] Figures 5A and 5B are sequence diagrams illustrating a session using a load-balancing cluster. Using the load-balancing cluster, publisher device 502 transmits streams to source server 503, which acts as a client receiver. Similarly, edge server 503 transmits streams to subscriber device 502. Data is transmitted between the source server and edge server within the server cluster using another low-latency protocol, such as the one described in the patent document cited above.

[0085] In Figure 5A, the initial proposal message is sent from the publisher or subscriber device to the load balancer server 500 (step 504). For the publisher device, the load balancer server 500 selects a source server 503 based on a load balancing algorithm to assign to the publisher device and sends a message to the selected source server. For the subscriber device, the load balancer server 500 selects an edge server 503 based on a load balancing algorithm to assign to the subscriber device and sends a message to the selected edge server assigned to the subscriber device. In some implementations, the load balancer server 500 manages a group of independent servers that are neither source devices nor edge devices. In this case, the load balancer selects a server and assigns it to either the publisher device or the subscriber device. In Figure 5A, a response from the source or edge server 503 is sent to the load balancer server 500, which is then sent to the corresponding publisher device or subscriber device (step 506) for further communication. The remaining steps of the optional Patch request and response (step 505) and the ICE configuration and negotiation with the STUN / TURN server 501 (step 508) are similar to the settings described in Figure 4A.

[0086] In the example of Figure 5B, after a data channel is established over the WebRTC connection, publisher device 502 can use the Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to send and receive any data over the data channel, and transmit any audio or video stream over the audio or video channel (step 510). Similarly, subscriber device 502 can use the Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to send and receive any data over the data channel, and receive any audio or video stream over the audio or video channel (step 510).

[0087] Similar to Figures 3 and 4B, data can be resent as shown in step 512. Furthermore, when the publisher or subscriber device 502 and the application on the source or edge server 503 cease communication, device 502 sends an HTTP Delete message to the URL of the WHIP / WHEP server instance enabled by the data channel of the load balancer server 500 (step 514). The load balancer server 500 coordinates the termination of the WebRTC connection with server 503 and sends an HTTP Delete response to device 502 (step 516). As shown in Figure 5B, this type of communication (steps 510 to 516) is proxied to the designated source or edge server 503 through the load balancer server 500.

[0088] In Figures 5C and 5D, another example implementation is shown where the load balancer initiates a connection, but subsequently, the publisher or subscriber device switches to using a direct connection with the selected source server or selected edge server after setup. In this implementation, the setup process of Figure 5C is similar to that shown in Figure 5A. The publisher or subscriber device sends an initial proposal message to the load balancer server 520 (step 524). The load balancer server 520 selects a source or edge server 523 and sends a message to the selected source or edge server. A response from the source or edge server 523 is sent to the load balancer server 520, which in turn sends the response to the corresponding publisher or subscriber device (step 526). The remaining steps of the optional patch request and response (step 525) and the ICE configuration and negotiation with the STUN / TURN server 521 (step 528) are similar to the setup process described in Figure 5A.

[0089] In the example of Figure 5D, after establishing a data channel over the WebRTC connection, publisher device 522 can use the Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to directly send and receive any data with source server 523 via the data channel (step 530), and transmit any audio or video stream via the audio or video channel. Similarly, subscriber device 522 can use the Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to directly send and receive any data with edge server 523 via the data channel (step 530), and receive any audio or video stream via the audio or video channel.

[0090] Similar to Figures 3 and 4B, data can be retransmitted as shown in step 532. Furthermore, when the publisher or subscriber device 522 and the application on the source or edge server 503 cease communication, device 522 sends an HTTP Delete message to the URL of the WHIP / WHEP server instance enabled by the data channel of load balancer server 500 (step 534). Load balancer server 500 coordinates the termination of the WebRTC connection with server 523 and sends an HTTP Delete response to device 522 (step 536).

[0091] Figure 6 is a sequence diagram illustrating a session using a data channel without streaming media. In this implementation, the Session Description Protocol (SDP) proposal and response do not specify any audio or video channels. This implementation can be applied to any configuration described in Figures 2 through 5A through 5D.

[0092] The application on publisher device 602 invokes a data channel-enabled WHIP / WHEP client layer, indicating one or more data channels, and no video or audio channels. The data channel-enabled WHIP / WHEP client layer sends a POST message to WebRTC server 600 (step 604). This message includes a proposal with an SDP-formatted session description, indicating one or more data channels (DCs) to be created. The data channel-enabled WHIP / WHEP server instance returns a response (step 606) including information about the data channels (DCs) and the WebRTC connection. Optional HTTP Patch requests and responses (step 605) and ICE negotiation and configuration with STUN / TURN server 601 (step 608) are similar to those in Figure 3. After a data channel is established on the WebRTC connection, publisher device 602 can use Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to send and receive any data over the data channel (step 610). As with other implementations, data can be retransmitted (step 612). In addition, when the application stops communicating, the HTTP Delete message (step 614) and the HTTP Delete response message (step 616) can be used to terminate the WebRTC connection.

[0093] WebRTC connections can be established using only one or more data channels, without any audio or video channels. These data channels are bidirectional, low-latency, and real-time, supporting a variety of communication applications that exchange data, such as text-based chat, data for sports applications, telemetry data from vehicles or aircraft, and more.

[0094] Figure 14 is a block diagram illustrating how such a system propagates data channel messages within a cluster system, as shown in Figures 4A and 4B, 5A and 5B, and 5C and 5D. Specifically, this block diagram describes how the system can send messages between all connected users. Notably, publisher devices 1402 or 1407 and subscriber devices 1422 or 1420 can bidirectionally exchange data over an established data channel via cluster 1450. Data 1400 can be sent from publisher device 1402 to source server 1404 located in the cluster, and then, optionally via one or more relay servers 1413, to edge server 1410 associated with subscriber device 1420. Edge server 1410 ultimately sends data 1419 to subscriber device 1420. Similarly, data can be sent from subscriber device 1422 to edge server 1412 located in the cluster, and optionally via one or more relay servers 1413 through cluster 1450 to source server 1408 associated with publisher device 1407, which ultimately sends data 1409 to publisher device 1407. Communication can be bidirectional; subscriber device 1420 can also send data to publisher devices 1402 and 1407, and also send data to subscriber device 1422. Furthermore, in some embodiments, publisher devices can send live streaming data, such as audio or video, or both, to subscriber devices. In some embodiments, data channels can route data to or from other devices, whether these devices are part of the cluster or external to the cluster.

[0095] It should be noted that in some implementations, a node cluster can distribute data channel messages throughout the cluster to maintain node synchronization, data and state consistency, or both. As another example, synchronization objects can be shared via data channels to maintain peer synchronization, data and state consistency, or both. In some implementations, a node cluster can distribute data channel messages throughout the cluster. Data is pushed throughout the cluster via message queues and then distributed to other clients that need to receive the data, and these clients implement the WHEP / WHIP protocol enabled by the data channel.

[0096] Data types: This type of data channel can transmit a variety of data types. Below is a set of non-restricted examples.

[0097] Key-Length-Value (KLV) data comprises data in any format consisting of a key field, a length field, and a value field. The key field identifies the data type being transmitted, the length field indicates the size of the data being transmitted, and the value field provides the data being transmitted. For video systems that support the SMPTE standard, KLV data is defined by the encoding standard for embedding information in video feeds. This standard is defined in SMPTE 336M-2007, and KLV data has the following format: Key field: 1 to 16 bytes; Length field: 1, 2, or 4 bytes; Value field: the data being transmitted.

[0098] JavaScript Object Notation (JSON) is an open format for data exchange. Its format is as follows: Key field: describes the data being transmitted; Value: the data being transmitted.

[0099] Remote Procedure Call (RPC) data can also be used. RPC has many standardized formats. For example, in gRPC, an RPC message consists of a request message and a response message. The request message typically includes the method name, input parameters, and optional metadata, while the response message includes the return value and optional metadata. In Apache Thrift and CORBA, RPC messages typically include information about the remote procedure to be called, any input parameters required by the procedure, and any output or return value returned by the procedure.

[0100] SCTE-35 is a standard describing the inline insertion of cue markers in a video stream. SCTE-35 data includes: program splicing information: data about timing, such as the next splicing event, the duration of the splicing event, and the type of splicing event (start, end, overlap); component splicing information: data about the content being spliced ​​or replaced, such as information about which audio and video streams will be spliced; and segment description information: metadata about segments, such as program ID, segment ID, and segment duration.

[0101] Binary, free-format, custom, or semi-structured data can also be used on the WebRTC data channel. Using this type of data, developers can define data structures to meet the needs of a given application or use case. Developers are able to design data structures in a way that they deem appropriate.

[0102] The following descriptions and related figures (Figures 7 through 13, and Figures 15A and 15B) illustrate different use cases for systems with a WHIP / WHEP layer enabled by a data channel, such as text messaging, file transfer, sports data, real-time gaming, and streaming statistics, including programs using pure data channels. Because the data channel is bidirectional, low-latency, and real-time, it allows for a wide variety of communication applications exchanging data, such as text-based chat, data for sports applications, telemetry data from vehicles or aircraft, spacecraft, ships, robots (including autonomous vehicles), and other applications. These use cases demonstrate the versatility and practicality of data channel support, whether alone or in conjunction with additional data channels. In the figures and descriptions below, the setup of a data channel connected via WebRTC, along with any associated audio or video streaming channels, is performed using the setup process described above in conjunction with Figures 2 through 6.

[0103] Adaptive bit rate, adaptive compression, and adaptive transcoding are illustrated in Figure 7. Information provided by the data channel can be used to select a transcoder that implements the desired bit rate or other characteristics of the encoded data, such as the data format that can be processed by downstream devices. Using the data channel in this way allows the system to provide more detailed flow condition information than that sent in the RTCP message used for receiver-estimated maximum bit rate (REMB) communication (see H. Alvestrand, ed., “RTCP message for Receiver Estimated Maximum Bitrate”, version 3.0, October 13, 2013, incorporated herein by reference).

[0104] Therefore, in Figure 7, the audio and video streaming system 700 provides audio and video streams to the transcoder 702. The transcoder 702 generates multiple streams 703', 703'', and 703''', representing multiple bitrate variants of the received audio and video streams, which are then provided to the cluster 704. The subscriber device 706 establishes a bidirectional data channel 710, through which it can transmit data channel messages 710 including advanced available bandwidth information. This information can be used by edge devices in the cluster 704 to which the subscriber device is assigned, selecting from bitrate variants 703', 703'', and 703''' for transmission to the subscriber device. The edge devices can then send various metadata to the subscriber device 706 via data channel 712.

[0105] In some implementations, the data channel can be used to collect and store transmission, bandwidth utilization, messages, and other data such as statistics to improve streaming quality and scale operations. In some implementations, various artificial intelligence and machine learning algorithms can be applied to the stored data to automatically adjust cluster and streaming settings in real time.

[0106] In some implementations, as shown in Figure 8, advertisement insertion or other data insertion can be achieved using the SCTE-35 standard. The SCTE-35 standard is a core signaling standard jointly issued by ANSI (American Institute of Cable and Telecommunications Engineers) for controlling advertising and content distribution for content providers and distributors. SCTE-35 signals can be used to identify advertising slots, advertising content, and program content—that is, a specific program or a segment within a program, or both. SCTE-35 signals can specify the timing of advertisement delivery by indicating the presentation time and duration of insertable digital content, as well as other information.

[0107] In some implementations, SCTE-35 data can be included as in-band data in any video content. The encoder can insert SCTE-35 signals as in-band data into the video content. For example, when using the Real-Time Messaging Protocol (RTMP), such SCTE-35 signals can be included as in-band data. In another example, when using the HTTP Live Streaming (HLS) protocol, such SCTE-35 signals can be included as part of a manifest.

[0108] In some implementations, SCTE-35 data is provided out-of-band. Using this method, servers in a live streaming cluster process the SCTE-35 data before receiving the video referenced by the SCTE-35 data (typically by timestamp). This prefetches digital content by the server, thus ensuring a higher chance of displaying the expected digital content to each client.

[0109] In some implementations, SCTE-35 data or similar data may be used in the system context described in, for example, U.S. Patent Application No. 17713401, filed April 5, 2022 (titled “SERVER-SIDE DIGITAL CONTENT INSERTION INAUDIOVISUAL STREAMS BROADCASTED THROUGH AN INTERACTIVE LIVE STREAMINGNETWORK”, which is hereby incorporated by reference.

[0110] In Figure 8, the WebRTC connection with a publisher device (not shown) may include one or more audio and video streaming channels and a data channel with a source device in cluster 804. As shown at 800, SCTE-35 metadata can be transmitted using the data channel, while video (and audio) are streamed using their respective channels. Ad data or other information streams 802 may also be provided to cluster 804 from the same or another publisher device (not shown) via the data channel (if not streaming data) or via one or more audio or video channels. Servers within cluster 804 can use SCTE-35 data and / or ad data 800 to access ad server 808 using API 806 to retrieve content specified by the SCTE-35 data. Servers within cluster 804 combine the content specified by the SCTE-35 data or other data with video stream 800 to create stream 810, which is transmitted to subscriber device 812 via the audio and video channels on the WebRTC connection to subscriber device 812. The data channel on the WebRTC connection can be used to transmit non-streaming data. The data channel over the WebRTC connection between subscriber device 812 and edge devices in cluster 804 can provide information from subscriber device 812, which can also be used to select content from ad server 808. The data channel with subscriber devices can also provide feedback from subscriber devices related to advertisements or other content.

[0111] In some implementations, as shown in Figure 9, the application providing chat functionality can run on two different end-user devices 902 and 904. In such an implementation, each end-user device 902 and 904 can be configured as a subscriber device and a publisher device connected to server cluster 900. Using the WHIP / WHEP protocol enabled by the data channel, audio and video channels are configured by the application for transmitting video or audio data, and a bidirectional data channel can be used to transmit other information 906, 908, such as text, still images, or other data. The audio and video channels, as well as the bidirectional channel, can be established over a single WebRTC connection.

[0112] Similar to chat applications, it can enable other types of communication applications, such as whiteboard applications, or collaboration applications, such as collaborative editing tools or video conferencing.

[0113] In some implementations, as shown in FIG10, application 1002 may include file transfer functionality. In this example implementation, application 1002 on an end-user device transfers a data file to server cluster 1000 via an established data channel. Then, the servers in cluster 1000 broadcast the data on the established data channel to other end-user devices 1004.

[0114] In some implementations, as shown in FIG11, applications on end-user devices 1100 and 1110 connected by cluster 1102 may include screen sharing functionality. In such applications, end-user device 1100, acting as a publisher device providing image data 1104 to cluster 1102, shares image data 1104 from the screen via a video channel. End-user device 1100 may also provide user video on another video channel on the connection. As shown in 1106, end-user device 1100 may receive control information from one or more users via a bidirectional data channel to control the content displayed on the shared screen. User video from another end-user device 1110 may also be transmitted in parallel on one or more media channels, as shown in 1108, where end-user device 1110 is configured as a publisher device. End-user devices 1100 and 1110 may be configured as subscriber devices to receive shared screen video sources. Other users' end-user devices may be configured similarly to end-user device 1110.

[0115] Metadata, such as sports metadata, is transmitted to client devices. In some implementations, as shown in FIG12, one or more applications running on one or more publisher devices 1200 can generate metadata 1214 related to sporting events or other events. In some applications, a server publisher device 1200 may be present. Video and audio (1216) from sporting events can be obtained through the application shown in FIG12, or through other systems, or through other real-time media channels within the system. As described above, metadata can be transmitted via a data channel established between the application on the publisher device and the server computer in cluster 1202 using a data channel-enabled WHIP / WHEP protocol. Data received by cluster 1202 can be shared with computational models, such as machine learning models, artificial intelligence models, or other software that generates additional metadata. The cluster can make an AI inference call 1212 to the AI ​​server 1210 that implements the computational model to request and receive such additional metadata. Metadata 1214 and additional metadata generated by AI server 1210 can be used by cluster 1306 or transmitted via a data channel established with subscriber device 1204. The received data can be displayed in various ways in the application on the subscriber's device, such as by displaying an overlay 1208 with real-time information synchronized with the displayed video and audio data 1206. Interactive data and other data from the subscriber device 1204 can also be transmitted to the cluster 1202 and used in conjunction with the AI ​​server 1210.

[0116] In some implementations, as shown in Figure 13, data transmission from client devices can also provide information in response to the real-time media and associated metadata they have received. The provided information can also be sent via an established data channel.

[0117] For example, as shown in Figure 13, one or more applications executing on one or more publisher devices 1300, and typically originating from multiple individual publisher devices 1300, can generate event-related metadata 1304. Video and audio (1302) from the event can be obtained through the application shown in Figure 13, or through other systems, or through other real-time media channels within the system. As described above, metadata 1304 can be transmitted via a data channel established between the application on the publisher device and the server computer in cluster 1306 using a data channel-enabled WHIP / WHEP protocol. Data received by cluster 1306 can be used to access server 1310 via API 1308, which receives and stores, and / or provides, additional metadata. Metadata 1304 and additional metadata generated by server 1310 can be transmitted via a data channel established with subscriber device 1312. The received data can be displayed in various ways in the application on the subscriber device, such as by displaying an overlay 1316 with real-time information synchronized with the displayed video and audio data 1314. User interactions can be transmitted to cluster 1306 via the established data channel and stored in server 1310.

[0118] In some implementations, as shown in Figures 15A and 15B, Digital Rights Management (DRM) technology can be supported using the WHIP / WHEP protocol enabled by a data channel. Figures 15A and 15B are sequence diagrams illustrating a session for transmitting DRM information using a data channel. Specifically, the sequence diagrams illustrate how a data channel can be used to apply DRM. An end-user device 1502, such as a publisher device or a subscriber device, can request DRM data (step 1509), such as a key, from a DRM key server 1503. As shown in step 1511, after receiving the DRM key, the end-user device 1502 can use this information to transmit data through the data channel in a manner similar to the data channel usage described above in conjunction with Figures 2 through 6. In the examples of Figures 15A and 15B, only a data channel is created.

[0119] The application on publisher device 1502 invokes a data channel-enabled WHIP / WHEP client layer, indicating one or more data channels, but without video or audio channels. The data channel-enabled WHIP / WHEP client layer sends a POST message to WebRTC server 1500 (step 1504). This message includes a proposal with an SDP-formatted session description, indicating one or more data channels (DCs) to be created. The data channel-enabled WHIP / WHEP server instance returns a response (step 1506) including information about the data channels (DCs) and the WebRTC connection. Optional HTTP Patch requests and responses (step 1505) and ICE negotiation and configuration with STUN / TURN server 1501 (step 1508) are similar to those in Figure 6.

[0120] After establishing a data channel over the WebRTC connection, publisher device 602 can use Datagram Transport Layer Security (DTLS) exchange of the WebRTC protocol to send and receive any data over the data channel (step 1510, FIG. 15B). This communication can be used to transmit request messages for protected video including a DRM key, and return messages indicating the start of DRM-protected content. As in other embodiments, data can be retransmitted (step 1512). Furthermore, HTTP Delete and HTTP Delete response messages 1515 and 1516 can be used to terminate the WebRTC connection when the application stops communicating. In some embodiments, the DRM key is rotated over time, and additional requests and responses with the DRM server can be performed (step 1513).

[0121] Remote Procedure Call (RPC), compression, encryption, and other features are included in some implementations. The data channel can be used to transmit RPC commands, data, and configuration data. This implementation allows RPC to be executed across a web browser on the client device. Because the data channel is bidirectional, it can carry request messages and any response messages.

[0122] In some implementations, information transmitted over a data channel can be compressed. In such implementations, the sending application can implement a compression type suitable for the transmitted information. The receiving application can implement corresponding decompression.

[0123] In some implementations, information transmitted over a data channel can be encrypted. In such implementations, the sending application can implement an encryption type suitable for the transmitted information. The receiving application can implement the corresponding decryption.

[0124] In some implementations, real-time streaming protocols such as RTMP, RTSP, SRT, SIZ, MPEG-TS, and MoQ (Media over QUIC) may include metadata. A receiving device for such a stream can extract the metadata from the streaming protocol and convert it into data transmitted over a separate data channel. Another device can receive the metadata and convert it into metadata within the real-time streaming protocol.

[0125] Several example implementations have now been described. Figure 16 illustrates an example of a general-purpose computing device that can be used as a node, server, or device, such as a user device, client device, publisher device, subscriber device, edge server or edge node, relay server or relay node, source server or source node, stream manager, media server, transcoder, application server, content server, or other computer systems described herein for providing a computer cluster. This is merely one example of a computer and is not intended to imply any limitation on the scope or functionality of such a computer. The aforementioned system can be implemented in one or more computer programs executing on one or more such computers as shown in Figure 16.

[0126] Figure 16 is a block diagram of a general-purpose computer that uses a processing system to process computer program code. Computer programs on a general-purpose computer typically include an operating system and application programs. The operating system is a computer program that runs on the computer and manages and controls the application programs' and the operating system's access to various computer resources, including controlling the execution and scheduling of computer programs. These resources typically include memory, storage, communication interfaces, input devices, and output devices. The operation's management of these resources typically involves processing input from these resources.

[0127] Examples of such general-purpose computers include, but are not limited to, large computer systems such as server computers, database computers, desktop computers, laptop computers and notebook computers, and mobile or handheld computing devices such as tablet computers, handheld computers, smartphones, media players, personal data assistants, audio or video recorders or wearable computing devices.

[0128] Referring to Figure 16, the example computer 1600 includes a processing system comprising at least one processing unit 1602 and a memory 1604. The computer may have multiple processing units 1602 and multiple devices implementing the memory 1604. The processing unit 1602 may include one or more independently operating processing cores (not shown). Additional coprocessing units, such as a graphics processing unit 1620, may also be present in the computer. The memory 1604 may include volatile devices (e.g., dynamic random access memory (DRAM) or other random access memory devices), non-volatile devices (e.g., read-only memory, flash memory, etc.), or some combination of both, and optionally include any memory available in the processing device. Other memory, such as dedicated memory or registers, may also reside in the processing unit. This memory configuration is shown in Figure 16 by dashed line 1604. The computer 1600 may include additional memory (removable or non-removable), including but not limited to magnetically or optically recorded disks or magnetic tapes. This additional memory is shown in Figure 16 by removable memory 1608 and non-removable memory 1610. The various components in Figure 16 are typically interconnected via interconnection mechanisms such as one or more buses 1630.

[0129] Computer storage media are any media in which a computer can store and retrieve data from an addressable physical storage location. Computer storage media include volatile and non-volatile storage devices, as well as removable and non-removable storage devices. Memory 1604, removable memory 1608, and non-removable memory 1610 are examples of computer storage media. Some examples of computer storage media are RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile optical disc (DVD) or other optical or magneto-optical recording storage devices, magnetic tape, magnetic tape, disk storage, or other magnetic storage devices. Computer storage media and communication media are mutually exclusive media categories.

[0130] Computer 1600 may also include communication connection 1612, which allows the computer to communicate with other devices via a communication medium. A communication medium typically transmits computer program code, data structures, program modules, or other data over wired or wireless media by propagating modulated data signals, such as carrier waves or other transmission mechanisms, over the medium. The term "modulated data signal" refers to a signal whose one or more characteristics are set or altered to encode information in the signal, thereby changing the configuration or state of the signal receiving device. By way of example and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media, including any non-wired communication medium that allows signals to propagate, such as acoustic, electromagnetic, electrical, optical, infrared, radio frequency, and other signals. Communication connection 1612 is a device, such as a network interface or a radio transmitter, that interacts with the communication medium to send and receive data from signals propagated over the communication medium.

[0131] A communication connection may include one or more radio transmitters for telephone communication over a cellular telephone network, or a wireless communication interface for wireless connection to a computer network. For example, a computer may have cellular connections, Wi-Fi connections, Bluetooth connections, and other connections. This connection supports communication with other devices, such as voice or data communication.

[0132] The computer 1600 may have various input devices 1614, such as various pointing (whether single-pointer or multi-pointer) devices, such as mice, tablets and pens, touchpads and other touch-based input devices, styluses, image input devices such as still and moving cameras, and audio input devices such as microphones. The computer may also have various output devices 1616, such as displays, speakers, printers, etc. These devices are well known in the art and need not be discussed in detail herein.

[0133] Various memories 1610, communication connections 1612, output devices 1616, and input devices 1614 can be integrated within the computer housing or connected via various input / output interface devices on the computer. In this case, reference numerals 1610, 1612, 1614, and 1616 can denote interfaces used for connecting to devices or the devices themselves.

[0134] A computer's operating system typically includes computer programs, often called drivers, that manage access to various memory devices 1610, communication connections 1612, output devices 1616, and input devices 1614. This access may include managing input and output from these devices. In the case of a communication connection, the operating system may also include one or more computer programs that implement communication protocols for transferring information between the computer and the device via the communication connection 1612.

[0135] Each component of a computer system, and that runs on one or more computers (also referred to as a "module" or "engine," etc.), can be implemented as computer program code processed by the processing system of one or more computers. Computer program code includes computer-executable instructions or computer-interpreted instructions, such as program modules, which are processed by the computer's processing system. These instructions define routines, programs, objects, components, data structures, etc., and when processed by the processing system, instruct the processing system to perform operations on data or configure the processor or computer to implement various components or data structures in computer storage. Data structures are defined in the computer program and specify how data is organized in computer memory, such as in memory devices or storage devices, so that the data can be accessed, manipulated, and stored by the computer's processing system.

[0136] It should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific embodiments described above. The specific embodiments described above are disclosed as examples only.

[0137] The claims are as follows:

Claims

1. A computer system, comprising: A processing system, comprising one or more processing devices and one or more computer storage devices, the processing system processing computer program instructions, the computer program instructions configuring the processing system to implement: an application program for processing data; A setup protocol client communicates with the application and responds to the application to request the establishment of a real-time streaming connection with another device using a real-time streaming protocol, the request including a request to establish one or more bidirectional data channels within the real-time streaming connection; and a real-time streaming client, which communicates with the application to receive data from the application and transmit the data to the other device via a bidirectional data channel connected to the real-time streaming connection.

2. The computer system according to claim 1, wherein, The application further utilizes the data to process streaming media data, and wherein the setup protocol client further responds to the application to request the establishment of one or more streaming media channels on the live streaming connection, and wherein the live streaming client also communicates with the application to further receive the streaming media data and data from the application, and transmits the streaming media data to the other device through one or more streaming media channels of the live streaming connection, while simultaneously engaging in bidirectional data communication with the application and the other device through the bidirectional data channel of the live streaming connection.

3. The computer system according to claim 2, wherein, The setup protocol client includes a WebRTC (WHIP) client for the HTTP Ingest protocol, and the real-time streaming client includes a WebRTC client that supports the WebRTC protocol.

4. The computer system according to claim 3, wherein, The computer system includes a WebRTC streaming server, which deploys WebRTC server instances and WHIP / WHEP server instances.

5. The computer system according to claim 3, wherein, The computer system includes a flow manager for a cluster of server computers, the cluster including multiple WebRTC server computers, wherein each WebRTC server computer includes a corresponding WebRTC server instance, and wherein the flow manager includes a WHIP / WHEP server instance and is configured to select a WebRTC server computer to provide a requested WebRTC connection in response to a request from the WHIP / WHEP server instance, and communicate with the WebRTC server instance of the selected WebRTC server computer.

6. The computer system according to claim 3, wherein, The computer system includes a load balancer for a set of server computers, the set of server computers including multiple WebRTC server computers, each WebRTC server computer including a corresponding WebRTC server instance and an associated WHIP / WHEP server instance, and wherein the load balancer is configured to select a WebRTC server computer for providing a requested WebRTC connection in response to a request, and to communicate with the WHIP / WHEP server instance of the selected WebRTC server computer, and wherein the WHIP / WHEP server instance of the selected WebRTC server computer is configured to respond to the setup protocol client.

7. The computer system according to claim 2, wherein, The data on the bidirectional data channel includes metadata that needs to be synchronized with the live streaming media data.

8. The computer system according to claim 1, wherein, The real-time streaming client is also configured to receive data from the other device via a bidirectional data channel of the real-time streaming connection and transmit the data to the application.

9. A computer system, comprising: A processing system, comprising one or more processing devices and one or more computer storage devices, the processing system processing computer program instructions, the computer program instructions configuring the processing system to implement: an application program for processing data; A protocol client is configured to communicate with the application and, in response to the application, request to establish a real-time streaming connection with another device using a real-time streaming protocol, the request including data specifying one or more bidirectional data channels for the real-time streaming connection; and a real-time streaming client, which communicates with the application to transmit data received from the other device to the application via a bidirectional data channel of the real-time streaming connection.

10. The computer system according to claim 9, wherein, The application further utilizes the data to process streaming media data, and wherein the setup protocol client further responds to the application to request the establishment of one or more streaming media channels on the live streaming connection, and wherein the live streaming client further communicates with the application to receive streaming media data from the other device through one or more streaming media channels of the live streaming connection, while simultaneously engaging in bidirectional data communication with the application and the other device through the bidirectional data channel of the live streaming connection, and transmitting the received streaming media data to the application.

11. The computer system according to claim 10, wherein, The setup protocol client includes a WebRTC client for the HTTP Egress protocol (WHEP), and the real-time streaming client includes a WebRTC client that supports the WebRTC protocol.

12. The computer system according to claim 11, wherein, The computer system includes a WebRTC streaming server, which deploys WebRTC server instances and WHIP / WHEP server instances.

13. The computer system according to claim 11, wherein, The computer system includes a flow manager for a plurality of server computers, the plurality of server computers including a plurality of WebRTC server computers, each WebRTC server computer including a corresponding WebRTC server instance, and wherein the flow manager includes a WHIP / WHEP server instance, and wherein the flow manager is configured to, in response to a request, select a WebRTC server computer for providing a requested WebRTC connection, and communicate with the WebRTC server instance of the selected WebRTC server computer.

14. The computer system of claim 11, wherein the computer system includes a load balancer for a plurality of server computers, the plurality of server computers including a plurality of WebRTC server computers, each WebRTC server computer including a corresponding WebRTC server instance and an associated WHIP / WHEP server instance, and wherein, The load balancer is configured to, in response to a request, select a WebRTC server computer for providing the requested WebRTC connection and communicate with the WHIP / WHEP server instance of the selected WebRTC server computer, wherein the WHIP / WHEP server instance of the selected WebRTC server computer responds to the setup protocol client.

15. The computer system according to claim 10, wherein, The data on the bidirectional data channel includes metadata that needs to be synchronized with the live streaming media data.

16. The computer system according to claim 9, wherein, The real-time streaming client is also configured to receive data from the application and send data to the other device via a bidirectional data channel connected to the real-time streaming connection.

17. A computer system, comprising: A processing system, comprising one or more processing devices and one or more computer storage devices, the processing system processing computer program instructions configuring the processing system to implement: a real-time streaming server instance, the real-time streaming server instance supporting the establishment of real-time streaming connections with one or more bidirectional data channels to transmit data through the one or more bidirectional data channels of the real-time streaming connection; A setup protocol server instance, in response to a request from another device to establish a real-time streaming connection with the real-time streaming server instance, the request including a request to establish one or more bidirectional data channels within the requested real-time streaming connection, wherein the setup protocol server instance, in response to a request from a device, sends a request to the real-time streaming server instance to establish the requested real-time streaming connection with the requested one or more bidirectional data channels, and responds to the device by returning connection information to complete the requested real-time streaming connection.

18. The computer system of claim 17, wherein the setup protocol server instance further responds to a request to establish one or more streaming media channels on the live streaming connection.

19. The computer system according to claim 18, wherein, The configuration protocol server instance includes a WebRTC (WHIP or WHEP) server instance for the HTTP Ingress or HTTP Egress protocol, and the real-time streaming server instance includes a WebRTC server instance that supports the WebRTC protocol.

20. The computer system according to claim 18, wherein, The data on the bidirectional data channel includes metadata that needs to be synchronized with the live streaming media data.

Images