Improving security of streaming media by converting secure media format to streaming media format

Abstract

The following detailed description presents a method for supporting Digital Rights Management (DRM) in real-time streaming. The proposed method reaches the real-time constraint by multiplexing the original encoded real-time stream as a carrier of encrypted data. A system for implementing the described method on a real-time streaming architecture is also specified.

Description

Cross-references to related applications

[0001] This application claims priority and benefit to U.S. Provisional Patent Application Serial No. 63 / 563,244, filed March 8, 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] This invention generally relates to digital rights management (DRM) in real-time streaming. Background Technology

[0003] Media streaming is by far the dominant application traffic on the Internet. In a typical streaming service, video and / or audio data is streamed as a sequence of data packets from a set of servers, where the stream is stored or captured to multiple user devices for playback. Media streaming providers typically offer different types of services, including live, streaming, and archived or video-on-demand (VoD). VoD streaming refers to the streaming of media from one or more servers storing an archived version of the stream to one or more users, without the time constraints of live or streaming services, allowing users to watch at any time. Attached Figure Description

[0004] A detailed description of embodiments of the present invention will be given and explained using the accompanying drawings.

[0005] Figure 1 It showcases a common architectural model with multiple data centers located at global access points and a media streaming service infrastructure that publishers and subscribers can connect to.

[0006] Figure 2 It showcases chunk-based content and streaming content with keyframes and predicted frames.

[0007] Figure 3 It demonstrates the process of DRM packaging and license issuance.

[0008] Figure 4 The WebRTC system architecture was demonstrated.

[0009] Figure 5A It demonstrates adaptive bitrate streaming of live streams for multiple subscribers.

[0010] Figure 5B This demonstrates live adaptive bitrate streaming under dynamic network conditions.

[0011] Figure 6 The bitrate selection is shown.

[0012] Figure 7 The H.264 access unit is shown.

[0013] Figure 8 The H.264 access unit is shown as a single-frame media segment of ISO-BMFF.

[0014] Figure 9 The H.264 access unit is shown as an encrypted ISO-BMFF single-frame media segment.

[0015] Figure 10 The process of converting H.264 AU into an encrypted ISO-BMFF single-frame media clip is demonstrated.

[0016] Figure 11 The Opus audio frame is shown as a single-frame media clip in ISO-BMFF format.

[0017] Figure 12 Opus audio frames are shown as encrypted ISO-BMFF single-frame media clip streams.

[0018] Figure 13 It demonstrates the progress from RTP to MSE in initialization and media segments.

[0019] Figure 14 This is a flowchart illustrating a method to improve the security of streaming media by converting secure media formats into streaming media formats without introducing latency.

[0020] Figure 15 It is a block diagram illustrating an example of a computer system in which at least some of the operations described herein can be implemented.

[0021] The technology described herein will become clearer to those skilled in the art by studying the specific embodiments in conjunction with the accompanying drawings. Embodiments or implementations describing various aspects of the invention are shown by way of example, and the same reference numerals may indicate similar elements. Although various embodiments are depicted in the drawings for illustrative purposes, those skilled in the art will recognize that alternative implementations may be employed without departing from the principles of the technology. Therefore, while specific embodiments are shown in the drawings, various modifications can be made to the technology. Detailed Implementation

[0022] This technology generally relates to digital rights management (DRM) for media streaming, and more specifically to DRM methods and apparatus suitable for real-time media streaming over packet-switched networks such as the Internet. DRM improves the security of streaming media because it helps prevent the illegal online sharing of content and protects content creators from digital piracy. DRM can be used for: • Restrict access to specific devices, locations, or IP addresses • Limit the number of times a user can access or use the content. • Set an expiration date for the content • Prevent users from editing, saving, or printing content • Prevent users from taking screenshots or capturing images of the screen. • Add watermarks to documents and works to establish ownership.

[0023] More specifically, this technology describes a method for supporting DRM in real-time media streaming. The key to maintaining the real-time constraints of streaming live content lies in multiplexing the original encoded real-time stream as a carrier of encrypted data. A system for implementing the described method on a real-time streaming architecture is also specified.

[0024] Given the time constraints of real-time delivery, implementing DRM is a challenge for real-time media streaming. DRM ensures the secure delivery of various types of content from content providers (CPs) to users, preventing users from unauthorized distribution of the provided content.

[0025] Various DRM solutions are available on the market, including FairPlay Streaming (Apple's solution for HLS streaming), Widevine (Google's solution for DASH and HLS streaming), PlayReady (Microsoft's solution for DASH and HLS streaming), and ClearKey Content Protection (an open DRM solution for DASH streaming). However, adopting a DRM solution for real-time media streaming can add complexity and increase latency to streaming processing due to factors such as end-user authentication, encryption / decryption processes, transmitting (larger) encrypted data over the network, and DRM license acquisition. Even an increase of a few milliseconds to a few seconds of latency can impact the normal operation of real-time streaming services.

[0026] Therefore, the disclosed technology provides systems and methods for supporting DRM while meeting the time constraints of real-time media streaming. The specification and associated drawings are illustrative examples and should not be construed as limiting. This disclosure provides certain details to enable a thorough understanding and implementation of these examples. However, those skilled in the art will understand that the invention can be practiced without many of these details. Similarly, those skilled in the art will understand that the invention may include well-known structures or features that have not been shown or described in detail to avoid unnecessarily obscuring the description of the examples. Terms and Concepts

[0027] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the scope of this disclosure. Where the context permits, words used in singular or plural forms may also include either the singular or plural forms, respectively.

[0028] As used herein, unless otherwise expressly stated, terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” and “generating” refer to the actions and processes of a computer or similar electronic computing device that manipulates data represented as physical (electronic) quantities in computer memory or registers and transforms that data into other data similarly represented as physical quantities in computer memory, registers, or other such storage media, transmission, or display devices.

[0029] As used herein, terms such as “connection” and “coupled” can refer to any direct or indirect connection or coupling between two or more elements. The coupling or connection between elements can be physical, logical, or a combination thereof.

[0030] The reference to "embodiment" or "one embodiment" means that a particular feature, function, structure, or characteristic described is included in at least one embodiment. The appearance of such phrases does not necessarily refer to the same embodiment, nor does it necessarily refer to mutually exclusive alternative embodiments.

[0031] Unless the context explicitly requires otherwise, the words “comprise” and “comprising” should be interpreted in an inclusive rather than an exclusive or exhaustive sense (i.e., in the sense of “including but not limited to”).

[0032] The term "based on" should also be interpreted in an inclusive rather than an exclusive or exhaustive sense. Therefore, unless otherwise stated, the term "based on" is intended to mean "at least partially based on".

[0033] The term "module" broadly refers to software components, hardware components, and / or firmware components. A module is typically a functional component that can generate useful data or other outputs based on specified inputs. Modules can be self-contained. A computer program may include one or more modules. Therefore, a computer program may include multiple modules responsible for performing different tasks or a single module responsible for performing multiple tasks.

[0034] The word "or" used when referring to a list containing multiple items is intended to cover all of the following interpretations: any one item in the list, all items in the list, and any combination of items in the list.

[0035] The sequence of steps performed in any of the processes described herein is exemplary. However, these steps can be performed in various sequences and combinations unless contrary to physical possibility. For example, steps can be added to or removed from the processes described herein. Similarly, steps can be substituted or reordered. Therefore, the description of any process is intended to be open-ended.

[0036] "Live streaming media" broadly refers to streaming content transmitted over the internet without prior recording and storage. This can include broadcast content generated by a source device, transmitted over the network, and rendered by a receiving device. Live video streaming services are highly tolerant of streaming latency, which is the time from when an event is streamed or when a specific action occurs to when it is delivered to subscribers. For live video streaming solutions such as Apple's HTTP Live Streaming (HLS), a few seconds of streaming latency is common and generally considered acceptable.

[0037] "Real-time media streaming" refers to the simultaneous online streaming of media as it is recorded and broadcast "as it happens" to one or more users, where the latency is low enough to allow for "natural" interaction between participants, enabling them to react to and / or interact with content in the same stream and with other participants "in real time." Examples of such latency, imperceptible to device users, could include 100 ms, 300 ms, or 500 ms. However, this definition is not limited to a specific latency threshold but is defined more generally to support natural user interaction. Therefore, users on client devices within a broadcast group can interact with or respond to the live media stream in near real-time (i.e., with latency imperceptible to users on client devices within the broadcast group).

[0038] In other words, the low streaming latency of these streaming services allows participants to react to and / or interact with the content of the same stream and with other participants. In one example, a participant subscribing to a live streaming program can directly communicate with and guide the performer's actions (e.g., interacting with a performer in a stand-up comedy show, perhaps by responding to a question). In another example, players in a live streaming card game can place bets as the cards are revealed. In yet another example, a performer can respond to requests from online viewers or to requests from players in a live streaming card game to place bets as the cards are revealed. Therefore, embodiments of this technology focus on live media streaming.

[0039] To provide real-time streaming content to various client devices, streaming devices (such as servers) can generate media streams and transmit them over a network to client devices. Live media streaming requires the use of codecs to compress (e.g., encode) the media and transport protocols to transmit the encoded media over the network. Examples of video compression standards include AVC H.264, H.265, VP8, and VP9. Examples of transport protocols include Real-Time Streaming Protocol (RTSP) / Real-Time Protocol (RTP), Real-Time Messaging Protocol (RTMP), and Apple HLS. Before content elements (e.g., ad elements, ad clips) can be inserted into a given media stream, they may need to be transcoded (converted) to a format suitable for the stream (e.g., H.264) and the client devices that will render the media stream. Content elements can include images, audio, video, text, or combinations thereof.

[0040] A media stream may include at least one keyframe (e.g., data encoded for rendering a complete frame by a client device) and a sequence of predicted or incremental frames representing differences relative to the keyframe. The media stream may include a bitrate representing the number of bits in a video frame, where an increased bitrate indicates higher resolution for the associated media (e.g., high-definition video). For a new client device subscribing to the streaming media, the client device processes the keyframe to render the media frame. After the keyframe is rendered, the client device may process subsequent predicted frames to render and output a series of frames for the media.

[0041] In one example, a series of keyframes and predicted frames can be arranged at various times along the media stream. Examples include intra-frames and inter-frames in VP8, and keyframes and P-frames in the Moving Picture Experts Group (MPEG) protocol. Keyframes can be decoded without referencing any other frames in the sequence; that is, the decoder reconstructs such frames from a default state. Keyframes provide random access (or search) points in the media stream. Keyframes are encoded with reference to previous frames, specifically, all frames preceding and including the most recent keyframe. Typically, correct decoding of predicted frames depends on the correct decoding of the most recent keyframe and all subsequent predicted frames. Therefore, decoding algorithms cannot tolerate keyframes being dropped. For example, in environments where frames may be dropped or corrupted, correct decoding cannot occur until the keyframe is correctly received. Furthermore, content elements (such as advertisements) are often inserted into or relative to keyframes in the live media stream.

[0042] RTP is a protocol designed for transmitting audio and video data over the Internet. RTP is used for real-time transmission of media streams, such as voice and video. RTP is responsible for packaging media data into small packets and transmitting them over the network. Each RTP packet contains a sequence number and a timestamp, which are used to ensure that packets are delivered in the correct order and at the correct time. RTP packets are transmitted via User Datagram Protocol (UDP), which provides low latency and is ideal for real-time communication. RTCP is a protocol designed to provide feedback on the Quality of Service (QoS) of RTP traffic. Therefore, RTCP is used to monitor network conditions (such as packet loss and latency) and provide feedback to the sender. RTCP packets are periodically sent to provide quality feedback on the RTP stream.

[0043] WebRTC is a standard platform for real-time streaming applications based on the web. WebRTC relies on RTP to transmit encoded video frames. The receiver receives RTP packets and uses a buffer to generate video frames. Furthermore, the receiver uses RTCP to send loss and latency reports to the sender, which the network controller analyzes to potentially adjust the sender's encoding rate accordingly.

[0044] ISO-BMFF (International Organization for Standardization Basic Media File Format) or MPEG4 is a container file format that defines a common structure for files containing time-based multimedia data. It is designed as a flexible and scalable format that facilitates the exchange, management, editing, and presentation of media.

[0045] Media Source Extensions (MSE) is a W3C specification that allows JavaScripts to send byte streams to media codecs within web browsers for media data playback. Browsers and devices that support MSE can play back video chunks (or request byte ranges for video segments within a single file), enabling both live streaming with adaptive bitrate and VoD playback, and also allowing for closed captions.

[0046] EME (Encrypting Media Extension) is a W3C specification that provides a communication channel between web browsers and Content Decryption Module (CDM) software that implements DRM. EME is a JavaScript API that enables HTML5-based DRM by controlling the playback of protected content through an application programming interface (API) that extends MSE. EME works by integrating CDM into the browser, device, or mobile operating system, allowing the browser or device to communicate directly with the license server.

[0047] CENC (Common Encryption Scheme) specifies standard encryption and key mapping methods that can be used by one or more digital rights management and key management systems to enable the decryption of the same file using different DRM systems. CENC details the standard encryption and key mapping techniques used to store DRM-related data from one or more DRM technologies along with compressed audio / video data. Managing multiple DRMs is crucial because most browsers or other devices only support one type of DRM, making multi-DRM support a necessity for most video producers. Supports real-time streaming services

[0048] While the disclosed embodiments describe a complete, workable method and system for supporting DRM on real-time services using specific protocols, it should be understood that other equivalent embodiments exist. Because many modifications and variations will occur to those skilled in the art upon which this disclosure is presented, the method and system are not limited to the exact construction and operation shown and disclosed. Therefore, this disclosure is intended to bring all suitable modifications and equivalents within the scope of the claims.

[0049] Figure 1 An architectural model 100 is illustrated, featuring multiple data centers located at global access points and a media streaming service infrastructure that publishers and subscribers can connect to. As shown, client device 102 communicates with media streaming source 104 to request and receive live media streams. Streaming source 104 is the media streaming service infrastructure that publishers and subscribers can connect to. Client device 102 can also communicate with data centers that share a common Point of Presence (PoP) with it. As shown, multiple data centers 106-1 to 106-6 (collectively referred to as "Data Center 106") exist worldwide. Once a data center is selected, client device 102 establishes and authenticates a persistent control channel with the streaming service of streaming source 104 before using this control channel to subscribe to the live media stream. Client device 102 attempting to subscribe to a live media stream typically first selects a nearby PoP. This selection takes a set of available regions as input and can be based on, for example, the geographic or network location of client device 102, or the service's response time to a minimum request. Encoded media stream

[0050] Figure 2 This demonstrates chunk-based content or streaming content with keyframes and predicted frames. For example, encoded media streams (e.g., chunk-based stream S1, encoded stream S2) can include data representing media content, such as live video streams.

[0051] The encoded media stream may include a series of keyframes (e.g., keyframes 202-1 to 202-3, 204-1 to 204-3) and subsequent prediction frames (e.g., prediction frame set 206, 208). A keyframe (e.g., the first keyframe 202-1) (or "i-frame") may represent a complete frame of an image in the video. As an example, a keyframe may be similar to an intraframe in VP8 or a keyframe in MPEG.

[0052] In operation, the client device can process first keyframes 202-1 and 204-1 to render the corresponding media content, such as video frames. Keyframes can be decoded without referencing any other frames in the sequence, where the decoder reconstructs such frames starting from the decoder's "default" state. In some embodiments, keyframes can provide random access (or search) points in the video stream.

[0053] The encoded media stream may also include multiple predicted frames 206, 208 (or “incremental frames”) that immediately follow the keyframe in time. Predicted frames 206, 208 can represent the differences between themselves and the keyframe, which can reduce the amount of data required to render the frame represented by the predicted frames. As an example, predicted frames 206, 208 can be similar to inter-frames in VP8 or P-frames in MPEG terminology. Predicted frames 206, 208 can be encoded with reference to previous frames, and in some embodiments, all previous frames can be encoded up to and including the most recent keyframe. In many cases, correct decoding of predicted frames 206, 208 depends on the correct decoding of the most recent keyframe and all subsequent predicted frames. Therefore, the decoding algorithm may not tolerate the case where the keyframe is dropped. In environments where frames may be dropped or corrupted, correct decoding may not be possible until the keyframe is correctly received.

[0054] Media streams can be encoded using encoding techniques such as chunked transfer encoding to create chunks (i.e., segments of the media stream that are not overlapping). Figure 2 The chunk-based stream S1 in the example can represent an example of a chunk-based stream. Chunks of the encoded media stream can be transmitted and received independently of each other. In some embodiments, neither the receiver nor the sender of the chunk-based stream may need to know about data streams outside of the chunk currently being processed.

[0055] Figure 3This demonstrates the DRM encapsulation and license issuance process. Typically, DRM for streaming media content in a web browser involves a combination of encryption, licensing, and secure playback. Media to be streamed is encrypted on the server side before transmission. When a user attempts to access DRM-protected content, the browser sends a request to a license server containing potentially additional information, such as the capabilities of the user's device. The license server evaluates the request and, if valid and / or certified, issues a license to the user. The license contains the decryption key required to decode the DRM-protected content and / or instructions on how to obtain the decryption key. Once the browser receives the encrypted content and the corresponding license, it decrypts the content using the decryption key, and the browser can then play back the content using the appropriate codecs. For example, in ISO-BMFF, the data contains encoded content and Common Encryption (CENC) DRM information for several DRMs. This information is delivered to an EME-compatible browser, which communicates with the appropriate license server for one of the DRMs and obtains the decryption key. The data is then played back via MSE after decryption.

[0056] Figure 4 The WebRTC system architecture is demonstrated. WebRTC is a standard platform for real-time streaming applications based on the web. WebRTC uses the RTP protocol to transmit audio and video. WebRTC has three components: a transmitter, a network controller, and a receiver. WebRTC relies on RTP to transmit encoded video frames. The transmitter obtains an inferred rate from the network controller and encodes the video frames captured by the camera at that rate. These encoded video frames are packaged into RTP packets and transmitted. The receiver receives the RTP packets and uses a buffer to generate video frames. RTP packets have a payload type field that describes which media codec can be used to process the packets. Furthermore, the receiver uses RTCP to send loss and latency reports to the transmitter, which the network controller analyzes to potentially adjust the transmitter's encoding rate accordingly.

[0057] WebRTC establishes a set of benchmark codecs that all compatible browsers must support, including H.264 and VP8 for video and Opus and G.711 for audio. Encoded streams intended for RTP transmission (such as H.264 or Opus) cannot be used for DRM encryption / decryption, which requires ISO-BMFF. Furthermore, ISO-BMFF cannot be transmitted via RTP as is.

[0058] Figure 5AThis demonstrates live adaptive bitrate streaming for multiple subscribers. As shown, multiple subscribers (e.g., client A (subscriber 1), client B (subscriber 2)) can subscribe to an encoded media stream. As an example, client A can subscribe to the encoded media stream at time T0. Client B can subscribe at time T1. In some embodiments, if the encoded media stream utilizes adaptive bitrate streaming, each distribution layer of the stream can be pre-encoded. However, during live streaming, when a new client (e.g., client B) subscribes to an already broadcast encoded media stream, it requests a new keyframe at a specific resolution (i.e., the distribution layer of the stream associated with that specific resolution). In many cases, new client B can request the encoded media stream at a time before the new keyframe arrives at a time-domain relevant time T2. The time interval between the new client's subscription time T1 and the time T2 for rendering the first frame can include a startup delay T3.

[0059] Figure 5B This demonstrates live adaptive bitrate streaming under dynamic network conditions. In some embodiments, in response to changing network conditions, the client can request a change in the bitrate of the encoded media stream to adapt to the changing network conditions.

[0060] As an example, such as Figure 5B As shown, the client can determine that higher bandwidth is available at the first time point T1. Determining that higher bandwidth is available can include the client identifying that the available bandwidth can accommodate encoded media streams with higher bitrates. Therefore, the client can request to subscribe to a distribution layer with a higher bitrate to improve the quality of the representative media.

[0061] However, in order to subscribe to a distribution layer with a higher bitrate, the client may have to wait until the new keyframe of the requested distribution layer arrives at time T2. Therefore, the duration between the first time T1 and time T2 represents the duration T3 during which bandwidth is not fully utilized. This waiting time T3 (or "first frame time") during which bandwidth is not fully utilized can lead to a degraded client experience because the quality presented by the decoded media's bitrate is lower than that produced by a higher bitrate stream.

[0062] In addition, such as Figure 5BAs shown, a bandwidth drop might occur at time T4 for the client. This bandwidth drop can include a reduction in available bandwidth, in which case the client may be unable to process the encoded media stream at the current bitrate. In this example, the client might request a lower bitrate stream, and a new keyframe at the lower bitrate might arrive at a later time (e.g., the first keyframe time T5). The duration from the bandwidth drop time T4 to the new keyframe time T5 could be the duration T6 of the rendering discontinuous time. During the rendering discontinuous time, the resulting media might experience interruptions / stuttering, or media, adaptive bitrates, and dynamic conditions might fail to display.

[0063] Figure 6 The diagram illustrates bitrate selection. As shown, the RTP protocol is used to transmit encoded frames (e.g., audio and video). The transmitter obtains an inferred bitrate from the network controller and encodes video frames captured by the camera at that bitrate. The encoded video frames are packaged into RTP packets and transmitted. The receiver receives the RTP packets and uses buffers (e.g., packet buffers, frame buffers) to generate video frames. RTP packets have a payload type field that describes which media codec can be used to process the packets. Furthermore, the receiver sends loss and latency reports to the transmitter using RTCP, which the network controller analyzes to potentially adjust the transmitter's encoding rate accordingly.

[0064] This disclosure describes a method for transparently supporting DRM in the WebRTC platform, which achieves real-time constraints by multiplexing the raw encoded real-time stream as a carrier of encrypted data. The method packages the raw encoded stream into an ISO-BMFF for encryption, and then embeds the encrypted ISO-BMFF payload into the raw encoded stream, disguising it as a valid H.264 or Opus fragment.

[0065] The disclosed technology currently uses H.264 as an example. H.264 is capable of encoding high-quality video at a lower bitrate than older compression standards. Figure 7 The H.264 Access Unit (AU) is illustrated. The structure of an H.264 AU represents a single video frame. A single AU can consist of multiple packets called NAL (Network Abstraction Layer) units, including the AU NAL, SPSNAL, PPS NAL, SEI NAL, and VCL NAL. The payload (frame data) is contained within the VCL or video coding layer; each AU can have one or more VCLs or video coding layers. In this case, another important NAL is the SEI NAL, or Supplemental Enhancement Information, which allows the inclusion of arbitrary data that the decoder will ignore.

[0066] These H.264 AUs are packaged as ISO-BMFF files, containing two types of fragments: initialization fragments and media fragments. For example, Figure 8 This demonstrates an H.264 AU as a single-frame media segment in ISO-BMFF. The initialization segment is a header that describes the content located within the media segment. The SEI NAL contains the initialization segment, which has a UUID identifying the source. Multiple initialization segments can be included.

[0067] The resulting ISO-BMFF fragments are encrypted. For example, Figure 9 This illustrates an H.264 access unit as an encrypted ISO-BMFF single-frame media segment. The initialization segment and media segment enlarge to accommodate DRM-specific data, including a key identifier from the DRM (a 16-byte value required to obtain the key). As shown, encryption can be partial, leaving the header unencrypted.

[0068] After stripping the original AU and transforming it into a "carrier" AU, the encrypted ISO-BMFF fragment is embedded in the original encoded stream, making it function as a valid H.264 unit. For example, Figure 10 This demonstrates the process of converting an H.264 AU into an encrypted ISO-BMFF single-frame media segment. The media segment is appended to the last stripped VCLNAL unit within a given AU. An initialization segment (IS) is inserted into a new SEI NAL unit. This SEI is inserted before the first VCL within the given AU.

[0069] The described embedding can be performed frame by frame without introducing any latency.

[0070] The proposed method is general and is included in a series of figures in this disclosure. Figures 11 to 13 An Opus audio frame is used as another example to demonstrate the disclosed technique. The audio frame is included in an ISO-BMFF clip.

[0071] Initialization fragments are periodically inserted because they are needed to provide synchronization points for clients, and these initialization fragments are merged with media fragments when inserted, so that a data packet will include the initialization fragment followed by the media fragment. Figure 11 ).

[0072] In this case, encryption works similarly, where the encrypted portion is the audio frame. The encrypted segment is then fully embedded into the Opus stream.

[0073] This method can be easily adapted to the need for frames larger than those that RTP packets can accommodate (e.g., higher audio bitrates), i.e., by moving the initialization fragment to its packet (increasing the number of packets per second).

[0074] On the client side, the received segments are unpacked and passed to a Media Source Extension in an MS-compatible browser for playback. For the audio example, unpacking the ISO-BMFF initialization segment and the media segment is straightforward; however, for the video example, some parsing is required.

[0075] Figure 14 This is a flowchart of a method to improve the security of streaming media by converting a secure media format into a streaming media format without introducing latency. In step 1400, the hardware or software processor described in this application can obtain a media segment representing a portion of media represented by a multimedia file format. The multimedia file format can be ISO-BMFF, which can be encrypted and is a secure media format. However, ISO-BMFF is not configured for streaming and does not conform to RTP, and therefore cannot be streamed as is. ISO-BMFF is a container file format that defines a general structure for files containing time-based multimedia data, such as video and audio.

[0076] A media segment can be a single frame in ISO-BMFF and can include multiple packets, including a first packet ignored by the decoder and a second packet containing the payload (i.e., the content of the media file format). The first packet can be SEI NAL, and the second packet can be VCL NAL.

[0077] In step 1410, the processor may obtain a key identifier that indicates the encryption key. The key identifier may indicate how the encryption key is calculated and / or where the encryption key is obtained. The memory occupied by the key identifier (e.g., 16 bytes) is smaller than the memory occupied by media represented by a multimedia file format.

[0078] In step 1420, the processor can transform media represented by a multimedia file format into media represented by a streaming format by performing the following steps. First, the processor can include a key identifier in a first data packet associated with a segment of the media represented by the streaming format, thereby causing the decoder to ignore the key identifier. The segment associated with the media represented by the streaming format can be an AU, which can represent a keyframe configured to be decoded without reference to another access unit in the media represented by RTP, or it can represent an incremental frame configured to be decoded together with a previous keyframe associated with the media represented by RTP. Second, the processor can encrypt the payload using an encryption key to obtain an encrypted payload. Third, the processor can include the encrypted payload in a second data packet associated with the segment. Finally, the processor can improve the security of streaming media represented by the multimedia file format by streaming the media represented by the streaming format without introducing latency, because the time required to decrypt the media represented by the streaming format is shorter compared to network latency.

[0079] Media segments may include additional data packets, which the processor may also encrypt. The processor can obtain a media segment representing a portion of media represented by a multimedia file format, wherein multiple data packets include Access Unit Network Abstraction Layer (AU NAL) packets, Sequence Parameter Set Network Abstraction Layer (SPS NAL) packets, Picture Parameter Set Network Abstraction Layer (PPS NAL) packets, a first data packet, a second data packet, and an initialization segment. Each of the multiple data packets may include a header and a body. The processor can transform a media segment associated with a multimedia file format into a segment associated with a streaming format by performing the following steps: First, the processor may add the initialization segment associated with the multimedia file format to the first data packet of the segment. Second, the processor may encrypt the body of each of the multiple data packets to obtain multiple encrypted bodies. Finally, the processor may add multiple headers associated with each of the multiple data packets and multiple encrypted bodies to the segment associated with the streaming format.

[0080] The processor can perform bitrate selection. Specifically, the processor can create multiple streams of media represented by a streaming format, wherein a first stream among the multiple streams has a higher bitrate than a second stream among the multiple streams. The processor can send the first stream to the client. The processor can receive a request from the client to reduce the bitrate. Upon receiving the request, the processor can send the second stream to the client.

[0081] A processor can send segments associated with media represented in a streaming format to a receiver over a network, where the network introduces network latency. The processor can cause the receiver to decrypt an encrypted payload, where decryption requires a first amount of time. The processor can also cause the receiver to decode segments associated with the streaming media, where decoding requires a second amount of time. Because the first amount of time is less than the sum of the network latency and the second amount of time, the processor can improve the security of streaming media represented in a multimedia file format without introducing latency. Specifically, the first amount of time required to decrypt the encrypted payload can be a few milliseconds (ms), for example, less than 10 ms. Similarly, the second amount of time required to decode a segment can be a few milliseconds. However, the network latency can be much larger, approximately 50 ms or longer. Therefore, the increased decryption latency is imperceptible compared to the network latency. Computer System

[0082] Figure 15 This is a block diagram illustrating an example of a computer system 1500 in which at least some of the operations described herein can be implemented. As shown, the computer system 1500 may include one or more processors 1502, main memory 1506, non-volatile memory 1510, network interface device 1512, video display device 1518, input / output device 1520, control device 1522 (e.g., keyboard and pointing device), drive unit 1524 including machine-readable (storage) medium 1526, and signal generation device 1530, these components being communicatively connected to bus 1516. Bus 1516 represents one or more physical buses and / or point-to-point connections connected via appropriate bridges, adapters, or controllers. For simplicity, Figure 15 Various common components (e.g., cache memory) are omitted. Instead, computer system 1500 is intended to demonstrate hardware devices on which components shown or described with respect to the accompanying drawings, as well as any other components described in this specification, may be implemented.

[0083] Computer system 1500 can take any suitable physical form. For example, computing system 1500 can share similar architectures with server computers, personal computers (PCs), tablet computers, mobile phones, game consoles, music players, wearable electronic devices, networked (“smart”) devices (e.g., televisions or home assistant devices), AR / VR systems (e.g., head-mounted displays), or any electronic device capable of executing a set of instructions specifying actions to be taken by computing system 1500. In some implementations, computer system 1500 can be an embedded computer system, a system-on-a-chip (SoC), a single-board computer system (SBC), or a distributed system (such as a mesh architecture of computer systems), or the computer system may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1500 can operate in real-time, near real-time, or batch processing modes.

[0084] Network interface device 1512 enables computing system 1500 to mediate data with external entities in network 1514 via any communication protocol supported by computing system 1500 and entities located outside computing system 1500. Examples of network interface devices 1512 include network adapter cards, wireless network interface cards, routers, access points, wireless routers, switches, multilayer switches, protocol converters, gateways, bridges, bridging routers, hubs, digital media receivers and / or repeaters, and all wireless elements mentioned herein.

[0085] Memory (e.g., main memory 1506, non-volatile memory 1510, machine-readable medium 1526) can be local, remote, or distributed. Although shown as a single medium, machine-readable medium 1526 can include multiple media (e.g., centralized / distributed databases and / or associated caches and servers) storing one or more instruction sets 1528. Machine-readable medium 1526 can include any medium capable of storing, encoding, or carrying instruction sets executable by computing system 1500. Machine-readable medium 1526 can be non-transitory or includes non-transitory devices. In this context, non-transitory storage media can include tangible devices, meaning that the device has a concrete physical form, although the device can change its physical state. Thus, for example, non-transitory means that the device remains tangible despite such a change in state.

[0086] Although the implementations are described in the context of a fully functional computing device, various examples can be distributed as program products in various forms. Examples of machine-readable storage media, machine-readable media, or computer-readable media include recordable media (such as volatile and non-volatile memory 1510), removable flash memory, hard disk drives, optical disks), and transmission media (such as digital and analog communication links).

[0087] Generally, routines executed to implement the examples herein may be implemented as part of an operating system or as part of a particular application, component, program, object, module, or sequence of instructions (collectively, a “computer program”). A computer program typically comprises one or more instructions (e.g., instructions 1504, 1508, 1528) set at various times in a computing device's various memories and storage devices. When read and executed by processor 1502, the instructions cause computing system 1500 to perform operations to execute elements relating to various aspects of this disclosure. Remark

[0088] The terms “example,” “implementation,” and “implementation” are used interchangeably. For example, a reference to “an example” or “example” in this disclosure may, but is not necessarily, a reference to the same implementation; and such a reference means at least one implementation. The phrase “in an example” does not necessarily refer to the same example, nor is it necessarily a separate or alternative example that is mutually exclusive with other examples. A feature, structure, or characteristic described in connection with an example may be included in another example in this disclosure. Furthermore, various features that some examples may possess but others do not are described. Similarly, various requirements that some examples may need to meet but others do not are described.

[0089] The terms used herein should be interpreted in their broadest and most reasonable manner, even when used in conjunction with certain specific examples of the invention. The terms used in this disclosure generally have their ordinary meaning in the relevant art, within the context of this disclosure, and in the specific context in which each term is used. The listing of alternative language or synonyms does not preclude the use of other synonyms. No term should be given a special meaning simply because it is described or discussed in detail herein. The use of highlighting has no effect on the scope and meaning of terms. Furthermore, it should be understood that the same thing can be expressed in more than one way.

[0090] Unless the context explicitly requires otherwise, throughout the specification and claims, the terms “comprise” and “comprising” should be interpreted in an inclusive rather than exclusive or exhaustive sense—that is, in the sense of “including but not limited to”. As used herein, the terms “connection,” “coupled,” and any variations thereof mean any direct or indirect connection or coupling between two or more elements; the coupling or connection between elements may be physical, logical, or a combination thereof. Additionally, the terms “this article,” “above,” “below,” and terms with similar meanings may refer to the entire application, rather than any particular part of it. Where the context permits, singular or plural terms used in the above detailed description may also include plural or singular, respectively. When referring to a list containing two or more items, the word “or” covers all of the following interpretations of the word: any one item in the list, all items in the list, and any combination of items in the list. The term “module” broadly refers to software components, firmware components, and / or hardware components.

[0091] While specific technical examples have been described above for illustrative purposes, various equivalent modifications are possible within the scope of this invention, as will be recognized by those skilled in the art. For example, although processes or blocks are presented in a given order, alternative implementations may execute routines with steps or employ systems with blocks in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternative combinations or sub-combinations. Each of these processes or blocks can be implemented in a variety of different ways. Furthermore, while processes or blocks are sometimes shown as being executed in series, these processes or blocks may alternatively be executed or implemented in parallel, or may be executed at different times. Further, any specific figures indicated herein are merely examples, and alternative implementations may employ different values ​​or ranges.

[0092] The details of the disclosed embodiments can vary considerably in the specific embodiments, but are still covered by the teachings disclosed. As noted above, specific terms used in describing features or aspects of the invention should not be construed as implying that such terms are redefined herein as limited to any particular characteristic, feature, or aspect of the invention associated with that term. Generally, unless such terms are explicitly defined in the foregoing specific embodiments, the terms used in the following claims should not be construed as limiting the invention to the specific examples disclosed herein. Therefore, the actual scope of the invention covers not only the disclosed examples but also all equivalent ways of practicing or implementing the invention under the claims. Some alternative embodiments may include more elements or fewer elements than those described above.

[0093] Any patents and applications mentioned above, as well as other references and any documents that may be listed in the accompanying application documents, are incorporated herein by reference in their entirety, except where there is any subject matter disclaimer or denial, or where the incorporated material is inconsistent with the express disclosure herein (in which case the statement in this disclosure shall prevail). Various aspects of the invention may be modified to incorporate the systems, functions, and concepts of the foregoing references, thereby providing further embodiments of the invention.

[0094] To reduce the number of claims, certain embodiments are presented below in specific claim forms, but the applicant may envision other forms for various aspects of the invention. For example, aspects of the claims may be expressed as means plus function or otherwise, such as embodied in a computer-readable medium. Claims intended to be interpreted as means plus function claims will use the phrase “means for…”. However, the use of the term “for” in any other context is not intended to provoke a similar interpretation. The applicant reserves the right to seek such additional claim forms in this application or a continuation thereof.

Claims

1. A non-transitory computer-readable storage medium comprising instructions recorded thereon, wherein, When executed by at least one data processor of the system, the instructions cause the system to perform the following operations: To obtain a media segment representing a portion of media encoded by the ISO Basic Media File Format (ISO-BMFF). The ISO-BMFF is not configured for streaming Web Real-Time Communication (WebRTC). The media segment includes multiple data packets, including Supplemental Enhanced Information Layer (SEI NAL) data packets and Video Coding Layer Network Abstraction Layer (VCL NAL) data packets. The data added to the SEI NAL data packet is ignored by the decoder. The VCL NAL data packet includes a payload; Obtain the key identifier that indicates the encryption key. The memory occupied by the key identifier is smaller than the memory occupied by the media represented by the ISO-BMFF; The media represented by ISO-BMFF is transformed into media represented by Real-Time Protocol (RTP) in the following manner: The key identifier is included in the SEINAL packet of the access unit associated with the media represented by the RTP, thereby causing the decoder to ignore the key identifier; The payload is encrypted using the encryption key to obtain an encrypted payload; The encrypted payload is included in the VCL NAL associated with the access unit; An initialization fragment describing the content of the media segment is added to the access unit; Without introducing time delay, the security of streaming media represented by ISO-BMFF is improved by streaming the media represented by RTP. The time required to decrypt the media represented by the RTP is shorter than the network latency.

2. The non-transitory computer-readable storage medium of claim 1, comprising instructions for performing the following operations: Obtain the media segment representing a portion of the media as represented by the ISO-BMFF. in, The multiple data packets include Access Unit Network Abstraction Layer (AU NAL) data packets, Sequence Parameter Set Network Abstraction Layer (SPS NAL) data packets, Picture Parameter Set Network Abstraction Layer (PPS NAL) data packets, the SEI NAL data packets, the VCL NAL data packets, and the initialization fragment data packets. Each of the plurality of data packets includes a header and a body; and The media segment associated with the ISO-BMFF is transformed into the access unit associated with the RTP in the following manner: Add the initialization fragment associated with the ISO-BMFF to the SEI NAL of the access unit; Encrypt the body of each of the plurality of data packets to obtain a plurality of encrypted bodies; and Multiple headers associated with each of the multiple data packets and the multiple encrypted bodies are added to the access unit associated with the RTP.

3. The non-transitory computer-readable storage medium of claim 1, comprising instructions for performing the following operations: The access unit is sent to the receiver via the network. in, The network causes network latency; The receiver then decrypts the encrypted payload. Decrypting the encrypted payload requires a first amount of time. The receiver decodes the access unit. Decoding the access unit requires a second time amount; and To improve the security of streaming media represented by ISO-BMFF without introducing time delay. Wherein, the first time quantity is less than the sum of the network latency and the second time quantity.

4. The non-transitory computer-readable storage medium of claim 1, comprising instructions for performing the following operations: Create multiple streams of the media represented by the RTP. in, The first stream of the plurality of streams has a higher bit rate than the second stream of the plurality of streams; Send the first stream to the client; Receive a request from the client to reduce the bitrate; as well as Upon receiving the request, the second stream is sent to the client.

5. The non-transitory computer-readable storage medium as described in claim 1, wherein, The media represented by ISO-BMFF includes video and / or audio.

6. The non-transitory computer-readable storage medium of claim 1, comprising instructions for performing the following operations: Obtain the media segment representing a portion of the media as represented by the ISO-BMFF. in, The plurality of data packets include Access Unit Network Abstraction Layer (AU NAL) data packets, Sequence Parameter Set Network Abstraction Layer (SPS NAL), Picture Parameter Set Network Abstraction Layer (PPS NAL), the SEI NAL, the VCL NAL, and the initialization fragment; and The media segment associated with the ISO-BMFF is transformed into the access unit represented by the RTP in the following manner: The initialization fragment associated with the ISO-BMFF is added to the SEI NAL associated with the RTP.

7. A method comprising: To obtain a media segment representing a portion of media represented by a multimedia file format. The multimedia file format was not configured for streaming. The media segment includes multiple data packets, including a first data packet ignored by the decoder and a second data packet containing a payload; Obtain the key identifier that indicates the encryption key. The memory occupied by the key identifier is smaller than the memory occupied by the media represented by the multimedia file format; The media represented by the multimedia file format is transformed into media represented by the streaming format in the following manner: The key identifier is included in the first data packet of the segment associated with the media represented by the streaming format, thereby causing the decoder to ignore the key identifier; The payload is encrypted using the encryption key to obtain an encrypted payload; Include the encrypted payload in the second data packet associated with the fragment; and Without introducing latency, the security of streaming media represented by the multimedia file format is improved by streaming the media represented by the streaming format. The time required to decrypt the media represented by the streaming format is shorter than the network latency.

8. The method of claim 7, comprising: Obtain the media segment representing a portion of the media represented by the multimedia file format. The plurality of data packets include Access Unit Network Abstraction Layer (AU NAL) data packets, Sequence Parameter Set Network Abstraction Layer (SPS NAL) data packets, Picture Parameter Set Network Abstraction Layer (PPS NAL) data packets, the first data packet, the second data packet, and an initialization fragment. Each of the plurality of data packets includes a header and a body; and The media segment associated with the multimedia file format is transformed into the segment associated with the streaming format in the following manner: The initialization fragment associated with the multimedia file format is added to the first data packet of the fragment; Encrypt the body of each of the plurality of data packets to obtain a plurality of encrypted bodies; and Multiple headers associated with each of the multiple data packets and the multiple encrypted bodies are added to the segment associated with the streaming format.

9. The method of claim 7, comprising: Create multiple streams of the media represented by the streaming format. The first stream among the plurality of streams has a higher bit rate than the second stream among the plurality of streams; Send the first stream to the client; Obtain a request from the client to reduce the bitrate; and Upon receiving the request, the second stream is sent to the client.

10. The method of claim 7, wherein, The media represented by the multimedia file format includes video and / or audio.

11. The method of claim 7, comprising: Obtain the media segment representing a portion of the media represented by the multimedia file format. The plurality of data packets include Access Unit Network Abstraction Layer (AU NAL) data packets, Sequence Parameter Set Network Abstraction Layer (SPS NAL) data packets, Picture Parameter Set Network Abstraction Layer (PPS NAL) data packets, the first data packet, the second data packet, and an initialization fragment; and The media segment associated with the multimedia file format is transformed into the segment represented by the streaming format in the following manner: The initialization fragment associated with the multimedia file format is added to the first data packet associated with the streaming format.

12. The method of claim 7, comprising: The segment associated with the media represented by the streaming format is sent to the receiver via a network. The network causes network latency; The receiver then decrypts the encrypted payload. Decrypting the encrypted payload requires a first amount of time. The receiver is then instructed to decode the segments associated with the media represented by the streaming format. Decoding the segment associated with the media represented by the streaming format requires a second amount of time; and To improve the security of streaming media represented by the multimedia file format without introducing the aforementioned latency. Wherein, the first time quantity is less than the sum of the network latency and the second time quantity.

13. A system comprising: At least one hardware processor; as well as At least one non-transitory memory stores instructions that, when executed by the at least one hardware processor, cause the system to perform the following operations: To obtain a media segment representing a portion of media represented by a multimedia file format. The multimedia file format was not configured for streaming. The media segment includes multiple data packets, including a first data packet ignored by the decoder and a second data packet containing a payload; Obtain the key identifier that indicates the encryption key. The memory occupied by the key identifier is smaller than the memory occupied by the media represented by the multimedia file format; The media represented by the multimedia file format is transformed into media represented by the streaming format in the following manner: The key identifier is included in the first data packet of the segment associated with the media represented by the streaming format, thereby causing the decoder to ignore the key identifier; The payload is encrypted using the encryption key to obtain an encrypted payload; Include the encrypted payload in the second data packet associated with the fragment; and Without introducing latency, the security of streaming media represented by the multimedia file format is improved by streaming the media represented by the streaming format. The time required to decrypt the media represented by the streaming format is shorter than the network latency.

14. The system of claim 13, further comprising instructions for performing the following operations: Obtain the media segment representing a portion of the media represented by the multimedia file format. in, The multiple data packets include Access Unit Network Abstraction Layer (AU NAL) data packets, Sequence Parameter Set Network Abstraction Layer (SPS NAL) data packets, Picture Parameter Set Network Abstraction Layer (PPS NAL) data packets, the first data packet, the second data packet, and the initialization fragment. Each of the plurality of data packets includes a header and a body; and The media segment associated with the multimedia file format is transformed into the segment associated with the streaming format in the following manner: The initialization fragment associated with the multimedia file format is added to the first data packet of the fragment; Encrypt the body of each of the plurality of data packets to obtain a plurality of encrypted bodies; and Multiple headers associated with each of the multiple data packets and the multiple encrypted bodies are added to the segment associated with the streaming format.

15. The system of claim 13, further comprising instructions for performing the following operations: Create multiple streams of the media represented by the streaming format. in, The first stream of the plurality of streams has a higher bit rate than the second stream of the plurality of streams; Send the first stream to the client; Receive a request from the client to reduce the bitrate; as well as Upon receiving the request, the second stream is sent to the client.

16. The system of claim 13, wherein, The media represented by the multimedia file format includes video and / or audio.

17. The system of claim 13, further comprising instructions for performing the following operations: Obtain the media segment representing a portion of the media represented by the multimedia file format. in, The plurality of data packets include Access Unit Network Abstraction Layer (AU NAL) data packets, Sequence Parameter Set Network Abstraction Layer (SPS NAL) data packets, Picture Parameter Set Network Abstraction Layer (PPS NAL) data packets, the first data packet, the second data packet, and the initialization fragment; as well as The media segment associated with the multimedia file format is transformed into the segment represented by the streaming format in the following manner: The initialization fragment associated with the multimedia file format is added to the first data packet associated with the streaming format.

18. The system of claim 13, wherein, The multimedia file format includes the ISO Basic Media File Format (ISO-BMFF), the streaming format includes the Real-Time Protocol (RTP), the first data packet includes a Supplemental Enhanced Information Network Abstraction Layer (SEI NAL) data packet, and the second data packet includes a Video Coding Layer Network Abstraction Layer (VCL NAL) data packet.

19. The system of claim 13, wherein, The segment representation associated with the media represented by the streaming format is configured to be a keyframe that is decoded without reference to another access unit in the streaming format, or the segment representation associated with the media represented by the streaming format is configured to be an incremental frame that is decoded together with a previous keyframe associated with the media represented by the streaming format.

20. The system of claim 13, further comprising instructions for performing the following operations: The segment associated with the media represented by the streaming format is sent to the receiver via a network. in, The network causes network latency; The receiver then decrypts the encrypted payload. Decrypting the encrypted payload requires a first amount of time. The receiver is then instructed to decode the segments associated with the media represented by the streaming format. Decoding the segment associated with the media represented by the streaming format requires a second amount of time; and To improve the security of streaming media represented by the multimedia file format without introducing the aforementioned latency. Wherein, the first time quantity is less than the sum of the network latency and the second time quantity.

Images