Signaling of location of protected data in coded bitstream

Abstract

In some embodiments, the method receives an encoded bitstream. Information of the encoded bitstream is determined in an encoding process. The information follows a syntax for describing protected data in the encoded bitstream. The method utilizes the information to determine a set of locations of the encoded bitstream to be encrypted according to the syntax. The method selectively encrypts a first set of portions of the encoded bitstream corresponding to the protected data based on the set of locations, thereby forming an encrypted bitstream. A second set of portions of the encoded bitstream corresponding to plaintext data is not encrypted. The method outputs the encrypted bitstream.

Description

[0001] Cross-references to related applications

[0002] This application is a continuation application entitled to and claim priority under 35 USC §120 of PCT application No. PCT / CN2025 / 091945, filed April 29, 2025, entitled “Signaling for the Location of Protected Data in a Coded Bitstream,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This invention relates to encoding, packaging, and encryption. Background Technology

[0004] In digital rights management (DRM)-based video encoding and transmission systems, the encoding, encapsulation, and encryption of video content enable secure distribution and adaptive streaming. A major challenge in video content encryption lies in identifying the specific bytes within the encoded bitstream that require encryption. Locating these bytes within the encoded bitstream is particularly complex. For example, this requires extensive parsing of the encoded bitstream, typically involving bit-by-bit analysis, to accurately determine the encryption location. This process significantly reduces the overall system efficiency. Summary of the Invention

[0005] In some embodiments, the method receives an encoded bitstream. Information about the encoded bitstream is determined during the encoding process. This information follows a syntax for describing protected data within the encoded bitstream. The method uses this information to determine a set of locations of the encoded bitstream to be encrypted, based on the syntax. Based on this set of locations, the method selectively encrypts a first portion of the encoded bitstream corresponding to the protected data, thereby forming an encrypted bitstream. A second portion of the encoded bitstream corresponding to plaintext data is not encrypted. The method outputs the encrypted bitstream. Attached Figure Description

[0006] The accompanying drawings are for illustrative purposes only and are intended to provide possible structural and operational examples of the disclosed inventive systems, apparatuses, methods, and computer program products. These drawings are not intended to limit any changes in form and detail that may be made by those skilled in the art without departing from the spirit and scope of the disclosed embodiments.

[0007] Figure 1 A simplified system for encoding, encapsulating, and encrypting bit streams is shown according to some embodiments.

[0008] Figure 2 A more detailed example of the encoding, encapsulation, and encryption pipeline according to some embodiments is shown.

[0009] Figure 3 A simplified flowchart of a method for processing source video to generate an encrypted bitstream, according to some embodiments, is shown.

[0010] Figure 4A The first class (category 1) of encoded bitstreams according to some embodiments is shown.

[0011] Figure 4B A second type (category 2) of encoded bitstreams according to some embodiments is shown.

[0012] Figure 5 Metadata syntax is shown for the encoder specification syntax of a Category 1 bitstream according to some embodiments.

[0013] Figure 6 Metadata syntax for an encoder specification method for a category 2 bitstream according to some embodiments is shown.

[0014] Figure 7 The metadata syntax of a second example of the Category 1 bitstream encoder specification syntax according to some embodiments is shown.

[0015] Figure 8 Metadata syntax for a second example of a Category 1 bitstream encoder specification method according to some embodiments is shown.

[0016] Figure 9 A simplified flowchart of an encoded bitstream encryption method according to some embodiments is shown.

[0017] Figure 10 A video streaming system according to one embodiment is shown, which communicates with multiple client devices via one or more communication networks.

[0018] Figure 11 A schematic diagram of a device for viewing video content and supplemental content is shown. Detailed Implementation

[0019] This document describes a technique for encoding, encapsulating, and encrypting systems. For ease of explanation, numerous examples and specific details are provided below to enable the reader to thoroughly understand certain embodiments. Certain embodiments as defined in the claims may individually include some or all of the features described in these examples, or may be used in combination with other features described below, and may also include modifications and equivalents of the features and concepts described herein.

[0020] Overview

[0021] This system employs an encryption-friendly syntax design, improving the parsing process to identify protected data within the encoded bitstream that requires encryption. This syntax reduces the complexity and necessity of bit-by-bit parsing of the encoded bitstream. Previously, the input source video was first encoded / compressed into an encoded bitstream. This encoded bitstream was then fed into the encapsulation and encryption system. To generate the encrypted bitstream, the parsing system required complex bit-by-bit parsing to identify the positions of protected bytes. Using the positions of these protected bytes, the encryption system encrypted the corresponding positions in the encoded bitstream, thus generating the encrypted bitstream. In contrast, this system only requires parsing a small amount of information (e.g., a few bytes of metadata) to determine the position of protected data (e.g., protected bytes), significantly improving the overall system efficiency.

[0022] In some embodiments, the encoder determines the data type in the bitstream being encoded. For example, the interleaving pattern of protected data and clear data in the encoded bitstream may differ in different types of bitstreams. Protected data may be bytes in the encoded bitstream that will be encrypted. Clear data may be bytes that will not be encrypted. The encoder derives the location information of the protected data based on information from the content encoding process (e.g., header information used to identify the type of data being encoded). The encoder then provides this information as metadata to the encapsulation and encryption system to determine the location of the protected data. This metadata can be provided in different ways. For example, the metadata may be included in the encoded bitstream. Alternatively, the metadata may be provided outside the encoded bitstream, for example, through a separate channel. The encapsulation and encryption system then uses this metadata to determine which portions of the encoded bitstream to encrypt and which to leave unencrypted. For example, the encapsulation and encryption system encrypts the location of the protected data while leaving the location of the clear data unencrypted.

[0023] system

[0024] Figure 1A simplified system 100 for encoding, encapsulating, and encrypting bitstreams according to some embodiments is illustrated. System 100 includes a server system 102 and a client device 104. The server system 102 can provide an encrypted bitstream for content to the client device 104. The client device 104 can decrypt and decode the encrypted bitstream. The encrypted bitstream can be an encoded bitstream containing encrypted byte portions and unencrypted byte portions. The client device 104 then plays the content through an interface 110. The content can be of different types, such as video, audio, images, etc. Video can be used for discussion purposes, but can also be used to enjoy other types of content.

[0025] In some embodiments, server system 102 may be implemented using one or more computing devices that interact with client device 104. In some embodiments, server system 102 may include multiple servers or other computing devices to provide the functionality described herein. Server system 102 may be designed to process content requests from multiple client devices and provide encrypted bitstreams as responses.

[0026] In some embodiments, client device 104 may be a user device that interacts with server system 102 to request content or other data. Client device 104 may be a computer, mobile phone, smartphone, tablet, living room device, or other device capable of communicating with the server system. In some embodiments, client device 104 may be configured to receive and display content provided by server system 102 on interface 110.

[0027] Server system 102 and client device 104 can communicate with each other via a network. In some embodiments, the communication between server system 102 and client device 104 can use encryption or other security measures to protect the transmitted data. Server system 102 and client device 104 can also be configured to communicate using different protocols or technologies.

[0028] Server system 102 includes encoding, encapsulation, and encryption processes 106 for performing encoding, encapsulation, and encryption. Encoding is the process of compressing raw video content into a specific codec format (such as High Efficiency Video Coding (HEVC) H.264 or H.265, Audio Video I (AV1), etc.) to reduce file size while maintaining video quality. By encoding video at multiple bitrates and resolutions, system 100 can support Adaptive Bitrate Streaming (ABR), automatically adjusting video quality to adapt to available bandwidth, ensuring a smooth playback experience for users regardless of network conditions.

[0029] After encoding, the video needs to be encapsulated into segments and then encapsulated using a suitable streaming media protocol format, such as DASH (Dynamic Adaptive Streaming over HTTP) or HLS (HTTP Live Streaming). This encapsulation process breaks the video into smaller segments, which are then transmitted incrementally over the network. Encapsulation also includes integrated Digital Rights Management (DRM) metadata, which associates the video content with encryption keys, ensuring that only authorized users with the decryption keys can view the content.

[0030] Digital rights management systems (DRMs) protect content by encrypting video streams and controlling access permissions. Encryption methods may rely on Advanced Encryption Standard (AES) to ensure the security of the encoded bitstream, while key exchange protocols protect access to the decryption key. Each DRM provider typically manages licenses and enforces policies to restrict copying, screen recording, and unauthorized distribution. DRM protection is applied to applications such as OTT (Over-the-Top) media, video-on-demand (VoD), and subscription streaming services.

[0031] The decryption and decoding process 108 receives the encrypted bitstream and decrypts the protected bytes to form an encoded bitstream. Then, the decryption and decoding process 108 can decode the encoded bitstream and play the content (e.g., video) through interface 110.

[0032] The encoding, encapsulation, and encryption process will be described in more detail below 106.

[0033] Encoding, packaging and encryption process

[0034] Figure 2 A more detailed example of an encoding, encapsulation, and encryption process 106 according to some embodiments is shown. The encoding, encapsulation, and encryption process 106 includes an encoder 202, a parser 204, and an encapsulation and encryption system 206.

[0035] Encoder 202 receives content, such as source video, audio, or other content. This discussion will focus on source video as an example. Encoder 202 encodes the source video into an encoded bitstream. Encoder 202 can use various techniques (such as compression) to reduce the size of the source video. Encoder 202 can also determine metadata or other information about the bitstream, such as information that can be used to determine the location of protected data. Metadata can be added to the encoded bitstream or to a separate stream depending on the signaling method used. The addition of metadata is based on syntax. Syntax refers to the structural rules used to convey the location of metadata values ​​for protected data. An example of syntax is shown below.

[0036] Parser 204 parses the metadata and determines the location of the protected data. The parsing process does not require parsing every single bit of the encoded bitstream. Instead, parser 204 uses the metadata to determine the location of the protected data. Compared to parsing the entire encoded bitstream, parser 204 may only need to parse the metadata (potentially only a few bytes), which significantly improves the overall efficiency of the system. Furthermore, the metadata may be located at the beginning of a frame; parser 204 can parse the metadata from the beginning of the frame to determine the location of the protected data within that frame and then jump to that frame. Parser 204 then passes the location information of the protected bytes to encapsulation and encryption system 206.

[0037] The encapsulation and encryption system 206 encapsulates the encoded bitstream. Encapsulation constructs the encoded video data into a container format to facilitate storage, retrieval, and streaming between devices with varying performance. Modern container formats, such as MPEG-DASH (HTTP-based Dynamic Adaptive Streaming) and HLS (HTTP Live Streaming), divide video data into small chunks that dynamically adjust based on the network conditions of the client device 104, ensuring smooth playback. The MPEG-4 (MP4) container format is a popular choice due to its compatibility and efficient handling of video, audio, and metadata within a compact structure. These formats support Adaptive Bitrate Streaming (ABR), which automatically adjusts quality based on available bandwidth to prevent buffering and interruptions.

[0038] Fragmented MP4 (fMP4) represents an evolution of the MP4 standard, dividing video content into incrementally loaded "segments" for seamless adaptive streaming playback. fMP4 can be used with MPEG-DASH and HLS, supporting efficient bitrate switching to ensure uninterrupted playback for end users even under fluctuating network conditions. Adaptive streaming technology uses algorithms to determine the optimal segment quality based on network bandwidth, ensuring an optimal balance between quality and continuity. Other protocols can also be used for encapsulation.

[0039] The encapsulation and encryption system 206 can encrypt the protected data of the segments. Different encryption standards can be used. Common Encryption (CENC) is an industry-standard encryption standard that enables interoperability between different DRM systems. CENC allows content providers to encrypt content once and then distribute it through multiple DRM systems, eliminating the need for multiple encrypted versions. This interoperability helps simplify the content distribution process while maintaining strong security. Combined with segmented MP4 encapsulation, CENC can be used to transmit encrypted video in adaptive streaming formats, especially suitable for multi-platform streaming scenarios.

[0040] The location of the protected data needs to be determined before encryption can be performed. These locations can be byte offsets in the encoded bitstream or other location information. The encapsulation and encryption system 206 uses this location information to encrypt the protected data while keeping the plaintext data unencrypted, thus forming an encrypted bitstream. The server system 102 sends the encrypted bitstream to the client device 104. High-level HTTP protocols (such as HTTP / 2 and HTTP / 3) can be used to improve the transmission efficiency of encapsulated video content. These protocols support features such as multiplexing, header compression, and faster data transmission, thereby significantly reducing latency. In adaptive streaming, these improvements support faster segment loading and better responsiveness, especially in the event of network congestion, thus ensuring a consistent viewing experience.

[0041] Low-latency streaming protocols, including Low-Latency HLS (LL-HLS) and Low-Latency DASH (LL-DASH), can be used to solve the latency problems existing in traditional live streaming. Just-in-Time (JIT) encapsulation technology can dynamically encapsulate content according to the viewer's device and network requirements, thereby further improving efficiency. These technological advancements can be used in applications with extremely high requirements for near-real-time playback, such as live sports events and interactive content, ensuring minimal latency between content generation and end-user playback.

[0042] The processes for generating encoded and encrypted bitstreams will be described in detail below.

[0043] Method and Flow

[0044] Figure 3A simplified flowchart 300 of a method for processing source video to generate an encrypted bitstream according to some embodiments is shown. In step 302, encoder 202 determines the type of bitstream to be encrypted. For example, there may be different types of bitstreams with different protected data and plaintext data patterns. These different patterns will be illustrated in Figures 4 to 5 below. Figure 8 As described in the text.

[0045] In step 304, encoder 202 derives the location information of the protected data based on the bitstream type. For example, encoder 202 may use contextual information during the encoding process, such as header information describing the type of information in the bitstream (e.g., each frame), to determine the location of the protected data. Other contextual information, such as frame type, may also be used.

[0046] In step 306, encoder 202 provides the parser 204 with the location information of the protected data. As will be discussed in detail below, the metadata of the protected data can be provided using different syntaxes and different signaling methods. For example, with certain signaling methods, the protected data can be provided in the encoded bitstream or via a metadata side channel. The different syntaxes will be described in detail below.

[0047] In step 308, parser 204 parses the metadata and determines the location of the protected data. The method of parsing the metadata to determine the location may vary depending on the syntax used, which will be described in detail below. Parser 204 may provide the location information to encapsulation and encryption system 206.

[0048] In step 310, the encapsulation and encryption system 206 encrypts the protected data based on the position information. For example, the encapsulation and encryption system 206 encrypts the bytes in the encoded bitstream that correspond to the position of the protected data, while leaving the bytes corresponding to the plaintext data unencrypted.

[0049] Derivation of the location of the protected byte

[0050] In some embodiments, there may be two types of bitstreams, each corresponding to a different position of the protected byte, such as... Figure 4A and 4B As shown. These two types are for illustrative purposes only; it is understood that other types may exist.

[0051] Figure 4AA first class (Category 1) of coded bitstreams according to some embodiments is depicted. In this coded bitstream, frame 1 is shown at 402, and frame 2 is shown at 404. The frame may be an image of a video. The coded bitstream may contain plaintext data and protected data. The plaintext data should not be encrypted, while the protected data should be encrypted. Different types of categories may have different patterns for plaintext and protected data. As described later, this pattern can be described using syntax. In Category 1, a frame contains only a segment (e.g., consecutive bytes) of a consecutive instance of plaintext data, and one or more segments of consecutive bytes of protected data. For example, for frame 1, plaintext data is contained at 406-1. Then, one or more segments of protected data are contained at 408-1. These protected data segments must not contain any interspersed plaintext data. In some embodiments, a single protected data segment or multiple consecutive protected data segments may be contained. This pattern restarts in frame 2. For example, in frame 2, plaintext data is contained at 406-2. Then, one or more instances of protected data are contained at 408-2. High-level video coding (AVC) bitstreams (containing one slice per frame), high-efficiency video coding (HEVC) bitstreams (containing one slice per frame), and audio-video 1 (AV1) bitstreams (containing one tile per frame) can be classified as Category 1.

[0052] Figure 4B A second type (Category 2) of coded bitstream according to certain embodiments is depicted. In this coded bitstream, frame 1 is shown at 402, and frame 2 is shown at 404. In Category 2 frames, multiple plaintext data segments and multiple protected data segments are contained. Plaintext data segments and protected data segments may alternate or follow other patterns. One difference in Category 2 is that, compared to Category 1 where there is only one plaintext data segment, this bitstream contains multiple plaintext data segments. For example, for frame 1, plaintext data is contained at 410-1. Then, protected data is contained at 412-1. This pattern repeats again, with plaintext data at 410-2 and protected data at 412-2. This pattern can continue until the end of the frame, where both plaintext and protected data may be contained simultaneously. Then, for frame 2, plaintext data, protected data, plaintext data, protected data, etc., appear in the same pattern as in frame 1. AVC bitstreams (containing multiple slices in one frame), HEVC bitstreams (containing multiple slices in one frame), and AV1 bitstreams (containing multiple tiles in one frame) can be classified into Category 2.

[0053] The encapsulation and encryption system 206 needs to identify the location of all protected data and plaintext data fragments, which can be complex if parsed bit by bit. The following uses a syntax to describe the locations to improve the efficiency of determining them.

[0054] Metadata

[0055] The following describes the different syntaxes used for different categories of metadata. In some embodiments, two signaling methods can be used: an encoder canonical method and an encoder non-canonical method. In the encoder canonical method, encoder 202 adds metadata to the encoded bitstream, for example, in frames (e.g., in the first segment at the beginning of a frame). Encoder 202 can insert metadata into the encoded bitstream in accordance with video codec standards. In the encoder non-canonical method, encoder 202 provides metadata about the location of protected or plaintext data through a separate channel (e.g., through sidecar metadata or custom supplementary information). This separate channel can be separate from the encoded bitstream because the metadata is not inserted into the bytes of the encoded bitstream based on an encoding protocol or specification. This method does not require modification of the video codec standard and can be used for solutions with complete control over encoder 202 and encapsulation and encryption system 206. It should be noted that some systems may not encode, encapsulate, and encrypt the source content, but may only encode the source content. Furthermore, some systems may only receive the encoded bitstream. In this case, it may not be possible to transmit metadata between different systems via a separate channel.

[0056] Different syntax examples can be used. Two examples are described below, but other examples may also be used. The following describes the use of the encoder canonical method. However, the same syntax can also be used for the encoder non-canonical method, except that the metadata is sent through a separate channel instead of being inserted into the encoded bitstream. Table 1 shows the first example of the bitstream syntax:

[0057]

[0058] Table 1

[0059] Encoder 202 inserts metadata into a specific location within a frame, such as the first plaintext data segment of each frame, according to the syntax shown in Table 1, although other locations may also be used. Inserting metadata into the first plaintext data segment reduces the amount of data that needs to be parsed to find the metadata. After reading the metadata, parser 204 can move to the next frame without parsing the rest of the frame. In Table 1, Num_of_entries is an integer (not less than 0) representing the number of entries. Entry_type can be a boolean value (0 or 1) indicating the type of entry, such as plaintext data or protected data. Entry_size (greater than 0) is a value used to determine the position of an entry, such as the number of bytes in the current entry. For example, the entry size can be used to determine how many bytes to encrypt or not encrypt. Furthermore, the entry size can also be a byte offset or other positional information.

[0060] This syntax can be used for both Category 1 and Category 2. Figure 5 The metadata syntax of the Category 1 bitstream encoder specification syntax according to some embodiments is shown. Figure 5 In the table, at positions 502-1 in frame 1 and 502-2 in frame 2, the metadata for the N, ET, and EZ entries corresponds to Num_of_entries, Entry_type, and Entry_size in the syntax of Table 1, respectively. An entry can represent a data fragment in a frame, such as plaintext data or protected data.

[0061] Here, a portion of the plaintext data at the beginning of each frame can contain metadata. An N entry is included at 504-1. Then, an ET1 entry is included at 506-1, and an EZ1 entry is included at 508-1. For frame 1, encoder 202 can determine the total size of frame 1 (Frame_size). Encoder 202 can then set the value of N to 1, indicating the inclusion of one entry. The value of ET1 is the plaintext type (e.g., a boolean value of 0 can be used to represent plaintext data), and the value of EZ1 is the size of the plaintext data. Parser 204 can determine the location of the protected data; for example, the size of the protected data is determined by Frame_size - EZ1. For example, if the frame size is 1000 bytes and the plaintext data is 100 bytes, then the protected data is 1000 bytes - 100 bytes = 900 bytes. The location of the plaintext data can be bytes 0-99, and the location of the protected data can be bytes 100-999. With this location information, encapsulation and encryption system 206 can encrypt the protected data.

[0062] In other embodiments, encoder 202 can insert not only the value of a single entry, but also the value of ET2 (representing the protection type, e.g., a Boolean value of 1 can be used to represent protected data), and the value of EZ2 represents the size of the protected data. Furthermore, depending on the number of protected data segments in frame 1, the values ​​of ET and EZ2 can have one or more instances. For example, another protected data segment might contain a protection type (e.g., a Boolean value of 1 can be used to represent protected data) in ET3, and the value of EZ3 might represent the size of the protected data. The value of N can increase based on the number of entries contained in the metadata. Frame 2 contains another instance of metadata describing the plaintext data and protected data in frame 2.

[0063] Figure 6 The metadata syntax of a Category 2 bitstream encoder specification method according to some embodiments is shown. Figure 6 In the code, at positions 602-1 in frame 1 and 602-2 in frame 2, the metadata for the N, ET, and EZ entries corresponds to Num_of_entries, Entry_type, and Entry_size in the syntax, respectively. Here, the plaintext data portion at the beginning of each frame can contain metadata. An N entry is included at position 604-1. Then, an ET1 entry is included at position 606-1, and an EZ1 entry is included at position 608-1. These entries can be repeated. For frame 1, encoder 202 can determine the total size of frame 1 (Frame_size). Encoder 202 then sets the value of N to K-1. N is set to K-1 because the frame size can be used to determine the size of the last entry. However, all entries may be included, and N is set to K. The value of ET1 is the plaintext type (e.g., a boolean value of 0 can be used to represent plaintext data), and the value of EZ1 is the size of the plaintext data. Encoder 202 can determine that the value of ET2 is the protected type (e.g., a boolean value of 1 can be used to represent protected data), and the value of EZ2 is the size of the protected data. Depending on the number of plaintext and protected data segments in Frame 1, there may be one or more instances of the ET and EZ values. For example, another plaintext data segment might contain a plaintext type in ET3 (e.g., a boolean value of 0 can be used to represent plaintext data), and the value of EZ3 is the size of the plaintext data. Another protected data segment might contain a protected type in ET4 (e.g., a boolean value of 1 can be used to represent protected data), and the value of EZ4 is the size of the protected data. Frame 2 contains another instance of metadata describing the plaintext and protected data in Frame 2.

[0064] Using the inserted metadata, parser 204 can directly deduce the locations of plaintext and protected data based on the entry type and size after parsing the inserted metadata. For example, parser 204 uses the size to determine the byte offset of each plaintext and protected data fragment. If parser 204 inserts the inserted metadata at the beginning of the first plaintext data instance, parser 204 does not need to parse any additional bits of the frame until the next frame. Alternatively, parser 204 can jump to the frame after reading the metadata and determining the location of the plaintext or protected data.

[0065] Table 2 shows a second example of bitstream syntax:

[0066]

[0067] Table 2

[0068] In Table 2, Num_of_entries (not less than 0) is an integer representing the number of entries. Entry_size (not less than 0) represents the number of bytes in the current entry. This scheme can be used respectively for Figure 7 and Figure 8 The examples show categories 1 and 2. In... Figure 7 and Figure 8 In this context, the N and EZ entries correspond to Num_of_entries and Entry_size, respectively.

[0069] The difference between the first and second syntaxes is that in the second syntax, the Entry_type corresponding to Entry_size is determined by the default pattern. The first pattern can be plaintext data, protected data, plaintext data, protected data, etc. The second pattern can be protected data, plaintext data, protected data, plaintext data, etc. The third pattern can be plaintext data, protected data, protected data, protected data. When using the Entry_type syntax, the encapsulation and encryption system 206 does not need to know the specific entry type. Instead, Entry_type can be determined based on the default pattern.

[0070] Figure 7 The metadata syntax of a second example of the Category 1 bitstream encoder specification syntax according to certain embodiments is shown. Figure 7In the first frame at 702-1 and the second frame at 702-2, the metadata for N entries and EZ entries corresponds to Num_of_entries and Entry_size in the syntax, respectively. In some embodiments, the default Entry_type is plaintext data, protected data, plaintext data, protected data, etc. If the frame size is known at the beginning, the value of N is 1, and the value of EZ1 is the size of the plaintext data. If the frame size is unknown, the value of N is 2. The values ​​of EZ1 and EZ2 correspond to the sizes of plaintext data and protected data, respectively. If there are no more protected data entries, there may be no more entries after EZ2. If there are more protected data entries, the EZ value corresponding to the plaintext data entry may be zero because there are no more plaintext data entries in category 1 of the frame. However, the alternating EZ value corresponding to protected data corresponds to the size of the protected data. In this case, the value of N can be increased.

[0071] In some embodiments, the default Entry_type is Protected Data, Plaintext Data, Protected Data, Plaintext Data, etc. If the frame size is known at the beginning, then N is 2, EZ1 is 0, because the first entry is plaintext data, so the size of the protected data is zero here. EZ2 is the size of the plaintext data. The frame size can be used to determine the size of the protected data as described above. If the frame size is unknown, then N can be 3. EZ1 is 0, and the values ​​of EZ2 and EZ3 correspond to the sizes of the plaintext data and the protected data, respectively. If there are more protected data entries, the EZ value corresponding to the plaintext data entry may be zero, because there are no more plaintext data entries in category 1 of the frame. However, the size of the alternating EZ entries corresponding to the protected data corresponds to the size of the protected data. The N value can also be increased.

[0072] Figure 8 The metadata syntax of a second example of an encoder specification method for a category 2 bitstream, according to some embodiments, is shown. Figure 8In the example, at positions 802-1 in frame 1 and 802-2 in frame 2, the metadata for the N and EZ entries corresponds to Num_of_entries and Entry_size in the syntax, respectively. In one embodiment, the default Entry_type is a pattern of plaintext data, protected data, plaintext data, protected data, etc. If the frame size is known at the beginning and there are K segments (including plaintext and protected data), then the value of N is K-1. The EZ1 entry is the size of the first plaintext data entry in the frame. The value of the EZ2 entry is the size of the first protected data entry in the frame. Subsequent entries are the sizes of plaintext and protected data entries until the last entry, which can be determined based on the frame size and the sizes of other entries. If the frame is unknown, then the value of N is K. The sizes of all plaintext and protected data entries are contained in the EZ values, which alternate between the sizes of plaintext and protected data entries.

[0073] In some embodiments, the default Entry_type is a pattern of protected data, plaintext data, protected data, plaintext data, etc. If the frame size is known at the beginning and there are K segments (including plaintext data and protected data), then N is K. The first EZ1 entry is 0 because the first entry is plaintext data, so the size of the protected data is zero here. The value of the EZ2 entry is the size of the first plaintext data in the frame. The value of the EZ3 entry is the size of the first protected data in the frame. N is K because the pattern begins with a protected data entry (size zero). If the frame size is unknown, then N is K+1. Additional entries are needed because the pattern begins with a protected data entry (size zero). If there are more plaintext and protected data entries after the first plaintext and first protected data entries, then after EZ1, EZ2, and EZ3 entries, the EZ value will alternate between the size of the protected data entry and the size of the plaintext data entry.

[0074] In some embodiments, the default Entry_type is a pattern of plaintext data, protected data, protected data, ..., protected data, etc. If the frame size is known at the beginning and there are K segments (including plaintext data and protected data), then the value of N is K–1. The EZ1 entry is the size of the first plaintext data entry in the frame. The value of the EZ2 entry is the size of the first protected data entry in the frame. Subsequent entries are the sizes of all protected data entries until the last entry, which can be determined based on the frame size and the sizes of other entries. If the frame is unknown, then the value of N is K. The sizes of all plaintext data and protected data entries are contained in the EZ values, which alternate between the sizes of plaintext data entries and protected entries.

[0075] Encryption process

[0076] Figure 9 A simplified flowchart 900 of a method for encrypting an encoded bitstream according to some embodiments is shown. In step 902, a parser 204 receives the encoded bitstream. In some embodiments, an encapsulation and encryption system 206 receives the encoded bitstream from an encoder 202, which may be located within the same process or in a separate process.

[0077] In step 904, parser 204 determines the type of signaling method used. For example, encapsulation and encryption system 206 determines which of the above-described signaling methods was used to transmit metadata of protected data in the encoded bitstream. In some embodiments, a parameter can be set in encapsulation and encryption system 206 to indicate the signaling method used. Alternatively, information in the encoded stream can also indicate the signaling method used. Alternatively, information can be transmitted via a separate channel to indicate the signaling method used.

[0078] In step 906, parser 204 determines metadata according to a signaling method. For example, metadata may be contained in an encoded bitstream or transmitted over a separate channel.

[0079] In step 908, parser 204 analyzes metadata to determine the location of protected data in the encoded bitstream. The method of determining the location may vary depending on the syntax and bitstream type used. In some embodiments, parser 204 determines the size of plaintext data and protected data in the source video frame.

[0080] In step 910, the encapsulation and encryption system 206 encrypts the location of the protected data in the encoder bitstream to generate an encrypted bitstream. The plaintext data portion is not encrypted. In some embodiments, the encapsulation and encryption system 206 may encrypt portions of the encoded bitstream using the size and entry type of the plaintext data. For example, for plaintext data, the encapsulation and encryption system 206 does not encrypt the size of the plaintext data. Then, for the protected data, the encapsulation and encryption system 206 encrypts a portion of the protected data. This process can be performed on the entire frame.

[0081] In step 912, the encapsulation and encryption system 206 outputs the encrypted bitstream. The encrypted bitstream can be sent to client device 104. Server system 102 can deliver the encrypted bitstream to client device 104. Client device 104 can decrypt the encrypted portion of the encrypted bitstream. The encrypted bitstream can be an encoded bitstream containing encrypted and unencrypted bytes. Client device 104 decodes the encoded bitstream after decryption. Then, client device 104 replays the content through interface 110.

[0082] Example

[0083] The following describes a method for determining the location of plaintext and protected data. In the AV1 video coding format, Open Bitstream Units (OBUs) are the basic structures used to encapsulate different video data elements; these OBUs are grouped into Temporal Units (TUs). Each OBU type in the AV1 bitstream has a specific purpose, designed to carry different types of information. An OBU consists of a 1- or 2-byte header specifically identifying the OBU type, followed by an optional length field and an optional payload. The presence and content of the payload depend on the OBU type. Depending on its type, an OBU can carry configuration information, metadata, or coded video data. In some embodiments, protected data is located at Frame OBUs to TileGroup OBUs. Encoder 202 can parse the sequence header OBU and frame header OBU to determine the location of protected data within a frame.

[0084] Different constraints can be applied to determine the protected data. In one embodiment, the following constraints are applied within the protected data:

[0085] • The protected data must be completely covered by one or more consecutive subsamples.

[0086] • All OBU headers and their associated obu_size fields must not be protected.

[0087] • The time separator OBU, sequence header OBU, frame header OBU (including intra-frame OBU), redundant frame header OBU, and padding OBU shall not be protected.

[0088] • Metadata OBU must not be protected.

[0089] • Tile group OBU and frame OBU must be protected.

[0090] • Only tile data in tile group OBU and frame OBU is protected.

[0091] Due to the aforementioned limitations, the derivation of protected data involves searching for tile data within the tile group OBU or frame OBU. However, this derivation is highly dependent on the derivation of the sequence header OBU and frame header OBU / frame header. Derivation of the frame header requires a very complex bitstream parsing process, which significantly limits the efficiency of the video transmission process. Therefore, using metadata to determine the location of protected data may be more efficient than bitstream parsing.

[0092] in conclusion

[0093] Therefore, metadata can be used to more effectively determine the location of protected data. Computational resources are saved because there is no need to parse every single bit of the encoded bitstream. Furthermore, since bitstream parsing only requires analyzing a small amount of information, the encapsulation and encryption processes can be completed much faster.

[0094] system

[0095] like Figure 10 As shown, the features and aspects disclosed herein can be used in conjunction with the video streaming system 1000 and communicate with multiple client devices via one or more communication networks. The description of the video streaming system 1000 is merely to provide an example of how this disclosure can be used to distribute and deliver prepared content. It should be understood that this technology is not limited to streaming video applications and can be applied to other applications and delivery mechanisms.

[0096] In one embodiment, a media program provider may include a media program library. For example, media programs may be aggregated and offered via a website, application, or browser. Users can access the media program provider's website or application and request media programs. Users may be restricted to requesting only media programs offered by the media program provider.

[0097] In system 1000, video data may be obtained from one or more sources (e.g., video source 1010) and used as input to video content server 1002. The input video data may contain raw or edited frame-based video data in any suitable digital format, such as Moving Picture Experts Group (MPEG)-1, MPEG-2, MPEG-4, VC-1, H.264 / Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), or other formats. Alternatively, the video may be provided in a non-digital format and converted to digital format using a scanner or transcoder. The input video data may contain various types of video clips or programs, such as television episodes, movies, and other content produced as primary content of consumer interest. The video data may also contain audio, or only audio.

[0098] The video streaming system 1000 may include one or more computer servers or modules 1002, 1004, and 1007 distributed across one or more computers. Each server 1002, 1004, and 1007 may include one or more data stores 1009, or be operatively connected to one or more data stores 1009, such as databases, indexes, files, or other data structures. The video content server 1002 has access to data storage (not shown) containing various video segments. The video content server 1002 may provide video segments according to instructions from a user interface controller communicating with client devices. As used herein, a “video segment” refers to a specific portion of frame-based video data, such as that used for watching television series, movies, recorded live performances, or other video content in a streaming video session.

[0099] In some embodiments, the video supplementary content (SC) server 1004 may access a data repository storing relatively short videos (e.g., 10-second, 30-second, or 60-second supplementary content) configured as advertisements for specific advertisers or information. The supplementary content may be provided to an entity in exchange for some form of payment, or it may contain promotional information, public service information, or other information from system 1000. The video supplementary content server 1004 may provide supplementary content segments according to instructions from a user interface controller (not shown).

[0100] The video streaming system 1000 may also include a server system 102.

[0101] The video streaming system 1000 may also include an integration and streaming component 1007 that integrates video content and supplementary content into streaming video segments. For example, the streaming component 1007 may be a content server or a streaming media server. A controller (not shown) may determine the selection or configuration of supplementary content in the streaming video based on any suitable algorithm or process. The video streaming system 1000 may include... Figure 10 Other modules or units not shown, such as management servers, business servers, network infrastructure, supplementary content selection engines, etc.

[0102] The video streaming system 1000 can be connected to a data communication network 1012. The data communication network 1012 may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a telephone network, a wireless network 1014 (e.g., a wireless cellular communication network (WCS)), or some combination of these networks or similar networks.

[0103] One or more client devices 1020 can communicate with the video streaming system 1000 via a data communication network 1012, a wireless network 1014, or other networks. Such client devices may include, for example, one or more laptops 1020-1, desktop computers 1020-2, "smart" mobile phones 1020-3, tablets 1020-4, IPTVs 1020-5, or combinations thereof, and can communicate via a router 1018 (for a local area network), a base station 1017 (for the wireless network 1014), or other connectivity methods. During operation, such client devices 1020 can send and receive data or instructions to the system 1000 in response to user input received from user input devices or other inputs. In response, the system 1000 can provide video clips and metadata to the client devices 1020 from a data repository 1009 based on media programs selected by the user. Client devices 1020 can use displays, projectors, or other video output devices to output video content from streaming video clips in a media player and receive user input to interact with the video content.

[0104] Audio and video data can be distributed from the streaming component 1007 to remote client devices in various ways, including computer networks, telecommunications networks, and combinations thereof. In streaming transmission, the content server continuously streams audio and video data to a media player component that runs at least partially on the client device, which can simultaneously play the audio and video data and receive streaming data from the server. While this article discusses streaming transmission, other transmission methods can also be used. The media player component can begin playing video data immediately after receiving the initial portion of the data from the content provider. Traditional streaming technologies use a single provider to deliver a data stream to a group of end users. Transmitting a single streaming media stream to a large number of users can require high bandwidth and processing power, and the bandwidth required by the provider increases with the number of end users.

[0105] Streaming media can be delivered on demand or live. Streaming supports immediate playback at any point in the file. End users can skip media files to begin playback or change the playback position to any point within the media file. Therefore, end users do not need to wait for the file to download incrementally. Typically, streaming media is transmitted by several dedicated servers with high bandwidth capabilities via dedicated equipment that accepts video file requests and transmits the amount of data required to play the video at the rate required for playback, based on the format, bandwidth, and structure information of these files. The streaming media server can also consider the transmission bandwidth and performance of the media player on the target client. The streaming media component 1007 can communicate with the client device 1020 using control messages and data messages to adapt to changing network conditions during video playback. These control messages can contain commands for enabling control functions such as fast forward, rewind, pause, or jumping to specific parts of the file on the client.

[0106] Because the streaming component 1007 transmits video data only at the required rate when needed, the number of streams served can be precisely controlled. Viewers cannot watch high-data-rate videos on low-data-rate transmission media. However, the streaming server (1) provides users with random access to video files, (2) allows monitoring of who is watching which video programs and for how long, (3) utilizes transmission bandwidth more efficiently because only the amount of data required to support the viewing experience is transmitted, and (4) the video files are not stored on the viewer's computer but are discarded by the media player, thus allowing for greater control over the content.

[0107] The streaming component 1007 can use TCP-based protocols such as Hypertext Transfer Protocol (HTTP) and Real-Time Messaging Protocol (RTMP). The streaming component 1007 can also provide live webcasting and support multicast, allowing multiple clients to listen to a single streaming media stream simultaneously, thus saving bandwidth. The streaming player can provide random access to any point in a media program without buffering the entire video. Instead, this is achieved by the media player sending control messages to the streaming server. Other protocols used for streaming playback include HTTP Live Streaming (HLS) or HTTP-based Dynamic Adaptive Streaming (DASH). HLS and DASH protocols transmit video over HTTP via a playlist containing multiple short segments, typically provided by one or more Content Delivery Networks (CDNs) at various bitrates. This allows the media player to switch bitrates and content sources segment by segment. This switching helps compensate for network bandwidth fluctuations and infrastructure failures that may occur during video playback.

[0108] Streaming video content can be delivered in several modes. One mode involves users paying to watch video programs, such as paying for access to a media library or limited media programming, or using a pay-per-view service. Another mode, widely adopted shortly after the advent of broadcast television, involves sponsors paying to broadcast media programs in exchange for the right to play supplementary content during or alongside the main program. In some modes, supplementary content is inserted at predetermined times during the video program; these times can be called "slots" or "breaks." For streaming video, media players can be configured so that client devices must play predetermined supplementary content within specified time slots before they can play the video.

[0109] See Figure 11 The figure shows a schematic diagram of a device 1100 for viewing video content and supplementary content. In some embodiments, the device 1100 may include a processor (CPU) 1102 operatively coupled to a processor memory 1104 that stores binary-coded functional modules for execution by the processor 1102. These functional modules may include an operating system 1106 for handling system functions such as input / output and memory access, a browser 1108 for displaying web pages, and a media player 1110 for playing videos. The memory 1104 may also store... Figure 11 Other modules not shown, such as those used to perform other operations described in other parts of this document.

[0110] Bus 1114 or other communication components may be used to support information communication within device 1100. Processor 1102 may be a dedicated microprocessor configured or operated to perform specific tasks according to the features and aspects disclosed herein, specifically by executing machine-readable software code that defines those specific tasks. Processor memory 1104 (e.g., random access memory (RAM) or other dynamic storage device) may be connected to bus 1114 or directly to processor 1102 and stores information and instructions to be executed by processor 1102. Memory 1104 may also store temporary variables or other intermediate information during the execution of such instructions.

[0111] The computer-readable medium in storage device 1124 may be connected to bus 1114 and store static information and instructions of processor 1102; for example, storage device (CRM) 1124 may store modules of operating system 1106, browser 1108, and media player 1110 when device 1100 is powered off, and load these modules into processor memory 1104 when device 1100 is powered on. Storage device 1124 may include non-transitory computer-readable storage medium that stores information, instructions, or combinations thereof, such that, for example, when processor 1102 executes these instructions, device 1100 will be configured or operable to perform one or more operations of the methods described herein.

[0112] Network communication interface 1116 may also be connected to bus 1114. Network communication interface 1116 provides or supports bidirectional data communication between device 1100 and one or more external devices (e.g., streaming media system 1000), which may be selectively implemented via router / modem 1126 and wired or wireless connection 1125. Alternatively, device 1100 may include a transceiver 1118 connected to antenna 1129, through which device 1100 can wirelessly communicate with the base station of a wireless communication system or router / modem 1126. Alternatively, device 1100 may communicate with video streaming media system 1000 via a local area network, virtual private network, or other network. Another option is that device 1100 may function as a module or component of system 1000 and communicate with other components via bus 1114 or other means.

[0113] Device 1100 may be connected to display unit 1128 (e.g., via bus 1114 and graphics processing unit 1120). Display 1128 may include any suitable configuration for displaying information to the operator of device 1100. For example, display 1128 may include or use a liquid crystal display (LCD), a touch screen LCD (e.g., a capacitive display), a light-emitting diode (LED) display, a projector, or other display device to present information to the user of device 1100 in a visual display manner.

[0114] One or more input devices 1130 (e.g., alphanumeric keypad, microphone, keypad, remote control, game controller, camera, or camera array) can be connected to bus 1114 via user input port 1122 to transmit information and commands to device 1100. In some embodiments, input device 1130 provides or supports control over cursor position. Such cursor control device (also referred to as a pointing device) can be configured as a mouse, trackball, touchpad, touchscreen, cursor arrow keys, or other device for receiving or tracking physical motion and converting that motion into electrical commands instructing cursor movement. The cursor control device can be integrated into display unit 1128, for example, using a touchscreen. The cursor control device can transmit directional information and command selection to processor 1102 and control cursor movement on display 1128. The cursor control device can have two or more degrees of freedom, for example, allowing the device to specify cursor position in a planar or three-dimensional space.

[0115] Some embodiments may be implemented in a non-transitory computer-readable storage medium for use by or connected to an instruction execution system, apparatus, system, or machine. The computer-readable storage medium contains instructions for controlling a computer system to perform the methods described in some embodiments. The computer system may include one or more computing devices. When executed by one or more computer processors, these instructions may be configured or operated to perform the operations described in some embodiments.

[0116] Unless the context clearly specifies otherwise, the terms “a,” “an,” and “the” in this description and the following claims shall include the plural form. Furthermore, unless the context clearly specifies otherwise, the term “among” as used in this specification and the following claims shall have the meaning of both “among” and “above”.

[0117] The foregoing specification describes various embodiments and illustrates how certain aspects of some embodiments can be implemented. The above examples and embodiments should not be considered as the only embodiments, but are provided to illustrate the flexibility and advantages of certain embodiments as defined in the following claims. Based on the foregoing disclosure and the following claims, other settings, embodiments, implementations, and equivalents may be adopted without departing from the scope defined in the claims.

Claims

1. A method comprising: Receive encoded bit stream; Determine information for the encoded bitstream determined during content encoding, wherein the information follows a syntax for protected data describing the encoded bitstream; Using the information, determine the location set of the encoded bit stream to be encrypted based on the syntax; Based on the location set, the first portion of the encoded bitstream corresponding to the protected data is selectively encrypted to form an encrypted bitstream, wherein the second portion of the encoded bitstream corresponding to the plaintext data is not encrypted; and Output the encrypted bit stream.

2. The method according to claim 1, further comprising: The content is encoded to generate the encoded bitstream, wherein the information of the encoded bitstream is determined during the encoding process.

3. The method of claim 1, wherein the information of the encoded bitstream is inserted into the encoded bitstream.

4. The method of claim 1, wherein the information of the encoded bitstream is inserted into a first plaintext data segment in the encoded bitstream.

5. The method according to claim 1, wherein, The information in the encoded bitstream is received separately from the encoded bitstream.

6. The method according to claim 1, wherein, Determining the location set using the information includes: Determine the number of entries in the information; For each entry: Determine the entry type, which indicates whether the entry pertains to protected data or plaintext data; and Determine the entry size, which is used to determine the location of the protected data or the plaintext data.

7. The method according to claim 6, wherein, The entry size is used to determine the size of the protected data or the plaintext data.

8. The method according to claim 6, wherein, The entry size is used to determine the byte offset of the protected data or the plaintext data.

9. The method according to claim 6, wherein: The entry type indicates that the plaintext data is located at the first position in the frame; The entry type of one or more entries indicates that the protected data is located at one or more second locations in the frame.

10. The method according to claim 6, wherein: The entry type of the first plurality of entries indicates that the plaintext data is located at a plurality of first positions in the frame; The entry type of the second plurality of entries indicates that the protected data is located in a plurality of second positions in the frame.

11. The method according to claim 1, wherein, Determining the location set using the information includes: Determine the number of entries in the information; For each entry: Determine the entry size used to locate the protected data or the plaintext data.

12. The method according to claim 11, wherein: Determine the default mode for the plaintext data and the protected data, and The default mode is used to determine that the plaintext data is located at a first position in the frame, and the protected data is located at one or more second positions in the frame.

13. The method according to claim 12, wherein: The default mode alternates between the plaintext data and the protected data. The plaintext data is located at the beginning of the default mode. After the entry size includes the plaintext data located at the first position, the entry size for any other plaintext data in the default mode is zero.

14. The method of claim 11, wherein: Determine the default mode for the plaintext data and the protected data, and The default mode is used to determine that the plaintext data is located at multiple first positions in the frame, and the protected data is located at multiple second positions in the frame.

15. The method of claim 14, wherein: The default mode alternates between the plaintext data and the protected data. The protected data is first in the default mode, and In the default mode, the first entry of the protected data entry size is zero.

16. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a computing device, enable the computing device to perform the following operations: Receive encoded bit stream; Determine information for the encoded bitstream determined during content encoding, wherein the information follows a syntax for protected data describing the encoded bitstream; Using the information, determine the location set of the encoded bit stream to be encrypted; Based on the location set, the first portion of the encoded bitstream corresponding to the protected data is selectively encrypted to form an encrypted bitstream, wherein the second portion of the encoded bitstream corresponding to the plaintext data is not encrypted; and Output the encrypted bit stream.

17. An apparatus comprising: One or more computer processors; as well as A computer-readable storage medium containing instructions for controlling the one or more computer processors to perform the following operations: Received content; The context is analyzed during the encoding of the content to determine the location of protected data and plaintext data; Determine the syntax for identifying information describing the location of protected data in the encoded bitstream; Information is used to determine the location set of the encoded bit stream to be encrypted, wherein the information is formatted using the syntax. as well as The location set information is provided to the encryption system using signaling methods.

18. The apparatus according to claim 17, wherein: The encryption system selectively encrypts a first portion of the encoded bitstream corresponding to the protected data based on the location set to form an encrypted bitstream, wherein a second portion of the encoded bitstream corresponding to the plaintext data is not encrypted.

19. The apparatus according to claim 17, wherein, Analyzing the context used to encode the bitstream includes: Analyze the header information to determine the location of the protected data and the plaintext data.

20. The apparatus according to claim 17, wherein, Providing the information for the location set using the signaling method includes: Add the information of the location set to the encoded bitstream, or The information of the location set is added to a channel that is separate from the encoded bitstream.

Images