Authentication of supplemental information in data streams

Abstract

An apparatus and method for checking plausibility of a data stream are described. According to an aspect, a data stream includes an indication that indicates whether one or more supplemental information messages should be included in a portion of the data stream that is to be used to perform a trustworthiness check of the data stream.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511524245.9 (22) Application Date 2025.10.23 (30) Priority Data 24208513.2 2024.10.23 EP (71) Applicant Fraunhofer Association for the Promotion of Applied Research Address Germany (72) Inventors Jonathan Pfaff, Tobias Hinz, Karlsten Zullingheiko Schwarz, Robert Scubin, Sánchez de la Fuente, Cornelius Helch, Thomas Scheldtler, Malpe, Thomas Wigand (74) Patent Agency: China Patent & Trademark Agency Co., Ltd. 11021 Patent Attorneys: Luo Songmei, Pan Jianying (51) Int.Cl. H04L 9 / 40 (2022.01) H04L 9 / 32 (2006.01) H04L 67 / 01 (2022.01) H04L 5 / 00 (2006.01) (54) Title of Invention: Authentication of Supplementary Information in a Data Stream (57) Abstract: This paper describes an apparatus and method for checking the credibility of a data stream. According to one aspect, the data stream includes an indication indicating whether one or more supplementary information messages should be included in a portion of the data stream, which will be used to perform a credibility check on the data stream. Claims 3 pages, Description 55 pages, Drawings 10 pages, CN 121923845 A 2026.04.24 CN 1 21 92 38 45 A 1. An apparatus (20) for decoding an audio data stream (14) in which an audio signal is encoded, the audio data stream comprising a plurality of payload packets (16) carrying payload data, and further comprising supplementary information packets (18) carrying supplementary information messages (19), wherein the apparatus is configured to: derive (21) an indication (51) from the audio data stream, the indication (51) indicating whether one or more supplementary information messages should be considered to verify (40) the audio data stream; if the indication (51) indicates that the one or more supplementary information messages should be considered to verify the audio data stream, then the predetermined portion (13) of the audio data stream is determined (30) by including the one or more supplementary information messages in a predetermined portion of the audio data stream. (21) Obtain (43) a digital signature (for verifying the predetermined portion (13) from the indication in the audio data stream. 2. The apparatus of claim 1, wherein the apparatus is configured to verify the predetermined portion by checking whether the predetermined portion in the audio data stream matches the digital signature.3. The apparatus of any one of claims 1 or 2, wherein the apparatus is configured to provide the predetermined portion and the digital signature to verify the predetermined portion and the digital signature. 4. The apparatus of any one of claims 1 to 3, wherein checking whether the predetermined portion (13) in the audio data stream matches the digital signature (43) comprises: applying a hash function to the predetermined portion (13) to obtain a hash value; and checking whether the hash value matches the digital signature (43). 5. The apparatus of claim 4, wherein checking whether the hash value matches the digital signature (43) comprises: forming a verification string (48) based on the hash value; decrypting the digital signature (43) to obtain a check value; and checking whether the verification string (48) matches the check value. 6. The apparatus of any one of claims 1 to 5, configured to: determine the predetermined portion (13) by including the one or more supplementary information messages in the predetermined portion (13) if the indication indicates that one or more supplementary information messages should be considered to verify the predetermined portion (13). 7. The apparatus according to any one of claims 1 to 6, configured to include all supplementary information packets in the predetermined portion (13), the supplementary information packets carrying any supplementary information messages to be included in the predetermined portion (13). 8. The apparatus according to any one of claims 1 to 7, configured to: determine whether to include supplementary information packets in the supplementary information packets in the predetermined portion (13) based on the result of checking whether a supplementary information payload packet includes supplementary information messages of any supplementary information message type from a set of supplementary information message types. 9. The apparatus according to any one of claims 1 to 8, wherein the indication is a flag having a first state and a second state, and wherein the apparatus is configured to determine the predetermined portion (13) in the audio data stream by: if the flag has a first state, determining the one or more supplementary information messages to be included in the predetermined portion (13) based on a set of predefined supplementary information messages in the supplementary information messages; and if the flag has a second state, avoiding including any of the supplementary information messages in the predetermined portion (13). 10. The apparatus of claim 9, wherein the group predefined supplementary information message includes all supplementary information messages in the supplementary information claim 1 / 3 page 2 CN 121923845 A group that reference the plurality of payload groups (16). 11. The apparatus of claim 9, wherein the group predefined supplementary information message includes all supplementary information messages of any type from a set of supplementary information message types.12. The apparatus according to any one of claims 1 to 11, further configured to: if the indication indicates that the one or more supplementary information messages should be considered to verify the audio data stream, derive an identifier of the one or more supplementary information messages to be included in the predetermined portion (13) from the audio data stream. 13. The apparatus according to claim 12, configured to: if the indication indicates that the one or more supplementary information messages should be considered to verify the audio data stream, derive a syntax element from the audio data stream, the syntax element identifying a set of supplementary information message types to be included in the predetermined portion (13) of the audio data stream. 14. The apparatus according to any one of claims 1 to 13, configured to derive a syntax element from the audio data stream, the syntax element indicating whether the one or more supplementary information messages should be considered to verify the audio data stream, and the syntax element signaling an identifier of the one or more supplementary information messages to be included in the predetermined portion (13). 15. The apparatus of claim 14, wherein the syntax element has a second state and a plurality of first states, and wherein the apparatus is configured to: if the syntax element has one of the first states, determine the one or more supplementary information messages to be included in the predetermined portion (13) depending on the state of the syntax element; and if the syntax element has the second state, avoid including any of the one or more supplementary information messages in the predetermined portion (13). 16. The apparatus of claim 12, configured to: if the indication indicates that the one or more supplementary information messages should be considered to verify the audio data stream, derive a first syntax element and a corresponding number of second syntax elements from the audio data stream, the first syntax element indicating a count of supplementary information message types to be considered for the predetermined portion (13), each second syntax element identifying a corresponding supplementary information message type to be included in the predetermined portion (13).17. An apparatus (10) for encoding an audio data stream (14) in which an audio signal is encoded, wherein the apparatus is configured to: enable the audio data stream to be trusted; and to encode a plurality of payload packets (16) carrying payload data into the audio data stream, and to also encode supplementary information packets (18) carrying supplementary information messages into the audio data stream, wherein the apparatus is configured to: insert an instruction (51) into the audio data stream (23), the instruction (51) indicating whether one or more supplementary information messages should be considered to verify the audio data stream; if the instruction indicates that the one or more supplementary information messages should be considered to verify the audio data stream, then determine (30') the predetermined portion (13) of the audio data stream by including one or more of the supplementary information messages into a predetermined portion (13) of the audio data stream that will be trusted to be verified; and obtain (41') a digital signature (43) based on the predetermined portion (13). 18. A method for decoding an audio data stream (14) in which an audio signal is encoded, the audio data stream comprising a plurality of payload packets (16) carrying payload data and further comprising supplementary information packets (18) carrying supplementary information messages (19), wherein the method comprises: deriving (21) an indication (51) from the audio data stream, the indication (51) indicating whether one or more supplementary information messages should be considered to verify (40) the audio data stream; if the indication (51) indicates that the one or more supplementary information messages should be considered to verify the audio data stream, determining (30) the predetermined portion (13) of the audio data stream by including the one or more supplementary information messages in the audio data stream; obtaining (21) a digital signature (43) for verifying the predetermined portion (13) from the indication in the audio data stream.19. A method for encoding an audio data stream (14) in which an audio signal is encoded, wherein the method comprises: enabling the audio data stream to be trusted; and encoding a plurality of payload packets (16) carrying payload data into the audio data stream, and also encoding supplementary information packets into the audio data stream, the supplementary information packets carrying supplementary information messages, wherein the method comprises: inserting (23) an indication into the audio data stream, the indication indicating whether one or more supplementary information messages should be considered to verify the audio data stream; if the indication indicates that the one or more supplementary information messages should be considered to verify the audio data stream, determining (30') the predetermined portion (13) of the audio data stream by including one or more of the supplementary information messages into a predetermined portion (13) of the audio data stream that would enable trusted checking; and obtaining a digital signature (43) based on the predetermined portion (13). 20. An audio data stream, in which data signals are encoded, the audio data stream being capable of credibility checking, the audio data stream comprising: a plurality of payload packets (16) carrying payload data, and further comprising supplementary information packets carrying supplementary information messages; and an indication for indicating whether to consider one or more supplementary information messages to check the credibility of the audio data stream. Claims 3 / 3 Page 4 CN 121923845 A Authentication Technical Field of Supplementary Information in Data Streams

[0001] Embodiments of the present invention relate to apparatus and methods for decoding data streams to be credibly checked. Other embodiments relate to apparatus and methods for enabling data streams to be credibly checked. Other embodiments relate to data streams capable of credibility checking. Specifically, embodiments relate to media data streams (e.g., video data streams or audio data streams) and decoders and encoders respectively for decoding or encoding such data streams. Background Art

[0002] Content authentication is crucial for preventing media tampering. Rapid AI development has led to the emergence of sophisticated deepfakes, blurring the lines between authentic and fake content and raising serious cybersecurity and copyright issues. Therefore, the ability to verify the authenticity of media has become crucial.

[0003] Examples of methods for performing such authentication include providing a digital signature to the media by first hashing the media asset and then signing it using the private key of a content generator, such that on the client side, given the content generator's public key, the client can compare the provided signature with a hash value calculated by itself based on the received media asset. If these values ​​match, the client can securely assume that the media has not been tampered with.

[0004] The complex structure of a data stream can complicate the authentication of that stream. This is especially true in scenarios where the data stream is modified or generated by an entity (which, for example, extracts a portion of the data stream), such as in scenarios where the data stream comprises multiple sub-streams or where an entity combines multiple data streams. If the entity is not the one that encoded the original data stream, and if recompiling the digital signature that allows verification of the data stream should be avoided, new solutions are needed to allow reliable verification of the data stream.

[0005] The object of embodiments of the present invention is to provide a concept for checking the trustworthiness of a data stream that offers an improved trade-off between high adaptability to complex streaming scenarios, low computational cost, and a low bit rate of signaling overhead required to provide the information needed for authentication.

[0006] This object is achieved by the subject matter of the independent claims. Summary of the Invention

[0007] Embodiments of the present invention relate to the verification of supplementary information messages for a data stream that provide supplementary information for payload data carried in payload packets of the data stream. The phrases “verification of the data stream” and “checking the credibility of the data stream” are used interchangeably below and, for example, refer to verifying whether the content of the data stream corresponds to content provided by a content provider that signs the content using a certificate. In some scenarios, including supplementary information messages in the data stream verification process may have disadvantages, namely, that intermediate entities in the data stream transmission chain may want to discard some parts of the data stream. For example, entities may discard supplementary information messages that are not needed for use cases, for which intermediate entities prepare the data stream. On the other hand, the information carried by the supplementary information messages may affect the content of the decoded data, making unauthorized modifications to the supplementary information messages possible to forge the data, which cannot be identified if the supplementary information messages are excluded from the verification process.

[0008] Embodiments of the invention rely on the idea of ​​including an indication in the data stream for indicating whether one or more supplementary information messages should be considered to verify the data stream. Using this indication, the receiver knows whether to include one or more supplementary information messages in the verification process, such that the receiver can determine the part of the data stream on which the verification will be based based on the indication. Therefore, this instruction allows the encoder (which enables the data stream to perform trust checks) to choose whether one or more supplementary information messages should be included in the trust check. Thus, a decision can be made between increasing the level of verification by including supplementary information messages in the trust check, or having greater flexibility in verifying supplementary information messages at later stages (e.g., by intermediate network entities) without having to recalculate the digital signature for the verification process.

[0009] An embodiment provides an apparatus for decoding a data stream, the data stream including a plurality of payload packets carrying payload data, and further including supplementary information packets carrying supplementary information messages. The apparatus is configured to: derive an indication from the data stream indicating whether to consider one or more supplementary information messages to verify the data stream; if the indication indicates to consider one or more supplementary information messages to verify the data stream, determine the predetermined portion by including one or more supplementary information messages in the predetermined portion of the data stream; and obtain a digital signature for verifying the predetermined portion from the indication in the data stream.

[0010] An embodiment provides an apparatus for decoding a data stream. The apparatus is configured to check the credibility of a data stream, the data stream including a plurality of payload packets carrying payload data, and further including supplementary information packets carrying supplementary information messages. The apparatus is configured to: derive an indication from a data stream, the indication indicating whether supplementary information messages should be considered to verify the data stream; if the indication indicates that supplementary information messages should be considered to verify the data stream, determine the predetermined portion by including one or more supplementary information messages in the data stream to be verified; obtain a digital signature based on the data stream; and check whether the predetermined portion in the data stream matches the digital signature.

[0011] An embodiment provides an apparatus for encoding a data stream. The apparatus is configured to: enable the data stream to be trusted, and encode multiple payload groups carrying payload data into the data stream, and also encode supplementary information groups carrying supplementary information messages into the data stream. The apparatus is configured to: insert an indication into the data stream, the indication indicating whether one or more supplementary information messages should be considered to verify the data stream; if the indication indicates that one or more supplementary information messages should be considered to verify the data stream, determine the predetermined portion by including one or more supplementary information messages in the data stream to be trusted; and obtain a digital signature based on the predetermined portion.

[0012] An embodiment provides a method for decoding a data stream, the data stream including a plurality of payload packets carrying payload data, and further including supplementary information packets carrying supplementary information messages, wherein the method includes: deriving an indication from the data stream for indicating whether to consider one or more supplementary information messages to verify the data stream; if the indication indicates to consider one or more supplementary information messages to verify the data stream, determining the predetermined portion by including one or more supplementary information messages in the predetermined portion of the data stream; and obtaining a digital signature for verifying the predetermined portion from the indication in the data stream.

[0013] An embodiment provides a method for decoding a data stream, wherein the method includes checking the credibility of the data stream, the data stream including multiple payload packets carrying payload data, and further including supplementary information packets carrying supplementary information messages, wherein the method includes: deriving an indication from the data stream for indicating whether to consider the supplementary information messages to verify the data stream; if the indication indicates that the supplementary information messages should be considered to verify the data stream, determining the predetermined portion by including one or more supplementary information messages in the data stream to be verified; obtaining a digital signature based on the data stream; and checking whether the predetermined portion in the data stream matches the digital signature.

[0014] An embodiment provides a method for encoding a data stream, wherein the method includes: enabling the data stream to perform credibility checks; encoding multiple payload groups carrying payload data into the data stream, and also encoding supplementary information groups (page 2 / 55 of specification 6 CN 121923845 A) into the data stream, the supplementary information groups carrying supplementary information messages, wherein the method includes: inserting an indication into the data stream, the indication indicating whether to consider one or more supplementary information messages to verify the data stream; if the indication indicates to consider one or more supplementary information messages to verify the data stream, determining the predetermined portion by including one or more of the supplementary information messages in the data stream to enable credibility checks; obtaining a digital signature based on the predetermined portion (e.g., and inserting the digital signature into the data stream).

[0015] An embodiment provides a data stream having data signals encoded therein, the data stream being capable of reliability checking, the data stream comprising: a plurality of payload packets carrying payload data, and further comprising supplementary information packets carrying supplementary information messages; and an indication for indicating whether to consider one or more supplementary information messages to check the reliability of the data stream.

[0016] Advantageous embodiments are defined by the subject matter of the dependent claims.

[0017] The embodiments of the present disclosure are described in more detail below with reference to the accompanying drawings, in which:

[0018] FIG1 illustrates an apparatus for decoding a data stream according to an embodiment;

[0019] FIG2 illustrates a verification module according to an embodiment;

[0020] FIG3 illustrates an apparatus for encoding a data stream according to an embodiment;

[0021] FIG4 illustrates an apparatus for decoding a data stream according to an embodiment;

[0022] FIG5 illustrates an apparatus for decoding a data stream with a type-wise indication having supplementary information messages to be verified according to an embodiment;

[0023] FIG6 illustrates an apparatus for decoding a data stream with a separate indication having supplementary information messages to be verified according to an embodiment;

[0024] FIG7 illustrates substream-level verification of segments of a data stream according to an embodiment;

[0025] FIG8 illustrates a video encoder according to an embodiment;

[0026] FIG9 illustrates a video decoder according to an embodiment;

[0027] FIG10 illustrates the subdivision of an image according to an embodiment. Detailed Description of Embodiments

[0028] Embodiments of the invention will now be described in more detail with reference to the accompanying drawings, wherein the same or similar elements or elements having the same or similar functions are assigned the same reference numerals or identified by the same names. In the following description, several details are set forth to provide a thorough description of embodiments of the present disclosure. However, it will be apparent to those skilled in the art that other embodiments may be implemented without these specific details. Furthermore, unless specifically indicated otherwise, features of the different embodiments described herein may be combined with each other.

[0029] In the following description, when referring to a data stream, embodiments of the invention include a data stream as a media data stream having media signals encoded therein, such as a video data stream having video encoded therein or an audio data stream having audio signals encoded therein. In general, embodiments of the invention can be used for any data stream transmitted in signal sequence, such as a data stream having sampled signals (e.g., sampled measurement signals) encoded therein. According to an embodiment, the data stream has waveform signals encoded therein, such as biomedical waveform signals. Although some aspects of the invention will be described below with respect to video data streams, these aspects can be applied equivalently to other types of data streams unless explicitly indicated otherwise. Specification 3 / 55 pages 7 CN 121923845 A

[0030] FIG1 illustrates an apparatus 20 for decoding data stream 14. Apparatus 20 may be referred to as decoder 20. Data stream 14 may be a media data stream, such as a video data stream or an audio data stream, or any of the above data stream types. For example, decoder 20 can decode media signals, such as video or audio signals, from data stream 14.Decoder 20 includes extractor 21. Extractor 21 obtains digital signature 43 based on data stream 14. According to an embodiment, extractor 21 derives digital signature 43 from data stream 14. In other words, digital signature 43 may be included in data stream 14. According to an alternative embodiment, data stream 14 includes an indication of a resource that includes digital signature 43, and extractor 21 obtains digital signature 43 from the resource indicated in data stream 14 using the indication of the resource.

[0031] For example, digital signature 43 is used to check the credibility of data stream 14. In other words, digital signature 43 may be used to verify data stream 14, for example, to verify the credibility of data stream 14.

[0032] Decoder 20 also includes a portion determiner 30 configured to determine a predetermined portion 13 of data stream 14 on which the credibility check of data stream 14 (also referred to below as credibility check) will be performed. Digital signature 43 may be associated with the predetermined portion. In other words, digital signature 43 may be used to verify predetermined portion 13.

[0033] For example, the verification or credibility check of data stream 14 can be performed on a partial basis (i.e., on a portion of data stream 14). Therefore, digital signature 43 can be used to check the credibility of a portion of the data stream, or in other words, to verify a portion of data stream 14.

[0034] The credibility check (i.e., checking the credibility of data stream 14 or a predetermined portion 13) includes checking whether a predetermined portion of data stream 14 matches or is consistent with digital signature 43.

[0035] According to an embodiment, decoder 20 includes verification module 41 configured to perform a credibility check. As shown in FIG1, verification module 41 receives digital signature 43 and predetermined portion 13 to perform a credibility check. As shown by the dashed lines in FIG1, verification module 41 is an optional feature of apparatus 20.

[0036] In other words, according to an embodiment, decoder 20 is configured to check the credibility of a predetermined portion 13 of data stream 14. In other words, according to an embodiment, decoder 20 is configured to check whether a predetermined portion 13 of data stream 14 matches or is consistent with digital signature.

[0037] As already mentioned, the verification module 41 is optional in the device 20. According to an alternative embodiment, the trustworthiness check may be performed externally relative to the decoder 20. In other words, the verification module 41 is not necessarily part of the decoder 20, but may be part of a separate entity, such as a device for checking the trustworthiness of the data stream 14. For example, in this case, the decoder 20 may provide or forward the digital signature 43 and the predetermined portion 13 for trustworthiness checking. For example, the decoder 20 may collect information for trustworthiness checking, such as the digital signature 43 and the predetermined portion 13, and provide the collected information for trustworthiness checking. For example, the decoder 20 may form a cascade of information for trustworthiness checking.These alternative embodiments (where verification module 41 is not part of device 20) can be embodied in combination with all embodiments of all aspects of the invention.

[0038] Further optional features of device 20 of FIG1 are described below, which can be applied to all embodiments described herein.

[0039] For example, data stream 14 may include multiple packets. Partition determiner 30 may determine predetermined portion 13 by packet-wise (e.g., by including one or more of the multiple packets).

[0040] For example, alternatively, data stream 14 may include multiple payload packets 16. Payload packets 16 may carry payload data, such as data to be transmitted by data stream 14. For example, in the case where data stream 14 is a media data stream, payload packets 16 may carry encoded media data, such as video data in the case where data stream 14 is a video data stream. In other words, payload packets 16 may be encoded video payload packets carrying video data (e.g., encoded video data), such as Video Coding Layer (VCL), Network Abstraction Layer (NAL) units. For example, video data may refer to information from which image sample values ​​of the video are reconstructed.

[0041] Data stream 16 may also optionally include supplementary information packets 18, which may alternatively be referred to as supplementary information payload packets, compared to encoded data payload packets carrying encoded data. For example, in the case of video data streams (specifically, in cases where H.264, H.265, and H.266 can format video data streams), supplementary information packets may be supplementary enhancement information (SEI) NAL units. For example, supplementary information packets 18 are distributed among payload packets 16. Supplementary information packets 18 carry supplementary information messages 19. For example, supplementary information messages 19 include information that is helpful to processes related to decoding, display, or other purposes, but which is not required by the decoding process to determine the values ​​of samples in the decoded images of the video.

[0042] In other words, for example, supplementary information packets may carry information about encoding options and information for the decoding process, but do not include encoded samples of signals encoded into the data stream.

[0043] For example, each of the supplementary information messages may be associated with one of the payload packets.

[0044] For example, each of the supplementary information messages is associated with a payload packet in the payload packets, for example, the associated payload packet is part of the same sample of the sampled signal encoded into the data stream, for example, part of the same picture unit or access unit as the supplementary information packet.

[0045] As described below with respect to embodiments of the first, fourth, and fifth aspects, the predetermined portion 13 may optionally include one or more supplementary information messages, which may be included in one or more supplementary information packets. In other words, the predetermined portion 13 may optionally include one or more supplementary information packets.

[0046] For example, the predetermined portion 13 includes all or a subset of the payload packets 16 of the segments of the data stream 14. For example, the predetermined portion 13 may include portions of the payload packets 16, such as a portion of each payload packet to be included in the predetermined portion, alternatively, the payload packets may be included as holes in them. The segments of the data stream 14 from which the predetermined portion 13 can be determined may, for example, be independent encoded sequences of the data stream 14, such as a coded video sequence (CVS) in the case of a video data stream. In other examples, the segment (from which the predetermined portion 13 is determined) may be indicated in the data stream 14, for example, by one or more indications (which associate packets of the data stream with the predetermined portion 13) and / or by indications such as supplementary information messages from which the predetermined portion 13 is selected (which indicate the start and end of the segment). For example, the start and end can be indicated by corresponding supplementary information messages.

[0047] According to an embodiment, for a predetermined portion 13, or for a segment of data stream 14 that includes the predetermined portion (e.g., segment 17 described with respect to FIG. 7), data stream 14 may include a set of verification parameters that include verification parameters for verifying the predetermined portion. For example, the set of verification parameters may indicate one or more of the following: a hash function (e.g., hash function 31 described with respect to FIG. 2) for performing a credibility check on the predetermined portion 13, a certificate for decrypting the digital signature 43 used to perform the credibility check on the predetermined portion 13, an identifier for associating the predetermined portion with a media asset, and an indication of how to derive the predetermined portion 13. The set of verification parameters may be signaled in a supplementary information message, which may correspond, for example, to the following DSC ISCI message. For example, indication 51 may be included in the set of verification parameters.

[0048] FIG. 2 illustrates an embodiment of a verification module 41, which may optionally be part of the decoder 20. Alternatively, the credibility check described with respect to the verification module 41 may be performed outside the decoder 20. According to the embodiment of FIG2, the credibility check, or checking whether a predetermined portion 13 of the data stream 14 matches the digital signature 43, includes applying a hash function 31 to the predetermined portion 13 to obtain a hash value 33. According to this embodiment, the verification module 41 further includes a verification block 39, which checks whether the hash value 33 matches the digital signature 43.

[0049] Hereinafter, further optional features of the verification module 41 are described.

[0050] According to an embodiment, verification block 39 includes a verification string generator 48 configured to form a verification string based on a hash value 33, such as IdString in the sample syntax described below. For example, verification string generator 48 may form a concatenation, as described on page 5 / 55 of the specification, CN 121923845 A. This concatenation includes hash value 33 and optional additional information, such as one or more of the following: a media asset identifier, identifying the media asset to which the media of the predetermined portion 13 belongs; an identifier of the algorithm of hash function 31; and another hash value, for example, obtained from a previous portion of data stream 14. According to this embodiment, verification block 39 includes a decryption block 59 that decrypts digital signature 43 to obtain check value 47, and verification block 39 checks whether verification string 48 matches check value 47.

[0051] For example, decryption module 59 may decrypt digital signature 43 using public key 45 of an asymmetric encryption scheme. For example, extractor 21 can derive public key 45 based on an indication (e.g., a resource identifier) ​​in data stream 14, which indicates a resource from which public key 45 can be derived, and derive public key 45 from the resource indicated in data stream 14.

[0052] In other words, according to an embodiment, verification block 39 checks whether hash value 33 matches digital signature 43 by forming a verification string based on hash value and optionally based on additional information and comparing the verification string with digital signature 43 using the public key (where comparing the verification string with digital signature may include decryption performed by decryption block 59).

[0053] According to an alternative embodiment, all or part of the additional information described above for deriving verification string 48 may be concatenated with predetermined portion 13 and processed by hash function 31 to obtain hash value 31. In other words, according to these alternative embodiments, all or part of the additional information (e.g., the hash function identifier of the previous segment or portion and / or an additional hash value or digital signature) is used to derive the verification string, and is thus reflected in the verification string by hashing the information together with the predetermined portion 13, thereby deriving the hash value 33 based on the information.

[0054] For example, the generation of the digital signature 43 can be performed on the encoder side by forming the verification string and signing it with the private key of an asymmetric encryption scheme.

[0055] For example, the signature may include another hashing process, i.e., hashing the verification string with another hash function to obtain another hash value and signing that other hash value. In this example, the decoder side may not be able to reconstruct the verification string from the digital signature 43, but can only check whether the check value formed using the hash value 33 matches the digital signature, for example by forming the verification string and hashing the verification string with another hash function to derive the check value.In other words, in this case, the verification of verification block 39 may include: hashing the verification string using another hash function to obtain another hash value, and checking whether the other hash value matches the digital signature, for example, by decrypting the digital signature using a public key and checking whether the obtained check value is equal to the other hash value.

[0056] In other words, according to an embodiment, checking whether hash value 33 matches or matches check value 47 may include: using hash value 33 to form a verification string (e.g., by concatenating hash value 33 with another piece of information (e.g., another hash value or hash function identifier as described below), and hashing the verification string, for example, using another hash function. Then, verification block 39 may check whether the hashed verification string is equal to the check value 47 decrypted from the digital signature. On the encoder side, according to this embodiment, a digital signature may be generated by: forming a verification string as on the decoder side, hashing the verification string using another hash function, and signing the hashed verification string to obtain digital signature 43.

[0057] According to an alternative embodiment, the check value 47 may correspond to a verification string, such as hash value 33 or a concatenation of hash value 33 with another piece of information (e.g., another hash value or hash function identifier). In other words, in this case, the decryption of the digital signature may produce hash value 33 as part of (or the entire check value 47). In this case, the digital signature may be larger due to the omission of another hashing process.

[0058] For example, whether to use one or another of the above alternatives may depend on the hash function selected.

[0059] FIG3 illustrates an apparatus 10 for encoding data stream 14. Apparatus 10 may be referred to as encoder 10. Apparatus 10 is configured to enable the data stream 14 to be trusted. Encoder 10 provides data stream 14 by inserting (e.g., encoding) multiple payload packets 60 carrying payload data and supplementary information packets 18 carrying supplementary information messages 90 into data stream 14. In other words, encoder 10 can provide data stream 14 as described in FIG. 1. For example, encoder 10 includes inserter 23, which inserts payload groups and supplementary information groups into data stream 14.

[0060] Encoder 10 includes verification module 41', which obtains digital signature 43 based on predetermined portions of data stream 14. Encoder 10 also includes portion determiner 30', which determines predetermined portion 13. For example, portion determiner 30' determines predetermined portion 13 based on data 14' to be inserted into data stream 14 by including a portion of data 14' (which corresponds to predetermined portion 13 determined by portion determiner 30) into predetermined portion 13.

[0061] Any description of device 20 in the sense that information derived by device 20 from data stream 14 can be inserted into data stream 14 by device 10 can optionally be applied to device 10. Furthermore, for example, any hash function used by verification module 41 (e.g., hash function 31) can be equivalent to the corresponding hash function used by verification module 41'. This also applies to the input of the corresponding hash function (e.g., hash function 31 used to derive hash value 33).

[0062] The interaction between the verification module of decoder 20 and the verification module 41' of encoder 10 has been briefly described above. Verification module 41' can be used to verify a string that includes a hash value obtained by applying a hash function to a predetermined portion 13. Verification module 41' can also sign the verification string, for example, using the private key of the aforementioned asymmetric encryption scheme, to generate a digital signature 43.

[0063] For example, inserter 23 may insert digital signature 43 into data stream 14. Alternatively, inserter 23 may insert an indication of a resource from which digital signature 43 can be derived into data stream 14.

[0064] Any description of device 20 may optionally be equivalently applied to device 10 in the sense that information derived by device 20 from data stream 14 can be inserted into data stream 14 by device 10. Furthermore, any hash function used by device 10 (e.g., hash function 31) may be equivalent to the corresponding hash function used by device 20. This also applies to the input of the corresponding hash function (e.g., hash function 31 used to derive hash value 33). The generation of digital signature 43 and the verification performed by device 20 using digital signature 43 may each be part of an asymmetric encryption scheme, and these steps may each be performed by a pair of private and public keys, wherein at least the private key is used for signing to generate digital signature 43, and wherein the public key is used for decryption in order to verify the verification string formed on the receiver side against digital signature 43.

[0065] Embodiments of the present invention are described below.

[0066] FIG4 illustrates an apparatus 20 for decoding a data stream 14 according to an embodiment of the present invention. The apparatus 20 of FIG4 may optionally correspond to the decoder 20 of FIG1, that is, the decoder 20 of FIG4 may be based on any embodiment described with respect to FIG1. ​​Furthermore, the following embodiments may optionally be combined with any embodiment described with respect to FIG1 to FIG3.

[0067] According to an embodiment of the present invention, the extractor 21 is configured to derive an indication 51 from the data stream 14, the indication 51 indicating whether to consider, use, or include one or more supplementary information messages to verify the data stream 14, for example, to check the credibility of the data stream.According to an embodiment, if instruction 51 indicates that the one or more supplementary information messages should be considered to verify data flow 14, then the portion determiner 30 determines the predetermined portion 13 by including the one or more supplementary information messages in the predetermined portion 13. The one or more supplementary information messages referenced by instruction 51 (i.e., instruction 51 for indicating whether to include supplementary information messages in the predetermined portion 13) may be referred to as a set of verification supplementary information messages.

[0068] Note that the final decision regarding whether to include supplementary information messages or groups in the predetermined portion or exclude them from the predetermined portion may depend on one or more additional criteria, such as those described with respect to the fourth aspect. In other words, instruction 51 may be one of a plurality of criteria for whether to include supplementary information messages in the predetermined portion 13.

[0069] In other words, an embodiment of the present invention is an apparatus 20 for decoding a data stream 14 (e.g., an apparatus for decoding a media signal from a media data stream), the data stream including a plurality of payload packets 16 (e.g., VCL NAL unit specification 7 / 55 page 11 CN 121923845 A), the payload packets carrying payload data (e.g., media data, e.g., video data) (e.g., encoded video data; e.g., information on sample values ​​of images from which the video is reconstructed), and also including supplementary information packets 18 (e.g., supplementary information payload packets) (e.g., SEI NAL units) (e.g., scattered among the payload packets), the supplementary information packets carrying supplementary information messages 19 (e.g., including information that helps in processes related to decoding, display, or other purposes but which the decoding process does not need to use to determine sample values ​​in the decoded images), wherein the apparatus is configured to: derive 21 instructions 51 (e.g., syntax elements, e.g., dsci_sei_messages_digitally_signed_) from the data stream. The instruction 51 indicates whether to consider (or use or include) one or more supplementary information messages to verify the 40 data stream (e.g., verify the credibility of the data stream; in other words, check the credibility of the data stream) (or, for example, whether to authenticate the predetermined portion 13 without considering any supplementary information); if the instruction 51 indicates to consider the one or more supplementary information messages to verify the data stream, the predetermined portion 13 in the 30 data stream is determined by including the one or more supplementary information messages in the predetermined portion (e.g., to verify or check the credibility of the predetermined portion); a digital signature 43 for verifying the predetermined portion 13 is obtained from the instruction in the data stream (e.g., using information derived from the data stream, such as deriving a digital signature from the data stream or deriving a digital signature from a resource indicated in the data stream) (e.g., the digital signature is derived based on the predetermined portion).

[0070] According to an embodiment, the apparatus is configured to verify a predetermined portion by checking whether a predetermined portion in a data stream matches a digital signature (e.g., to check the credibility of the data stream or a predetermined portion thereof).

[0071] According to an embodiment, the apparatus is configured to provide (or forward) the predetermined portion and the digital signature to verify the predetermined portion (e.g., performed by another apparatus or entity) (e.g., verification includes checking whether a predetermined portion in a data stream matches a digital signature).

[0072] In other words, the embodiment of the invention described with respect to FIG4 is an apparatus 20 for decoding a data stream 14 (e.g., an apparatus for decoding a media signal from a media data stream), wherein the apparatus is configured to check the credibility of the data stream (e.g., an apparatus for verifying the data stream), the data stream including a plurality of payload packets 16 (e.g., carrying encoded payload data) (e.g., VCL NAL units), the payload packets 16 carrying payload data (e.g., encoded video data; for example, the data is information for reconstructing sample values ​​of an image from it), and also including supplementary information packets 18 (e.g., supplementary information payload packets) (e.g., SEI NAL units) (e.g., scattered among the payload packets), the supplementary information packets carrying supplementary information messages 19 (e.g., including information that helps in processes related to decoding, display, or other purposes but which the decoding process does not need to use to determine sample values ​​in the decoded image), wherein the apparatus is configured to: derive 21 instructions 51 (e.g., syntax elements, such as dsci_sei_messages_digitally_signed_) from the data stream. The instruction 51 indicates whether to consider (or use or include) supplementary information messages (e.g., one or more supplementary information messages) to verify the data stream (or check the credibility of the data stream) (or whether to authenticate a predetermined portion without considering any supplementary information); if the instruction 51 indicates that supplementary information messages should be considered to verify the data stream, then the predetermined portion 13 in the data stream to be verified (or checked for credibility) is determined by including one or more supplementary information messages 19 in the predetermined portion 13; a digital signature 43 is obtained based on the data stream 14 (e.g., using information derived from the data stream, such as deriving a digital signature from the data stream or from a resource indicated in the data stream); and the predetermined portion 13 of the data stream is checked to see if it matches (or is consistent with) the digital signature 43.

[0073] Further optional features of the embodiments will be described below.

[0074] According to an embodiment, the verification set of supplementary information messages includes a subset of all supplementary information messages of the data stream 14. In other words, instruction 51 can indicate whether supplementary information messages belonging to the verification set should be included in or excluded from the predetermined section 13.

[0075] In other words, according to an embodiment, instruction 51 indicates whether supplementary information messages belonging to the supplementary information message verification set should be included in the predetermined section 13 or excluded from the predetermined section 13. For example, alternatively, the data stream may include additional supplementary information messages that do not belong to the verification set, which should always be included in the verification, i.e., in the predetermined section.

[0076] According to an embodiment, instruction 51 indicates whether the one or more supplementary information messages should be considered or included in the predetermined section 13, or whether any supplementary information messages should not be considered when verifying the data stream, for example, not included in the predetermined section 13. In other words, instruction 51 can distinguish between considering including the verification set in the predetermined section and not including any supplementary information messages (e.g., any segment referenced by digital signature 43) in the predetermined section. In other words, indication 51 indicates whether supplementary information messages (i.e., supplementary information messages of the verification set) should be included in or excluded from the predetermined section 13, i.e., whether supplementary information messages should be considered when verifying the data stream, or whether any supplementary information messages should not be considered (e.g., no supplementary information messages should be considered at all) when verifying the data stream or the predetermined section.

[0077] The verification set may include all supplementary information messages or a portion thereof of the segment referenced by the digital signature 43 of the data stream. In the latter case, the verification set may be predefined, for example, by message type, or may be signaled in the data stream 14, for example, by indicating message types to be included or excluded, or by individually identifying these messages or groups of corresponding messages. Further optional details will be described in more detail below.

[0078] The verification set of supplementary information messages may be defined by supplementary information message type, or by individually identifying supplementary information messages to be included in the verification set of supplementary information messages, or by grouping, or by any combination of these criteria. Depending on the group selection, complete supplementary information groups can be included in or excluded from a predetermined portion, for example, by individually identifying groups to be included in or excluded from a predetermined portion, or by group type. In other words, in the group selection, all supplementary information messages included in the selected supplementary information group are included in the verification set of supplementary information messages.

[0079] According to an embodiment, if instruction 51 indicates that one or more supplementary information messages should be considered to verify predetermined portion 13, portion determiner 30 determines predetermined portion 13 by including the one or more supplementary information groups in predetermined portion 13.

[0080] In other words, including supplementary information messages in the predetermined portion 13 can be done in groups, that is, the supplementary information messages to be considered to change the predetermined portion 13 can be identified or indicated by identifying or indicating one or more supplementary information messages in the supplementary information groups of the data stream 14 to be included in the predetermined portion 13.

[0081] For example, according to an embodiment, the portion determiner 30 can include all supplementary information groups 18 carrying any supplementary information messages to be included in the predetermined portion 13 (e.g., any one of the one or more supplementary information messages of the verification set) in the predetermined portion 13.

[0082] According to an embodiment, the portion determiner 30 decides whether to include supplementary information groups in the supplementary information groups in the predetermined portion 13, the decision being based on the result of checking whether the supplementary information group includes supplementary information messages of any supplementary message type in a set of supplementary message types.

[0083] In other words, the identification of supplementary information messages belonging to the verification set can be performed based on the supplementary information message type by including certain types of supplementary information messages. Additionally, the inclusion or exclusion of supplementary information messages can be performed by including each supplementary information group, each supplementary information group including at least one supplementary information message of one type of supplementary information message in that group of supplementary information message types.

[0084] According to an embodiment, indication 51 is a flag having a first state and a second state. According to this embodiment, if the flag has a first state, the portion determiner determines a predetermined portion 13 of the data stream 14 by determining one or more supplementary information messages to be included in the predetermined portion 13 based on a predefined set of supplementary information messages, and if the flag has a second state, any supplementary information message in the predefined set of supplementary information messages is avoided from being included in the predetermined portion. For example, if the flag has a first state, the portion determiner 30 may include all supplementary information messages in the predefined set of supplementary information messages, or may include at least one instance of all supplementary information messages in the predefined set of supplementary information messages, as will be described in more detail below.

[0085] For example, indication 51 may be a flag having only a first state and a second state. In other words, indication 51 may be binary.

[0086] According to an embodiment, the set of predefined supplementary information messages includes all supplementary information messages in the supplementary information group that reference multiple payload groups.

[0087] According to another embodiment, the set of predefined supplementary information messages includes all supplementary information messages of any type in a set of supplementary information message types. For example, the set of supplementary information message types is a subset of multiple supplementary information message types (e.g., multiple possible supplementary information message types), which can be defined according to the type of data stream 14.For example, the set of supplementary information message types can be predefined. That is, for example, the set of supplementary information message types may be known to decoder 20, for example, and need not be sent in data stream 14.

[0088] According to an embodiment, the identifier of the one or more supplementary information messages to be included in a predetermined portion is signaled in data stream 14. If indication 51 indicates that the one or more supplementary information messages should be considered to verify the data stream.

[0089] Hereinafter, embodiments of how to identify the verification set of supplementary messages in data stream 14 will be described with reference to FIG5 and FIG6.

[0090] FIG5 shows an embodiment of decoder 20 according to the invention. For example, decoder 20 of FIG5 may be an embodiment of decoder 20 of FIG1, that is, any previously described details may optionally be applied to decoder 20 of FIG5. According to FIG5, data stream 14 includes syntax element 53, which identifies a set of supplementary information message types to be included in predetermined portion 13. In other words, according to this embodiment, the verification set can be identified based on the supplementary information message types. For example, in the illustrative example of FIG5, supplementary information group 18' includes supplementary information message 19' of the type indicated by syntax element 53. According to this embodiment, extractor 21 derives syntax element 53 from data stream 14. Part determiner 30 can use the information of syntax element 53 to determine predetermined part 13 because supplementary information messages of the type included in the group of supplementary information message types indicated by syntax element 53 are included in predetermined part 13, or because supplementary information groups including supplementary information messages of the type included in the group of supplementary information message types indicated by syntax element 53 are included in predetermined part 13, as shown in FIG5.

[0091] Note that in embodiments where supplementary information group includes only one supplementary information message, the type of supplementary information message can be considered as the type of supplementary information group. In other words, there is not necessarily a difference between supplementary information message type and supplementary information group type, but the type of supplementary information group can be considered equivalent to the type of supplementary information message carried in the corresponding supplementary information group.

[0092] According to an embodiment, syntax element 53 is signaled in a supplementary information message (e.g., in the supplementary information message, which will be referred to below as a verification parameter message). For example, the verification parameter message carries the aforementioned set of verification parameters. For example, the supplementary information message carrying syntax element 53 may be included in a supplementary information group.

[0093] According to an embodiment, instruction 51 and syntax element 53 are included in the same supplementary information message (e.g., verification parameter message).

[0094] According to an embodiment, instruction 51 is signaled by syntax element 53 (e.g., by a predetermined state of syntax element 53).

[0095] For example, according to an embodiment, syntax element 53 has a second state and a plurality of first states.According to each of the first states, one or more supplementary information messages are to be included in the predetermined portion 13, wherein the first state distinguishes different verification sets. In other words, the first state identifies supplementary information messages to be included in the predetermined portion. In other words, each of the first states indicates a not necessarily appropriate subset of the one or more supplementary information messages to be included in the predetermined portion 13. According to the second state, none of the one or more supplementary information messages are included in the predetermined portion 13.

[0096] Therefore, according to the embodiment, if the syntax element 53 has one of the first states, the portion determiner 13 determines the one or more supplementary information messages to be included in the predetermined portion 13 based on the state of the syntax element 53. If the syntax element has the second state, the portion determiner 30 avoids including any of the one or more supplementary information messages in the predetermined portion 13.

[0097] According to an embodiment, if instruction 51 indicates that one or more supplementary information messages should be included in the predetermined section 13, extractor 21 derives a first syntax element (e.g., dsci_num_sei_message_types_digitally_signed_minus1 hereinafter) from data stream 14. This first syntax element indicates a count of the supplementary information message types to be considered for the predetermined section 13, and extractor 21 also derives a corresponding number of second syntax elements from data stream 14, each second syntax element identifying a corresponding supplementary information message type to be included in the predetermined section 13. For example, the corresponding number of second syntax elements ([e.g., dsci_digitally_signed_sei_type[i] hereinafter) corresponds to the count indicated by the first syntax element. In other words, for example, the validation set can be indicated according to the supplementary information message type, the size of the validation set is variable, and it is signaled in data stream 14. Therefore, the number of supplementary information messages to be included in the predetermined section can be adaptively set by the encoder.

[0098] For example, supplementary information message types that may be discarded during data stream processing (e.g., removed by a file parser or an entity extracting substreams from the data stream) may be excluded from trust checks to avoid the need to recalculate digital signatures. On the other hand, supplementary information messages may include information that affects the output of the decoded data stream, thus verification may be desirable. Allowing the size of the verification set to be variable and including or excluding individual types from the verification set allows the decoder to adjust trade-offs between these aspects individually for the data stream. At the same time, the indication of supplementary information message types provides a group identifier for supplementary information messages to be included in a predetermined section, thereby avoiding the need for a supplementary information message identifier for the verification set.

[0099] According to an embodiment, the first syntax element and the second syntax element may be signaled in the above-described verification parameter message, for example, together with the indication 51.

[0100] FIG6 illustrates another embodiment of the decoder 20. The decoder 20 of FIG6 may optionally be an example of the decoder 20 of FIG1 or FIG4. According to the embodiment of FIG6, the part determiner 30 determines whether to include the supplementary information payload group in the supplementary information payload group 18 in the predetermined portion 13 based on the result of checking whether the supplementary information payload group includes a supplementary information message (e.g., as a prefix of another supplementary information message (e.g., another supplementary information message within the same group)). The supplementary information message indicates that the supplementary information payload group is included in the predetermined portion 13. For example, in FIG6, each of the payload groups 18' includes a supplementary information message 55, which indicates that the corresponding supplementary information group 18' is included in the predetermined portion 13, while the supplementary information group 18'' does not include such an indication.

[0101] Note that, optionally, the part determiner 30 may determine whether to include the supplementary information group 18' in the predetermined portion 13 based on one or more additional conditions. In other words, according to the embodiment, supplementary information group 18' (indication 55 indicates that the group should be included in the predetermined portion 13) is not necessarily included in the predetermined portion 13, but the decision may depend on other conditions. For example, in the case of repeated supplementary information messages, only one instance of each message may be included in the predetermined portion.

[0102] Alternatively, according to the embodiment, the portion determiner 30 includes supplementary information group 18', which includes supplementary information message 55, which indicates that the supplementary information group should be included in the predetermined portion.

[0103] Another condition for the portion determiner 30 to decide whether to include or exclude supplementary information groups in the predetermined portion 13 may be whether the corresponding supplementary information group belongs to a set of predetermined supplementary information groups that are always or never included in the predetermined portion 13.

[0104] Therefore, according to an embodiment, if a supplementary information group does not include a supplementary information message 55 indicating that the supplementary information group should be included in a predetermined portion, then if the corresponding supplementary information group belongs to that group of predetermined supplementary information messages, the portion determiner 30 excludes it, and if the corresponding supplementary information group does not belong to that group of predetermined supplementary information groups, the portion determiner 30 excludes it. Note that this decision may optionally depend on other conditions such as the already mentioned dependency on repetition.

[0105] For example, supplementary information message 55 may be a prefix supplementary information message preceding all supplementary information messages within a supplementary information group, or it may be another supplementary information message.

[0106] Continuing below, the description of decoder 20 of FIG4 continues, describing alternative schemes indicating the verification set.

[0107] According to an embodiment, the decision regarding whether to include a supplementary information packet in the predetermined portion 13 depends on checking whether there is a prefix supplementary information packet preceding the supplementary information packet, which indicates that subsequent supplementary information packets (e.g., immediately following supplementary information packets) should be included in the predetermined portion 13. In other words, instead of supplementary information message 55, a prefix supplementary information packet can be used for this indication, compared to the embodiment described with respect to FIG6. All additional details described with respect to FIG6 may also optionally apply to this embodiment.

[0108] According to another embodiment, the decision regarding whether to include a supplementary information message in the predetermined portion 13 depends on checking whether there is a prefix supplementary information message preceding the supplementary information message, which indicates that subsequent supplementary information messages (e.g., immediately following supplementary information messages) should be included in the predetermined portion 13. In other words, according to this embodiment, inclusion or exclusion can be done on a message-by-message basis, rather than, for example, on a packet-by-packet basis. The details described with respect to FIG6 may also optionally apply to this embodiment.

[0109] According to another embodiment, the decision regarding whether to include a supplementary information group in a predetermined portion 13 depends on the result of checking whether the supplementary information group includes a nested supplementary information message that indicates that the supplementary information group should be included in the predetermined portion.

[0110] For example, a nested supplementary information message may include a set of one or more supplementary information messages. Based on the fact that the set of supplementary information messages is signaled within a nested supplementary information message, the portion determiner 30 can conclude that a supplementary information group including a nested supplementary information message will be considered to be included in the predetermined portion 13. In other words, according to this embodiment, inclusion in or exclusion from a predetermined portion can be done on a group basis, and an indication of whether to include a group can be signaled by a nested supplementary information message.

[0111] According to another embodiment, a nested supplementary information message is used to signal whether a supplementary information message is included in or excluded from a predetermined portion on a message basis. According to this embodiment, the decision regarding whether to include a supplementary information message depends on the result of checking whether a supplementary information message is included in a nested supplementary information message that indicates that the supplementary information message should be included in the predetermined portion 13.

[0112] As mentioned above, the inclusion or exclusion of supplementary information packets may also depend on whether they include supplementary information messages of a type belonging to a set of supplementary information message types to be included in the predetermined portion. In other words, for example, the set of supplementary information message types includes supplementary information message types to be included in the predetermined portion but not explicitly signaled in data stream 14. In other words, the set of types may be predefined.The set of predefined supplementary information messages can be included in the predetermined section along with supplementary information message types that are adaptively signaled.

[0113] Supplementary information messages belonging to the set of predetermined supplementary information message types to be included in the predetermined section can be independent of instruction 51, or alternatively, can depend on instruction 51. In other words, supplementary information messages belonging to the set of predetermined types can always be included in the predetermined section. Independent of instruction 51, or alternatively, if instruction 51 indicates that supplementary information messages should be considered for use in verifying the data stream, then the set of predetermined types of supplementary information messages are considered to be included in the predetermined section 13, and optionally, additional supplementary information messages indicated in the data stream 14 are considered to be included in the predetermined section. If instruction 51 indicates that the one or more supplementary information messages are not considered for use in verifying the data stream, then according to one embodiment, the set of predetermined types of supplementary information messages are considered to be included in the predetermined section 13, without considering the inclusion of additional supplementary information messages in the predetermined section 13. According to another embodiment, if instruction 51 indicates that the one or more supplementary information messages are not considered for use in verifying the data stream, then neither are supplementary information messages of the predetermined type considered for inclusion in the predetermined portion 13, nor are any additional supplementary information messages considered for inclusion in the predetermined portion 13.

[0114] Supplementary information messages can be included and excluded by using the group of predetermined types for identification.

[0115] According to an embodiment, if a supplementary information group includes any supplementary information message of the predetermined type, then the supplementary information group is considered to be included in the predetermined portion. According to another embodiment, if a first supplementary information message belongs to the group of predetermined supplementary information message types, then the supplementary information group is considered to be included in the predetermined portion.

[0116] For example, according to an embodiment, the portion determiner 30 determines whether to include a supplementary information group in the predetermined portion based on the result of checking whether the supplementary information group includes any supplementary information message of one type of a group of supplementary information message types to be included in the predetermined portion 13.

[0117] For example, if instruction 51 indicates that supplementary information messages should be considered to verify the data stream, the part determiner 30 may check for supplementary information packets in supplementary information packets (e.g., for each of the supplementary information packets) whether the supplementary information packet includes any supplementary information message of the type to be included in the predetermined part, and if the supplementary information packet includes any supplementary information message of the type to be included in the predetermined part, then the supplementary information packet is included in the predetermined part.

[0118] According to an embodiment, the part determiner 30 determines whether to include a supplementary information group in a predetermined part based on the result of checking whether a first supplementary information message in one or more supplementary information messages carried in a supplementary information group is one of a set of supplementary information message types to be included in a predetermined part.

[0119] For example, if the instruction indicates that supplementary information should be considered to verify a predetermined part, the first supplementary information message in one or more supplementary information messages carried in a supplementary information group (e.g., for each of the supplementary information groups) is checked to see if it is one of a set of supplementary information message types to be included in a predetermined part, and if the first supplementary information message in one or more supplementary information messages carried in a supplementary information group is one of a set of supplementary information message types to be included in a predetermined part, then the supplementary information group is included in the predetermined part.

[0120] Hereinafter, another alternative for identifying the verification set is described, which utilizes a part-based verification system implemented in a trust check. In other words, the verification substream described below with respect to FIG. 7 can be used to identify supplementary information messages to be included in a trust check.

[0121] FIG. 7 illustrates another embodiment of the decoder 20. Among the features described with respect to FIG7, they may optionally be combined with any previously described embodiments of decoder 20. According to the embodiment of FIG7, credibility checks are performed on a per-part basis, 13', 13''. In other words, according to this embodiment, data stream 14 comprises multiple parts, and data stream 14 may be verified on a per-part basis. For example, data stream 14 includes a corresponding digital signature for each of these parts. Optionally, data stream 14 may include the aforementioned set of verification parameters for each of these parts. Alternatively, verification parameters for all parts of a segment or a subset of these parts may be included in a common set of verification parameters. The aforementioned predetermined part 13 may be one of these multiple parts. Each of these parts may be defined by assigning one or more packets or data included in the packets to it. Specification 13 / 55 pages 17 CN 121923845 A

[0122] For example, in FIG7, payload packet 16' is assigned to part 13', and payload packet 16'' is assigned to part 13''. Performing verification on a segment-by-segment basis for data stream 14 may include identifying individual segments, such as segments 13' and 13'' in Figure 7, and performing a credibility check 41 on each segment. For this purpose, data stream 14 may indicate corresponding digital signatures for these segments, such as digital signature 43' for segment 13' and digital signature 43'' for segment 13'' in Figure 7. For example, the segments described in Figure 7 may be referred to as substreams or verification substreams of data stream 14.

[0123] For example, in the case of a video data stream, different layers of the layered video data stream can be assigned to different sub-streams. For example, different layers can carry different representations of the video encoded into the video data stream 14 (e.g., having different spatial resolutions, carrying different types of data (e.g., texture and depth)), or can carry different views of the scene. As another example, different sub-streams can be associated with different temporal layers of the media data stream, each temporal layer carrying samples for forming representations of the encoded media signal at different temporal resolutions, for example, so that a higher temporal resolution is obtained when different temporal layers are combined. However, it should be noted that these examples are merely examples of organizing the data of the media data stream into different sub-streams, and the grouping and association between sub-streams can depend on the encoder or entity that enables the data stream 14 to perform credibility checks.

[0124] In other words, for each of the plurality of parts (in which the data stream can be verified), the data stream 14 can include a corresponding digital signature, or alternatively, an indication of the corresponding digital signature.

[0125] According to an embodiment, in addition to verifying the data stream in units of portions in the sense of sub-streams, credibility checks can also be performed in units of segments 17 in the time series of segments of data stream 14. A segment can be referred to as a verification period. In other words, within a segment of data stream 14, multiple sub-streams can be defined, such as the aforementioned portions 13', 13'', and the segment can be verified in units of these multiple sub-streams. Packets or data belonging to each sub-stream do not necessarily form a continuous portion of data stream 14, but within a segment, packets or data fragments can be individually assigned to one of the sub-streams. Hereinafter, only one segment 17 is considered, that is, for example, when referring to a portion, it can refer to the sub-streams within the aforementioned segment 17.

[0126] According to an embodiment, for example, within a segment, data stream 14 includes an indication of the count of sub-streams of the data stream; for example, only one segment of the data stream is considered a data stream.

[0127] According to an embodiment, the extractor 21 extracts the number of portions from the data stream 14, for example, by indicating the number of portions extracted from the data stream 14, such as the count of substreams of the data stream. According to this embodiment, the portion determiner 30 may assign each of the payload groups 16 to one of these portions, for example, to one or more of these portions, i.e., not necessarily to each of these portions.

[0128] Hereinafter, an embodiment according to the invention will be described, which utilizes the substream concept to verify supplementary information messages.

[0129] According to an embodiment, portions of a plurality of sequentially ordered portions define, for example, a hierarchical order among them, each of these portions having a rank within that order.

[0130] According to an embodiment, verification of these portions is performed according to the lowest to highest rank among these portions.

[0131] According to an embodiment, the verification of a part depends on the lower-ranking neighbor of that part in the order of these parts. For example, a hash value derived from a lower-ranking part can be included in the verification string used to verify that part.

[0132] According to an embodiment, the credibility check of a part may optionally depend on one or more lower-ranking parts. For example, one or more hash values ​​derived from one or more lower-ranking parts (to which the credibility check of the part depends) can be included in the verification string used to verify that part.

[0133] In other words, these parts can provide part-by-part verification, wherein hash values ​​already derived for a certain part can be reused to verify higher-ranking parts.

[0134] According to an embodiment, the verification set of supplementary information messages to be considered for credibility checks is to be verified as part of a predetermined substream in a substream. For example, the supplementary information messages to be verified can be gathered in a substream with a predetermined rank within the order defined between these parts.

[0135] According to an embodiment where the verification subset is identified based on the type of supplementary information message, the portion determiner 30 may include any type of supplementary information message of the set of supplementary information message types to be included in the credibility check in a predetermined portion 13, according to this embodiment, the supplementary information message having a predetermined ranking within an order defined among the plurality of portions.

[0136] For example, the predetermined ranking is the highest ranking within an order defined among the plurality of portions.

[0137] Alternatively, supplementary information messages to be included in the credibility check may be included in a portion assigned a payload group associated with the corresponding supplementary information message. As described above, each of the supplementary information messages may be associated with one of the payload groups. Supplementary information messages to be included in the credibility check may be assigned to a portion assigned a payload group associated with that supplementary information message.

[0138] According to an embodiment, the data stream 14 includes the instruction, for example, indicating the distinction between: including all supplementary information messages of the type to be included in the credibility check in a substream of a predetermined ranking; or including all supplementary information messages of the type to be included in the credibility check in a portion associated with their respective assigned payload groups.

[0139] According to another embodiment, the plurality of portions includes a first portion, each first portion having a payload group associated therewith, and each first portion having a corresponding second portion associated therewith in the plurality of portions, wherein supplementary information messages to be considered for credibility checks and assigned to the payload group associated with the corresponding first portion are included in the corresponding second portion.In other words, the plurality of parts includes a first part and a second part, each first part being associated with one of the second parts, the first part being used to verify payload packets, and the second part being used to verify supplementary information messages assigned to payload packets respectively associated with the first part.

[0140] More generally, according to an embodiment, each of the supplementary information messages is associated with one of the payload packets, and wherein the means is configured to assign each of the supplementary information messages of the data stream having any type of a set of supplementary information message types to one of the plurality of parts, the one part being associated with another of these parts, the other of these parts being associated with the payload packet associated with the corresponding supplementary information message.

[0141] According to an embodiment, each of the supplementary information packets is associated with one of the payload packets, and wherein the means is configured to assign each of the supplementary information packets of the data stream carrying any type of a set of supplementary information information types to one of the plurality of parts, the one part being associated with another of these parts, the other of these parts being associated with the payload packet associated with the corresponding supplementary information packet.

[0142] According to an embodiment, the apparatus 20 is configured to derive a syntax element from a data stream, the syntax element indicating a count of the plurality of parts, the syntax element having a value, wherein the apparatus is configured to infer that the count of the plurality of parts corresponds to the value plus 1 and multiplied by 2.

[0143] According to an embodiment, the apparatus 20 is configured to derive a syntax element from a data stream, the syntax element indicating a count of the plurality of parts, the syntax element having a value, wherein the apparatus is configured to infer that the count of the plurality of parts corresponds to the value plus 1.

[0144] According to an embodiment, one of the plurality of parts and another part are consecutive to each other in a hierarchical order defined among the plurality of parts, for example, one of the plurality of parts is located after the other part in the hierarchical order.

[0145] According to an embodiment, the position index of the one part in the hierarchical order defined among the parts corresponds to the position index of the other part in the hierarchical order plus half the count of the plurality of parts.

[0146] Although the description of Figures 1 to 7 relates to an apparatus, the block diagrams in these figures can alternatively be regarded as flowcharts of the corresponding methods, as described on pages 15 / 55 of CN 121923845 A, where each block represents a step of the corresponding method. Therefore, Figures 1 to 7 also provide an illustration of the corresponding methods.

[0147] Hereinafter, aspects of the invention will again be described in other ways; in other words, specific embodiments and additional embodiments of the invention will be described.The embodiments described with reference to Figures 1 through 7 can be considered as a generalization of the embodiments described below. However, the following description may also include additional embodiments of the invention, which may be implemented independently of the previously described embodiments. Any features and details described with respect to the following embodiments may optionally be integrated into the previous embodiments.

[0148] Although the following description relates to video data streams, it is obvious that the same concepts can be equivalently applied to any type of media data stream, or any type of data stream that includes sampled data. Another example of sampled data may be a waveform signal. In addition to audio signals, another example of a waveform signal is a biomedical waveform signal.

[0149] Authenticating a video requires providing the client with information about how the hashing was performed (including the hashing method used and how the data was organized, as well as the content that was actually hashed and how the hashing was performed), a certificate carrying a public key that can be used for verification, and one or more signatures for the video.

[0150] In the following, existing solutions for content verification are described. Subsequent embodiments of the invention may be described as modifications to existing solutions, such that details of the existing solutions may form features of embodiments of the invention.

[0151] To this end, the existing solution is to include three SEI messages in each encoded video sequence (CVS) of the video bitstream:

[0152] 1. Digital Signature Content Initialization SEI message: It carries information about the hashing method used, the URL or C2PA list used to obtain the certificate (which contains information such as the public key and UUID of the content used when calculating the signature, so that different bitstreams that belong to the same content and are signed separately (e.g., different audio and video streams) can be identified as belonging to the same content (so audio with different content cannot be used for a specific video, avoiding the use of content spoken by someone at different times for a new video)), and how many substreams are used to generate the signature. The latter refers to splitting the content into different substreams, each with a signature, so that if less important parts of the video are discarded (e.g., when there is network congestion, for a high-resolution enhancement layer of layered coding), the received parts can still be authenticated.

[0153] 2. Digital Signature Content Selection SEI message: It identifies the substream ID to which a slice of the image (i.e., a VCL NAL unit) belongs.

[0154] 3. Digital Signature Content Verification SEI Message: It provides a corresponding signature for each substream.

[0155] In the following, the existing solution for these three SEI messages and the verification process is reproduced. Unless otherwise stated, the details of this solution may optionally be implemented in embodiments of the invention, for example, in terms of modifications to the solution.

[0156] Digital Signature Content Initialization SEI Message: Specification 16 / 55 pages 20 CN 121923845 A

[0157]

[0158] dsci_hash_method_type indicates a secure hash algorithm used to compute message digests of a subset of the SPS, PPS, APS, PH, and VCL NAL units of the encoded video sequence. Based on these message digests and the digital signature content, the decoder verifies the presence of a digital signature in the SEI message. The decoder can then verify that the encoded video was created by the content creator indicated by the syntax elements dsci_key_source_uri, dsci_use_key_register_idx_flag, and (if the dsci_use_key_register_idx_flag flag is equal to 1) dsci_key_register_idx. Supported values ​​for the syntax element dsci_hash_method_type, the block size used to compute the message digest, and the size of the computed message digest are specified in Table 1. Values ​​of dsci_hash_method_type not listed in Table 1 are reserved for future use by ITU-T|ISO / IEC and should not be present in payload data conforming to this version of the specification. The decoder should ignore trusted initialization SEI messages that include reserved values ​​of dsci_hash_method_type. The secure hashing algorithms listed in Table 1 are specified in the "Secure Hash Standard" FIPS PUB 180-4.

[0159] Table 1 - Supported Values ​​of dsci_hash_method_type

[0160] Specification 17 / 55 pages 21 CN 121923845 A

[0161] dsci_key_source_uri contains a URI with syntax and semantics as specified in IETF Internet Standard 66. If dsci_key_retrieval_mode_idc equals 0, then dsci_key_source_uri specifies a C2PA manifest store as specified in the C2PA technical specification.If dsci_key_retrieval_mode_idc equals 1, then the following applies:

[0162] - If dsci_use_key_register_idx_flag equals 0, then the URI identifies the content provider's certificate, which can be used to verify the signature present in subsequent digital signature content verification SEI messages;

[0163] - Otherwise (if dsci_use_key_register_idx_flag equals 1), the URI identifies the certificate registry and the content provider's certificate, which can be used to verify the signature present in subsequent digital signature content verification SEI messages indicated by dsci_key_register_idx.

[0164] dsci_num_verification_substreams_minus1 plus 1 indicates the number of substreams for which message digests have been calculated, and the signatures can be present in subsequent digital signature content verification SEI messages.

[0165] The variable NumVerificationSubstream is exported as:

[0166] NumVerificationSubstream = dsci_num_verification_substreams_minus1 + 1.

[0167] dsci_key_retrieval_mode_idc equal to 0 indicates that the URI contained in dsci_key_source_uri specifies a C2PA manifest store as specified in the C2PA technical specification. dsci_key_retrieval_mode_idc equal to 1 indicates that the URI contained in dsci_key_source_uri and dsci_key_register_idx (if present) specify a certificate. In this version of the specification, the value of dsci_key_retrieval_mode_idc should be in the range of 0 to 1. The decoder should also allow other values ​​of dsci_key_retrieval_mode_idc, but should ignore the contents of the digital signature content initialization SEI message, the associated digital signature content selection SEI message, and the associated digital signature content verification SEI message.

[0168] The dsci_use_key_register_idx_flag being equal to 1 indicates that the URI contained in dsci_key_source_uri specifies the registry of the certificate, and that the syntax element dsci_key_register_idx exists in the SEI message.The dsci_use_key_register_idx_flag being equal to 0 indicates that the URI contained in dsci_key_source_uri specifies the certificate, and the syntax element dsci_key_register_idx is not present in the SEI message.

[0169] When dsci_key_retrieval_mode_idc is equal to 0, the media asset for which the Active Manifest, as specified in the C2PA technical specification, provides content binding is the digitally signed content initialization SEI message. The following constraints apply to the C2PA manifest store identified by dsci_key_source_uri:

[0170] - The Active Manifest should contain only one c2pa.hash.data as specified in the C2PA technical specification, which is hard-bound to the content assertion.

[0171] - The exclusion scope indicated in c2pa.hash.data should match the dsci_key_source_uri bytes in the digitally signed content initialization SEI message.

[0172] dsci_key_register_idx, when present, contains an index in the certificate registry indicated by dsci_key_source_uri, which specifies the content provider's certificate, which can be used to verify the signature present in subsequent digital signature content verification SEI messages.

[0173] The certificate indicated by the syntax elements dsci_key_retrieval_mode_idc, dsci_use_key_register_idx_flag, dsci_key_source_uri, and (if dsci_use_key_register_idx_flag equals 1) dsci_key_register_idx should specify the digital signature method and associated parameters (if applicable) and the content provider's public key. When dsci_key_retrieval_mode_idc equals 1, the format for providing this information is outside the scope of this specification. It is recommended to use a digital signature algorithm conforming to the "Digital Signature Standard" FIPS 186-5.

[0174] dsci_content_uuid_present_flag equal to 1 indicates that the syntax element dsci_content_uuid exists. (See specification 18 / 55, page 22, CN 121923845 A). dsci_content_uuid_present_flag equal to 0 indicates that the syntax element dsci_content_uuid does not exist.When dsci_key_retrieval_mode_idc equals 0, dsci_content_uuid_present_flag should equal 1.

[0175] dsci_content_uuid (if present) indicates the identifier of the video content and should have a value designated as UUID according to the procedure in Annex A of ISO / IEC 11578:1996.

[0176] When a digitally signed content initialization SEI message exists in the AU, the calculation of the NumVerificationSubstream message digest is initialized according to the specification for the specified dsci_hash_method_type in FIPS PUB 180-4. Each SPS, PPS, APS, PH, and VCLNAL unit after the digitally signed content initialization SEI message is associated with one of the NumVerificationSubstream message digests; the verification substream id is indicated by the digitally signed content selection SEI message, or is inferred to be equal to 0 if the PU does not have a digitally signed content selection SEI message. The message used to compute the k-th message digest (where k is in the range of 0 to dsci_num_verification_substreams_minus1, inclusive) is obtained by concatenating all SPS, PPS, APS, and VCL NAL units associated with the k-th verification substream. The message digest computation is block-based, where the block size depends on the value of dsci_hash_method_type specified in Table 1. For each SPS, PPS, APS, PH, and VCL NAL unit, the associated message digest is updated according to the algorithm specified in FIPS PUB 180-4 for the specified dsci_hash_method_type. Note that because the message digest is computed by concatenation of all SPS, PPS, APS, PH, and VCL NAL units for the verification substream, some processing blocks may typically span two or more consecutive NAL units.

[0177] Digital Signature Content Selection SEI Message:

[0178]

[0179] dscs_verification_substream_id indicates the verification substream to which the SPS, PPS, APS, PH, and VCL NAL units of the currently encoded image should be assigned. If a digital signature content initialization SEI message exists in the current encoded video sequence, but a digital signature content selection SEI message does not exist in the encoded image, the value of dscs_verification_substream_id is inferred to be 0.The value of dscs_verification_substream_id should be in the range of 0 to dsci_num_verification_substreams_minus1, inclusive.

[0180] As specified below, the message digest of the verification substream with id equal to dscs_verification_substream_id is updated using the SPS, PPS, APS, PH, and VCL NAL units of the currently encoded image, based on the dsci_hash_method_type specified in the previous digital signature content initialization SEI message.

[0181] Digital signature content verification SEI message:

[0182]

[0183] dscv_verification_substream_id indicates the verification substream to which the SEI message applies.

[0184] The length of the syntax element dscv_signature_length_in_octets_minus1 plus 1 is specified in the specification of dscv_signature, page 19 / 55, 23 CN 121923845 A, in octet units (an octet consists of 8 bits).

[0185] dscv_signature contains the digital signature of the verification substream indicated by dscv_verification_substream_id.

[0186] The verification of the bitstream signature consists of the following ordered steps:

[0187] 1. The message digest (referred to as CurrDigest) is calculated as follows:

[0188] - According to the specification in FIPS PUB 180-4, the concatenation of the SPS, PPS, APS, PH, and VCL NAL units of the verification substream with id equal to dscv_verification_substream_id is filled. Note that filling only the last NAL unit of the verification substream is sufficient.

[0189] - The calculation of the message digest CurrDigest is performed according to the specification in FIPS PUB 180-4. The length of the message digest (in bits) is given in Table 1.

[0190] 2. The reference message digest RefDigest is determined as follows:

[0191] - If dscv_verification_substream_id is greater than 0, then the reference message digest RefDigest is the last calculated message digest of the verification substream whose id is equal to dscv_verification_substream_id-1.The requirement for bitstream consistency is that any digitally signed content verification SEI associated with the verification substream with ID equal to dscv_verification_substream_id-1 must exist before the digitally signed content verification SEI message of the verification substream with ID equal to dscv_verification_substream_id.

[0192] - Otherwise, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID equal to 0 in the encoded video sequence, and the previously encoded video sequence does not contain any digitally signed content initialization SEI message (this includes the case where the current encoded video sequence is the first encoded video sequence in the bitstream), then RefDigest is set to be equal to a bit string consisting of DigestSize bits equal to 1, where DigestSize is the size of the message digest specified in Table 1.

[0193] - Otherwise, the reference message digest RefDigest is the last computed message digest of the verification substream with ID equal to 0.

[0194] 3. The identifier string IdString is constructed by concatenating the binary indications of the reference message digest RefDigest, the current message digest, dsci_hash_method_type, and dsci_content_uuid2 (if present).

[0195] The number of bits in RefDigest is determined by the value of dsci_hash_method_type valid when calculating the RefDigest value, the number of bits in CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented by 8 bits, and the value of dsci_content_uuid (if present) is represented by 128 bits.

[0196] 4. The identifier string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and (if dsci_use_key_register_idx_flag equals 1) dsci_key_register_idx.

[0197] Note 1: Since the bit string used for signature verification includes RefDigest, it is possible not only to verify that the SPS, PPS, APS, PH, and VCL NAL units used to calculate the current message digest are correct, but also to additionally verify that neither additional SPS, PPS, APS, PH, and VCL NAL units were added to the bitstream nor removed from the bitstream.

[0198] Note 2: When the decoder is tuned to the bitstream, the IdString constructed for the first digital signature content verification SEI message cannot be verified because the value of RefDigest cannot be calculated correctly. However, starting from the second digital signature content verification SEI message, the signature can be verified.

[0199] After verification, the message digest of the verification substream with id equal to dscv_verification_substream_id is reinitialized according to the specification for the specified dsci_hash_method_type in FIPS PUB 180-4, based on page 24 of the specification 20 / 55.

[0200] Embodiments of the invention are described below.

[0201] Verifying data alone is insufficient; the verification data directly affects the image decoded by the decoder (i.e., VCL NAL units). Some processes may link to the video data using metadata describing the post-processing steps, which may radically alter the content. An example is the SEI message of a neural network post-filter, which can output an image very different from the image output by the decoder using only VCL NAL units.

[0202] Furthermore, some descriptive metadata (e.g., copyright or content tags) may be modified, which does not affect the output sample itself but affects key information that may have other meanings.

[0203] Another attack angle could be generative face technology, which can be manipulated and abused to fraudulently alter decoded images.

[0204] Furthermore, masking / hiding attacks (i.e., hiding decoded samples in the output) could be carried out by introducing fraudulent film granules or manipulating display overlays. Additionally, all frame-packed SEI messages are potentially manipulated in their respective applications for fraudulent purposes to mask or hide decoded samples.

[0205] Moreover, user data registration and unregistered SEI messages should be able to be signed to ensure the authenticity of application-specific solutions.

[0206] It may not matter if such metadata is always signed using VCL NAL units, as some metadata may be irrelevant, and it would prevent some systems that do not need such metadata from discarding it, which violates the design of the video coding standard, which places such data in a less important bitstream and allows it to be discarded without affecting the decoding process.

[0207] The same applies to the repetition of SEI messages, which is a robustness measure in the video coding standard, where it is allowed to repeatedly place SEI messages with the same content within certain portions of the bitstream. Any such repetition allows for resistance to packet loss affecting redundant data, and any solution for media authenticity must have the same degree of robustness, i.e., the loss of redundant data should not prevent the ability to test the authenticity of received data.

[0208] As a first embodiment, information is added to the bitstream that allows identification of whether such metadata (i.e., SEI messages) is considered when calculating the hash / signature. Examples are shown below.

[0209] Digital Signature Content Initialization SEI Message Specification 21 / 55 Page 25 CN 121923845 A

[0210]

[0211] dsci_sei_messages_digitally_signed_flag equal to 1 specifies that the non-VCL NAL unit corresponding to the CVS SEI message is used to calculate the message digest. dsci_sei_messages_digitally_signed_flag equal to 0 specifies that the non-VCL NAL unit corresponding to the CVS SEI message is not used to calculate the message digest.

[0212] There are different options regarding how to handle multiple SEI message types.

[0213] In one embodiment, when signaling indicates that SEI messages should be considered to sign the video bitstream via dsci_sei_messages_digitally_signed_flag, all SEI messages are included in the message digest.

[0214] Alternatively, this processing may be applied only to a predefined set of SEI messages identified as being associated with a signature.For example, in one embodiment, the video coding standard may define a list of RelevantForSigningSEIMessages containing one or more SEI message types, which are listed as follows:

[0215] NN post-filter related: NNPFC, NNPFA SEI messages,

[0216] Film grain synthesis: FGS

[0217] SEI processing order SEI message indicating the post-processing order of SEI messages: SPO

[0218] Masking related: FGS SEI message, Display overlay SEI message,

[0219] Face-generative related: GFV SEI message, GFVE SEI message

[0220] Packing related: Frame packing arrangement SEI message, Packing region information SEI message, Packing by region SEI message, Constructing rectangle SEI message

[0221] Registered and unregistered user-data SEI messages

[0222] Legal, regulatory and annotation related SEI messages: Copyright, Content tag or text description SEI message, AI usage restriction SEI message

[0223] In an alternative embodiment, the SEI messages to be included in the authenticity processing specification (pages 22 / 55, CN 121923845 A) can also be identified by the value range of the SEI message type, which can also reserve space for future SEI messages.

[0224] As another alternative embodiment, signaling can indicate different possible combinations of SEI messages to be signed. For example, this can be achieved by adding syntax elements pointing to different combinations, as follows:

[0225]

[0226] dsci_sei_messages_digitally_signed_flag equal to 1 indicates that the message digest is calculated using a non-VCL NAL unit corresponding to the CVS SEI message. dsci_sei_messages_digitally_signed_flag equal to 0 indicates that the message digest is not calculated using a non-VCL NAL unit corresponding to the CVS SEI message.

[0227] dsci_sei_messages_digitally_signed_idc identifies the SEI message used to calculate the message digest, as specified in Table X. The value of dsci_sei_messages_digitally_signed_idc should be in the range of 0 to Y, inclusive. Specification 23 / 55, page 27, CN 121923845 A

[0228]

[0229] As an alternative to using two syntax elements, only idc can be used, where a value of 0 indicates that no SEI message is used for the message digest.

[0230] As another alternative, the SEI type used to calculate the message digest can be sent as a signal in the bitstream.Example as follows: Specification 24 / 55 Page 28 CN 121923845 A

[0231]

[0232] dsci_num_sei_message_types_digitally_signed_minus1 plus 1 specifies the number of SEI message types used to calculate the message digest.

[0233] dsci_digitally_signed_sei_type[i] identifies the i-th SEI type used to calculate the message digest.

[0234] Note that if the type value is greater than 255, the SEI message type can use more than 8 bits. Therefore, the above syntax can be extended to have 16 bits instead of 8 bits. Alternatively, another syntax element (e.g., dsci_digitally_signed_sei_type_length_inbytes_minus1) can be added, which indicates the length of the syntax element dsci_digitally_signed_sei_type[i] (e.g., 8 * (dsci_digitally_signed_sei_type_length_inbytes_minus1 + 1) bits).

[0235] Alternatively, instead of identifying in advance which SEI messages are used for message digest, a prefix SEI message can be added before each SEI message used for message digest, or nested SEI messages including any SEI message can be used to indicate that such SEI messages are used for message digest.

[0236] Note that SEI messages are encapsulated in a single SEI NAL unit that may contain more than one SEI message. Currently, authentication is performed based on NAL units (the entire NAL unit). For SEI messages, in one embodiment only SEI messages may be considered, but as an alternative, the entire SEI NAL unit may be considered. Therefore, it would be beneficial to identify whether to use the SEI NAL unit based on the first SEI message (page 25 / 55, CN 121923845 A). For this purpose, the following constraint is applied: any SEI message with an SEI message type that will not be verified is not included in the same NAL unit that includes SEI messages with the SEI message type to be verified.

[0237] Another aspect to consider is the assignment of SEI messages to specific substreams. Different options exist:

[0238] 1. The SEI message belongs to the highest indicated substream ID.

[0239] 2. The SEI message belongs to the same substream ID as the PU that includes such an SEI message.

[0240] 3. When the SEI message is indicated for message digest / for signature generation, the number of substreams is twice the number of signals sent, and the substreams of the SEI message are indicated by the SEI message selected by the digital signature content.

[0241] For option 1, the following changes will be required.

[0242] 1.2 Digital Signature Content Selection SEI Message

[0243] 1.2.1 Digital Signature Content Selection SEI Message Syntax

[0244]

[0245] 1.2.2 Digital Signature Content Selection SEI Message Semantics

[0246] The Digital Signature Content Selection SEI message provides a mechanism for associating an encoded picture with one of the verification substreams indicated in the Digital Signature Content Initialization SEI message.

[0247] [...]

[0248] dscs_verification_substream_id indicates the verification substream to which the SPS, PPS, APS, PH, and VCL NAL units of the current encoded picture are assigned. When a Digital Signature Content Initialization SEI message exists in the current encoded video sequence, but a Digital Signature Content Selection SEI message does not exist in the encoded picture, the value of dscs_verification_substream_id for the SPS, PPS, APS, PH, and VCL NAL units is inferred to be equal to 0. When `dsci_sei_messages_digitally_signed_flag` equals 1, it is inferred that the value of `dscs_verification_substream_id` is equal to `dsci_num_verification_substream_minus1` of the SEI message specified by the DSCI SEI message (here, the subset of SEI messages used for message digests may be specified in the DSCI SEI message). The value of `dscs_verification_substream_id` should be in the range of 0 to `dsci_num_verification_substreams_minus1`, inclusive.

[0249] As an alternative, in order to allow signaling to use either Option 1 (using the highest substream for SEI messages) or Option 2 (SEI messages being the same as the substream of the VCL NAL unit of the encoded image), the syntax of the DSCI SEI message can be extended as follows: Specification 26 / 55 pages 30 CN 121923845 A

[0250]

[0251] dsci_sei_messages_digitally_signed_highest_substream_flag equal to 1 indicates that the SEI message used to compute the message digest is assigned to the highest verified substream.The fact that dsci_sei_messages_digitally_signed_highest_substream_flag equals 0 indicates that the SEI message used to compute the message digest is assigned to the same verification substream as the VCL NAL unit of the associated encoded picture.

[0252] 1.2.2 Semantics of Digital Signature Content Selection SEI Message

[0253] The Digital Signature Content Selection SEI message provides a mechanism for associating an encoded picture with one of the verification substreams indicated in the Digital Signature Content Initialization SEI message.

[0254] [...]

[0255] dscs_verification_substream_id indicates the verification substream to which the SPS, PPS, APS, PH, and VCL NAL units of the current encoded picture are assigned. When a Digital Signature Content Initialization SEI message exists in the current encoded video sequence, but a Digital Signature Content Selection SEI message does not exist in the encoded picture, it is inferred that the value of dscs_verification_substream_id of the SPS, PPS, APS, PH, and VCL NAL units is equal to 0. The value of dscs_verification_substream_id should be in the range of 0 to dsci_num_verification_substreams_minus1, inclusive of 0 and dsci_num_verification_substreams_minus1.

[0256] Another alternative would be to operate as indicated above in option 3. This can be instantiated in the standard as follows: Specification 27 / 55 Page 31 CN 121923845 A

[0257]

[0258] dsci_sei_messages_digitally_signed_separate_substream_flag equal to 1 indicates that the SEI message used for calculating the message digest is assigned to a separate verification substream as specified in the semantics of the DSCS SEI message. dsci_sei_messages_digitally_signed_separate_substream_flag equal to 0 indicates that the SEI message used for calculating the message digest is assigned to the same verification substream as the VCL NAL unit of the associated encoded picture.

[0259] 1.2.2 Semantics of Digital Signature Content Selection SEI Message

[0260] The Digital Signature Content Selection SEI message provides a mechanism for associating an encoded image with one of the verification substreams indicated in the Digital Signature Content Initialization SEI message.

[0261] [...]

[0262] dscs_verification_substream_id indicates the verification substream to which the SPS, PPS, APS, PH, and VCL NAL units of the currently encoded image are assigned. When dsci_sei_messages_digitally_signed_flag equals 1 and dsci_sei_messages_digitally_signed_highest_substream_flag equals 0, the SEI messages used for message digests belong to the same verification substream. When dsci_sei_messages_digitally_signed_flag equals 1 and dsci_sei_messages_digitally_signed_highest_substream_flag equals 1, the SEI messages used for message digests belong to the verification substream with an id equal to dscs_verification_substream_id plus dsci_num_verification_substreams_minus1 plus 1. When a digitally signed content initialization SEI message exists in the current encoded video sequence, but a digitally signed content selection SEI message does not exist in the encoded image, it is inferred that the value of dscs_verification_substream_id of the SPS, PPS, APS, PH, and VCL NAL units is equal to 0. When dsci_sei_messages_digitally_signed_flag equals 1, dsci_sei_messages_digitally_signed_highest_substream_flag equals 1, and the encoded image does not contain a digitally signed content selection SEI message, the SEI message used for message digest belongs to the verification substream with an id equal to dsci_num_verification_substreams_minus1 plus 1. The value of dscs_verification_substream_id should be in the range of 0 to dsci_num_verification_substreams_minus1, inclusive.

[0263] Alternatively, syntax elements that distinguish between options 1, 2, and 3 above may be introduced.

[0264] Video Coding Scheme

[0265] Hereinafter, a video coding scheme will be described in which embodiments of the present invention may optionally be implemented.In other words, the apparatus 20 according to any of the foregoing embodiments can be a video decoder implemented according to any of the embodiments of the decoder 20 described below. Similarly, the apparatus 10 can optionally be a video encoder according to any of the embodiments of the encoder 10 described below.

[0266] The following description of the figures begins with the presentation of a description of a block-based predictive coding encoder and decoder used to encode images of a video to form an example of an encoding framework in which embodiments of the invention can be built. Figures 8, 9, and 10 depict the respective encoders and decoders. In the following, although embodiments are described using the following Figures 8 and 9 and their subsequent figures, the description of embodiments of the inventive concept, together with a description of how these concepts are respectively incorporated into the encoders and decoders of Figures 8 and 9, can also be used to form encoders and decoders that do not operate according to the underlying encoding framework of the encoders and decoders of Figures 8, 9, and 10.

[0267] Figure 8 illustrates an apparatus for predictively encoding an image 12 to a data stream 14 using transform-based residual coding, exemplarily. This apparatus or encoder is indicated by reference numeral 10. Figure 9 illustrates the corresponding decoder 20, i.e., the apparatus 20 configured to also predictively decode the image 12' from the data stream 14 using transform-based residual decoding, wherein the apostrophe has been used to indicate that the image 12' reconstructed by the decoder 20 deviates from the image 12 initially encoded by the apparatus 10 in terms of the encoding loss introduced by the quantization of the predictive residual signal. Figures 8 and 9 exemplarily use transform-based predictive residual coding, but embodiments of this application are not limited to this kind of predictive residual coding. As will be outlined below, so too are other details described for Figures 8 and 9.

[0268] The encoder 10 is configured to perform a spatial-to-spectral transformation on the predictive residual signal and encode the resulting predictive residual signal into the data stream 14. Similarly, the decoder 20 is configured to decode the predictive residual signal from the data stream 14 and perform a spectral-to-spatial transformation on the resulting predictive residual signal.

[0269] Internally, encoder 10 may include a prediction residual signal former 22, which generates a prediction residual 24 to measure the deviation of the prediction signal 26 from the original signal (i.e., from image 12). The prediction residual signal former 22 may, for example, be a subtractor that subtracts the prediction signal from the original signal (i.e., from image 12). Encoder 10 then includes a transformer 28 that performs a space-to-spectral transform on the prediction residual signal 24 to obtain a spectral domain prediction residual signal 24', which is then quantized by a quantizer 32, which is also included in encoder 10. The thus quantized prediction residual signal 24'' is encoded into bitstream 14.For this purpose, encoder 10 may optionally include entropy encoder 34, which entropy-encodes the prediction residual signal that is transformed and quantized into data stream 14. Prediction signal 26 is generated by prediction stage 36 of encoder 10 based on prediction residual signal 24'' that is encoded into data stream 14 and can be decoded from data stream 14. To this end, as shown in Figure 8, prediction stage 36 may internally include a dequantizer 38 that dequantizes the prediction residual signal 24” to obtain a spectral domain prediction residual signal 24''' corresponding to the signal 24 excluding the quantization loss. It then includes an inverse transformer 40 that performs an inverse transform (i.e., a spectral-to-spatial transform) on the latter's prediction residual signal 24'''' to obtain a prediction residual signal 24'''' corresponding to the original prediction residual signal 24 excluding the quantization loss. The combiner 42 of prediction stage 36 then recombines the prediction signal 26 and the prediction residual signal 24''''', for example, by addition, to obtain a reconstructed signal 46, i.e., a reconstruction of the original signal 12. The reconstructed signal 46 may correspond to signal 12'. The prediction module 44 of prediction stage 36 then generates the prediction signal 26 based on signal 46 using, for example, spatial prediction (i.e., intra-picture prediction) and / or temporal prediction (i.e., inter-picture prediction).

[0270] Similarly, as shown in FIG9, decoder 20 may internally consist of components corresponding to prediction stage 36 and interconnected in a manner corresponding to prediction stage 36. Specifically, entropy decoder 50 of decoder 20 may entropy decode the quantized spectral domain prediction residual signal 24'' from the data stream, and then the dequantizer 52, inverse transformer 54, combiner 56, and prediction module 58, interconnected and cooperating with respect to prediction stage 36 in the manner described above, recover the reconstructed signal based on the prediction residual signal 24'', so that, as shown in FIG9, the output of combiner 56 produces the reconstructed signal (i.e., picture 12').

[0271] Although not specifically described above, it is readily apparent that encoder 10 can set encoding parameters, including prediction modes, motion parameters, etc., according to some optimization scheme, such as optimizing some rate and distortion-related criteria (i.e., encoding cost). For example, encoder 10, decoder 20, and various modules 44 and 58 can support different prediction modes, such as intra-frame coding modes and inter-frame coding modes, respectively. The granularity of switching between these prediction mode types by encoder and decoder can correspond to subdividing images 12 and 12' into coding segments or coding blocks, respectively. For example, based on these coding segments, images can be subdivided into blocks that are intra-frame coded and blocks that are inter-frame coded. As outlined in more detail below, intra-frame coded blocks are predicted based on the spatially encoded / decoded neighborhood of the corresponding blocks.Several intra-coding modes may exist and be selected for each intra-coding mode, including directional or angular intra-coding modes, according to which segments are filled by extrapolating sample values ​​of the neighborhood along a direction specific to each directional intra-coding mode into each intra-coded segment. For example, the intra-coding modes may also include one or more additional modes, such as: a DC coding mode, according to which the prediction of each intra-coded block assigns DC values ​​to all samples within each intra-coded segment; and / or a planar intra-coding mode, according to which the prediction of each block is approximated or determined as the spatial distribution of sample values ​​at sample locations of each intra-coded block, described by a two-dimensional linear function, having a driving tilt and offset of a plane defined by the two-dimensional linear function based on adjacent samples. In contrast, inter-coded blocks can be predicted temporally, for example. For inter-coded blocks, a motion vector can be signaled within the data stream, indicating the spatial displacement of said portion of a previously encoded image of the video to which image 12 belongs, where the previously encoded / decoded image is sampled to obtain the prediction signal for the corresponding inter-coded block. This means that, in addition to the residual signal encoding included in data stream 14 (e.g., the entropy-coded transform coefficient level representing the quantized spectral domain prediction residual signal 24''), data stream 14 may have already encoded therein coding mode parameters for assigning coding modes to various blocks, prediction parameters for some blocks (e.g., motion parameters for inter-frame coded segments), and optional other parameters, such as parameters for controlling and signaling the subdivision of images 12 and 12' into segments, respectively. Decoder 20 uses these parameters to subdivide the images in the same manner as encoder, assigning the same prediction modes to segments, and performing the same predictions to produce the same prediction signals.

[0272] Figure 10 illustrates the relationship between the reconstructed signal (i.e., reconstructed image 12') on one side and the combination of the prediction residual signal 24'''' and prediction signal 26 signaled in data stream 14 on the other side. As described above, this combination can be additive. Prediction signal 26 is shown in Figure 10 as subdividing the image region into intra-frame coded blocks illustratively indicated by shaded lines and inter-frame coded blocks without shaded lines. The subdivision can be any subdivision, such as regularly subdividing the image region into rows and columns of square blocks or non-square blocks, or subdividing the image 12 from a root block into multiple leaf blocks of variable size, such as quadtree subdivision, etc., wherein Figure 10 shows its blend, in which the image region is first subdivided into rows and columns of root blocks, and then further subdivided into one or more leaf blocks according to the recursive multitree subdivision.

[0273] Similarly, data stream 14 may have intra-coding modes encoded therein for intra-coding blocks 80, which assign one of the several supported intra-coding modes described on pages 34 of specification 30 / 55, CN 121923845 A, to the respective intra-coding blocks 80. For inter-coding blocks 82, data stream 14 may have one or more motion parameters encoded therein. In general, inter-coding blocks 82 are not limited to being encoded temporally. Alternatively, inter-coding blocks 82 may be any block predicted from previously encoded portions beyond the current picture 12 itself (e.g., previously encoded pictures of the video to which picture 12 belongs, or pictures of another view, or lower layers in the case of scalable encoders and decoders).

[0274] The prediction residual signal 24'''' in FIG. 10 is also shown as subdividing the picture region into blocks 84. These blocks may be referred to as transform blocks in order to distinguish them from coding blocks 80 and 82. In practice, Figure 10 shows that encoder 10 and decoder 20 can subdivide images 12 and 12' into blocks using two different subdivisions: one image is subdivided into coded blocks 80 and 82, while the other image is subdivided into transform blocks 84. The two subdivisions can be the same, i.e., each coded block 80 and 82 can simultaneously form transform block 84. However, Figure 10 shows cases where, for example, subdividing into transform block 84 extends the subdivision into coded blocks 80 and 82, such that any boundary between the two blocks 80 and 82 overlaps the boundary between the two blocks 84; or alternatively, each block 80, 82 either coincides with one of the transform blocks 84 or with a cluster of transform blocks 84. However, the subdivisions can also be determined or selected independently of each other, such that transform block 84 can alternatively span the block boundaries between blocks 80 and 82. Regarding the subdivision into transform blocks 84, statements similar to those regarding subdivision into blocks 80 and 82 are therefore true, i.e., block 84 can be the result of regularly subdividing the image region into blocks (arranged or not arranged in rows and columns), the result of recursively subdividing the image region into multiple trees, or a combination thereof or any other type of subdivision method. Incidentally, it should be noted that blocks 80, 82, and 84 are not limited to square, rectangular, or any other shape.

[0275] FIG10 also shows that the combination of prediction signal 26 and prediction residual signal 24'''' directly produces reconstructed signal 12'. However, it should be noted that, according to alternative embodiments, more than one prediction signal 26 can be combined with prediction residual signal 24'''' to produce image 12'.

[0276] In FIG10, transform blocks 84 should have the following meaning. Transformer 28 and inverse transformer 54 perform their transforms in units of these transform blocks 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84.Some codecs allow skipping transforms, so that for some transform blocks 84, the predicted residual signal is directly encoded in the spatial domain. However, according to the embodiments described below, encoder 10 and decoder 20 are configured to support several transforms. For example, the transforms supported by encoder 10 and decoder 20 may include:

[0277] oDCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform

[0278] oDST-IV, where DST stands for Discrete Sine Transform

[0279] oDCT-IV

[0280] oDST-VII

[0281] oIdentity Transform (IT)

[0282] Of course, while transformer 28 will support all forward versions of these transforms, decoder 20 or inverse transformer 54 will support their respective backward or inverse versions:

[0283] oInverse DCT-II (or Inverse DCT-III)

[0284] oInverse DST-IV

[0285] oInverse DCT-IV

[0286] oInverse DST-VII

[0287] oIdentity Transform (IT)

[0288] The following description provides more details about the transforms that encoder 10 and decoder 20 may support. In any case, it should be noted that the set of supported transformations may include only one transformation, such as a spectrum-to-space or space-to-spectrum transformation.

[0289] As outlined above, Figures 8, 9, and 10 have been presented as examples in which the inventive concepts further described below can be implemented to form specific examples of encoders and decoders according to this application. In this regard, the encoders and decoders of Figures 8 and 9 may respectively represent possible implementations of the encoders and decoders described herein. However, Figures 8 and 9 are merely examples. However, the encoder of an embodiment of this application may perform block-based encoding on image 12 using a concept outlined in more detail below and different from that of the encoder of Figure 8 (e.g., a still image encoder instead of a video encoder, which does not support inter-frame prediction, or the way it is subdivided into blocks 80 is different from that illustrated in Figure 10). Similarly, the decoder according to embodiments of this application may perform block-based decoding of image 12' from data stream 14 using the coding concept further outlined below, but may differ from, for example, decoder 20 of FIG. 9, in that it is not a video decoder but a still image decoder, does not support intra-frame prediction, or subdivides image 12' into blocks in a manner different from that described with respect to FIG. 10, and / or does not derive prediction residuals from data stream 14 in the spatial domain but derives them, for example, in the transform domain.

[0290] Embodiments provide an H.264 / AVC decoder, an H.264 / AVC video data stream, a video encoder for providing an H.264 / AVC video data stream, or an H.265 / HEVC decoder, an H.265 / HEVC video data stream, a video encoder for providing an H.265 / HEVC video data stream, or an H.266 / VVC decoder, an H.266 / VVC video data stream, a video encoder for providing an H.266 / VVC video data stream.

[0291] Hereinafter, embodiments of the invention will be described with reference to FIGS. 8 to 10, which may optionally be implemented in or combined with the above framework.

[0292] All embodiments of the invention disclosed herein may optionally be combined with any of the four video coding variants described below. That is, data stream 14 may be a data stream according to any of these variants, apparatus 20 may be configured to decode video from data stream 14 according to any of the three variants, and apparatus 14 may be configured to provide a data stream according to any of the four variants. The first variant and the second variant can be optimally combined with each other.

[0293] According to the first variant of the two variants of the invention, video is decoded from the video data stream by decoding the predicted residual data of the residual block into / from the video data stream, and by decoding the residual block from the video data stream using block-based predicted residual data and transform-based residual decoding.

[0294] According to the first variant, the decoding of the predicted residual data of the residual block is performed using context-adaptive variable-length decoding by using the following:

[0295] - a first syntax element and a tail-1 quantity, the first syntax element indicating the total number of non-zero transform coefficients in the transform block representing the residual block, the tail-1 quantity indicating the number of non-zero transform coefficients with an absolute value of 1 when traversing the coefficients in scan order;

[0296] - one or more second syntax elements indicating the sign of the non-zero transform coefficients with an absolute value of 1 when traversing the coefficients in scan order;

[0297] - one or more third syntax elements indicating the value of the non-zero transform coefficients other than the number of non-zero transform coefficients with an absolute value of 1 when traversing the coefficients in scan order;

[0298] - a fourth syntax element indicating the total number of zero-value transform coefficient levels in the transform block starting from the first non-zero transform coefficient encountered in scan order; and

[0299] - One or more fifth syntax elements indicate the position of non-zero transform coefficients along the scan order by indicating the number of consecutive zero-value transform coefficients in the scan order between consecutively encountered non-zero transform coefficients, or

[0300] As an alternative to using context-adaptive variable-length decoding, the decoding specification for the predictive residual data of the residual block, page 32 / 55, CN 121923845 A, is performed using context-adaptive binary arithmetic decoding in the following manner:

[0301] - The importance graph indicating the position of non-zero transform coefficients in the transform block representing the residual block is decoded by: decoding an importance flag indicating whether a non-zero transform coefficient is located at the current position during a forward scan of the transform coefficients of the transform block, and if so, and if the current position is not the last position in the forward scan, decoding a final importance flag indicating whether the non-zero transform coefficient at the current position is the last non-zero transform coefficient in the forward scan order, and

[0302] The values ​​of the non-zero transform coefficients are decoded sequentially according to the reverse scan order, which reverses the forward scan order.

[0303] According to the first variant of the two variants of the invention, the video is encoded into the video data stream by encoding the predicted residual data of the residual blocks into the video data stream through block-based prediction and transform-based residual coding.

[0304] According to the first variant, the encoding of the prediction residual data of the residual block is performed using context-adaptive variable-length encoding by using the following:

[0305] - a first syntax element and a tail-1 quantity, the first syntax element indicating the total number of non-zero transform coefficients in the transform block representing the residual block, the tail-1 quantity indicating the number of non-zero transform coefficients with an absolute value of 1 when the coefficients are traversed in scan order;

[0306] - one or more second syntax elements indicating the sign of the non-zero transform coefficients with an absolute value of 1 when the coefficients are traversed in scan order;

[0307] - one or more third syntax elements indicating the value of the non-zero transform coefficients other than the number of non-zero transform coefficients with an absolute value of 1 when the coefficients are traversed in scan order;

[0308] - a fourth syntax element indicating the total number of zero-value transform coefficient levels in the transform block starting from the first non-zero transform coefficient encountered in scan order; and

[0309] - One or more fifth syntax elements indicate the position of non-zero transform coefficients along the scan order by indicating the number of consecutive zero-value transform coefficients in the scan order between consecutive non-zero transform coefficients encountered in the scan order, or

[0310] As an alternative to using context-adaptive variable-length coding, the encoding of the prediction residual data of the residual block is performed by using context-adaptive binary arithmetic coding in the following manner:

[0311] - The importance graph indicating the position of non-zero transform coefficients in the transform block representing the residual block is encoded by encoding an importance flag in the forward scan of the transform coefficients of the transform block indicating whether a non-zero transform coefficient is at the current position, and if so, and if the current position is not the last position in the forward scan, encoding a final importance flag indicating whether the non-zero transform coefficient at the current position is the last non-zero transform coefficient in the forward scan order, and

[0312] - The values ​​of non-zero transform coefficients are encoded sequentially in the reverse scan order, which reverses the forward scan order.

[0313] According to the second variant, the video decoder (or means for decoding video from a video data stream) includes an encoded picture buffer (CPB) and a decoded picture buffer (DPB). The video decoder is configured to:

[0314] receive a data stream as a sequence of access units (AUs) having pictures of video encoded therein in an encoded order (e.g., the term "access unit" refers to a portion of the video data stream that includes encoded video data or information associated with a time frame of the video).

[0315] The current AU removed from the CPB is decoded using inter-picture prediction of reference pictures stored in the DPB to obtain a decoded picture, and

[0316] the decoded picture is inserted into the DPB,

[0317] each reference picture stored in the DPB is assigned a category as one of short-term reference picture, long-term reference picture, and picture not used for reference,

[0318] DPB mode information is read from the current AU,

[0319] if the DPB mode information indicates a first mode, one or more reference pictures classified as short-term pictures are removed from the DPB according to a first-in-first-out (FIFO) strategy,

[0320] if the DPB mode information indicates a second mode, memory management control information including at least one command in the current AU is read and the at least one command is executed to change the category assigned to at least one of the reference pictures stored in the DPB, and the category of the reference pictures in the DPB is used to manage the reference pictures removed from the DPB.

[0321] According to the second variant, the video data stream should be decoded by feeding it to a decoder including an encoded picture buffer (CPB). According to the second variant, the video encoder (or means for encoding video into a video data stream) is configured to:

[0322] encode pictures of the video encoded in the encoding order of the access unit (AU) sequence into the data stream,

[0323] wherein the means is configured to: when encoding AUs,

[0324] encode the current picture from reference reference pictures stored in the decoded picture buffer (DPB) into the current AU using inter-picture prediction, and

[0325] insert the decoded version of the current picture in the DPB into the DPB,

[0326] assign each reference picture stored in the DPB to a classification as one of short-term reference pictures, long-term reference pictures, and pictures not used for reference,

[0327] write DPB mode information into the current AU,

[0328] if the DPB mode information indicates a first mode, remove one or more reference pictures classified as short-term pictures from the DPB according to a FIFO policy,

[0329] If the DPB mode information indicates a second mode, memory management control information including at least one command is written to the current AU, which indicates a change in the classification of at least one of the reference pictures stored in the DPB, wherein the classification of the reference pictures in the DPB is used to manage reference pictures removed from the DPB.

[0330] According to the third variant, video is decoded from the video data stream by block-based predictive decoding and transform-based residual decoding in the following manner: context-adaptive binary arithmetic decoding is performed using the quantization index of the transform coefficients of the transform block representing the residual block, and the quantization index is sequentially dequantized, according to which the value of the current transform coefficient depends on the parity of the quantization index of the previous quantization index.

[0331] According to the third variant, video is encoded into the video data stream by block-based predictive coding and transform-based residual coding in the following manner: context-adaptive binary arithmetic coding is performed using the quantization index of the transform coefficients of the transform block representing the residual block, and the transform coefficients are sequentially quantized to obtain quantization indexes, according to which the quantizer quantizes the current transform coefficient based on the parity of the quantization index of the previous quantization index.

[0332] All three variants relate to a video encoder, a video decoder, a method for decoding video, a method for encoding video, and a video data stream obtained by the corresponding encoding method.

[0333] Further optional details and features of the first and second variants will be described below.

[0334] Embodiments of the first and second variants may conform to the H.264 / AVC standard. Embodiments of the third variant may conform to the H.266 / VVC standard.

[0335] One aspect of the first and second variants relates to the processing of the decoded picture and its buffering in the decoded picture buffer specification, pages 34 / 55, CN 121923845 A (DPB).

[0336] According to the embodiment, two types of reference pictures can be distinguished: short-term and long-term. The encoder does the same thing at each time point during decoding when simulating the DPB filling state of the decoder. When the predictive reference is no longer needed, the reference picture can be marked as "not for reference". The transition between these three states (short-term, long-term, and not for reference) is controlled by the decoded reference picture marking process. There are two alternative decoded reference picture marking mechanisms, an implicit sliding window process and an explicit memory management control operation (MMCO) process. For each currently decoded picture or each currently decoded AU, it is signaled in the data stream to which process will be used for DPB management. When the number of reference frames equals the given maximum number (max-num-ref-frames in SPS), the sliding window procedure marks short-lived reference frames as "not for reference". Short-lived reference frames are stored in a first-in, first-out (FIFO) manner, ensuring that the most recently decoded short-lived frame remains in the DPB. The explicit MMCO procedure is controlled via multiple MMCO commands. If this mode is selected for the current AU or the currently decoded frame, the bitstream contains one or more of these commands for that AU.The MMCO command can be any of the following: 1) marking one or more short-term or long-term reference pictures as "not for reference", 2) marking all pictures as "not for reference", or 3) marking the current reference picture or an existing short-term reference picture as a long-term picture and assigning the long-term picture index to that long-term picture. The reference picture marking operation, along with any output (for presentation) and the removal of pictures from the DPB, can be performed after the pictures have been decoded.

[0337] Some possible but optional details of the reference picture marking mechanism are discussed below. 1) The first aspect relates to gaps in the frame number and non-existent pictures. Although not stated above, each reference picture in the DPB can be associated with a frame number. Typically, the frame number is incremented by 1 for each reference picture, but gaps in the frame number can be allowed by setting the corresponding high-level (e.g., sequence-level) flag (which may be referred to as parameter-gaps-in-frame-num-allowed-flag) to 1, for example, to allow the encoder or MAME (Media Aware Network Element) to transmit a bitstream in which the frame number of the reference picture is incremented by more than one relative to the previous reference picture in the decoding order. This can be advantageous to support time scalability. AU sequences with gaps in frame numbers can be received, and non-existent pictures can be created to fill the gaps. The non-existent picture is assigned a frame number value in the gap and is treated as a reference picture during the decoding of the reference picture marker, but will not be used in the output (and therefore will not be displayed). The non-existent picture ensures that the state of the DPB regarding the frame number of the picture residing in it is the same for the decoder that received the picture and the decoder that did not receive the picture.

[0338] Another possible aspect of the first and second variations involves the loss of a reference picture when using a sliding window. When a reference picture is lost, the picture can be hidden, and the loss can be reported to the encoder if the feedback channel is available when the loss is detected. If gaps in frame numbers are not allowed, the discontinuity of the frame number value indicates the accidental loss of the reference picture. If gaps in frame numbers are allowed, the discontinuity of the frame number value may be due to the deliberate removal of a time layer or subsequence or accidental picture loss, and the decoder should only infer the picture loss if the non-existent picture is inferred during the inter-frame prediction process. The picture order count of the hidden picture may be unknown, which may cause the decoder to use an incorrect reference picture when decoding the B picture without detecting any error.

[0339] Another possible aspect of the first and second variations involves the loss of reference pictures with MMCO. When a reference picture containing an MMCO command that marks a short-term reference picture as "not for reference" is lost, the reference picture status in the DPB becomes incorrect, and therefore the list of reference pictures for a small number of pictures after the loss of the picture may become incorrect.If a picture containing MMCO commands associated with long-term reference pictures is lost, there is a risk that the number of long-term reference pictures in the DPB may differ from the original number when the picture is received, resulting in an "incorrect" sliding window process for all subsequent pictures. That is, the encoder and decoder may contain different numbers of short-term reference pictures, resulting in different walks of the sliding window process. To make matters worse, the decoder may not be aware that the sliding window process is out of sync. Specification 35 / 55 pages 39 CN 121923845 A

[0340] Hereinafter, MMCO commands are shown. In different embodiments, one or more or all of these commands may be applied to produce:

[0341]

[0342] Another option for the implementation of the decoder and encoder according to the first and second variants is now described, which may optionally be combined with the previously described options regarding DPB management and involves entropy decoding of a certain syntax element (e.g., residual data in the form of transform coefficients) into a bitstream. Lossless entropy encoding of lossy quantized transform coefficients is a key part of efficient video codecs. One such method is called Context-Adaptive Variable-Length Coding (CAVLC), in which the encoder switches between different variable-length code (VLC) tables for various syntax elements, depending on the values ​​of previously transmitted syntax elements in the same slice in the context-adaptive manner. Both the encoder and decoder can use CAVLC. Because each syntax element is encoded into the bitstream by writing a corresponding codeword (which has been selected from the context-adaptive code table for that syntax element) into the bitstream, each CAVLC-encoded bit in the bitstream can be associated with a single syntax element. Therefore, when using CAVLC, relevant information about the levels of transform coefficients appearing in the bitstream in scan order can be obtained as syntax elements in a directly accessible form. The encoder and decoder can use CAVLC to signal the transform coefficients in the bitstream.The following syntax elements may be used, such as syntax elements with the following semantics:

[0343] - A syntax element (indicated by CoeffToken) indicating the total number of non-zero transform coefficient levels in a transform block

[0344] - One or more syntax elements indicating the number of a trailing transform coefficient level (indicated by CoeffToken), for example, a series of syntax elements and their symbols (indicated by trailing_ones_sign_flag) appearing at the end of the scan sequence of syntax elements until the last non-zero syntax element, they are all one

[0345] - One or more syntax elements for each non-zero transform coefficient except for a trailing transform coefficient, which indicate the transform coefficient level value

[0346] - A syntax element indicating the total number of zero-value transform coefficient levels Specification 36 / 55 pages 40 CN 121923845 A

[0347] - A syntax element indicating the number of consecutive transform coefficient levels with zero values ​​in the scan sequence forward from the current scan position before encountering a non-zero-value transform coefficient level.

[0348] Alternatively or additionally, the encoder may select between the use of CABAC (and thus context-adaptive binary arithmetic coding) and the use of CAVLC and signal this selection in the bitstream, and the decoder may read this signal and use the indicated decoded residual data in a manner that is appropriate.

[0349] Another option for implementing the decoder and encoder according to the first and second variants is now described, which may optionally be combined with any of the previously described options regarding DPB management and CAVLC, and involves a quarter-pixel interpolation filter. In order to allow inter-frame prediction at a finer granularity than the regular full-pixel sample grid, a sample interpolation process is used to derive sample values ​​at sub-pixel sample locations, which can range from half-pixel locations to quarter-pixel locations. The encoder and decoder may use a method for performing quarter-pixel interpolation, and the method is as follows. First, sample values ​​are generated at half-pixel locations using a 6-tap FIR filter, and then the generated half-pixel location sample values ​​are averaged by interpolation to generate sample values ​​at quarter-pixel locations for the luminance component.

[0350] Further embodiments according to the first and second variants are described below:

[0351] According to an embodiment, the apparatus may further include a decoded picture buffer (DPB) and is configured to decode the current AU from reference pictures stored in the DPB using inter-picture prediction to obtain a decoded picture, and to insert the decoded picture into the DPB, assign each reference picture stored in the DPB to a classification as one of short-term reference pictures, long-term reference pictures, and pictures not used for reference, read DPB mode information from the current AU, and if the DPB mode information indicates a first mode, remove one or more reference pictures classified as short-term pictures from the DPB according to a FIFO policy, and if the DPB mode information indicates a second mode, read memory management control information including at least one command from the current AU and execute the at least one command to change the classification of at least one of the reference pictures stored in the DPB, and use the classification of the reference pictures in the DPB for managing the removal of 424 reference pictures from the DPB.

[0352] In an embodiment, the apparatus may be configured to read from the current AU an indication of whether a decoded image is not used for inter-frame prediction; if the decoded image is not indicated as not being used for inter-frame prediction or is not directly output, then inserting the decoded image into the DPB is performed; and if the decoded image is indicated as not being used for inter-frame prediction, then the decoded image is directly output without buffering the decoded image in the DPB.

[0353] According to an embodiment, the apparatus may be configured to: assign a frame index to each reference image classified as a long-term image in the DPB, and if the frame index assigned to the predetermined reference image is referenced in the current AU, then the predetermined reference image classified as a long-term image in the DPB is used as the reference image for reference in the DPB.

[0354] In an embodiment, the apparatus may be configured to perform one or more of the following operations: if at least one command in the current AU is a first command, then reclassify reference images in the DPB that are classified as short-term reference images as images not used for reference; if at least one command in the current AU is a second command, then reclassify reference images in the DPB that are classified as short-term images as long-term reference images not used for reference; if at least one command in the current AU is a third command, then reclassify reference images in the DPB that are classified as short-term images as long-term reference images, and assign frame indexes to the reclassified reference images; if at least one command in the current AU is a fourth command, then set a frame index limit according to the fourth command, and reclassify all reference images in the DPB that are classified as long-term images and have been assigned frame indexes exceeding the frame index limit as images not used for reference; if at least one command in the current AU is a sixth command, then classify the current image as a long-term image, an image not used for reference, and assign frame indexes to the reclassified reference images.

[0355] According to an embodiment, the apparatus can be configured to remove any reference images from the DPB that are classified as not for reference and are no longer output.

[0356] In an embodiment, the apparatus can be configured to: read an entropy coding mode indicator from a data stream, and if the entropy coding mode indicator indicates a context-adaptive variable-length coding mode, decode the prediction residual data from the current AU using the context-adaptive variable-length coding mode; and if the entropy coding mode indicator indicates a context-adaptive binary arithmetic coding mode, decode the prediction residual data from the current AU using the context-adaptive binary arithmetic coding mode.

[0357] According to an embodiment, the apparatus can be configured to derive half-pixel values ​​based on motion vectors in the current AU and using a 6-tap FIR filter, and to derive quarter-pixel values ​​in the referenced reference image by averaging adjacent half-pixel values.

[0358] In an embodiment, the apparatus may be configured to:

[0359] decode the current AU using inter-frame prediction from reference images stored in the DPB to obtain a decoded image, and insert the decoded image into the DPB;

[0360] assign each reference image stored in the DPB to a classification as one of a short-term reference image, a long-term reference image, and an image not used for reference;

[0361] read DPB mode information from the current AU;

[0362] if the DPB mode information indicates a first mode, remove one or more reference images classified as short-term images from the DPB according to a first-in-first-out (FIFO) policy;

[0363] if the DPB mode information indicates a second mode, read memory management control information including at least one command in the current AU, execute the at least one command to change the classification assigned to at least one of the reference images stored in the DPB, and use the classification of the reference images in the DPB to manage the reference images removed from the DPB.

[0364] According to an embodiment, the apparatus may be configured to:

[0365] read from the current AU an indication of whether a decoded image is not used for inter-image prediction;

[0366] if the decoded image is not indicated as not being used for inter-image prediction or is not directly output, insert the decoded image into the DPB, and if the decoded image is indicated as not being used for inter-image prediction, output the decoded image directly without buffering the decoded image in the DPB.

[0367] In an embodiment, the apparatus may be configured to:

[0368] assign a frame index to each reference image classified as a long-term image in the DPB, and

[0369] if the frame index assigned to a predetermined reference image is referenced in the current AU, use the predetermined reference image classified as a long-term image in the DPB as a reference image for reference in the DPB.

[0370] According to an embodiment, the apparatus can be configured to perform one or more of the following operations:

[0371] If at least one command in the current AU is a first command, then

[0372] a reference picture classified as a short-term reference picture in the DPB is reclassified as a picture not used for reference;

[0373] If at least one command in the current AU is a second command, then

[0374] a reference picture classified as a long-term reference picture in the DPB is reclassified as a picture not used for reference;

[0375] If at least one command in the current AU is a third command, then

[0376] a reference picture classified as a short-term picture in the DPB is reclassified as a long-term reference picture, and a frame index is assigned to the reclassified reference picture;

[0377] If at least one command in the current AU is a fourth command, then

[0378] (Page 42, CN 121923845 A) The frame index upper limit is set according to the fourth command, and all reference pictures in the DPB that are classified as long-term pictures and have been assigned frame indexes exceeding the frame index upper limit are reclassified as unused reference pictures.

[0379] If at least one command in the current AU is the sixth command, then

[0380] the current picture is classified as a long-term picture, an unused reference picture, and the frame index is assigned to the reclassified reference picture.

[0381] In an embodiment, the apparatus may be configured to:

[0382] remove any reference pictures from the DPB that are classified as unused reference pictures and are no longer output.

[0383] Next, additional optional details and features of the third variant are described.

[0384] Multiple Reference Lines (MRL): For intra-frame prediction, not only adjacent lines of adjacent samples can be used, but also one of two non-adjacent reference lines can be used as the reference line for intra-frame prediction of luminance samples, which corresponds to two or three lines far from the current block.

[0385] Adaptive MV Resolution (AMVR): Performs selection of MV resolution at the coding unit (CU) level. For inter-frame predicted CUs, the selected MV resolution is indicated and can be one-quarter, one-half, one-time, or four times the luminance sample resolution. If half-luminance sample resolution is selected, the half-sample locations in the block use alternative luminance interpolation filters; that is, different interpolation filters are used when the MV resolution is one-quarter or half.

[0386] History-based MV prediction (HMVP): In addition to spatial and temporal neighborhood MV predictions, new candidate types are added for MV predictions in the merge mode and AMVP candidate lists. A five-entry table is used to build HMVP candidates, which is maintained and updated using a first-in-first-out (FIFO) rule. The motion vector candidate list is generated using spatial and temporal neighbors and HMVP candidates.

[0387] Affine Motion: Luminance uses an affine motion model with CU-level signaling. CU-level affine motion can be a 4-parameter model or a 6-parameter model. The 4-parameter model uses two MVs, which correspond to two control points located at the upper left and upper right corners of the CU, respectively, and the 6-parameter model uses three MVs, which correspond to three control points located at the upper left, lower left, and upper right corners, respectively. When the CU is encoded in affine motion mode, the luminance block of the CU is divided into 4×4 sub-blocks, and the MV at the center sample position of each sub-block is calculated according to the affine motion model and set as the sub-block MV based on the control points. During the calculation process, the sub-block MV is rounded to 1 / 16 of the luminance sample precision, and a set of 6-tap interpolation filters is applied to generate the prediction for each sub-block. For non-affine motion cases, the prediction and encoding of affine motion parameters use a merge mode and an AMVP mode.

[0388] Coefficient Encoding: When encoding coefficient levels, an indicator flag (SigFlag) is first used to specify whether the coefficient level is not 0. When the flag is equal to 1 (coefficient level is not 0), another flag (Gt1-Flag) is indicated, which indicates whether the absolute level is greater than 1. In this case (absolute level is greater than 1), there are two additional flags: the parity-Flag, which specifies the parity of the transform coefficient level; and another flag (Gt3-Flag), which specifies whether the absolute value of the transform coefficient level is greater than 3.

[0389] Video decoding according to the fourth variant is performed by decoding the prediction residual data of intra-frame prediction blocks in the video data stream by using context-adaptive binary arithmetic decoding, and decoding the video from the video data stream by block-based prediction and transform-based residual decoding.

[0390] The coordinates of the position of the last non-zero transform coefficient encountered when traversing the transform coefficients of the transform block along a predetermined scan order in the transform block representing the prediction residual data are decoded, and

[0391] The values ​​of the transform coefficients included between the last non-zero transform coefficient and the first scanned transform coefficient and ranked along the predetermined scan order are decoded sequentially, and the specification 39 / 55 pages 43 CN 121923845 A

[0392] Depending on the intra-prediction mode of the intra-prediction block, a predetermined scan order is selected among the diagonal scan order, horizontal scan order and vertical scan order by utilizing a mapping that maps each of the various intra-prediction modes to the corresponding scan order in the diagonal scan order, horizontal scan order and vertical scan order.

[0393] The video coding according to the invention according to the fourth variant is performed by encoding the prediction residual data of the intra-prediction block in the video data stream using context-adaptive binary arithmetic coding, and encoding the video into the video data stream by block-based prediction and transform-based residual coding.

[0394] The coordinates of the position where the last non-zero transform coefficient is encountered when traversing the transform coefficients of the transform block along a predetermined scan order in the transform block representing the prediction residual data are encoded, and

[0395] The values ​​of the transform coefficients included between the last non-zero transform coefficient and the first scanned transform coefficient and ranked along the predetermined scan order are encoded sequentially, and

[0396] Depending on the intra-prediction mode of the intra-prediction block, the predetermined scan order is selected among the diagonal scan order, horizontal scan order and vertical scan order by utilizing a mapping that maps each of the various intra-prediction modes to the corresponding scan order in the diagonal scan order, horizontal scan order and vertical scan order.

[0397] The present invention includes a video encoder, a video decoder, a method for decoding video according to the above-described video decoding variant, a method for encoding video, and a video data stream obtained by a corresponding encoding method according to the above-described fourth variant of the video.

[0398] Hereinafter, additional optional details and features of specific video encoding and video decoding variants of embodiments of the present invention are described.

[0399] Embodiments of the fourth variant may conform to H.265 / HEVC.

[0400] PCM

[0401] Bypasses prediction, transform, quantization, and entropy coding, and samples are directly represented by a predefined number of bits. When PCM is enabled, the number of bits used to represent luminance and chrominance samples (minus 1) is indicated, respectively. In addition, the minimum block size and maximum block size (as the difference from the minimum value) of the 2Nx2N block most suitable for using PCM are also indicated. When parsing a 2Nx2N block, if the size of the block is between the described minimum and maximum values, a flag indicates whether PCM mode is used. When using PCM mode, PCM samples are byte-aligned, and the luminance samples of a block are directly represented / parsed in raster scan with the indicated PCM luminance bit length, and then the chrominance samples of a PCM block are represented / parsed in raster scan with the indicated PCM chrominance bit length (which is the first half of the chrominance sample Cb and the remaining Cr samples). Furthermore, the CABAC decoding engine terminates each time the PCM flag is parsed and initialized after decoding the PCM syntax.

[0402] Intra-mode dependent coefficient scanning order

[0403] The scanning order of coefficients in an intra-coded block changes dynamically based on the transform block size and intra-mode.Each 4×4 sub-block / region of the transform block is scanned (e.g., one coefficient region is used for a 4×4 transform block size, four coefficient regions are used for an 8×8 transform block, and 16 regions are used for a 16×16 transform block size). The choice of scanning method for the intra-mode depends on the transform block size. For 16x16 and 32x32 transform block sizes, scanning is performed diagonally (starting from 0,0->1,0->0,1->2,0->1,1->0,2->3,0, and so on: for example, each step decreases the y value and increases the x value by 1, and when y equals 0, it restarts from y with a value of x+1 and sets x to 0, while starting from a value of 0,0). For 4×4 (luminance or chrominance) and 8×8 (luminance) transform block sizes, the coefficient scanning order depends on the intra-mode associated with the predicted intra-direction of the adjacent blocks. Vertical scanning is used when the prediction direction is close to horizontal, and horizontal scanning is used when the prediction direction is close to vertical. For other prediction directions, diagonal vertical scanning is used.

[0404] Transform Quantization Bypass Mode Specification 40 / 55 pages 44 CN 121923845 A

[0405] For each coding unit or coding block, it can be indicated whether to bypass both transform and quantization, indicating that the residual signal from inter-frame or intra-frame picture prediction is directly entropy-coded for the block. In this mode, the in-loop filter is skipped.

[0406] Additional Embodiments

[0407] Hereinafter, embodiments of the invention will be described again in more general terms. The following embodiments may optionally be combined with any of the features described above.

[0408] In the following additional embodiments, the following option may be applicable to all embodiments: the data stream may be a video data stream or an audio data stream. The means for decoding the data stream may be a means for decoding video from a video data stream or a means for decoding audio signals from an audio data stream. The means for encoding the data stream can be means for encoding video into a video data stream or means for encoding audio signals into an audio data stream. The payload packet can be an encoded video payload packet carrying video data (e.g., encoded video data). The term supplementary information packet can be used equivalently to the term supplementary information payload packet. The expressions “checking the credibility of the data stream” and “verifying the data stream” are used interchangeably.

[0409] The different aspects are composable, that is, any feature defined for any aspect can be combined with any other aspect.

[0410] 1. An apparatus 20 for decoding a data stream 14, the data stream including a plurality of payload packets 16 carrying payload data, and further including supplementary information packets 18 carrying supplementary information messages 19, wherein the apparatus is configured to:

[0411] derive an indication 51 from the data stream, the indication 51 indicating whether one or more supplementary information messages should be considered to verify the data stream 40;

[0412] if the indication 51 indicates that one or more supplementary information messages should be considered to verify the data stream, then include one or more supplementary information messages into a predetermined portion of the data stream to determine the predetermined portion 13;

[0413] obtain a digital signature 43 for verifying the predetermined portion 13 from the indication in the data stream.

[0414] 2. The apparatus according to embodiment 1, wherein the apparatus is configured to verify the predetermined portion by checking whether the predetermined portion in the data stream matches the digital signature.

[0415] 3. The apparatus according to any one of Embodiment 1 or 2, wherein the apparatus is configured to provide a predetermined portion and a digital signature to verify the predetermined portion and the digital signature.

[0416] 4. An apparatus 20 for decoding a data stream 14, wherein the apparatus is configured to check the credibility of the data stream, the data stream including a plurality of payload packets 16 carrying payload data, and further including supplementary information packets 18 carrying supplementary information messages 19, wherein the apparatus is configured to:

[0417] derive an indication 51 from the data stream, the indication 51 indicating whether to consider the supplementary information messages to verify the data stream;

[0418] if the indication 51 indicates to consider the supplementary information messages to verify the data stream, determine a predetermined portion 13 by including one or more supplementary information messages 19 in a predetermined portion 13 to be verified in the data stream;

[0419] obtain a digital signature 43 based on the data stream 14;

[0420] check whether the predetermined portion 13 in the data stream matches the digital signature 43.

[0421] 5. The apparatus according to any one of embodiments 1 to 4, wherein checking whether a predetermined portion 13 in the data stream matches the digital signature 43 includes:

[0422] applying a hash function to the predetermined portion 13 to obtain a hash value; and

[0423] checking whether the hash value matches the digital signature 43. Specification 41 / 55 pages 45 CN 121923845 A

[0424] 6. The apparatus according to embodiment 5, wherein checking whether the hash value matches the digital signature 43 includes:

[0425] forming a verification string 48 based on the hash value;

[0426] decrypting the digital signature 43 to obtain a check value; and

[0427] checking whether the verification string 48 matches the check value.

[0428] 7. The apparatus according to any one of embodiments 1 to 6 is configured to:

[0429] determine the predetermined portion 13 by including one or more supplementary information messages in the predetermined portion 13 if the indication indicates that one or more supplementary information messages should be considered to verify the predetermined portion 13.

[0430] 8. The apparatus according to any one of embodiments 1 to 7 is configured to include all supplementary information packets carrying any supplementary information messages to be included in the predetermined portion 13 in the predetermined portion 13.

[0431] 9. The apparatus according to any one of embodiments 1 to 8 is configured to: determine whether to include supplementary information packets in the supplementary information packets in the predetermined portion 13 based on the result of checking whether the supplementary information payload packet includes supplementary information messages of any supplementary information message type in a set of supplementary information message types.

[0432] 10. The apparatus according to any one of embodiments 1 to 9, wherein the indication is a flag having a first state and a second state, and wherein the apparatus is configured to determine a predetermined portion 13 in a data stream by:

[0433] if the flag has a first state, determining one or more supplementary information messages to be included in the predetermined portion 13 based on a set of predefined supplementary information messages in the supplementary information messages; and

[0434] if the flag has a second state, avoiding including any supplementary information messages in the predetermined portion 13.

[0435] 11. The apparatus according to embodiment 10, wherein the set of predefined supplementary information messages includes all supplementary information messages in the supplementary information group that refer to a plurality of payload groups 16.

[0436] 12. The apparatus according to embodiment 10, wherein the set of predefined supplementary information messages includes all supplementary information messages of any type in a set of supplementary information message types.

[0437] 13. The apparatus according to any one of embodiments 1 to 12 is further configured to:

[0438] if the indication indicates that one or more supplementary information messages should be considered to verify the data stream, then derive an identifier of one or more supplementary information messages to be included in the predetermined portion 13 from the data stream.

[0439] 14. The apparatus according to embodiment 13 is configured to: if the indication indicates that one or more supplementary information messages should be considered to verify the data stream, then derive a syntax element from the data stream, the syntax element identifying a set of supplementary information message types to be included in the predetermined portion 13 from the data stream.

[0440] 15. The apparatus according to any one of embodiments 1 to 14 is configured to derive a syntax element from the data stream, the syntax element indicating whether one or more supplementary information messages should be considered to verify the data stream and signaling an identifier of one or more supplementary information messages to be included in the predetermined portion 13.

[0441] 16. The apparatus according to embodiment 15, wherein the syntax element has a second state and a plurality of first states, and wherein the apparatus is configured to:

[0442] if the syntax element has one of the first states, determine one or more supplementary information messages to be included in the predetermined portion 13 based on the state of the syntax element, and

[0443] if the syntax element has a second state, avoid including any of the one or more supplementary information messages in the predetermined portion 13.

[0444] 17. The apparatus according to embodiment 13, configured to: if the indication indicates that one or more supplementary information messages should be considered to verify the data stream, derive a first syntax element and a corresponding number of second syntax elements from the data stream, the first syntax element indicating a count of supplementary information message types to be considered for the predetermined portion 13, each second syntax element identifying a corresponding supplementary information message type to be included in the predetermined portion 13.

[0445] 18. The apparatus according to Embodiment 7 is further configured to:

[0446] determine whether to include the supplementary information group in the supplementary information group in the predetermined portion 13 based on the result of checking whether there is a prefix supplementary information group before the supplementary information group, the prefix supplementary information group indicating that subsequent supplementary information groups are included in the predetermined portion 13.

[0447] 19. The apparatus according to Embodiment 7 is further configured to:

[0448] determine whether to include the supplementary information group in the supplementary information group in the predetermined portion 13 based on the result of checking whether the supplementary information group includes a supplementary information message, the supplementary information message indicating that the supplementary information group is included in the predetermined portion 13.

[0449] 20. The apparatus according to Embodiment 7 is further configured to:

[0450] determine whether to include the supplementary information message in the supplementary information message in the predetermined portion based on the result of checking whether there is a prefix supplementary information message before the supplementary information message, the prefix supplementary information message indicating that subsequent supplementary information messages are included in the predetermined portion 13.

[0451] 21. The apparatus according to Embodiment 7 is further configured to:

[0452] determine whether to include the supplementary information group in the supplementary information group in a predetermined portion based on the result of checking whether the supplementary information group includes a nested supplementary information message, the nested supplementary information message indicating that the supplementary information group is included in the predetermined portion 13.

[0453] 22. The apparatus according to Embodiment 7 is further configured to:

[0454] determine whether to include the supplementary information message in the supplementary information message in a predetermined portion based on the result of checking whether the supplementary information message is included in a nested supplementary information message, the nested supplementary information message indicating that the supplementary information message is included in the predetermined portion 13.

[0455] 23. The apparatus according to any one of embodiments 1 to 22 is configured to: when determining the predetermined portion 13,

[0456] decide whether to include the supplementary information group in the supplementary information group in the predetermined portion based on the result of checking whether the supplementary information group includes any supplementary information message of one type among a set of supplementary information message types to be included in the predetermined portion 13.

[0457] 24. The apparatus according to any one of embodiments 1 to 23 is configured to: when determining the predetermined portion,

[0458] decide whether to include the supplementary information group in the supplementary information group in the predetermined portion based on the result of checking whether a first supplementary information message of one or more supplementary information messages carried in the supplementary information group is of one type among a set of supplementary information message types to be included in the predetermined portion 13.

[0459] 25. The apparatus according to any one of embodiments 1 to 24 is configured to:

[0460] derive a plurality of portions from a data stream, capable of validating the data stream on a per-part basis, wherein the plurality of portions includes a predetermined portion 13; and

[0461] assign each of the payload groups 16 to one of the plurality of portions.

[0462] 26. The apparatus according to embodiment 25 is configured to: when determining the predetermined portion 13,

[0463] if the indication indicates that one or more supplementary information messages should be considered to validate the data stream, include any type of supplementary information message having a set of supplementary information message types in the predetermined portion 13, wherein the predetermined portion 13 has a predetermined ranking within an order defined among the plurality of portions.

[0464] 27. The apparatus according to embodiment 26, wherein the predetermined ranking is the highest ranking within an order defined among the plurality of portions. Specification page 43 / 55 47 CN 121923845 A

[0465] 28. The apparatus according to embodiment 25 is configured to: when determining a predetermined portion 13,

[0466] include a payload packet 16 assigned to the predetermined portion 13 in the predetermined portion 13; and

[0467] if the indication indicates that one or more supplementary information messages are to be considered to verify the data stream, then supplementary information messages of the data stream having any type of a set of supplementary information message types and associated with the payload packet 16 assigned to the predetermined portion 13 are included in the predetermined portion 13.

[0468] 29. The apparatus according to embodiment 25 or 28, wherein each of the supplementary information packets is associated with one of the payload packets 16, and wherein the apparatus is configured to assign each of the supplementary information packets carrying supplementary information of any type of a set of supplementary information types to a portion of one or more portions that has a payload packet associated with the corresponding supplementary information packet.

[0469] 30. The apparatus according to embodiment 25, 28, or 29 is configured such that, upon determining a predetermined portion 13,

[0470] if the indication indicates that supplementary information should be considered to verify the data stream, a decision is made based on the indication in the data stream between the following operations:

[0471] including all supplementary information of any type in the data stream having a set of supplementary information types in the predetermined portion 13, wherein the predetermined portion 13 has a predetermined ranking, such as a highest ranking, within an order defined among multiple portions, and

[0472] including all supplementary information of any type in the data stream having a set of supplementary information types and associated with the payload group 16 assigned to the predetermined portion 13 in the predetermined portion 13.

[0473] 31. The apparatus according to embodiment 25, wherein each of the supplementary information messages is associated with one of the payload packets 16, and wherein the apparatus is configured to assign each of the supplementary information messages of any type in a set of supplementary information message types of the data stream to one of a plurality of portions, the one portion being associated with another of these portions, the other of these portions being associated with the payload packet 16 associated with the corresponding supplementary information message.

[0474] 32. The apparatus according to embodiment 25, wherein each of the supplementary information packets is associated with one of the payload packets 16, and wherein the apparatus is configured to assign each of the supplementary information packets of the data stream carrying any type in a set of supplementary information information types to one of a plurality of portions, the one portion being associated with another of these portions, the other of these portions being associated with the payload packet 16 associated with the corresponding supplementary information packet.

[0475] 33. The apparatus according to embodiment 31 or 32 is configured to derive a syntax element from a data stream, the syntax element indicating a count of the plurality of parts, the syntax element having a value, wherein the apparatus is configured to:

[0476] infer that the count of the plurality of parts corresponds to the value plus 1 and multiplied by 2.

[0477] 34. The apparatus according to embodiment 31 or 32 is configured to derive a syntax element from a data stream, the syntax element indicating a count of the plurality of parts, the syntax element having a value, wherein the apparatus is configured to:

[0478] infer that the count of the plurality of parts corresponds to the value plus 1.

[0479] 35. The apparatus according to embodiment 33 or 34, wherein one of the parts and the other part are consecutive to each other within a hierarchical order defined among the plurality of parts, for example, in a hierarchical order, one of the parts follows the other part.

[0480] 36. The apparatus according to embodiment 33 or 34, wherein the position index of the one portion within the hierarchical order defined among these portions corresponds to the position index of the other portion within the hierarchical order plus half the count of the plurality of portions. Specification 44 / 55 pages 48 CN 121923845 A

[0481] 37. The apparatus according to any one of embodiments 1 to 36, wherein each of the supplementary information messages is associated with one of the payload groups 16.

[0482] 38. The apparatus according to any one of embodiments 1 to 37, wherein each of the supplementary information messages is associated with an associated payload group in the payload group 16, the associated payload group being part of the same picture unit or the same access unit as the supplementary information group.

[0483] 39. The apparatus according to any one of embodiments 1 to 38 is configured to: when determining the predetermined portion 13, check whether one or more supplementary information messages to be included in the predetermined portion 13 include multiple instances of the same supplementary information message; and

[0484] if one or more supplementary information messages to be included in the predetermined portion 13 include multiple instances of the same supplementary information message, then only one instance of the multiple instances of the same supplementary information message is included.

[0485] 40. The apparatus according to embodiment 39 is configured to: when checking whether one or more supplementary information messages to be included in the predetermined portion 13 include multiple instances of the same supplementary information message, perform a check on a single image unit or access unit.

[0486] 41. The apparatus according to embodiment 39 is configured to position one instance of the multiple instances of the same supplementary information message at a predetermined position within the predetermined portion 13.

[0487] 42. The apparatus according to embodiment 41, wherein the predetermined position is defined relative to:

[0488] the start or end position of the image unit included in the predetermined portion 13, or

[0489] relative to a predetermined payload group included in the predetermined portion 13, such as a predetermined payload group or a predetermined supplementary information group.

[0490] 43. The apparatus according to any one of embodiments 39 to 42, configured to: locate one or more supplementary information messages to be included in the predetermined portion 13 according to a predefined order among one or more supplementary information messages within the predetermined portion 13.

[0491] 44. An apparatus 10 for encoding a data stream, wherein the apparatus is configured to: enable the data stream to be trusted; and to encode a plurality of payload packets 16 carrying payload data into the data stream, and also to encode supplementary information packets 18 carrying supplementary information messages into the data stream, wherein the apparatus is configured to:

[0492] insert an instruction 51 into the data stream 23, the instruction 51 being used to indicate whether one or more supplementary information messages should be considered to verify the data stream;

[0493] if the instruction indicates that one or more supplementary information messages should be considered to verify the data stream, then determining a predetermined portion 13 30' by including one or more of the supplementary information messages into a predetermined portion 13 in the data stream that would enable trusted checking;

[0494] obtaining a digital signature 43 41' based on the predetermined portion 13.

[0495] 45. The apparatus according to embodiment 44 is configured to: upon obtaining digital signature 43,

[0496] apply a hash function to a predetermined portion 13 to obtain a hash value; and

[0497] obtain digital signature 43 based on the hash value.

[0498] 46. The apparatus according to embodiment 45 is configured to: upon obtaining digital signature 43,

[0499] form a verification string 48 based on the hash value; and

[0500] sign the verification string 48 to obtain digital signature 43.

[0501] 47. The apparatus according to any one of embodiments 44 to 46 is configured to: Specification 45 / 55 pages 49 CN 121923845 A

[0502] if the instruction indicates that one or more supplementary information messages are to be considered to verify predetermined portion 13, then predetermined portion 13 is determined by including one or more supplementary information groups into predetermined portion 13.

[0503] 48. The apparatus according to any one of embodiments 44 to 47 is configured to include all supplementary information groups in a predetermined portion 13, the supplementary information group carrying any supplementary information message to be included in the predetermined portion 13.

[0504] 49. The apparatus according to any one of embodiments 44 to 48 is configured to: determine whether to include supplementary information groups in a supplementary information group in the predetermined portion 13 based on the result of checking whether a supplementary information group includes supplementary information messages of any supplementary information message type in a set of supplementary information message types.

[0505] 50. The apparatus according to any one of embodiments 44 to 49, wherein the indication is a flag having a first state and a second state, and wherein the apparatus is configured to decide between operations:

[0506] setting the flag to the first state and determining one or more supplementary information messages to be included in the predetermined portion 13 based on a set of predefined supplementary information messages in the supplementary information messages; and

[0507] setting the flag to the second state and avoiding including any supplementary information messages in the predetermined portion 13.

[0508] 51. The apparatus according to embodiment 50, wherein the set of predefined supplementary information messages includes all supplementary information messages in the supplementary information group that refer to a plurality of payload groups 16.

[0509] 52. The apparatus according to embodiment 50, wherein the set of predefined supplementary information messages includes all supplementary information messages of any type in a set of supplementary information message types.

[0510] 53. The apparatus according to any one of embodiments 44 to 52 is further configured to:

[0511] If the indication indicates that one or more supplementary information messages should be considered to verify the data stream, then an identifier of one or more supplementary information messages to be included in the predetermined portion 13 is inserted into the data stream.

[0512] 54. The apparatus according to embodiment 53 is configured to: If the indication indicates that one or more supplementary information messages should be considered to verify the data stream, then a syntax element is inserted into the data stream, the syntax element identifying a set of supplementary information message types to be included in the predetermined portion 13 of the data stream.

[0513] 55. The apparatus according to any one of embodiments 44 to 54 is configured to insert a syntax element into the data stream, the syntax element indicating whether one or more supplementary information messages should be considered to verify the data stream, and the syntax element signaling an identifier of one or more supplementary information messages to be included in the predetermined portion 13.

[0514] 56. The apparatus according to embodiment 55, wherein the syntax element has a second state and a plurality of first states, and wherein the apparatus is configured to decide between operations:

[0515] setting the syntax element to one of the first states according to one or more supplementary information messages to be included in the predetermined portion 13; and

[0516] setting the syntax element to the second state and avoiding including any supplementary information message in the predetermined portion 13.

[0517] 57. The apparatus according to embodiment 53 is configured to: if the indication indicates that one or more supplementary information messages should be considered to verify the data stream, insert a first syntax element and a corresponding number of second syntax elements into the data stream, the first syntax element indicating a count of supplementary information message types to be considered for the predetermined portion 13, each second syntax element identifying a corresponding supplementary information message type to be included in the predetermined portion 13.

[0518] 58. The apparatus according to embodiment 53 is further configured to:

[0519] determine whether to include supplementary information packets in the predetermined portion based on the result of checking whether there is a prefix supplementary information packet before the supplementary information packet, the prefix supplementary information packet indicating that subsequent supplementary information packets should be included in the predetermined portion 13. Specification page 46 / 55 50 CN 121923845 A

[0520] 59. The apparatus according to embodiment 47 is further configured to:

[0521] determine whether to include the supplementary information group in the supplementary information group in the predetermined portion 13 based on the result of checking whether the supplementary information group includes a supplementary information message, the supplementary information message indicating that the supplementary information group is included in the predetermined portion 13.

[0522] 60. The apparatus according to embodiment 47 is further configured to:

[0523] determine whether to include the supplementary information message in the supplementary information message in the predetermined portion based on the result of checking whether there is a prefix supplementary information message before the supplementary information message, the prefix supplementary information message indicating that the subsequent supplementary information message is included in the predetermined portion 13.

[0524] 61. The apparatus according to embodiment 47 is further configured to:

[0525] determine whether to include the supplementary information group in the supplementary information group in the predetermined portion based on the result of checking whether the supplementary information group includes a nested supplementary information message, the nested supplementary information message indicating that the supplementary information group is included in the predetermined portion 13.

[0526] 62. The apparatus according to embodiment 47 is further configured to:

[0527] determine whether to include the supplementary information message in the supplementary information message in a predetermined portion based on the result of checking whether the supplementary information message is included in a nested supplementary information message, the nested supplementary information message indicating that the supplementary information group is included in the predetermined portion 13.

[0528] 63. The apparatus according to any one of embodiments 44 to 62 is configured to: when determining the predetermined portion 13,

[0529] determine whether to include the supplementary information group in the supplementary information group in the predetermined portion based on the result of checking whether the supplementary information group includes any supplementary information message of one type from a set of supplementary information message types to be included in the predetermined portion 13.

[0530] 64. The apparatus according to any one of embodiments 44 to 63 is configured to: when determining the predetermined portion 13,

[0531] decide whether to include the supplementary information group in the supplementary information group in the predetermined portion based on the result of checking whether a first supplementary information message in one or more supplementary information messages carried in the supplementary information group is one of a set of supplementary information message types to be included in the predetermined portion 13.

[0532] 65. The apparatus according to any one of embodiments 44 to 64 is configured to provide supplementary information groups in such a way that no supplementary information group includes both a supplementary information message of one type of supplementary information group type to be included in the predetermined portion 13 and a supplementary information message of a type not included in that set of types.

[0533] 66. The apparatus according to any one of embodiments 44 to 65 is configured to:

[0534] instruct a plurality of portions in a data stream capable of validating the data stream on a per-part basis, wherein the plurality of portions includes a predetermined portion 13; and

[0535] assign each of the payload groups 16 to one of the plurality of portions.

[0536] 67. The apparatus according to embodiment 66 is configured to: when determining the predetermined portion 13,

[0537] if the instruction indicates that one or more supplementary information messages should be considered to validate the data stream, include any type of supplementary information message having a set of supplementary information message types in the predetermined portion 13, wherein the predetermined portion 13 has a predetermined ranking within an order defined among the plurality of portions.

[0538] 68. The apparatus according to embodiment 67, wherein the predetermined ranking is the highest ranking within an order defined among the plurality of portions. Specification page 47 / 55 51 CN 121923845 A

[0539] 69. The apparatus according to embodiment 66 is configured to: when determining a predetermined portion 13,

[0540] include a payload packet 16 assigned to the predetermined portion 13 in the predetermined portion 13; and

[0541] if the indication indicates that one or more supplementary information messages are to be considered to verify the data stream, then supplementary information messages of the data stream having any type of a set of supplementary information message types and associated with the payload packet 16 assigned to the predetermined portion 13 are included in the predetermined portion 13.

[0542] 70. The apparatus according to embodiment 66 or 67, wherein each of the supplementary information packets is associated with one of the payload packets 16, and wherein the apparatus is configured to assign each of the supplementary information packets carrying supplementary information of any type of a set of supplementary information types to a portion of one or more portions that has a payload packet associated with the corresponding supplementary information packet.

[0543] 71. The apparatus according to embodiment 66, 67, or 68 is configured such that, upon determining a predetermined portion 13,

[0544] if the indication indicates that supplementary information should be considered to verify the data stream, an indication is inserted into the data stream, the indication distinguishing between:

[0545] including all supplementary information of any type in the data stream having a set of supplementary information types in the predetermined portion 13, wherein the predetermined portion 13 has a predetermined ranking, such as a highest ranking, within an order defined among multiple portions, and

[0546] including all supplementary information of any type in the data stream having a set of supplementary information types and associated with the payload group 16 assigned to the predetermined portion 13 in the predetermined portion 13.

[0547] 72. The apparatus according to embodiment 66, wherein each of the supplementary information messages is associated with one of the payload packets 16, and wherein the apparatus is configured to assign each of the supplementary information messages of any type in a set of supplementary information message types of the data stream to one of a plurality of portions, the one portion being associated with another of these portions, the other of these portions being associated with the payload packet 16 associated with the corresponding supplementary information message.

[0548] 73. The apparatus according to embodiment 33, wherein each of the supplementary information packets is associated with one of the payload packets 16, and wherein the apparatus is configured to assign each of the supplementary information packets of the data stream carrying any type in a set of supplementary information information types to one of a plurality of portions, the one portion being associated with another of these portions, the other of these portions being associated with the payload packet 16 associated with the corresponding supplementary information packet.

[0549] 74. The apparatus according to embodiment 72 or 73, configured to insert a syntax element into a data stream, the syntax element indicating a count of the plurality of parts, the syntax element having a value, wherein the apparatus is configured to:

[0550] set the value to the count of the plurality of parts divided by 2 and then subtracted by 1.

[0551] 75. The apparatus according to embodiment 72 or 73, configured to derive a syntax element from a data stream, the syntax element indicating a count of the plurality of parts, the syntax element having a value, wherein the apparatus is configured to:

[0552] set the value to the count of the plurality of parts minus 1.

[0553] 76. The apparatus according to embodiment 74 or 75, wherein one of the parts and the other part are consecutive to each other within a hierarchical order defined among the plurality of parts, for example, in a hierarchical order, one of the parts follows the other part.

[0554] 77. The apparatus according to embodiment 74 or 75, wherein the position index of the one part within the hierarchical order defined among the plurality of parts corresponds to the position index of the other part within the hierarchical order plus half the count of the plurality of parts. Specification 48 / 55 pages 52 CN 121923845 A

[0555] 78. The apparatus according to any one of embodiments 44 to 77, wherein each supplementary information message is associated with one of the payload groups 16.

[0556] 79. The apparatus according to any one of embodiments 44 to 78, wherein each of the supplementary information messages is associated with an associated payload group in the payload group 16, the associated payload group being part of the same picture unit or the same access unit as the supplementary information group.

[0557] 80. The apparatus according to any one of embodiments 44 to 79 is configured to: when determining a predetermined portion 13, check whether one or more supplementary information messages to be included in the predetermined portion 13 include multiple instances of the same supplementary information message; and

[0558] if one or more supplementary information messages to be included in the predetermined portion 13 include multiple instances of the same supplementary information message, then only one instance of the multiple instances of the same supplementary information message is included.

[0559] 81. The apparatus according to embodiment 80 is configured to: when checking whether one or more supplementary information messages to be included in the predetermined portion 13 include multiple instances of the same supplementary information message, perform a check on a single image unit or access unit.

[0560] 82. The apparatus according to embodiment 80 is configured to position one instance of the multiple instances of the same supplementary information message at a predetermined position within the predetermined portion 13.

[0561] 83. The apparatus according to embodiment 82, wherein the predetermined position is defined relative to:

[0562] the start or end position of the image unit included in the predetermined portion 13, or

[0563] relative to a predetermined payload group included in the predetermined portion 13, such as a predetermined payload group or a predetermined supplementary information group.

[0564] 84. The apparatus according to any one of embodiments 80 to 83, configured to: locate one or more supplementary information messages to be included in the predetermined portion 13 according to a predefined order among one or more supplementary information messages within the predetermined portion 13.

[0565] 85. The apparatus according to any one of embodiments 44 to 84, configured to: if a supplementary information group includes multiple instances of the same supplementary information group, insert the multiple instances into the data stream without interleaving with another supplementary information group.

[0566] 86. The apparatus according to any one of embodiments 44 to 85 is configured to provide a data stream in such a way that if supplementary information messages of the data stream include multiple instances of a first supplementary information message and multiple instances of a second supplementary information message, then each instance of the first supplementary information message and the second supplementary information message is included in one of the supplementary information packets.

[0567] 87. A method for decoding a data stream 14, the data stream including multiple payload packets 16 carrying payload data, and further including supplementary information packets 18 carrying supplementary information messages 19, wherein the method includes:

[0568] deriving an indication 21 51 from the data stream, the indication 51 indicating whether one or more supplementary information messages should be considered to verify the data stream 40;

[0569] if the indication 51 indicates that one or more supplementary information messages should be considered to verify the data stream, then including one or more supplementary information messages into a predetermined portion of the data stream to determine a predetermined portion 13;

[0570] obtaining a digital signature 43 21 for verifying the predetermined portion 13 from the indication in the data stream.

[0571] 88. A method for decoding a data stream 14, wherein the method includes checking the credibility of the data stream, the data stream including a plurality of payload packets 16 carrying payload data, and further including supplementary information packets 18, the supplementary information packets carrying supplementary information messages 19, wherein the method includes:

[0572] deriving an indication 51 from the data stream, the indication 51 indicating whether to consider the supplementary information messages to verify the data stream;

[0573] if the indication 51 indicates to consider the supplementary information messages to verify the data stream, determining a predetermined portion 13 to be verified by including one or more supplementary information messages 19 in the data stream;

[0574] obtaining a digital signature 43 based on the data stream 14;

[0575] checking whether the predetermined portion 13 in the data stream matches the digital signature 43.

[0576] 89. A method for encoding a data stream, wherein the method includes: enabling the data stream to be trusted; and encoding a plurality of payload packets 16 carrying payload data into the data stream, and also encoding supplementary information packets carrying supplementary information messages into the data stream, wherein the method includes:

[0577] inserting an indication 23 into the data stream, the indication indicating whether one or more supplementary information messages should be considered to verify the data stream;

[0578] if the indication indicates that one or more supplementary information messages should be considered to verify the data stream, determining a predetermined portion 13 30' by including one or more of the supplementary information messages into a predetermined portion 13 in the data stream that would enable trusted checking;

[0579] obtaining a digital signature 43 based on the predetermined portion 13.

[0580] 90. A data stream having data signals encoded therein, the data stream being capable of reliability checking, the data stream comprising:

[0581] a plurality of payload packets 16 carrying payload data, and further comprising supplementary information packets carrying supplementary information messages; and

[0582] an indication for indicating whether to consider one or more supplementary information messages to check the reliability of the data stream.

[0583] 91. The data stream according to embodiment 90, wherein the data stream is provided using the method of embodiment 89.

[0584] 92. An apparatus according to any one of embodiments 1 to 43, wherein a data stream has video encoded therein, and wherein the apparatus is configured to decode the video from the data stream by block-based prediction and transform-based residual decoding in such a way that:

[0585] the predictive residual data of the residual blocks are decoded from the data stream using context-adaptive variable-length decoding by using:

[0586] a first syntax element and a tail-1 quantity, the first syntax element indicating the total number of non-zero transform coefficients in the transform block representing the residual block, the tail-1 quantity indicating the number of non-zero transform coefficients having an absolute value of 1 when traversing the coefficients in scan order,

[0587] one or more second syntax elements indicating the sign of the non-zero transform coefficients having an absolute value of 1 when traversing the coefficients in scan order,

[0588] one or more third syntax elements indicating the value of the non-zero transform coefficients other than the number of non-zero transform coefficients having an absolute value of 1 when traversing the coefficients in scan order,

[0589] The fourth syntax element indicates the total number of zero-value transformation coefficient levels in the transform block, starting from the first non-zero transform coefficient encountered in the scan order, and

[0590] one or more fifth syntax elements indicate the position of the non-zero transform coefficients in the scan order by indicating the number of consecutive zero-value transform coefficients in the scan order between consecutive non-zero transform coefficients encountered in the scan order.Specification page 50 / 55 54 CN 121923845 A

[0591] 93. The apparatus according to any one of embodiments 1 to 43, wherein the data stream has video encoded therein, and wherein the apparatus is configured to decode the video from the data stream by block-based prediction and transform-based residual decoding in the following manner:

[0592] Decoding the prediction residual data of the residual block from the data stream using context-adaptive binary arithmetic decoding in the following manner:

[0593] Decoding the saliency map, which indicates the position of non-zero transform coefficients in the transform block representing the residual block, by decoding a saliency flag in a forward scan traversing the transform coefficients of the transform block, the saliency flag indicating whether the non-zero transform coefficient is at the current position, and if so, and if the current position is not the last position in the forward scan, decoding a last saliency flag indicating whether the non-zero transform coefficient at the current position is the last non-zero transform coefficient in the forward scan order, and

[0594] The non-zero transform coefficients are decoded sequentially in a reverse scan order, which is the opposite of the forward scan order.

[0595] 94. The apparatus according to any one of embodiments 1 to 43, wherein the data stream has video encoded therein, and wherein the apparatus is configured to decode the video from the data stream by means of block-based predictive coding and transform-based residual decoding:

[0596] Decoding the prediction residual data of an intra-prediction block from the data stream by means of context-adaptive binary arithmetic decoding:

[0597] Decoding the coordinates of the position of the last non-zero transform coefficient encountered in the transform coefficients of the transform block, which represents the prediction residual data, while traversing the transform coefficients of the transform block along a predetermined scan order, and

[0598] Sequentially decoding the values ​​of transform coefficients included between the last non-zero transform coefficient and the transform coefficients scanned first and ordered along the predetermined scan order, and

[0599] Selecting a predetermined scan order among the diagonal scan order, horizontal scan order and vertical scan order by means of a mapping that maps each of the various intra-prediction modes to the corresponding scan order among the diagonal scan order, horizontal scan order and vertical scan order, depending on the intra-prediction mode of the intra-prediction block.

[0600] 95. The apparatus according to any one of embodiments 1 to 43, wherein the data stream has video encoded therein, and wherein the apparatus is configured to decode the video from the data stream by means of block-based predictive coding and transform-based residual decoding:

[0601] Decoding the predictive residual data of the residual block from the data stream by using context-adaptive binary arithmetic decoding of the quantization index of the transform coefficients of the transform block representing the residual block and sequential dequantization of the quantization index, wherein the value of the current transform coefficient depends on the parity of the quantization index of the previous quantization index according to the quantization index.

[0602] 96. The apparatus according to any one of embodiments 44 to 86, wherein the apparatus is an encoder configured to encode video into a data stream by means of block-based predictive coding and transform-based residual coding,

[0603] using context-adaptive variable-length coding to encode predictive residual data of residual blocks into the data stream by means of:

[0604] a first syntax element and a tail-1 quantity, the first syntax element indicating the total number of non-zero transform coefficients in the transform block representing the residual block, the tail-1 quantity indicating the number of non-zero transform coefficients having an absolute value of 1 when traversing the coefficients in scan order,

[0605] one or more second syntax elements indicating the sign of the non-zero transform coefficients having an absolute value of 1 when traversing the coefficients in scan order,

[0606] one or more third syntax elements indicating the value of the non-zero transform coefficients other than the number of non-zero transform coefficients having an absolute value of 1 when traversing the coefficients in scan order.

[0607] A fourth syntax element indicates the total number of zero-value transformation coefficient levels in the transform block, starting from the first non-zero transform coefficient encountered in the scan order, and

[0608] one or more fifth syntax elements indicate the position of the non-zero transform coefficients in the scan order by indicating the number of consecutive zero-value transform coefficients in the scan order between consecutive non-zero transform coefficients encountered in the scan order.

[0609] 97. The apparatus according to any one of embodiments 44 to 86, wherein the apparatus is an encoder configured to encode video into a data stream by means of block-based predictive coding and transform-based residual coding,

[0610] encoding the predictive residual data of the residual block into the data stream by means of context-adaptive binary arithmetic coding:

[0611] encoding a saliency map that indicates the position of non-zero transform coefficients in a transform block representing the residual block by: encoding a saliency flag indicating whether a non-zero transform coefficient is located at the current position during a forward scan of the transform coefficients of the transform block, and if so, encoding a last saliency flag indicating whether the non-zero transform coefficient at the current position is the last non-zero transform coefficient in the forward scan order if the current position is not the last in the forward scan, and

[0612] encoding the values ​​of the non-zero transform coefficients sequentially in a reverse scan order, the reverse scan order being the opposite of the forward scan order.

[0613] 98. The apparatus according to any one of embodiments 44 to 86, wherein the apparatus is configured to encode video into a data stream by means of block-based predictive coding and transform-based residual coding,

[0614] encoding the prediction residual data of an intra-prediction block into the data stream by means of context-adaptive binary arithmetic coding:

[0615] encoding the coordinates of the position where the last non-zero transform coefficient is encountered when traversing the transform coefficients of the transform block along a predetermined scan order in the transform block representing the prediction residual data, and

[0616] sequentially encoding the values ​​of transform coefficients included between the last non-zero transform coefficient and the first scanned transform coefficient and ordered along the predetermined scan order, and

[0617] selecting a predetermined scan order among the diagonal scan order, horizontal scan order and vertical scan order by means of a mapping that maps each of the various intra-prediction modes to the corresponding scan order among the diagonal scan order, horizontal scan order and vertical scan order, depending on the intra-prediction mode of the intra-prediction block.

[0618] 99. The apparatus according to any one of embodiments 44 to 86, wherein the apparatus is configured to encode video into a data stream by means of block-based predictive coding and transform-based residual coding:

[0619] The predictive residual data of the residual block is encoded into the data stream by using context-adaptive binary arithmetic coding of the quantization index of the transform coefficients representing the residual block and sequentially quantizing the transform coefficients to obtain a quantization index, wherein, according to the quantization index, the quantizer quantizes the current transform coefficient according to the parity of the quantization index of the previous quantization index.

[0620] 100. The apparatus according to any one of Embodiments 1 to 43, and Embodiments 44 to 86, wherein the data stream has video encoded therein by block-based predictive coding and transform-based residual coding in the following manner:

[0621] The predictive residual data of the residual block is encoded into the data stream of specification page 52 / 55, CN 121923845 A using context-adaptive variable-length coding by using the following:

[0622] A first syntax element and a tail-1 quantity, the first syntax element indicating the total number of non-zero transform coefficients in the transform block representing the residual block, the tail-1 quantity indicating the number of non-zero transform coefficients having an absolute value of 1 when traversing the coefficients in scan order,

[0623] One or more second syntax elements indicating the sign of the non-zero transform coefficients having an absolute value of 1 when traversing the coefficients in scan order,

[0624] One or more third syntax elements indicating the value of the non-zero transform coefficients other than the number having an absolute value of 1 when traversing the coefficients in scan order,

[0625] The fourth syntax element indicates the total number of zero-value transformation coefficient levels in the transform block, starting from the first non-zero transform coefficient encountered in the scan order, and

[0626] one or more fifth syntax elements indicate the position of the non-zero transform coefficients in the scan order by indicating the number of consecutive zero-value transform coefficients in the scan order between consecutive non-zero transform coefficients encountered in the scan order.

[0627] 101. The apparatus according to any one of Embodiments 1 to 43, Embodiments 44 to 86, wherein the data stream has video encoded therein by block-based predictive coding and transform-based residual coding in the following manner:

[0628] The predictive residual data of the residual block is encoded into the data stream by context-adaptive binary arithmetic coding in the following manner:

[0629] A saliency map is encoded in the following manner, the saliency map indicating the position of non-zero transform coefficients in the transform block representing the residual block: in a forward scan traversing the transform coefficients of the transform block, a saliency flag is encoded indicating whether a non-zero transform coefficient is at the current position, and if so, and if the current position is not the last in the forward scan, a last saliency flag is encoded indicating whether the non-zero transform coefficient at the current position is the last non-zero transform coefficient in the forward scan order, and

[0630] the values ​​of the non-zero transform coefficients are encoded sequentially in a reverse scan order, the reverse scan order being the opposite of the forward scan order.

[0631] 102. The apparatus according to any one of Embodiments 1 to 43, Embodiments 44 to 86, wherein the data stream has video encoded therein by block-based predictive coding and transform-based residual coding in the following manner:

[0632] The prediction residual data of the intra-prediction block is encoded into the data stream by context-adaptive binary arithmetic coding in the following manner:

[0633] The coordinates of the position where the last non-zero transform coefficient is encountered when traversing the transform coefficients of the transform block along a predetermined scan order in the transform block representing the prediction residual data are encoded, and

[0634] The values ​​of the transform coefficients included between the last non-zero transform coefficient and the first scanned transform coefficients and ordered along the predetermined scan order are encoded sequentially, and

[0635] Depending on the intra-prediction mode of the intra-prediction block, the predetermined scan order is selected among the diagonal scan order, horizontal scan order and vertical scan order by utilizing a mapping that maps each of the various intra-prediction modes to the corresponding scan order in the diagonal scan order, horizontal scan order and vertical scan order.

[0636] 103. The apparatus according to any one of Embodiments 1 to 43 or Embodiments 44 to 86, wherein the data stream has video encoded therein by block-based predictive coding and transform-based residual coding in the following manner:

[0637] The predictive residual data of the residual block is encoded into the data stream by using context-adaptive binary arithmetic coding of the quantization index of the transform coefficients of the transform block representing the residual block and sequentially quantizing the transform coefficients to obtain a quantization index, wherein, according to the quantization index, the quantizer quantizes the transform coefficients according to the parity of the quantization index of the previous quantization index.

[0638] 104. A data stream generated using the method of Embodiment 89.

[0639] 105. A computer program for implementing the method of Embodiment 87, or Embodiment 88, or Embodiment 89 when executed on a computer or signal processor.

[0640] Alternative Embodiments

[0641] Although some aspects have been described as features in the context of the apparatus, it should be understood that such description can also be regarded as a description of corresponding features of the method. Specifically, a block diagram illustrating the function of the apparatus can also be understood as an illustration of a corresponding method, which includes functions described by the boxes in the block diagram as steps of the method.

[0642] Data signals or data streams (e.g., media data streams, video data streams, audio data streams) provided by embodiments of the present invention can be stored on a digital storage medium (e.g., a non-transitory or transient digital storage medium) or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium. In other words, another embodiment provides a computer product, such as a data stream product or a bit stream product (e.g., a non-transitory digital storage medium), which includes (e.g., having stored thereon) data signals or data streams according to any embodiment described herein.

[0643] Another embodiment provides a method for storing data, the method comprising the step of storing a data stream on a digital storage medium (e.g., a non-transitory digital storage medium) carrying data. For example, the data stream conforms to any embodiment described herein. For example, data is encoded into the data stream according to any encoding method described herein.

[0644] Another embodiment provides a method for transmitting a data stream of any embodiment described herein.

[0645] Features described regarding means for receiving or processing signals (e.g., receiver, decoder) should be understood as serving as a description of corresponding features of means for providing signals (e.g., encoder), and vice versa, and as a description of features of the corresponding signal (e.g., data stream). Specifically, those skilled in the art will understand that any information (e.g., data type, structure, item to be received by the receiver or derived from the signal by the receiver) is inserted into the signal by the corresponding provider, and vice versa.

[0646] Some or all of the method steps may be performed by (or using) hardware devices (such as microprocessors, programmable computers, or electronic circuits). In some embodiments, one or more method steps may be performed by such means.

[0647] Embodiments of the invention may be implemented in hardware or software, or at least partially in hardware, or at least partially in software, depending on certain implementation requirements. The implementation can be performed using a digital storage medium (e.g., floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory) having electronically readable control signals stored thereon, which cooperate with (or are capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage medium can be computer-readable.

[0648] Some embodiments of the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0649] Generally, embodiments of the invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product is run on a computer. This program code can, for example, be stored on a machine-readable medium.

[0650] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0651] In other words, embodiments of the methods of the present invention are therefore computer programs having program code that, as described on pages 54 / 55 of the specification, CN 121923845 A, performs one of the methods described herein when the computer program is run on a computer.

[0652] Thus, other embodiments of the methods of the present invention are data carriers or digital storage media or computer-readable media on which a computer program is recorded for performing one of the methods described herein. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory.

[0653] Thus, another embodiment of the methods of the present invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0654] Another embodiment includes a processing means, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0655] Another embodiment includes a computer on which a computer program is installed to perform one of the methods described herein.

[0656] Another embodiment of the invention includes an apparatus or system configured to transmit a computer program to a receiver (e.g., electronically or optically) to perform one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0657] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0658] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0659] The methods described herein may be performed using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0660] In the foregoing detailed embodiments, it can be seen that various features are combined together as examples for the purpose of simplifying this disclosure. This disclosure should not be interpreted as reflecting an intention that the claimed example requires more features than those expressly recited in each claim. Rather, as reflected in the appended claims, the subject matter may have fewer features than all the features of a single disclosed example.Therefore, the appended claims are incorporated into the detailed description, wherein each claim may stand alone as a separate example. While each claim may stand alone as a separate example, it should be noted that although dependent claims may relate to a particular combination with one or more other claims in the claim, other examples may also include combinations of dependent claims with the subject matter of each other dependent claim or combinations of each feature with other dependent or independent claims. Such combinations are presented herein unless it is stated that a particular combination is not intended to be used. Furthermore, even if a claim does not directly depend on an independent claim, it is intended to include features of any other independent claim.

[0661] The above embodiments are merely illustrative of the principles of this disclosure. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Therefore, it is intended to be limited only by the scope of the appended patent claims and not by the specific details given by way of the description and explanation of the embodiments herein.Instruction manual page 55 / 55, page 59, CN 121923845 A, Figure 1; Instruction manual figure 1 / 10 page, page 60, CN 121923845 A, Figure 2; Instruction manual figure 2 / 10 page, page 61, CN 121923845 A, Figure 3; Instruction manual figure 3 / 10 page, page 62, CN 121923845 A, Figure 4; Instruction manual figure 4 / 10 page, page 63, CN 121923845 A, Figure 5; Instruction manual figure 5 / 10 page, page 64, CN 121923845 A, Figure 6; Instruction manual figure 6 / 10 page, page 65, CN 121923845 A, Figure 7; Instruction manual figure 7 / 10 page, page 66, CN 121923845 A, Figure 8; Instruction manual figure 8 / 10 page, page 67, CN 121923845 A, Figure 9; Instruction manual figure 9 / 10 page, page 68, CN 121923845 A, Figure 10 Figure 10 / 10 of the specification, page 69 CN 121923845 A Abstract Apparatuses and methods for checking a data stream on trustworthiness are described.

Claims

1. An apparatus (20) for decoding an audio data stream (14) in which audio signals are encoded, the audio data stream comprising a plurality of payload packets (16) carrying payload data, and further comprising supplementary information packets (18) carrying supplementary information messages (19), wherein, The device is configured to: Derive (21) an indication (51) from the audio data stream, the indication (51) indicating whether one or more supplementary information messages should be considered to verify (40) the audio data stream; If the instruction (51) indicates that the one or more supplementary information messages should be considered to verify the audio data stream, then the predetermined portion (13) of the audio data stream is determined (30) by including the one or more supplementary information messages in a predetermined portion of the audio data stream. (21) Obtain (43) a digital signature (21) for verifying the predetermined part (13) from the indication in the audio data stream.

2. The apparatus according to claim 1, wherein, The device is configured to verify the predetermined portion by checking whether the predetermined portion in the audio data stream matches the digital signature.

3. The apparatus according to any one of claims 1 or 2, wherein, The device is configured to provide the predetermined portion and the digital signature to verify the predetermined portion and the digital signature.

4. The apparatus according to any one of claims 1 to 3, wherein, Checking whether the predetermined portion (13) in the audio data stream matches the digital signature (43) includes: A hash function is applied to the predetermined portion (13) to obtain a hash value; and Check whether the hash value matches the digital signature (43).

5. The apparatus according to claim 4, wherein, Checking whether the hash value matches the digital signature (43) includes: A verification string (48) is formed based on the hash value. The digital signature (43) is decrypted to obtain the check value; and Check whether the verification string (48) matches the check value.

6. The apparatus according to any one of claims 1 to 5, configured as follows: If the instruction indicates that one or more supplementary information messages should be considered to verify the predetermined part (13), then the predetermined part (13) is determined by including the one or more supplementary information groups into the predetermined part (13).

7. The apparatus according to any one of claims 1 to 6, configured to include all supplementary information groups in the predetermined portion (13), the supplementary information groups carrying any supplementary information messages to be included in the predetermined portion (13).

8. The apparatus according to any one of claims 1 to 7 is configured to: determine whether to include the supplementary information packet in the predetermined portion (13) based on the result of checking whether the supplementary information payload packet includes supplementary information messages of any supplementary information message type in a set of supplementary information message types.

9. The apparatus according to any one of claims 1 to 8, wherein, The indication is a flag having a first state and a second state, and wherein the device is configured to determine the predetermined portion (13) in the audio data stream by: If the flag has a first state, then the one or more supplementary information messages to be included in the predetermined portion (13) are determined based on a set of predefined supplementary information messages in the supplementary information messages; and If the flag has a second state, then avoid including any of the supplementary information messages in the predetermined part (13).

10. The apparatus according to claim 9, wherein, The group of predefined supplementary information messages includes all supplementary information messages in the supplementary information group that reference the plurality of payload groups (16).

11. The apparatus according to claim 9, wherein, The group of predefined supplementary information messages includes all supplementary information messages of any type within a set of supplementary information message types.

12. The apparatus according to any one of claims 1 to 11, further configured to: If the instruction indicates that the one or more supplementary information messages should be considered to verify the audio data stream, then the identifiers of the one or more supplementary information messages to be included in the predetermined portion (13) are derived from the audio data stream.

13. The apparatus of claim 12, configured to: if the indication indicates that the one or more supplementary information messages should be considered to verify the audio data stream, derive a syntax element from the audio data stream, the syntax element identifying a set of supplementary information message types to be included in the predetermined portion (13) of the audio data stream.

14. The apparatus according to any one of claims 1 to 13, configured to derive a syntax element from the audio data stream, the syntax element indicating whether the one or more supplementary information messages should be considered to verify the audio data stream, and the syntax element signaling an identifier of the one or more supplementary information messages to be included in the predetermined portion (13).

15. The apparatus according to claim 14, wherein, The syntax element has a second state and multiple first states, and the device is configured to: If the syntax element has one of the first states, then the one or more supplementary information messages to be included in the predetermined section (13) are determined based on the state of the syntax element. If the syntax element has the second state, then avoid including any of the one or more supplementary information messages in the predetermined part (13).

16. The apparatus of claim 12, configured to: if the indication indicates that the one or more supplementary information messages should be considered to verify the audio data stream, derive a first syntax element and a corresponding number of second syntax elements from the audio data stream, the first syntax element indicating a count of supplementary information message types to be considered for the predetermined portion (13), each second syntax element identifying a corresponding supplementary information message type to be included in the predetermined portion (13).

17. An apparatus (10) for encoding an audio data stream (14) in which audio signals are encoded, wherein, The apparatus is configured to: enable the audio data stream to undergo credibility checks; and encode multiple payload packets (16) carrying payload data into the audio data stream, and also encode supplementary information packets (18) carrying supplementary information messages into the audio data stream, wherein the apparatus is configured to: Insert an instruction (51) into the audio data stream (23), the instruction (51) being used to indicate whether one or more supplementary information messages should be considered to verify the audio data stream; If the instruction indicates that the one or more supplementary information messages should be considered to verify the audio data stream, then the predetermined portion (13) of the audio data stream is determined (30') by including one or more of the supplementary information messages in a predetermined portion (13) of the audio data stream that will enable a credibility check. The digital signature (43) is obtained based on the predetermined part (13) (41').

18. A method for decoding an audio data stream (14) in which audio signals are encoded, the audio data stream comprising a plurality of payload packets (16) carrying payload data, and further comprising supplementary information packets (18) carrying supplementary information messages (19), wherein, The method includes: Derive (21) an indication (51) from the audio data stream, the indication (51) indicating whether one or more supplementary information messages should be considered to verify (40) the audio data stream; If the instruction (51) indicates that the one or more supplementary information messages should be considered to verify the audio data stream, then the predetermined portion (13) of the audio data stream is determined (30) by including the one or more supplementary information messages in a predetermined portion of the audio data stream. (21) Obtain (43) a digital signature (21) for verifying the predetermined part (13) from the indication in the audio data stream.

19. A method for encoding an audio data stream (14) in which audio signals are encoded, wherein, The method includes: enabling the audio data stream to undergo credibility checks; and encoding multiple payload packets (16) carrying payload data into the audio data stream, and also encoding supplementary information packets carrying supplementary information messages into the audio data stream, wherein the method includes: Insert an instruction into (23) the audio data stream, the instruction being used to indicate whether one or more supplementary information messages should be considered to verify the audio data stream; If the instruction indicates that the one or more supplementary information messages should be considered to verify the audio data stream, then the predetermined portion (13) of the audio data stream is determined (30') by including one or more of the supplementary information messages in a predetermined portion (13) of the audio data stream that will enable a credibility check. A digital signature (43) is obtained based on the predetermined portion (13).

20. An audio data stream, wherein data signals are encoded therein, the audio data stream being capable of reliability checking, the audio data stream comprising: Multiple payload packets (16) carrying payload data, and also including supplementary information packets carrying supplementary information messages; as well as An indication is provided to indicate whether one or more supplementary information messages should be considered to check the credibility of the audio data stream.

Images