Method and device for signing an encoded video sequence

Abstract

To provide a device and a method of signing an encoded video sequence.SOLUTION: A method includes: (502) obtaining an encoded video sequence; (504) generating a set of one of more frame fingerprints for each encoded image frame; (506) generating a document comprising a header of a supplemental information unit, and a representation of the generated sets of one or more frame fingerprints; (510) generating a document signature by digitally signing the document; (512) generating the supplemental information unit consisting only of the document, the document signature and an indication of an end of the supplemental information unit; and (514) signing the encoded video sequence by associating the generated supplemental information unit with the encoded video sequence.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to the field of security arrangements for protecting video sequences against unauthorized activity. In particular, it proposes a method and device for signing encoded video sequences. [Background technology]

[0002] A video sequence is an orderly sequence of video frames, which are made up of pixels. When a typical scene is imaged, successive video frames will be highly correlated. The ability to predict one video frame given a frame over another is the underlying premise of predictive encoding. Predictive encoding is sometimes described as a data compression technique specifically adapted for video sequences.

[0003] Reference for predicting other frames (reference)A video frame used as a reference frame is called a reference frame. A frame encoded without information from other frames is called an intra-coded frame, intraframe, I-frame, or keyframe. A frame that uses prediction from one or more reference frames is called an intra-coded frame or interframe. A P-frame is an interframe that uses prediction from a single preceding reference frame (or a single frame for prediction of each region), and a B-frame is an interframe that uses prediction from two reference frames, one preceding and one succeeding. Frames are sometimes called pictures. Recommendation ITU-T H.264 (08 / 2021) "Advanced video coding for generic audiovisual services," International Telecommunication Union, specifies a video coding standard in which both forward-predicted and bidirectionally predicted frames are used.

[0004] Figure 1 shows a segment of a predictively encoded video sequence composed of I-frames and P-frames. As described above, an I-frame is a data structure having independently decodable image data that can be decoded into a decoded video frame (or a decoded portion of a video frame) using a predetermined, associated decoding operation. A P-frame is a data structure for which the associated decoding operation references not only the image data of the P-frame itself but also at least one other I-frame or P-frame. Conceptually, and somewhat simply, the image data in a P-frame indicates change or motion relative to the video frame that the preceding I-frame or P-frame encodes. Upon successful completion of the decoding operation, video frames decoded from P-frames and I-frames are generally indistinguishable.

[0005] Interframe dependency, This is shown in FIG. 1A as a negative-pointing arc-shaped arrow in JPEG0007789652000001.jpg7170. In the illustrated sample example, each P frame references the immediately preceding I or P frame. If a first P frame references a second P frame, the second P frame necessarily references at least one additional I or P frame. In this disclosure, the first P frame is considered to directly reference the second P frame and indirectly reference at least one additional I or P frame. Because the image data in an I-frame is independently decodable, the chain of references (arc-shaped arrow) does not continue beyond the I-frame. The combination of an I-frame and subsequent P-frames that directly or indirectly reference that I-frame may be referred to as a group of pictures, which in some encoding formats are referred to as a GOP. Four GOPs are shown in FIG. 1A: IPP, IPPPP, IPPP, and IPPP.

[0006] Digital signatures provide an additional layer of validation and security for digital messages sent over insecure channels. Using a digital signature, the recipient of the digital message can verify the authenticity or integrity of the message. Of course, video data, e.g., a video sequence, can be digitally signed as a special case, providing an additional layer of validation and security for the video data. Video data may be signed by signing a fingerprint of the video data rather than the video data itself. Frame-by-frame signatures The use of JPEG0007789652000002.jpg8170 is shown at the bottom of Figure 1, where each fingerprint may be a hash of the image data of that video frame, or a hash of the image of that video frame in combination with any further information. The signatures S1, S2, S3, S4 for those GOPs are obtained by digitally signing the combination of the fingerprints in that GOP. The security of the video sequence is further ensured by: If JPEG0007789652000003.jpg8170 is signed as both part of its own GOP and as part of the signature of the preceding GOP, as shown in Figure 1 A, then the entire GOP may be protected from unauthorized replacement, removal, or insertion. Note that the straight downward arrows in the top row represent hash functions, and the arrows in the bottom row represent digital signatures.

[0007] However, even if a video sequence is digitally signed as described above, the video sequence may still be subject to tampering because data may be added to the video, for example, before the first GOP in the video sequence. Unless JPEG0007789652000004.jpg9170 is signed both as part of its own GOP and as part of the signature of the preceding GOP, one or more GOPs may be removed, added, or replaced, with the risk of such tampering going undetected.

[0008] US2008 / 0291999A1 discloses a method and apparatus for marking individual video frames of a digital video stream conforming to the H.264 / AVC standard or an equivalent digital video stream. Each video frame in an H.264 / AVC video stream is conventionally divided into NAL units. Typically, many NAL units exist for each video frame. Supplemental Enhancement Information (SEI) types are defined in the H.264 / AVC standard. SEI types include a User Data Unregistered type, which can contain arbitrary data. In the disclosed method and apparatus, an NAL unit of this type is provided at the beginning of each video frame, preceding other NAL units associated with that video frame. This is schematically illustrated in FIG. 2. The payload data contained in the special SEI unit, sometimes referred to as SEI data, is typically control information for downstream control of the use of the video content. Examples of SEI data include stream positioning data (e.g., video frame number), stream bitrate (e.g., normal, fast-forward), decryption data (e.g., decryption key or key derivation seed), and validation elements (e.g., checksum or hash function value or signature). Thus, SEI data may include decryption data, e.g., decryption key or key derivation seed, necessary to decrypt encrypted video frames. Additionally, SEI data may include validation data, e.g., checksum or hash function value or signature, necessary to verify signed video frames.

[0009] US2008 / 0291999A1 states that the SEI NAL unit may itself be encrypted and / or signed (verified) to prevent the information contained therein from being easily accessible to unauthorized users, which may be understood as either 1) the entire SEI may be encrypted and / or signed, or 2) the information contained in that SEI may be encrypted and / or signed.

[0010] If this statement is understood as 1) that the entire SEI may be encrypted and / or signed, US 2008 / 0291999 A1 does not explain how encrypting and / or signing (validating) an SEI is achieved, nor does it explain how the information contained therein is easily accessible or stored anywhere, nor how to transmit the SEI decryption data and / or SEI validation data necessary to decrypt and / or verify an encrypted and / or signed SEI. Furthermore, if the entire SEI is encrypted, a recipient of such encrypted SEI would not be able to recognize it as an SEI because it is encrypted. Furthermore, the recipient would not be able to ignore the encrypted SEI and process the next NAL unit in the sequence because the recipient would have no knowledge of the length of the encrypted SEI. Therefore, this statement cannot be understood as the entire SEI being encrypted or signed.

[0011] Therefore, this statement should be understood as 2) "The information included in the SEI may be encrypted and / or signed." The information included in the SEI is understood as SEI data, and therefore, it is considered that the SEI data may be encrypted and / or signed (verified). If the SEI data is encrypted and / or signed, similar to encrypting and signing the video frame, it is assumed that the SEI may include decryption and / or validation data necessary for decrypting and verifying the encrypted and / or signed SEI data in its payload. Figure 3 schematically illustrates an SEI according to the prior art. The prior art SEI consists of a header, information about the payload size, a universally unique identifier (UUID), a payload, e.g., the SEI data, and a stop bit indicating the end of the SEI. Alternatively, it is assumed that the decryption and / or validation data are transmitted to the recipient via a separate transmission channel. Therefore, it is envisioned that the payload of the SEI disclosed in US2008 / 0291999A1 may include, in addition to the encrypted and / or signed SEI data, decryption data and / or validation data necessary for decrypting and / or verifying the encrypted and / or signed SEI data, or that such decryption data and / or validation data is transmitted via a separate transmission channel.

[0012] A security risk arises from such a solution in which only a portion of the payload, i.e., the SEI data, is encrypted and / or signed, in that it would be possible to tamper with other portions of the SEI, e.g., to increase the payload size and thereby introduce additional data into the SEI, without it being known when encrypted and / or verified using the signed SEI data and the encryption and / or validation data.

[0013] These examples illustrate the shortcomings of available methods for signing, eg, digitally signing, encoded video sequences. Summary of the Invention

[0014] One object of the present disclosure is to make available methods and devices for signing, e.g., digitally signing, encoded video sequences with mechanisms for protecting against unauthorized substitution, removal, or insertion of video frames. A further object is to provide mechanisms for detecting the occurrence of tampering with encoded video sequences.

[0015] These and other objects are achieved by the present invention as defined by the independent claims.

[0016] In a first aspect of the present invention, there is provided a method for signing an encoded video sequence, the method comprising obtaining an encoded video sequence consisting of encoded image frames, and signing a signature for each encoded image frame. Ruichi or multiple frame fingerprints One set of and generating a header of the ancillary information unit and Taichi or multiple frame fingerprints Multiple sets of a representation of the encoded video sequence; digitally signing the document to generate a document signature; generating an ancillary information unit consisting only of the document, the document signature, and an indication of the end of the ancillary information unit; and signing the encoded video sequence by associating the generated ancillary information unit with the encoded video sequence.

[0017] In this disclosure, the term "fingerprint" should be understood as a unique identifier of a data item. A fingerprint of a data item may be obtained by hashing the data item or a subset thereof, i.e., by performing a hashing operation on it. Alternatively, this fingerprint may be obtained by performing a different operation on the data item or a subset thereof, for example a checksum operation. A further alternative to obtaining a fingerprint of a data item is to digitally sign the data item or a subset thereof. The data item may be image data, which encodes a video frame or a portion of the image data. Optionally, the data item may be combined with other data, for example a cryptographic salt or a timestamp.

[0018] Furthermore, the term "frame fingerprint" in this disclosure should be understood as a fingerprint of a frame, e.g., an encoded image frame. Since an encoded image frame may consist of multiple units, e.g., Network Abstraction Layer (NAL) units in the H.26x encoding format or Open Bitstream Units (OBUs) in the AV1 encoding format, a frame fingerprint may be generated for each of those multiple units, and thus ,one or multiple such frame fingerprints One set of may be generated for each encoded image frame.

[0019] Each NAL unit consists of a header that specifies the type of the NAL unit, for example, if the NAL unit relates to an encoded image frame, e.g., an intraframe or an interframe, or if it relates to an ancillary information unit. In addition to the header, each NAL unit contains a payload. Each OBU has a header that provides identification information about the data (payload) contained in that OBU (AV1 Bitstream & Decoding Process Specification, https: / / aomediacodec.github.io / av1-spec / av1-spec.pdf).

[0020] In this disclosure, the term "document" should be understood as a text file or another data structure, such as a byte stream or bit stream. The term "document signature" refers to a digital signature for a document, which can be used to verify the authenticity of the document. Furthermore, the term "extras information unit (SIU)" should be understood in this disclosure as a unit or message configured to contain extraneous information about or related to an encoded video sequence. An extraneous information unit may be, for example, a Supplemental Enhancement Information (SEI) message in the H.26x encoding format or a Metadata Open Bitstream Unit (OBU) in the AV1 encoding format. The header of an extraneous information unit contains extraneous data, such as an indication of the extraneous information unit, i.e., an indication that the extraneous information unit is an extraneous information unit. By reading this indication, a decoder knows what type of unit to decode.

[0021] Generated Taichi or multiple frame fingerprints Multiple sets of Representation is ,raw done Taichi or multiple frame fingerprints All sets of From, centre REM Fingerprint Set of or List, or ,raw done Taichi or multiple frame fingerprints All sets of Some frame fingerprints from Set of or Alternatively, this representation may be a list of all generated Taichi or multiple frame fingerprints All sets of one or more hashes of frame fingerprints from ,raw done Taichi or multiple frame fingerprints All sets of The frame fingerprint may be or may include one or more hashes of portions of the frame fingerprint from

[0022] By the expression "signing the encoded video sequence by associating the generated extrinsic information unit with the encoded video sequence," it should be understood that a generated extrinsic information unit including a document and a document signature is associated with the encoded video sequence to sign, e.g., digitally sign, the encoded video sequence. This may also be expressed as "a signature is provided by associating the generated extrinsic information unit including a document and a document signature with the encoded video sequence." A recipient of the signed encoded video sequence and the associated extrinsic information unit will verify the encoded video sequence using the document and document signature of the extrinsic information unit. The document and document signature may be referred to as a signature for the encoded video sequence. As described in this disclosure, a generated extrinsic information unit may be associated with an encoded video sequence in various ways. By generating an ancillary information unit consisting only of the document, the document signature, and an indication of the end of the ancillary information unit, and associating the generated ancillary information unit with the encoded video sequence and thereby signing the encoded video sequence, it is possible to prevent undetected tampering with the ancillary information unit, for example, using the document, because if the document is altered after the document signature is generated, the document cannot be correctly verified using the document signature. For example, if the document and the document signature are generated on the device and included by the device in the ancillary information unit associated with the encoded video sequence before transmission to a recipient, and the SEI is altered after transmission but before it reaches the recipient, the recipient will not be able to verify the authenticity of the document using the received document signature.If the authenticity of the document cannot be verified, the recipient may understand that the encoded video sequence contains some unauthorized activity and may take necessary action.

[0023] Further details regarding embodiments of the first aspect of the invention are set out in the detailed description and the dependent claims.

[0024] In a second aspect of the present invention, there is provided a device configured to perform the method disclosed herein. Broadly speaking, the second aspect of the present invention shares the effects and advantages of the first aspect, and it may be implemented with a corresponding degree of technical variation.

[0025] A third aspect of the present invention relates to a computer program comprising instructions for causing a computer to perform the method(s) described in this disclosure. This computer program may be stored on or distributed by a data carrier. As used herein, a "data carrier" may be a transitory data carrier, such as a modulated electromagnetic or light wave, or a non-transitory data carrier. Non-transitory data carriers include volatile and non-volatile memory, such as permanent and non-permanent storage media of the magnetic, optical, or solid-state type. Still within the scope of "data carrier," such memory may be fixedly mounted or may be portable.

[0026] In general, all terms used in the claims are to be interpreted according to their ordinary meaning in the art unless expressly defined herein. Any reference to "a / an / the element, device, component, means, step, etc." is openly interpreted as referring to at least one instance of that element, device, component, means, step, etc. The steps of any method disclosed herein do not have to be performed in the exact order described, unless expressly stated.

[0027] Aspects and embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0028] [Figure 1] 1 shows a schematic representation of a segment of a prediction-based encoded video sequence, along with frame-by-frame fingerprints and signatures generated from them. [Figure 2] 1 shows a schematic representation of a segment of a video sequence with each video frame preceded by an SEI according to the prior art; [Figure 3] 1 shows a schematic diagram of an SEI according to the prior art. [Figure 4] 1 illustrates a schematic diagram of a device configured to sign an encoded video sequence according to an embodiment of the present invention; [Figure 5] 1 is a flowchart of a method for signing an encoded video sequence according to an embodiment of the present invention. [Figures 6A-6D] 1 shows a schematic representation of an encoded video sequence, an ancillary information unit, an example first document, and an example second document, respectively. [Figures 7A-7E] 1 illustrates a schematic diagram of an embodiment of a signed encoded video sequence. DETAILED DESCRIPTION OF THE INVENTION

[0029] Aspects of the present disclosure are described in more detail below with reference to the accompanying drawings, which show specific embodiments of the present invention. These aspects may, however, be embodied in many different forms and should not be construed as limiting. Rather, these embodiments are provided for illustrative purposes so that this disclosure will be thorough and complete, and will fully convey the full scope of the aspects of the present invention to those skilled in the art. Like reference numerals refer to like elements throughout this specification.

[0030] To achieve this objective, the present invention describes a method and device for signing an encoded video sequence. In particular, an encoded video sequence is signed by associating the encoded video sequence with a unique ancillary information unit, which is generated to consist only of a document, a document signature, and an indication of the end of the ancillary information unit. The ancillary information unit may be an SEI message in the H.26x encoding format or a Metadata OBU in the AV1 encoding format. Because the ancillary information unit contains only these three components and no other component(s), tampering with the content of the ancillary information unit, e.g., modifying the document, cannot be performed without detection, because any tampering would result in the document not being validated successfully. Therefore, a recipient of the ancillary information unit will understand that tampering with the content of the ancillary information unit has occurred. The method and device for signing an encoded video sequence are described in more detail below.

[0031] Referring to FIG. 5, a method 500 for signing an encoded video sequence is described. The method 500 may be applied to any encoded video sequence consisting of encoded image frames encoded using prediction-based encoding. For example, the encoded image frames may include at least one intra frame and one or more predicted inter frames. The at least one intra frame and one or more predicted inter frames may be encoded according to a video encoding format that defines temporal video compression. The intra frames may be I frames encoded according to the H.26x compression format, or intra frames or key frames encoded according to the AV1 compression format. The predicted inter frames may be forward-predicted inter frames (P frames) or bidirectionally predicted inter frames (B frames) encoded according to the H.26x compression format, or inter frames encoded according to the AV1 compression format.

[0032] The method 500 for signing encoded video sequences may be performed by a suitably configured general programmable computer (e.g., with associated input and output interfaces), in particular using a device 400 as schematically illustrated in FIG. 4 . The device 400 includes a processing circuit 410 configured to perform the actions of the method 500. For example, the processing circuit 410 may include a generating component 412 and a signing component 414 configured to perform the actions of the method 500. The device 400 further includes a memory 420 and an input / output interface 430 adapted for two-way communication with an external memory 490, which may, in some embodiments, store the encoded video sequence to be signed during some operational phases. This may be the case in embodiments in which the device 400 is included in a video management system configured to offer signing stored encoded video sequences as an external service. The device 400 and the external memory 490 may, in turn, be owned and operated by different entities or by a common entity. The (internal) memory 420 of the device 400 may be suitable for storing a program 421 containing software instructions for performing the method 500, cryptographic information (e.g., private keys) for generating signatures, as well as logs, configuration files, and data supporting various internal management procedures. The computer program 421 may contain instructions that, when executed by a computer, cause the computer to perform the actions of the method 500. The device 400 may be provided as a local computer or server, or it may be implemented in a distributed manner based on networked (cloud) processing resources. To provide local signing of encoded video sequences, the device 400 may be integrated into a camera 440, e.g., a digital video camera, such as a monitoring camera adapted for monitoring and / or surveillance applications.In some embodiments in which device 400 is included in a camera 440, e.g., a camera that captures video sequences, device 400 may be configured to communicate with encoder 450 of camera 440 and may be configured to receive encoded video sequences directly from encoder 450 and sign the encoded video sequences before transmitting the signed, encoded video sequences to a recipient, e.g., client device 460 including decoder 470, or to external memory 490 for storage. Client device 460 may be configured to verify and decode the signed, encoded video sequences. It should be understood that other configurations of device 400 are possible and that encoder 450 may be an external encoder configured separately from, but in communication with, camera 440 and device 400.

[0033] It should be noted that the camera 440 shown in FIG. 4 may also include other components related to image capture and image processing, for example, that are common in conventional camera systems and whose purpose and operation are familiar to those skilled in the art. Such components have been omitted from the illustration and description of FIG. 4 for clarity. The encoded video sequence may be encoded according to a conventional video encoding format. Some common video encoding formats that work with various embodiments of the present invention include High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2; Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 10; Versatile Video Coding (VVC), also known as H.266, MPEG-I Part 3, and Future Video Coding (FVC); VP9, ​​VP10, and AOMedia Video1 (AV1), as just a few examples. The H.264, H.265, and H.266 coding formats are sometimes referred to in this disclosure as H.26x coding formats. The terms "coding format" and "compression format" are used interchangeably in this disclosure. The above-mentioned video encoding formats define temporal video compression implemented in terms of intra-image frames and inter-image frames.

[0034] The method 500 for signing an encoded video sequence begins with action 502, where an encoded video sequence consisting of encoded image frames of the video sequence is obtained. FIG. 6A schematically illustrates an encoded video sequence including three intra-frames, denoted I1 through I3, and eight inter-frames, denoted P1 through P8. The device 400 may be configured to obtain, e.g., retrieve or receive, the encoded video sequence from an external memory 490 or from the encoder 450. Thus, the act of obtaining the encoded video sequence may include gaining access to a memory (e.g., external memory 490) where the video data is stored, downloading the video data, and / or receiving a transmission of the video data.

[0035] For each encoded image frame Ruichi or multiple frame fingerprints Set of is generated in action 504. As previously mentioned, a fingerprint is a unique identifier for a data item, and thus a frame fingerprint is a fingerprint for an image frame, and in particular for an encoded image frame. A particular frame fingerprint may be used by a recipient to verify the authenticity of the particular encoded image frame from which the particular frame fingerprint was generated.

[0036] A frame fingerprint for an encoded image frame may be obtained by hashing the encoded image frame, or a portion thereof, i.e., performing a hashing operation on it. A cryptographic salt may be added to the data being hashed, which may reduce the number of hashing operations required. An alternative to a hashing operation is to perform a different operation, for example a checksum operation, on the encoded image frame, or a portion thereof, to obtain the fingerprint. A further alternative to obtaining a fingerprint for an encoded image frame is to digitally sign the encoded image frame, or a portion thereof.

[0037] Since an encoded image frame may consist of multiple units, e.g., NAL units or OBUs, a frame fingerprint may be generated for each such unit, which is , complex Number of Frame Fingerprints One set of However, even if an encoded image frame is composed of several units, several frame fingerprints need not be generated for that encoded image frame; In contrast, raw Generated frame fingerprint Set of It should be understood that the frame fingerprint may include only one frame fingerprint. ,one or multiple frame fingerprints Set of , for example, using generating component 412 of processing circuitry 410.

[0038] In an embodiment where the encoded video sequence includes at least one intraframe and one or more predicted interframes, the generated Taichi or multiple frame fingerprints Multiple sets ofA representation must have at least one intraframe generated JPEG0007789652000005.jpg8170 and one or more predicted inter frames, JPEG0007789652000006.jpg8170.

[0039] For each of the intraframe and the predicted interframe, JPEG0007789652000007.jpg8170 can be generated using the same hash function or two different hash functions. JPEG0007789652000008.jpg8170 is generated using a relatively secure hash function (e.g., 1024 bits) to predict the interframe JPEG0007789652000009.jpg8170 is generated using a relatively simple hash function (e.g., 256 bits) that is computationally less expensive. If JPEG0007789652000010.jpg8170 relies on an intra-frame fingerprint, it can be generated computationally more simply using a simpler hash function without significantly reducing the overall level of security.

[0040] The encoded video sequence may include a first group of photos and a second group of photos. In such an embodiment, the method may include the sub-act of identifying the first and second groups of photos. Taichi or multiple frame fingerprints Multiple sets of The first group of photos was generated as an intraframe. JPEG0007789652000011.jpg7170 and the generated predicted interframes of the first group of photos JPEG0007789652000012.jpg7170 and the intraframe of the second group of photos JPEG0007789652000013.jpg8170.

[0041] Each predicted interframe is generated JPEG0007789652000014.jpg9170 may be generated by hashing a combination of data derived from the predicted Inter frame and data derived from an Intra frame to which the predicted Inter frame directly or indirectly references. The fingerprint of the predicted Inter frame may be independent of any further predicted Inter frames to which the predicted Inter frame directly or indirectly references. Furthermore, the data derived from the predicted Inter frame includes one of image data for the predicted Inter frame and a generated first fingerprint of the image data for the predicted Inter frame. Furthermore, the data derived from the Intra frame is a generated first fingerprint for the Intra frame.

[0042] The fingerprint calculation may be expressed as follows: JPEG0007789652000015.jpg9170where, JPEG0007789652000016.jpg8170 is a hash function, JPEG0007789652000017.jpg7170 is data resulting from an intra-frame, JPEG0007789652000018.jpg is data derived from the 7170 interframe. The hash function (or one-way function) may be a cryptographic hash function that provides a level of security deemed appropriate, taking into account the need for protection of the video data being signed and / or the value at stake if the video data is tampered with by an unauthorized party. Three examples include SHA-256, SHA3-512, and RSA-1024. The hash function is predetermined (e.g., it is reproducible) so that the fingerprint can be reproduced when the fingerprint is verified. Bracket annotations JPEG0007789652000019.jpg7170 refers to general data combination operations, which may include concatenating data linearly (parallel) or in various staggered arrangements. Combination operations may also include arithmetic operations on the data, such as bitwise OR (XOR), multiplication, division, or modulo operations.

[0043] A further development deals with the case where an interframe directly or indirectly references two intraframes. The fingerprint of the interframe is then generated from both of those intraframes. JPEG0007789652000020.jpg8170 based on the combination of the interframe itself JPEG0007789652000021.jpg7170 will be generated in addition to: JPEG0007789652000022.jpg12170

[0044] In action 506, the header of the ancillary information unit and the generated Taichi or multiple frame fingerprints Multiple sets ofA document including a header 611 and a representation of a frame fingerprint 612 is generated. A document 610 is schematically shown in FIG. 6C , including a header 611 and a representation of a frame fingerprint 612. The representation of a frame fingerprint 612 may be referred to as the payload of the document 610 shown in FIG. 6C . As previously described, the document may be a text file or another data structure, and the extrinsic information unit may be a unit or message that includes extrinsic information. The header of the extrinsic information unit includes extrinsic data, for example, an indication of the extrinsic information unit, indicating that it is an extrinsic information unit. As described below, the extrinsic information unit is generated in action 512. Therefore, it should be noted that although the extrinsic information unit has not yet been generated in action 506, it is still possible to have knowledge of the header of the extrinsic information unit, i.e., the header of the extrinsic information unit to be generated, in action 506. For example, in action 506, the device 400 may set a header to include an indication of the extrinsic information unit. As mentioned above, the indication of an ancillary information unit indicates the ancillary information unit as an ancillary information unit, and it should be understood that this is the same for all ancillary information units. Taichi or multiple frame fingerprints Multiple sets of Representation is ,raw done Taichi or multiple frame fingerprints All sets of From or part of ,centre REM Fingerprint Set of or Alternatively, this representation may be a list of Taichi or multiple frame fingerprints All sets ofThe representation may be or may include one or more hashes of frame fingerprints from, or portions thereof. Alternatively, the representation may be a combination of frame fingerprints and hashes of those frame fingerprints. Device 400 is configured to generate the document using, for example, a generation component of processing circuitry 410.

[0045] In some embodiments, the document includes additional information in addition to the header and representations described above. In such embodiments, the document may include one or more of: a payload size value for the payload included in the ancillary information unit; an identifier for the payload; metadata about the encoded video sequence; camera information about the camera capturing the video sequence; and a public key of an asymmetric cryptographic key pair. The identifier may identify the type of payload included in the document. For example, the identifier may identify or indicate that the payload relates to a signed, encoded video sequence. Furthermore, the identifier may be a unique identifier, such as a universally unique identifier (UUID). An example of a document 610 including all of the additional information described above in addition to a header 611 and representations 612 is shown in FIG. 6D. As shown, such a document 610 includes a header 611, a representation 612, a payload size value 613, an identifier ID 614, metadata 615 about the encoded video sequence, camera information 616, and a public key 617. The identifier ID 614 is an identifier of the payload included in the ancillary information unit, which may be a UUID. The metadata 615 may include information describing how to interpret or the syntax of the camera information 616 and / or the frame fingerprint representation 612, for example. In some embodiments, the representation 612, the metadata 615, the camera information 616, and / or the public key 617 are included in a sub-document. The sub-document, possibly including the identifier ID 614, may be referred to as a payload of the document 610 shown in FIG. 6D . As such, the sub-document may be a text file or another data structure. In such an embodiment, document 610 may include a header 611, a payload size value 613, an identifier 614, and sub-documents.

[0046] In some embodiments, a document fingerprint for the document is generated in act 508. The document fingerprint is a fingerprint for the document. This document fingerprint may be obtained by hashing the document, i.e., by performing a hashing operation on the document, i.e., the content of the document, or a portion thereof. Alternatively, this fingerprint may be obtained by performing a different operation, e.g., a checksum operation, on the document or a portion thereof. A further alternative to obtaining a document fingerprint is to digitally sign the document or a portion thereof. Processing circuitry 410 may be configured to generate the document fingerprint, for example, using generation component 412.

[0047] At action 510, a document signature is generated by digitally signing the document. In some embodiments that include generating a document fingerprint as described above at action 508, the document signature generated at action 510 is generated by digitally signing the document fingerprint. The document signature may be referred to as a signature of an encoded video sequence or a segment of an encoded video sequence. A segment of an encoded video sequence may be a group of photographs. The document signature may be generated, for example, using asymmetric encryption, i.e., using a private key from a key pair in which the public key is pre-shared with a recipient so that the recipient can verify the signature. Device 400 is configured to generate the document signature, for example, using generation component 412 of processing circuit 410.

[0048] In action 512, an ancillary information unit is generated, consisting only of the document, the document signature, and an indication of the end of the ancillary information unit. The ancillary information unit is of the same type as the units that make up an encoded image frame. That is, the ancillary information unit is a NAL unit in the H.26x encoding format and an OBU in the AV1 coding format. FIG. 6B shows the generated ancillary information unit 600, consisting of the document 610, the document signature 620, and an indication 630 of the end of the ancillary information unit 600. The indication of the end of the ancillary information unit may be an explicit indication, such as a stop bit, or it may be an implicit indication, such as reaching a predetermined number of bits in the ancillary information unit. It should be understood that the document, the document signature, and the indication of the end of the ancillary information unit are the only components of the ancillary information unit. This achieves the detection of any tampering with the content of the ancillary information unit. The device 400 is configured to generate the auxiliary information unit using, for example, the generating component 412 of the processing circuit 410.

[0049] As mentioned above, the ancillary information unit may be an SEI message in the H.26x encoding format or a metadata OBU in the AV1 encoding format. Generally, the SEI contains information that is not necessary for decoding the encoded image frame. However, the ancillary information unit according to the present invention is necessary for verifying the authenticity of the encoded image frame. As mentioned above, the ancillary information unit is generated to include a document, a document signature, and an end indication. The document is generated by Taichi or multiple frame fingerprints Multiple sets ofSince the document contains a representation, it is necessary to verify the encoded image frames. Due to manipulation of the document content, it may be impossible to verify the authenticity of the encoded image frames without being able to use the document signature to verify the document. Therefore, if a single ancillary information unit is generated for the entire encoded video sequence, the document may be generated for the entire encoded video sequence. one or multiple frame fingerprints Multiple sets of

[0003] The document contains representations, and therefore the document is required to verify the authenticity of those encoded image frames. Due to manipulation of the document content, it may be impossible to verify the authenticity of the encoded image frames without being able to use the document signature to verify the document. Correspondingly, if one ancillary information unit is generated for each group of photos, the document for each ancillary information unit must be successfully verified to be able to verify the authenticity of the encoded image frames for that group of photos.

[0050] At act 514, the encoded video sequence is signed by associating the generated extrinsic information unit with the encoded video sequence. Device 400 is configured to sign the encoded video sequence, for example, using signature component 414 of processing circuitry 410. To sign the encoded video sequence, the generated extrinsic information unit may be associated with the encoded video sequence in various ways, some of which are described below.

[0051] In some embodiments, the generated extrinsic information unit is associated with the encoded video sequence by a reference to the encoded video sequence and transmitted on a channel different from the channel transmitting the encoded video sequence. The generated extrinsic information unit and the encoded video sequence may be associated with each other using a timestamp. For example, if the generated extrinsic information unit and the encoded video sequence have the same or corresponding timestamps, they are considered to be associated with each other. Thus, the generated extrinsic information unit may be signed with the encoded video sequence even if it is not transmitted together with the encoded video sequence, as long as a recipient can receive both the generated extrinsic information and the encoded video sequence and verify the signature. The channel transmitting the (smaller) extrinsic information unit may be a first communication channel, e.g., a secure communication path, and the channel transmitting the (larger) encoded video sequence may be a second communication channel, e.g., any communication path. Alternatively, the generated ancillary information units may be associated with the encoded video sequence by an association between the encoded video sequence and the ancillary information units outside of the data format, for example by forming a data structure (container) that includes both the encoded video sequence and the ancillary information units and in which the encoded video sequence is transmitted to the intended recipient. These alternatives are advantageous in that they do not require the power to modify the encoded video sequence (e.g., authorization granted by the owner of the video data) and they may therefore be performed by entities that do not have written access rights to the encoded video sequence.

[0052] In some alternative embodiments, the generated extrinsic information unit is associated with the encoded video sequence by adding the generated extrinsic information to the end of the encoded video sequence. This is shown schematically in Figure 7A, where an extrinsic information unit SIU has been added to the end of the encoded video sequence IPPPIPPPPIPP.

[0053] An encoded video sequence sometimes contains several groups of photos. In such an embodiment, the signature of the encoded video sequence may be considered to consist of multiple sub-signatures, i.e., multiple ancillary information units, which are provided for different segments of the encoded video sequence. This allows for continuous signature verification during playback. This also supports signing video data encoding live video streams, which is particularly useful for video monitoring applications.

[0054] For example, an encoded video sequence may include a first portion of encoded image frames associated with a first group of photos and a second, immediately subsequent portion of encoded image frames associated with a second group of photos. For such an encoded video sequence, an ancillary information unit may be generated for each group. Accordingly, a generated ancillary information unit may be generated for the first portion of encoded image frames. Furthermore, the generated ancillary information unit may be associated with the encoded video sequence by adding the generated ancillary information unit following the first group of photos associated with the first portion of encoded image frames. Although referred to as a group of photos in this disclosure, other terms, such as a group of frames, a frame group, a photo, etc., may also be used. Set of , and frame Set ofIt should be understood that the first and second groups of pictures may be referred to as the first GOP and the second GOP in some encoding formats. This is, for example, schematically shown in Figures 7B, 7C, 7D, and 7E. Here, the side information unit SIU1 generated for the first portion of the encoded image frames associated with the first group of pictures GOP1 is associated with the encoded video sequence by adding it after the first group of pictures GOP1. The same is true for the side information units SIU0, SIU2, and SIU3 generated for the groups of pictures GOP0 (not shown), GOP2, and GOP3, respectively.

[0055] In addition to being added following the first group of pictures as described above, the generated ancillary information unit may be associated with the encoded video sequence by adding it to the encoded video sequence before a second group of pictures associated with a second portion of the encoded image frames. Thus, the ancillary information unit may be added after the first group of pictures but before the second group of pictures. In other words, the generated ancillary information unit may be added to the encoded video sequence between the first and second groups of pictures. This may also be expressed as inserting the ancillary information unit into the encoded video sequence after the first group of pictures, e.g., after the last Inter frame, but before the second group of pictures, e.g., before the subsequent Intra frame. This is schematically illustrated in FIG. 7B, where ancillary information unit SIU1 is added between groups of pictures GOP1 and GOP2. The same is true for ancillary information unit SIU2 added between groups of pictures GOP2 and GOP3.

[0056] Alternatively, and in addition to being subsequently added to the first group of pictures as described above, the generated ancillary information unit may be associated with the encoded video sequence by adding the generated ancillary information to the encoded video sequence as part of a second group of pictures associated with a second portion of the encoded image frames. This ancillary information unit may thus be included in the second group of pictures. This is shown schematically in Figures 7C and 7D, where ancillary information unit SIU1 is included in group of pictures GOP2.

[0057] Because the ancillary information units are of the same type as the units that make up the encoded image frames, the ancillary information units can be added to the encoded video sequence at any desired location, e.g., sequentially in the encoded video sequence, without the need for a dedicated decoder to decode the encoded video sequence at the recipient's end.

[0058] A recipient holding an encoded video sequence that has been signed as described above can verify its authenticity by following these steps: 1. The document signature included in the accompanying information unit is verified by the sender using the public key of the private key with which the document signature was generated. Thus, if the received document signature is successfully verified using the public key and the received document, the content of the received document can be confirmed as authentic. For example, if the received document signature is successfully verified using the public key and a hash of the received document, the content of the received document can be confirmed as authentic. 2. Once the document signature has been successfully verified and the document is thereby proven to be authentic, e.g., not tampered with, verify the fingerprint contained in the document. 3. If all fingerprints in the document are successfully verified, conclude that the encoded video sequence associated with the ancillary information unit is authentic (validation).

[0059] Typically, verifying in step 2 involves replicating the fingerprinting operation supposedly performed by the sender, i.e., recalculating the fingerprint. For its part, verifying in step 1 typically involves an asymmetric signature setup, where signing and verifying are distinct cryptographic operations corresponding to private / public keys. Other combinations of symmetric and / or asymmetric verification operations are possible without departing from the scope of the invention.

[0060] Document signatures, as described herein, are digital signatures. In some embodiments, document signatures work by verifying that a digital message or document has not been intentionally or unintentionally altered since it was signed. Document signatures do this by generating a unique hash of the message or document and encrypting it using the sender's private key. The generated hash is unique to that message or document, and changing any part of it completely changes the hash. Once that is complete, the message or digital document is digitally signed and sent to the recipient. The recipient then generates their own hash of the message or digital document and decrypts the sender's hash (included in the original message) using the sender's public key. The recipient compares their generated hash with the sender's decrypted hash to see if they match, verifying that the message or digital document has not been altered, and that the sender is trusted.

[0061] Aspects of the present disclosure have been primarily described above with reference to certain embodiments, however, as will be readily apparent to those skilled in the art, other embodiments different from those disclosed above are equally possible within the scope of the invention as defined by the claims.

Claims

1. Obtaining an encoded video sequence (502) consisting of encoded image frames; generating (504) a set of one or more frame fingerprints for each encoded image frame, wherein a frame fingerprint is a unique identifier for the encoded image frame, and the frame fingerprints include: performing a hashing or checksum operation on said encoded image frame or a subset thereof; or digitally signing said encoded image frames or a subset thereof using asymmetric encryption; and generating (506) a document including a header of the ancillary information unit and a representation of the plurality of sets of the generated frame fingerprint(s); generating a document signature by digitally signing the document (510), wherein the document signature is a digital signature for the document, the digital signature being generated by applying asymmetric encryption to the document; generating (512) the auxiliary information unit consisting only of the document, the document signature, and an indication of the end of the auxiliary information unit; providing a signature for the encoded video sequence by associating the generated extrinsic information unit with the encoded video sequence (514); A method comprising:

2. generating a document fingerprint for the document (508); generating (510) the document signature for the document includes generating the document signature by digitally signing the document fingerprint; The method of claim 1.

3. The document a payload size value for a payload included in the auxiliary information unit; an identifier of the payload; and metadata about the encoded video sequence; and camera information regarding a camera capturing the video sequence; a public key of an asymmetric cryptographic key pair; The method of claim 1 , further comprising one or more of:

4. 2. The method of claim 1, wherein the generated extrinsic information unit is associated with the encoded video sequence by a reference to the encoded video sequence and transmitted on a channel different from a channel transmitting the encoded video sequence.

5. The method described in claim 1, wherein the generated ancillary information unit is associated with the encoded video sequence by adding the generated ancillary information to the end of the encoded video sequence.

6. The encoded video sequence a first portion of encoded image frames associated with a first group of photographs; a second, immediately following portion of the encoded image frames associated with a second group of photographs; Including, the generated extrinsic information unit is generated for the first portion of the encoded image frame; the generated ancillary information unit is associated with the encoded video sequence by subsequently adding the generated ancillary information unit to the first group of photographs associated with the first portion of the encoded image frames. The method of claim 1.

7. The method described in claim 6, wherein the generated ancillary information unit is associated with the encoded video sequence by adding the generated ancillary information to the encoded video sequence before the second group of photographs associated with a second portion of the encoded image frames.

8. The method described in claim 6, wherein the generated ancillary information unit is associated with the encoded video sequence by adding the generated ancillary information to the encoded video sequence as part of the second group of photographs associated with a second portion of the encoded image frame.

9. 2. The method of claim 1, wherein the encoded image frames include at least one intra-frame and one or more predicted inter-frames, the at least one intra-frame and the one or more predicted inter-frames being encoded according to a video encoding format that defines temporal video compression.

10. The representations of the plurality of sets of one or more generated frame fingerprints, The at least one intra-frame is generated and, a generated predicted inter frame for each of the one or more predicted inter frames; and, 10. The method of claim 9, comprising:

11. The representations of the plurality of sets of one or more generated frame fingerprints, The intraframes of the first group of photographs are generated. and, The generated predicted interframes of the first group of pictures and, The generated intraframes of the second group of photographs and, The method of claim 6, comprising:

12. The generated predicted interframe signal for each predicted interframe. is generated by hashing a combination of data resulting from the predicted Inter frame and data resulting from an Intra frame that the predicted Inter frame directly or indirectly references; the fingerprint of the predicted Inter frame is independent of any further predicted Inter frames to which the predicted Inter frame directly or indirectly refers; the data resulting from the predicted Inter frame includes one of image data of the predicted Inter frame and a generated first fingerprint of the image data of the predicted Inter frame; the data resulting from the intra-frame is a generated first fingerprint of the intra-frame; The method of claim 10.

13. the intraframe is an I-frame encoded according to the H.26x compression format, or an intraframe or keyframe encoded according to the AOMedia Video 1 (AV1) compression format; The predicted Inter frame is a forward predicted Inter frame (P frame) or a bidirectionally predicted Inter frame (B frame) encoded according to the H.26x compression format, or an Inter frame encoded according to the AV1 compression format.

10. The method of claim 9.

14. An imaging device for capturing a video sequence; an encoder for encoding the video sequence and outputting an encoded video sequence comprising encoded image frames; 1. A processor, comprising: generating a set of one or more frame fingerprints for each encoded image frame, wherein a frame fingerprint is a unique identifier for the encoded image frame, and the frame fingerprints include: performing a hashing or checksum operation on said encoded image frame or a subset thereof; or digitally signing said encoded image frames or a subset thereof using asymmetric encryption; and generating a document including a header of the ancillary information unit and a representation of the plurality of sets of the generated frame fingerprint(s); generating a document signature by digitally signing the document, the document signature being a digital signature for the document, the digital signature being generated by applying asymmetric encryption to the document; generating the ancillary information unit consisting solely of the document, the document signature, and an indication of the end of the ancillary information unit; providing a signature for the encoded video sequence by associating the generated extrinsic information unit with the encoded video sequence; a processor for performing a transmitter for transmitting the encoded video sequence with the signature; A device (400) comprising:

15. A non-transitory computer-readable storage medium on which a computer program (421) is stored, the computer program including instructions that, when executed by a computer, cause the computer to perform a method according to any one of claims 1 to 13.

Images