Editable video data signed on uncompressed data

Abstract

To provide editable video data signed on uncompressed data.SOLUTION: A video bitstream with signature contains a data unit, an associated signature unit, and a digital signature of a bit string derived from a fingerprint, and is obtained by predictive coding of a video sequence calculated from macro-blocks reconstructed from the data unit associated with the signature unit. The video stream with signature is edited by receiving a request to replace a region, reconstructing a first set of macro-blocks containing the region and a second set of macro-blocks referring thereto, adding an archive object containing a fingerprint of the first set of macro-blocks, editing a first set of data units and encoding them, and re-coding the second set of macro-blocks and adding a new data unit to the video bitstream with signature.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to the field of security devices for protecting video data from unauthorized activity, particularly in connection with data storage and transmission. The present disclosure proposes methods and devices for editing signed video bitstreams and verifying signed video bitstreams that may result from such editing. [Background technology]

[0002] Digital signatures provide an additional layer of verification and security for digital messages transmitted over insecure channels, allowing the authenticity or integrity of a message to be verified and ensuring non-repudiation. With regard to video coding in particular, there are secure and highly efficient methods described in the prior art for digitally signing predictively coded video sequences. See, for example, our previous published patent applications, EP 4164173 and EP 4164230. See also U.S. Patent Application Publication No. 20140010366, which proposes a cryptographic video verification technique specifically adapted for predictively coded video data with a group-of-pictures structure.

[0003] Video sequences may need to be edited after being signed. In addition to visual enhancement, editing can be aimed at ensuring privacy through cropping, masking, blurring, or similar image processing that makes visual features less recognizable. In most available methods, this requires re-encoding and re-signing the entire edited frame. The re-encoding and re-signing preferably extends to several neighboring frames as well, so as not to disrupt any predictive coding dependencies (inter-frame / intra-frame references) that may exist, even if the neighboring frames are not directly affected by the edit. These steps can consume significant computational resources and introduce annoying delays for the user.

[0004] U.S. Patent No. 7,437,007 discloses a method for performing region of interest editing of a video stream in the compressed domain. The compressed video stream includes compressed video stream frames representing video stream frames having unwanted portions and region of interest portions. According to the method, the compressed video stream frames are edited to modify the unwanted portions while maintaining the original structure of the video stream, and to obtain compressed video stream frames including the region of interest portions. To achieve this, the editing includes skipping macroblocks located above, below, and to the right of the region of interest portions in predictively coded (P) frames and bidirectionally predictively coded (B) frames. The video stream under consideration in U.S. Patent No. 7,437,007 is not a signed video stream. Summary of the Invention

[0005] One object of the present disclosure is to make available a method for editing a signed video bitstream obtained by predictive coding of a video sequence that largely avoids the need to re-sign the bitstream outside of the portions affected by the edit, as is the case with some available methods. A particular object is to make available such a video editing method that preserves the signatures of all macroblocks except those that are edited. A further object is to enable video editing without significantly compromising the data security of the original signed video bitstream. A further object is to provide a method for verifying a signed video bitstream obtained by predictive coding of a video sequence. It is a still further object to provide a device and computer program for these purposes.

[0006] At least some of these objects are achieved by the invention as defined by the independent claims. The dependent claims relate to advantageous embodiments of the invention.

[0007] In a first aspect of the present disclosure, a method for editing a signed video bitstream obtained by predictive coding of a video sequence is provided. The signed video bitstream is intended to include data units and associated signature units. Each data unit represents (i.e., encodes) at most one macroblock in a video frame of the predictively coded video sequence. Each signature unit includes a digital signature of a bit string derived from multiple fingerprints, each fingerprint calculated from a macroblock reconstructed from one data unit associated with the signature unit. The signature unit may optionally include the bit string to which the digital signature pertains (a "document approach"). For a signed video bitstream having these characteristics, the method includes receiving a request to replace a region of at least one video frame; reconstructing a first set of macroblocks in which the region is contained and a second set of macroblocks that directly or indirectly reference the macroblocks in the first set; adding an archival object to the signed video bitstream, the archival object including a fingerprint calculated from the reconstructed first set of macroblocks; editing the first set of macroblocks in accordance with the replacement request and encoding the edited first set of macroblocks as a first set of new data units; re-encoding the second set of macroblocks as a second set of new data units; and adding the first and second sets of new data units to the signed video bitstream.

[0008] Because data units and macroblocks have a one-to-one relationship, a data unit does not represent multiple macroblocks (nor parts of multiple macroblocks), and therefore the direct effect of editing some macroblocks is limited to one or more data units (a first set) of the edited macroblock. Furthermore, because each fingerprint is calculated from a macroblock reconstructed from one associated data unit, the need to re-sign data units after editing is limited. More precisely, the method according to the first aspect preserves any predictive coding dependencies connecting pairs or groups of macroblocks by re-encoding a set (a second set) of data units representing macroblocks that directly or indirectly reference the edited macroblock. When re-encoding is limited to this second set of data units, the method efficiently utilizes available computational resources. This allows the method to be executed with good performance on conventional processing devices.

[0009] The method according to the first aspect includes an additional benefit on the recipient side. Because of the archiving object from which fingerprints associated with the first set of data units can be obtained, the recipient can verify all data units of the signed video bitstream that have not been affected by the editing. This allows a significant portion of the existing signatures to be preserved, and in this respect, the video editing method according to the first aspect is said to be minimally destructive. Verification at the recipient side is described in detail within the second aspect of this disclosure. Importantly, fingerprints associated with the second set of macroblocks do not need to be archived, since re-encoding is an operation that should not change the visual appearance of these macroblocks.

[0010] In some embodiments, the data security of an edited signed video bitstream is improved by adding one or more signature units associated with the first set of edited macroblocks to the bitstream. This prevents scenarios of unauthorized modification of the first set of edited macroblocks. The one or more signature units may be new signature units added to the bitstream or edited (alternative) versions of signature units included in the signed video bitstream before editing. The integrity of the second set of macroblocks is protected by digital signatures already present in the signature units in the bitstream. Indeed, since fingerprints are calculated from reconstructed macroblocks and the re-encoding operation should negligibly or not at all change their visual appearance, a recipient can verify the integrity of the second set of macroblocks using the existing digital signatures.

[0011] In some embodiments, the archive object further includes the locations of the first set of macroblocks. The locations may refer to the positions of the macroblocks within the frame, for example, in frame coordinates. If static macroblock partitioning is used, the positions of the macroblocks may be expressed as macroblock sequence numbers or another identifier. This provides one way of assisting a recipient of an edited video bitstream to determine whether a particular macroblock has been modified and, therefore, to select an appropriate method for obtaining a fingerprint of the data unit representing that macroblock.

[0012] In some embodiments, the second set of macroblocks is losslessly re-encoded, thereby obtaining new data units. The use of lossless encoding ensures that macroblocks subsequently reconstructed from these new data units will not differ from the second set of macroblocks. As a result, because the fingerprint is calculated from the reconstructed pixel / plaintext data, the new data units remain consistent with the signature units already present in the video bitstream before editing. In other embodiments, the second set of macroblocks is re-encoded using reduced data compression, and the fingerprint of the second set of macroblocks in the signed video bitstream includes a robust hash. Robust hashing refers to a class of algorithms that have a tolerance such that the algorithm accepts a data set as authentic even if it is subject to small differences. The algorithm may be optimized for hashing image or video data. The algorithm's tolerance may be configurable and may be set so that errors corresponding to tampering are detected while normal errors expected from data compression are not (they are small differences in the sense described above).

[0013] In some embodiments, the second set of macroblocks are recoded nonpredictively. Nonpredictive coding may correspond to using only I-frames. This is a simple and robust method of recoding the second set of macroblocks. If editing is expected to be relatively infrequent, the additional memory or bitrate costs are unlikely to be significant. In other embodiments, the second set of macroblocks are recoded predictively with reference to the first set of edited macroblocks. For example, the same Group of Pictures (GoP) structure can be preserved for continuity. However, these embodiments may not perform as well from a data compression perspective because they involve an attempt to predict across discontinuities introduced by editing. If the discontinuities are large, B or P frames may be larger than usual or may experience localized degradation in quality.

[0014] In some embodiments, the first set of edited macroblocks is losslessly and / or encoded using reduced data compression and / or non-predictively. Each of these data compression techniques limits or avoids further loss of image quality beyond that caused by the original image encoding operation. Because the edited macroblocks are likely to be used or studied more carefully than the rest of the video sequence, the additional computational and / or memory costs for this encoding are likely justified. Furthermore, for important use cases involving video surveillance, the size of the edited portion is typically negligible compared to the amount of data generated by continuous surveillance.

[0015] In some embodiments, each time a fingerprint is calculated in connection with encoding a macroblock, a fingerprinting operation is performed on image data (e.g., plaintext data, pixel data) in a reference buffer of the encoder. This may be implemented in an encoder device that prepares a signed video bitstream. It may also be implemented in an editing tool that performs the editing method according to the first aspect. Because a reference buffer is a necessary component of most encoder implementations (i.e., to ensure correct predictive coding), the reconstructed video data used for fingerprinting can be obtained without additional computational cost.

[0016] In some embodiments, the second set of macroblocks are non-predictively re-encoded, while in other embodiments, the second set of macroblocks are predictively re-encoded with reference to the edited first set of macroblocks.

[0017] According to a generalization of the first aspect, there is provided a video editing method performed on a signed video sequence including data units and associated signature units. Each macroblock in a video frame of a predictively coded video sequence is coded by at most M data units. Edits made to a macroblock 110 directly affect at most M data units 120, where M is a small integer such as 1, 2, 3, 4, 5, or at most 10. Each signature unit includes a digital signature of a bit string derived from a plurality of fingerprints, each fingerprint calculated from a macroblock reconstructed from each of at most M associated data units, which optionally also include the bit string. The video editing method includes receiving a request to replace a region of at least one video frame, reconstructing a first set of macroblocks in which the region is included and a second set of macroblocks that directly or indirectly reference the macroblocks in the first set, adding an archive object to a signed video bitstream that includes a fingerprint calculated from the reconstructed first set of macroblocks, editing the first set of macroblocks in accordance with the replacement request, re-encoding the second set of macroblocks, and adding the first and second sets of new data units thus obtained to the signed video bitstream.

[0018] According to a further generalization of the first aspect, there is provided a video editing method performed on a signed video sequence comprising data units and associated signature units. Each data unit represents (encodes) at most N macroblocks within a video frame of the predictively coded video sequence. This means that the effect of edits made to a macroblock is not limited to the macroblock itself, but may require re-encoding and / or re-signing one or more data units that the edited macroblock shares with further macroblocks, where N is a small integer such as 1, 2, 3, 4, 5, or at most 10. Each signature unit comprises a digital signature of a bit string derived from a plurality of fingerprints, each associated with exactly one associated data unit, and optionally a bit string. The video editing method includes receiving a request to replace a region of at least one video frame, reconstructing a first set of macroblocks in which the region is included and a second set of macroblocks that directly or indirectly reference the macroblocks in the first set, adding an archive object to a signed video bitstream that includes a fingerprint calculated from the reconstructed first set of macroblocks, editing the first set of macroblocks in accordance with the replacement request, re-encoding the second set of macroblocks, and adding the first and second sets of new data units thus obtained to the signed video bitstream.

[0019] In a second aspect of the present disclosure, a method for verifying a signed video bitstream obtained by predictive coding of a video sequence is provided. It is understood that the signed video bitstream includes data units, signature units associated with some of the data units, and an archival object. Each data unit represents a macroblock within a frame of the predictively coded video sequence. Each signature unit includes a bit string and, optionally, a digital signature for the bit string itself. It is expected that the verification method will at least partially verify that the bit string is derived from multiple fingerprints, each fingerprint being calculated from a macroblock reconstructed from one data unit associated with the signature unit. Finally, the archival object includes at least one archived fingerprint. The method for verifying the signed video bitstream includes reconstructing macroblocks from the data units associated with the signature unit, calculating respective fingerprints from at least some of the reconstructed macroblocks, obtaining the at least one archived fingerprint from the archival object, deriving bit strings from the calculated and obtained fingerprints, and verifying the data units associated with the signature unit using the digital signature in the signature unit.

[0020] Verification of the data unit may involve verifying the derived bit string using the digital signature. Alternatively (the "document approach"), the verification step involves verifying the bit string in the signature unit using the digital signature and then comparing the derived bit string with the verified bit string.

[0021] Although the archive object may have been added by performing the editing method according to the first aspect, the method according to the second aspect can be implemented without reliable knowledge of such prior processing. Thus, the method according to the second aspect achieves verification of the authenticity of the video sequence in that it verifies that the digital signature (and any bit sequence) carried in the signature unit actually matches the fingerprint for the associated data unit. Therefore, since this means that the corresponding reconstructed macroblock has been modified, the data unit may also have been unchanged.

[0022] The method according to the second aspect includes two options for obtaining fingerprints: by direct calculation from reconstructed macroblocks or by recovery from the archive object. This supports a process that minimizes the destruction of existing fingerprints during the editing phase (first aspect). The fact that each data unit represents one macroblock tends to limit the number of fingerprints that need to be archived for a given replacement request, and therefore the size of the archive object.

[0023] In some embodiments, the archive object further indicates the location of the macroblock represented by the data unit to which the archived fingerprint relates. In other words, the archived fingerprint is a fingerprint of a macroblock reconstructed from the data unit, and the location of the macroblock is indicated in the archive object. During execution of the method according to the second aspect, to obtain a fingerprint for the data unit, it is determined whether to calculate the fingerprint or obtain the fingerprint from the archive object based on the location indicated by the archive object.

[0024] As with the first aspect of the present disclosure, the second aspect can be generalized to the cases discussed above, i.e., a numerical ratio of macroblocks to corresponding data units of 1:M or N:1.

[0025] A third aspect of the present disclosure relates to devices configured to perform the methods of the first and / or second aspects. These devices may be incorporated into systems with different primary purposes (e.g., video recording, video content management, video playback), or may be dedicated to the aforementioned editing and verification, respectively. Devices within the third aspect of the present disclosure generally share the effects and advantages of the first and second aspects, which may be embodied with an equivalent degree of technical variation.

[0026] In a fourth aspect, a signed video bitstream includes data units and associated signature units, each data unit representing at most one macroblock in a video frame of a predictively coded video sequence, and each signature unit including a digital signature s(H1) of a bit string H1 derived from a plurality of fingerprints, each fingerprint h1, h2, ... being calculated from a macroblock reconstructed from one or more data units associated with the signature unit. In particular, a signed video bitstream includes data units and associated signature units, each data unit representing exactly one macroblock in a video frame of a predictively coded video sequence, and each signature unit including a digital signature s(H1) of a bit string H1 derived from a plurality of fingerprints, each fingerprint h1, h2, ... being calculated from a macroblock reconstructed from exactly one data unit associated with the signature unit. As noted above, if a portion of a video bitstream is expected to be edited at a later point in time, a video bitstream having this format can provide certain advantages, including the ability to reuse existing signature units for the purpose of verifying the unedited portion. Since the signature is attached to the decrypted image data, there is no need to handle dependencies between macroblocks. Yet another advantage is that fingerprinting (e.g., hashing) can be implemented at a very fine granularity, such as at the level of a single pixel.

[0027] The present invention further relates to a computer program comprising instructions for causing a computer to carry out the above-described method. The computer program can be stored or distributed on a data carrier. As used herein, a "data carrier" may be a transitory data carrier, such as a modulated electromagnetic or light wave, or a non-transitory data carrier. Non-transitory data carriers include volatile and non-volatile memories, such as permanent and non-permanent storage media of the magnetic, optical, or solid-state type. Still within the scope of "data carrier," such memories may be fixedly attached or mobile.

[0028] In a further aspect of the present disclosure, a signed video bitstream is provided that includes data units and associated signature units. Each data unit represents at most N macroblocks in a video frame of a predictively coded video sequence, where N is a small integer, such as 1, 2, 3, 4, 5, or at most 10. Each signature unit includes a digital signature of a bitstring derived from multiple fingerprints, each associated with exactly one associated data unit, and optionally the bitstring itself. The signed video bitstream is edit-friendly because the direct effect of editing one macroblock is limited to at most N data units of the edited macroblock, and each fingerprint is a fingerprint of exactly one associated data unit. This limits the propagation of edits to a limited number of data units, resulting in fewer data units needing to be re-signed after editing.

[0029] It should be noted that, as used in this disclosure, a "macroblock" may advantageously be a coded macroblock. However, the present invention is also applicable to non-predictively coded video, and in a more generalized sense, a macroblock can therefore be any group of contiguous pixels. Because video signatures are performed on decoded frames, pixel groupings need not be limited to any coded group divisions.

[0030] In general, all terms used in the claims should be interpreted according to their ordinary meaning in the art unless otherwise expressly defined herein. All references to a / an / the element, apparatus, component, means, step, etc. should be interpreted open-endedly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless otherwise specified. The steps of any method disclosed herein do not have to be performed in the exact order described, unless explicitly stated.

[0031] Aspects and embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0032] [Figure 1] 1 illustrates two exemplary patterns of intra-frame referencing within macroblocks in a video frame. [Figure 2] 1 shows a sequence of frames representing one Group of Pictures (GoP) with an exemplary pattern of inter-frame referencing. [Figure 3] 1 shows four exemplary correspondence patterns between data units and the macroblocks they represent. [Figure 4] It shows editing operations that directly affect some macroblocks in the first frame (first column) and lead to consequent changes in further frames (second and third columns) until the end of the GoP. [Figure 5] Shown from top to bottom are a signed video bitstream, macroblocks reconstructed from the bitstream, and the effects of edit operations performed on portions of the video bitstream. [Figure 6] 1 is a flowchart of a method for editing a signed video bitstream according to an embodiment of the present specification. [Figure 7]1 is a flowchart of a method for verifying a signed video bitstream according to an embodiment herein. [Figure 8] 8 shows a device suitable for carrying out the methods shown in FIGS. 6 and 7. [Figure 9] Several such devices are shown connected via local and / or wide area networks. [Figure 10A] (Document Approach) The signed video bitstream and some operations within the method are shown in FIG. [Figure 10B] The signed video bitstream and some operations within the method shown in FIG. 7 are shown. [Figure 11] 6 and 7 show the signal processing operations and data flows that occur during the execution of the methods shown in FIGS. 6 and 7, as well as functional units suitable for performing the signal processing operations. DETAILED DESCRIPTION OF THE INVENTION

[0033] Aspects of the present invention will be described more fully hereinafter with reference to the accompanying drawings, in which specific embodiments of the invention are shown. These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of all aspects of the present invention to those skilled in the art. Like numerals refer to like elements throughout the description.

[0034] In the terminology of this disclosure, a "video bitstream" includes any substantially linear data structure that may resemble a sequence of bit values. A video bitstream can be carried by a transitory medium (e.g., modulated electromagnetic or optical waves), as in some streaming use cases, or the video bitstream can be stored in a non-transitory medium, such as volatile or non-volatile memory.

[0035] A video bitstream represents a video sequence, which can be understood as a sequence of video frames played sequentially at a nominal time interval. Each video frame can be divided into macroblocks. Further in this disclosure, a "macroblock" can be a transform block, a prediction block, or both within a frame of a video sequence. The use of "frame" and "macroblock" herein is intended to be consistent with the H.26x video coding standard or similar specifications. A "macroblock" can also be a coding block. As noted above, although the term macroblock is used, the grouping of pixels need not be limited to any partitioning used for coding; rather, a macroblock can be any group of adjacent pixels. When applied to the case of prediction-based coding, it can be advantageous to use the same partitioning used for coding.

[0036] 1A and 1B show an exemplary partition of a video frame 100 into a uniform 4x4 arrangement of macroblocks 110. Note that this is a simplification for illustrative purposes. In practice, a video frame 100 is typically divided into a much larger number of macroblocks; for example, macroblocks may be 8x8, 16x16, or 64x64 pixels. Curly arrows are used consistently herein to indicate intra-frame or inter-frame references used in predictive coding. Without departing from the scope of this disclosure, the partition into macroblocks seen in FIGS. 1A and 1B can be significantly modified to include non-square arrangements and / or arrangements of macroblocks 110 that are not homogeneous with respect to the video frame 100 and / or mixed arrangements of different macroblocks 110 having different sizes or shapes. It will be understood that some video coding formats support dynamic macroblock partitioning, i.e., partitions may be different for different video frames within a sequence. This is true, for example, for H.265.

[0037] In FIG. 1A , the intraframe reference pattern is limited to a single row of macroblocks 110. In fact, each macroblock 110 points to the macroblock immediately to its left (if present in frame 100) in the sense that it is represented by a data unit that predictively represents the image data in the macroblock; that is, the image data in this macroblock is represented relative to the image data in the macroblock to the left. Conceptually, and somewhat simplified, the data units represent the macroblock 110 in terms of change or movement relative to the left macroblock. Another possible understanding is that the data units represent corrections to a given prediction operation that derives the macroblock 110 from the left macroblock. In alternative examples within the scope of this disclosure, a macroblock 110 may point to another macroblock 110 to its right, or above or below it.

[0038] In FIG. 1B, the intraframe reference pattern is denser. Here, each macroblock 110 points to its immediately left macroblock (if present in frame 100) and its immediately above macroblock (if present in frame 100). Thus, macroblock 110 is represented by a data unit that predictively represents image data within the macroblock, e.g., in terms of differences or corrections to predictions of this image data, based on a predetermined interpolation operation (or some other predetermined combination) that operates on image data in the macroblocks to the left and above. The interpolation may include post-processing operations such as smoothing. A further alternative within the scope of the present disclosure is to use intraframe references in the opposite direction to that shown in FIG. 1B, i.e., starting from the bottom row.

[0039] Image / video formats with a predefined intraframe reference pattern can be associated with a specified scan order, which represents a possible sequence for decoding macroblocks. In Figure 1A, the macroblock scan order is not unique, i.e., each row can be decoded independently. In the case of the pattern according to Figure 1B, macroblocks can be decoded row-wise from the top or column-wise from the left. Video formats with this reference pattern can specify a column-wise or row-wise scan order so that any reconstruction errors can be anticipated at the encoder side, which benefits coding efficiency. Furthermore, video formats with arbitrary macroblock orders or so-called slicing exist.

[0040] FIG. 2 illustrates a video sequence V including a sequence of frames 100. The video sequence V includes independently decodable frames (I-frames) and predicted frames, including unidirectionally predicted frames (P-frames) and bidirectionally predicted frames (B-frames). The International Telecommunications Union's ITU-T Recommendation H.264 (06 / 2019) "Advanced video coding for generic audiovisual services" specifies a video coding standard in which both forward and bidirectionally predicted frames are used. As can be seen in FIG. 2, the independently decodable frames do not reference any other frames. The unidirectionally predicted frames (P-frames) in FIG. 2 are forward predicted in that they directly reference at least one other preceding or immediately preceding frame. The bidirectionally predicted frames (B-frames) can further directly reference subsequent or immediately succeeding frames within the video sequence V. A first frame indirectly references a second frame if the video sequence includes a third frame (or a subsequence of frames) that the first frame directly references and that directly references the second frame. In predictive video coding, a Group of Pictures (GoP) is defined as a subsequence of video frames that does not reference any video frames outside the subsequence, and that can be decoded without referencing any other I, P, or B frames. The video frames in Figure 2 form a GoP. The GoP in Figure 2 is the smallest because it cannot be subdivided into further GoPs.

[0041] In a simpler implementation, a video sequence V may consist only of independently decodable frames (I) and unidirectionally predicted frames (P). Such a video sequence may have the appearance IPPIPPPPIPPPIPPP, with each P-frame pointing to the previous I-frame or P-frame. In this example, GoPs such as IPP, IPPPP, IPPP, IPPP, and so on may be identified.

[0042] There are several options for coordinating inter- and intra-frame predictive coding. For example, if static (fixed) macroblock partitions are used in all video frames, inter-frame references, such as those illustrated, may be defined at the level of one macroblock position at a time (e.g., the top-left macroblock in Figures 1A and 1B). Some video formats allow dynamic macroblock partitions; for example, a macroblock can be predicted from corresponding pixels in a preceding video frame or from spatially shifted pixels in a preceding frame. Alternatively, inter-frame references are defined for the entire video frame. I-frames consist of only I-blocks, and P-frames can consist of only P-blocks or a mix of I- and P-blocks.

[0043] Referring now to Figure 3, attention is directed to a video bitstream encoding the video sequence under consideration. Sub-Figures 3A, 3B, 3C, and 3D illustrate different correspondence patterns between the data units 120 and the macroblocks 110 that they represent (encode). For purposes of this disclosure, the data units may have any suitable format and structure, and no assumption is made other than that the data units can be separated (or extracted) from the video bitstream to, for example, enable processing without the need to decode that data unit or any surrounding data units. In addition to data units, a signed video bitstream further includes signature units that are separable from the signed video bitstream in the same or similar manner. More details regarding signature units are presented below with reference to Figure 5.

[0044] 3A, data units 120 correspond one-to-one to macroblocks 110 (dashed lines). This correspondence pattern ensures that each macroblock 110 is always reconstructed from one data unit 120. Furthermore, when a macroblock 110 is edited (which may in fact require changes to signature units, metadata, etc.), no other data units 120 other than the corresponding data unit 120 need be modified.

[0045] Alternatively, as shown in Figure 3B, each macroblock 110 is encoded by multiple data units 120, where each data unit 120 represents at most one macroblock 110. Thus, if each macroblock 110 is encoded by at most M data units 120, then any macroblock 110 can be reconstructed from at most M data units 120. Edits made to a macroblock 110 directly affect at most M data units 120, where M is a small integer such as 1, 2, 3, 4, 5, or at most 10.

[0046] In a further option, as shown in FIG. 3C , each data unit 120 encodes multiple macroblocks 110. This means that the effects of edits made to a macroblock 110 are not limited to the macroblock 110 itself, and it may be necessary to re-encode and / or re-sign data units 120 that the edited macroblock 110 shares with additional macroblocks 110. Because performing re-signing on unnecessarily large data sets can be computationally wasteful, this correspondence pattern, while quite feasible to implement, is not applied in the highest-performance embodiments of this disclosure. If a data unit 120 is allowed to be shared by at most a predetermined number N of macroblocks 110, the total amount of additional computation can remain limited. Again, N can be specified to be a small integer, such as 1, 2, 3, 4, 5, or at most 10.

[0047] 3B and 3C are possible within the scope of this disclosure. As a result, the techniques proposed herein may be applied to video bitstreams in which the ratio of macroblocks 110 to data units 120 may be 2:1, 1:1, 2:2, 3:3, 4:2, 2:4, or 4:4.

[0048] 3D, each data unit 120 can represent any number of macroblocks 110 in a video sequence, and each macroblock 110 can be encoded by any number of data units 120. This correspondence pattern, as suggested by the dashed line, can mean that in the worst case, even limited editing operations on macroblocks 110 require a complete re-encoding and re-signing of the video sequence. The techniques disclosed herein should not be implemented on video sequences having the structure shown in FIG. 3D.

[0049] FIG. 4 illustrates the editing operations described below.

[0050] FIG. 5 illustrates a section of a video sequence V including a series of macroblocks 110 belonging to one or more video frames. For example, the macroblocks 110 may occupy fixed positions within successive video frames (e.g., the top-left macroblocks in FIGS. 1A and 1B). The macroblocks 110 are assumed to be predictively coded according to the references indicated by the curly arrows. It can be seen that the first five macroblocks 110 belong to one Group of Prediction (GoP), and the following five macroblocks 110 belong to the next Group of Prediction (GoP). The video sequence V is encoded as a signed video bitstream B including data units 120 and signature units 130. For illustrative purposes and not limitation, FIG. 5 illustrates the data units 120 as encoding the video macroblocks 110 according to the corresponding pattern shown in FIG. 3A, i.e., a one-to-one relationship between the macroblocks 110 and the data units 120. In other words, each data unit 120.n is created by applying the encoder Enc to the corresponding macroblock 110.n. Data units 120 may conform to a proprietary or standardized video coding format, such as ITU-T H.264, H.265, or AV1. Bitstream B may further include additional types of units (e.g., dedicated metadata units) without departing from the scope of this disclosure.

[0051] Each signature unit 130 may be associated with multiple data units 120. While it is understood that in FIG. 5 , the data units 120 between two consecutive signature units 130 are associated with the subsequent signature unit 130, this is not an essential feature of the present invention, and other arrangements are possible without departing from the scope of the present disclosure. A signature unit 130 may be associated with a set of data units 120 that are all contained in one Group of Pictures (GoP), although other association patterns are possible. Furthermore, the set of data units 120 associated with a signature unit 130 is preferably selected taking into account the applicable macroblock scanning order. For example, the set of data units 120 associated with a signature unit 130 may represent the number of macroblocks to be sequentially scanned during decoding, thereby minimizing the number of macroblocks that need to be re-referenced if a signature unit 130 fails verification.

[0052] A signature unit 130 includes at least one bit string (e.g., H1) and a digital signature for that bit string (e.g., s(H1)). As suggested by the use of dashed lines, the presence of a bit string is optional. If a signature unit 130 includes multiple bit strings, the signature unit 130 can have one digital signature for all of these bit strings, or multiple digital signatures, each for a single bit string or each for a subgroup of bit strings. The bit string on which the digital signature is formed may be a combination of fingerprints calculated based on a macroblock 111 reconstructed from the data unit 120 associated with the signature unit 130, or the bit string may be a fingerprint of the combination of fingerprints. More precisely, the fingerprint is the fingerprint of the reconstructed macroblock, which can be obtained by reading a so-called reference buffer in the encoder or by performing an independent decoding operation. The combination of fingerprints (or "references") may be a list or other concatenation of string representations of fingerprints. In the ITU-T H.264 and H.265 formats, the signature unit may be included as a Supplemental Enhancement Information (SEI) message in the video bitstream. In the AV1 standard, the signature may be included in a Metadata Open Bitstream Unit (OBU).

[0053] Each fingerprint may be a hash or a salted hash. A salted hash may be a hash of a data unit (or a portion of a data unit) combined with a cryptographic salt; the presence of the salt can prevent an unauthorized party with access to multiple hashes from guessing which hash function is being used. Potentially useful cryptographic salts include the value of an active internal counter, a random number, and the time and location of the signature. The hash may be generated by a hash function (or one-way function) h, which is a cryptographic function that provides a level of security deemed appropriate given the confidentiality of the video data being signed and / or the value that would be compromised if the video data were manipulated by an unauthorized party. Three examples are SHA-256, SHA3-512, and RSA-1204. The hash function should be predefined (e.g., reproducible) so that the fingerprint can be regenerated when the recipient attempts to verify it. In the example of Figure 5, the bit string is given by: H1=h([h1, h2, h3, h4, h5]) and H2=h([h6, h7, h8, h9, h 10 ]), where h1, h2, ... are hashes of macroblocks 111.1, 111.2, ... and [·] represents concatenation. The concatenation operation may be linear (side-by-side) or may provide a staggered arrangement of the data. The successive operations may further include arithmetic operations on the data, such as bitwise OR, XOR, multiplication, division, or modulo operations. An exemplary salted hash may be defined as follows: TIFF0007789738000001.tif5170 or TIFF0007789738000002.tif5170 where σ is a cryptographic salt. In the first example, the hash function h has a parametric dependence on the second argument, which is assigned a salt σ.

[0054] In some embodiments, each of the fingerprints h1, h2... is calculated from macroblocks (e.g., pixel values ​​or other plaintext data) reconstructed from data unit 120. The fingerprints are calculated as follows: h1 = h(Y 111.1 ) or h1=h([Y 111.1 , σ]) or h1 = h(Y 111.1 , σ), where Y 111.1 represents data from the first one of the reconstructed macroblocks 111, and σ is an optional cryptographic salt. In a third option, the hash function h has a parametric dependence on the second argument to which the salt σ is assigned. The fingerprint can be calculated from the entire macroblock or from a subset thereof extracted according to a pre-agreed rule. In a variant of these embodiments, the fingerprints h1, h2, ... are calculated not at the plaintext level but from intermediate reconstructed data derived from the data units. More precisely, when an encoder is used that includes a frequency domain transform (e.g., DCT, DST, DFT, wavelet transform) followed by a coding process (e.g., entropy, Huffman, Lempel-Ziv, run-length, binary or non-binary arithmetic coding, e.g., context-adaptive variable length coding, CAVLC, context-adaptive binary arithmetic coding, CABAC), the transform coefficients are usually available as intermediate reconstructed data at the decoder side. The transform coefficients can be recovered from the coded representation. If the encoder further includes a quantization process immediately downstream of the transform, the quantized transform coefficients will be available at the decoder side. In more complex codecs, there may be more types of intermediate reconstructed data at more successive processing stages, which can be used for the fingerprint calculation. It is particularly convenient to use types of intermediate reconstructed data that appear identically in the encoding process as the quantized transform coefficients. Common to all embodiments considered in this paragraph, the fingerprint relates to exactly one data unit 120, one of the data units 120 associated with the signature unit 130.

[0055] Optionally, fingerprints can be linked together in sequence to detect unauthorized removal or insertion of data units. That is, each fingerprint depends on the next or previous fingerprint, e.g., the input to the hash includes the hash of the next or previous fingerprint. The linking can be done, for example, by: h1 = h(Y 111.1 ), h2=h([h1, Y 111.2 ]), h3=h([h2, Y 111.3 ]) and Y 111.1 , Y 111.2 , Y 111.3 denotes data from the first, second, and third macroblocks of the reconstructed macroblock 111. Another way to link fingerprints is by using h1 = h(Y 111.1 ), h 12 =h([Y 111.1 , Y 111.2 ]), h 13 =h([Y 111.2 , Y 111.3 ]) etc.

[0056] 5, a cryptographic element (not shown) with a pre-stored private key can be utilized to generate the digital signature s(H1). The recipient of the signed video bitstream can be expected to hold a public key belonging to the same key pair (see also FIG. 10), which allows the recipient to verify that the signature generated by the cryptographic element is authentic but not generate a new signature. The public key can also be included as metadata in the signed video bitstream, in which case it need not be stored at the recipient's end.

[0057] Next, referring to FIG. 6, a method 600 for editing a signed video bitstream B obtained by predictive coding of a video sequence V will be described. It is assumed that the non-optional steps of method 600 are performed after the original signing of the video bitstream. For example, if the signed video bitstream is originally generated on a recording device, editing method 600 can be performed on a video management system (VMS). Another exemplary use case is when the signed video bitstream is generated on a device, stored in memory, and then re-referenced for editing using the same device. Editing may also be performed at a later point in time, for example, after the need to perform privacy masking is known.

[0058] As mentioned above, a device that performs editing method 600 may be a dedicated application or system, but may have the basic functional structure shown in FIG. 8. As shown, device 800 includes processing circuitry 810, memory 820, and external interface 830. Memory 820 may be suitable for storing a computer program 821 having instructions that implement editing method 600. External interface 830 may be a communications interface that allows device 800 to communicate with similar devices (not shown) maintained by a recipient and / or video content author (e.g., a recording device), or may enable read and write operations in external memory 890 suitable for storing video bitstreams.

[0059] 9 illustrates the case where a bitstream is transferred between multiple devices. Note that the device performing the editing method 600 may be connected to the recipient device via a local area network (connecting lines in the bottom half of FIG. 9) or a wide area network 990. An attack on bitstream B can occur over either type of network, which justifies the signature.

[0060] 6, one embodiment of method 600 begins with step 612 of receiving a request to replace a region of at least one video frame 100 in video sequence V. The request may be received via a human-machine interface or in an automated manner, for example, in a message from a controlling application running on the same device 800 or remotely. The region to be replaced may be a set of replacement pixel values, such as a privacy mask, for replacing similarly located pixels in video sequence V.

[0061] For the avoidance of doubt, it should be noted that the video sequence V to be edited is encoded by predictive coding as a signed video bitstream B comprising data units 120 and associated signature units 130, each data unit representing at most one macroblock 110 in a video frame 100 of the predictively coded video sequence V, and each signature unit comprising a digital signature of a bit string derived from a plurality of fingerprints, each associated with exactly one associated data unit. Such a bitstream format is illustrated with reference to Figure 5.

[0062] In the next step 614 of the method 600, a first set of macroblocks containing the region and a second set of macroblocks that directly or indirectly reference the macroblocks in the first set are determined and reconstructed. In FIG. 5, the reconstruction corresponds to the arrow symbolizing the decoding operation Dec. Recalling that bidirectionally predicted frames (B-frames) may be defined in some video coding formats, it is understood that the second set of macroblocks may be located before or after the first set of macroblocks, or may occupy both of these positions. It is understood that the first and second sets are defined to be disjoint. For example, it may be defined that a macroblock belongs to the second set only if it does not belong to the first set, i.e., if this macroblock is not needed to form the set of macroblocks containing the region to be replaced. Thus, if the first set of macroblocks extends to a Group of Pictures (GoP) boundary, the second set of macroblocks is typically empty. Furthermore, it is understood that the second set of macroblocks may contain macroblocks in more than one P-frame or more than two B-frames, since additional frames may be used as references for the region to be replaced, depending on the video encoder initially used. In step 614, as a result of inter- or intra-frame prediction references between macroblocks, it may be necessary to reconstruct more macroblocks than just the first and second sets. More precisely, one or more macroblocks located earlier in the chain of prediction references leading to the first and second sets of macroblocks may have to be reconstructed first.

[0063] If the region to be replaced is limited to a single video frame, the first set of macroblocks can be determined by referencing only the macroblock partitions of the frame. More precisely, the first set is all macroblocks with which the region overlaps (i.e., macroblocks with which the region has a non-empty intersection in pixel space). If the region extends to multiple frames, this operation is repeated for each frame. In the special case where the region is repeated identically in all video frames and the macroblock partitions are constant across all such frames, the first set of macroblocks is a copy of that determined for the initial frame (by the overlap criterion) for each subsequent frame. The second set of macroblocks can be determined based on the first set and the pattern of intra- and inter-frame references within the signed predictive coded video sequence. Because such references by definition do not extend beyond Group of Picture (GoP) boundaries, the search for macroblocks to be included in the second set can be limited to the Group of Picture (GoP) to which the first set of macroblocks belong or to those Groups of Picture (GoP).

[0064] Possible results of step 614 are shown in FIG. 4, where each column represents one video frame of video sequence V and each row represents one macroblock at a specific position within the frame (e.g., the top-left macroblock). Furthermore, in FIG. 4, inter-macroblock references are shown as curly arrows, and the boundary between two consecutive GoPs, GoP1 and GoP2, is shown as a dashed vertical line. Note that inter-frame references are defined at the level of one macroblock position in FIG. 4. Furthermore, the diagonally hashed macroblocks are macroblocks directly affected by the request to replace a region; they are all located in the first frame and form a first set of macroblocks 401. The macroblocks with dotted shading are all macroblocks that directly (in the second frame) or indirectly (in the third frame) reference macroblocks in the first set; these are identified as a second set of macroblocks 402. Consistent with expectations, the second set of macroblocks does not extend beyond a GoP boundary.

[0065] It should be noted that the arrangement of the first and second sets of macroblocks seen in Figure 4 may, but need not necessarily, be changed when intra-frame referencing is introduced: for example, if a macroblock position corresponding to a first row references a macroblock position corresponding to a second row, the first and second sets of macroblocks 401, 402 remain unchanged.

[0066] In FIG. 5, the first set consists of macroblocks 111.2 and 111.3, and the second set consists of macroblocks 111.4 and 111.5.

[0067] In the next step 616, an archival object 140 is added to the signed video bitstream B. The archival object 140 includes fingerprints h2 and h3 calculated from the first set of reconstructed macroblocks. At the level of the signed video bitstream B, the archival object 140 may have a format similar to that of the data units 120 and the signature units 130, in that the archival object 140 can be separated from the video bitstream without decoding. The fingerprints h2 and h3 are not necessarily calculated by the entity executing the method 600. Indeed, in the case of a bitstream B according to the "document approach," in which the signature units 130 include bit strings, these fingerprints are already available from one of the signature units 130. Furthermore, it should be noted that if the signature units 130 in bitstream B include such bit strings, step 616 can be performed as soon as the first set of macroblocks is determined, i.e., before the completion of step 614.

[0068] Optionally, each archival object 140 may include digital signatures of these fingerprints, or of a combination of these fingerprints within that archival object 140, or may include digital signatures of the fingerprints of the combination. Further optionally, archival object 140 may also include the locations of the first and second sets of macroblocks whose signatures are archived. The locations may refer to the positions of the macroblocks within the frame, for example, in frame coordinates, which correspond to the positions in the bit string. If static macroblock partitions are used, the positions of the macroblocks 111 may be expressed as macroblock sequence numbers or another identifier. For example, the bit string may be formed by concatenating the fingerprints in the same order as the macroblock sequence within the frame.

[0069] A further step 618 of method 600 is shown with reference to the lower left portion of Figure 5. In this step, an edit operation Edit transforms a first set of macroblocks 111.2, 111.3 into an edited first set of macroblocks 112.2, 112.3. The editing of the first set of macroblocks is in accordance with a request to replace a region of the video frame. Then, in a further step 618, the edited first set of macroblocks 112.2, 112.3 are encoded as a first set of new data units 121.2, 121.3.

[0070] The first set of edited macroblocks 112.2, 112.3 can be encoded using normal encoder settings, a normal Group of Pictures (GoP) pattern, etc., i.e., in the same manner as video bitstream B was created. Optionally, the resulting first set of edited macroblocks 112.2, 112.3 after substitution is instead encoded as independently decodable data units. Each independently decodable data unit may correspond to an I-frame in the H.264 or H.265 encoding specification, a coded macroblock that does not reference other macroblocks, or an equivalent data unit. This is consistent with the inventors' recognition that substitution introduces abrupt temporal changes in a video sequence. Specifically, edited macroblock 112.2 is likely to be significantly different from the immediately preceding, unedited macroblock 111.1, which may degrade predictive coding performance. A further option is to encode the first set of edited macroblocks 112.2, 112.3 losslessly and / or using reduced data compression. This should be understood against the background that the video sequence V is encoded at a predetermined normal level of data compression. More precisely, the first set of edited macroblocks 112.2, 112.3 is expected to be encoded at a reduced data compression level compared to the normal data compression.

[0071] Step 620 of method 600 is optional and will be described separately.

[0072] In the next, non-optional step 622, the second set of macroblocks 112.4, 112.5 are re-encoded as a second set of new data units 121.4, 121.5. The re-encoding preferably minimally alters the visual appearance of the second set of macroblocks 112.4, 112.5. Ideally, the image data (e.g., pixel data or other plaintext data) obtained by decoding the new data units 121.4, 121.5 is identical to or visually inseparable from the second set of macroblocks 112.4, 112.5. However, to achieve this, the re-encoding operation in step 622 may modify predictive coding settings and / or modify the encoding process. In particular, the encoding process may be modified with respect to the level of data compression, with lossy coding (used during normal data compression) being replaced by lossy coding (reduced data compression) or lossless coding. Lossless encoding may involve representing the second set of macroblocks 112.4, 112.5 as unencoded "raw" blocks, such as a list of the original values ​​for each position within the macroblocks in an appropriate color space. If some type of lossy encoding is used for the second set of macroblocks 112.4, 112.5, it may be advantageous to combine this with robust hashing, and in particular robust hash verification. In this way, macroblocks reconstructed from the new data units 121.4, 121.5 can be accepted as authentic with respect to the original signature unit 130, even if the image quality of these macroblocks is slightly degraded.

[0073] Regarding changes in predictive coding settings, in important use cases (e.g., masking, blurring), it may be advantageous to use nonpredictive coding. Again, the substitution introduces an abrupt temporal change to the video sequence, in that edited macroblock 112.3 is likely to be significantly different from the immediately following unedited macroblock 112.4, potentially degrading coding performance if predictive coding is applied. When performance-based coding is not a primary concern, or when the editing operation is of a less obtrusive nature (e.g., filtering, enhancement), predictive coding can be used. For predictive reference (curly arrows in FIG. 5), here, image data in the second set of macroblocks 112.4, 112.5 is expressed relative to image data in the first set of edited macroblocks 112.2, 112.3, which would normally require updating. As a result, the contents of the second set of new data units 121.4, 121.5 differ from data units 120.4, 120.5.

[0074] Then, in step 624, the first and second sets of new data units 121.2, 121.3, 121.4, 121.5 are added to signed video bitstream B. At the same time, the corresponding original data units 120.2, 120.3, 120.4, 120.5 may be removed from video bitstream B. In some embodiments, step 624 is the last operation in editing method 600.

[0075] In some embodiments, the method 600 uses fingerprints calculated from the first set of edited macroblocks 112.2, 112.3. The step 620 further includes adding TIFF0007789738000003.tif5170 to the video bitstream B. The fingerprint is calculated by adding the signature unit 130 associated with the data units 120.2, 120.3 encoding the first set of macroblocks to at least the calculated fingerprint TIFF0007789738000004.tif5170 by replacing it with a substitute signature unit (not shown) that contains a digital signature of a bit string derived from TIFF0007789738000004.tif5170. For example, the bit string may be the calculated fingerprint The fingerprint of TIFF0007789738000005.tif5170 and one or more unedited macroblocks can be derived, so that the complete frame 100 or a portion can be conveniently verified using a single signature unit. Alternate signature units can be obtained by editing an existing signature unit, in particular by extending it with a further digital signature. Alternatively, the calculated fingerprint At least one new signature unit 131 containing a bit string derived from the digital signature of TIFF0007789738000006.tif5170 is added to video bitstream B. The new signature unit 131 may have the same structure as the signature unit 130 described above.

[0076] Optionally ("document approach"), the substitute signature unit and the new signature unit 131 may further comprise the digitally signed bit string itself.

[0077] Note that it is typically not necessary to compute and include fingerprints for the second set of macroblocks, as these remain susceptible to verification using appropriate ones of the existing signature units 130 in video bitstream B. Optional step 620 can be performed at any point in method 600 after the edited first set of macroblocks 112.2, 112.3 is available.

[0078] In still other embodiments, method 600 further includes an initial step 610 of providing at least one signature unit 130. It is understood that in use cases believed to be of primary interest, step 610 is performed by a different entity than steps 612, 614, 616, 618, 620, 622, and 624 of method 600 and / or step 610 is performed at an earlier point in time. In either approach, step 610 is separated from subsequent steps 612, 614, 616, 618, 620, 622, and 624 by relatively insecure data transfer and / or a retention period that justifies the signatures to ensure a desired level of data security.

[0079] Optional step 610 may include substep 610.1 of reconstructing multiple macroblocks from respective data units 120 associated with the signature unit; substep 610.2 of calculating multiple fingerprints from the respective reconstructed macroblocks; substep 610.3 of deriving a bit string from the calculated fingerprints, which is a combination of the multiple fingerprints or a fingerprint of the combination; and substep 610.4 of obtaining a digital signature for the bit string. Suitable implementations of fingerprint calculation 610.2, bit string derivation 610.3, and digital signature 610.4 are described in detail above. In particular, the bit string to which the digital signature in signature unit 130 relates may be a combination of fingerprints of the associated data units 120, or may be a fingerprint of the combination of fingerprints of the associated data units 120. The combination (or "document") may be a list or another concatenation of the respective string representations of the fingerprints.

[0080] Having described the editing method 600, attention is now directed to the receiver side. More precisely, a method 700 for verifying a signed video bitstream B will be described with reference to the flowchart of FIG. 7. It is again assumed that the signed video bitstream B was obtained by predictive coding of a video sequence V and, optionally, by subsequent editing operations. It is not essential that the signed video bitstream B has been processed according to the editing method 600. Furthermore, it is assumed that the signed video bitstream includes data units 120, associated signature units 130, and an archive object 140. Here, each data unit 120 represents one macroblock 110 in a frame 100 of the predictively coded video sequence V, each signature unit 130 includes a digital signature (e.g., s(H1), s(H2)) of a bitstring (e.g., H1, H2) and optionally the bitstring itself, and the archive object 140 includes at least one fingerprint, which may be an archived fingerprint for a data unit not currently present in the bitstream B and / or which has undergone editing. Irrelevant to the verification method 700, it is typically not possible for the recipient to determine whether a particular signature unit 130 was added in connection with an edit (e.g., by the editing method 600) or whether it was part of the original, unedited bitstream B.

[0081] In a first step 710 of the method 700, a macroblock 113 is reconstructed from data units 120, 121 associated with a signature unit 130. The reconstruction involves a decoding process symbolized by the top row of downward arrows in Figures 10A and 10B.

[0082] The next step 712 of the method 700 is to extract from at least some of the reconstructed macroblocks 113 their respective fingerprints. TIFF0007789738000007.tif5170 is calculated. Because of the editing in method 600, fingerprints are not calculated from the second and third macroblocks 113.2, 113.3 (first set), which are different from the corresponding original macroblocks in video sequence V. The fact that the second and third macroblocks 113.2, 113.3 belong to the first set may be indicated in metadata in the edited video bitstream B or may be apparent from the encoder timestamps on the corresponding data units 121.1, 121.3. Yet another option may be to include the original edit request (a request to replace a region of a video frame) in the edited video bitstream B, from which a recipient can determine which macroblocks are changed.

[0083] In a third step 714, which in principle could be performed before or overlapping with step 712, at least one archived fingerprint is obtained from archive object 140. In the example shown in Figures 10A and 10B, fingerprints h2, h3 for the second and third macroblocks 113.2, 113.3 are obtained in this way. Note that only TIFF0007789738000008.tif5170 can fail verification in the following fourth step 716, in which case failure would suggest that incorrect manipulation of bitstream B has occurred.

[0084] In step 716, after fingerprints for all macroblocks have been obtained in steps 712 and 714, a bit string TIFF0007789738000009.tif5170 is derived. This can be done according to pre-agreed rules, for example by a procedure similar to that described in step 610.3. It is recalled that the bit string may be a combination of the obtained fingerprints or a fingerprint of said combination of fingerprints.

[0085] In a next step 718, the data units 120, 121 associated with the signature unit 130 are verified using the digital signature s(H1) in the signature unit 130. For the avoidance of doubt, it should be noted that the verification in step 718 of the data units is indirect, without any necessary processing acting on the data units themselves.

[0086] In embodiments where the signature unit 130 does not include the bit string H1, step 718 generates a bit string derived using the digital signature s(H1). This is done by examining TIFF0007789738000010.tif5170. For example, the derived bit string TIFF0007789738000011.tif5170 can be verified using the public key that belongs to the same key pair as the private key used to generate the digital signature s(H1). In Figure 10B, this is the derived bit string TIFF0007789738000012.tif5170 and the digital signature s(H1) to a cryptographic entity 1001 containing the public key, which outputs a binary result W1 representing the outcome of the verification.

[0087] Alternatively, in embodiments where signature unit 130 includes bit string H1 (the "document approach"), step 718 may be performed in two substeps. In the first substep, bit string H1 is verified, as described above, for example, using a public key belonging to the same key pair as the private key used to generate digital signature s(H1). This is illustrated in FIG. 10A by function block 1001 and binary result V2. Then, in the second substep, the verified bit string H1 is compared with the bit string derived in step 716. TIFF0007789738000013.tif5170. The comparison may be a bitwise equality check, as suggested by function block 1002 in Figure 10A, resulting in an output V2 of either a true or false value. If both results V1, V2 are true, then it can be concluded that the signed video bitstream 100 is authentic as far as this signing unit 130 is concerned.

[0088] Execution of method 700 may then include repeating the relevant ones of steps 710, 712, 714, 716, 718 described above for any additional signature units 130 in signed video bitstream B. If the result is positive for all signature units 130, signed video bitstream B is concluded to be valid, and it may be consumed or processed further. Otherwise, signed video bitstream B shall be deemed inauthentic and may be quarantined from further use or processing.

[0089] As previously mentioned, the steps of any method disclosed herein do not have to be performed in the exact order described unless explicitly stated. This is particularly illustrated by verification method 700, where step 714 can clearly be performed before, between, or after steps 710 and 712, as desired.

[0090] Note that verification of the data units in the first set is based on a different trust relationship than verification of the data units in the second set. The data units in the first set are verified by trusting the entity that created the digital signature s(H1), i.e., the owner of the private key if asymmetric key cryptography is used. The data units in the second set are verified by trusting the entity that compiled the signed bitstream B and created the archive object.

[0091] In some embodiments of verification method 700, deriving 716 the bit string includes deciding 716.1 whether to calculate the required fingerprint from the reconstructed macroblock 111 or obtain it from archival object 140. This decision can be guided by location information in archival object 140, which indicates the location of the macroblock 110 represented by the data unit 120 to which the archived fingerprint pertains. Having access to these macroblock locations allows the receiver to perform a reliable integrity check based on the assumption that any macroblock 110 in video frame 100 that cannot be reconstructed from data unit 120 in signed video bitstream B is encoded by another data unit whose fingerprint can necessarily be obtained from archival object 140. If archival object 140 does not indicate the location of these macroblocks, the receiver can, for example, insert missing fingerprints, i.e., fingerprints that cannot be calculated from data units 120 in signed video bitstream B, by a trial-and-error approach. A trial and error approach may involve performing steps 714 and 716 for each possible way of inserting the archived fingerprint from archive object 140 (each such way of insertion may be inferred to be a permutation of the location of the missing macroblocks), and concluding that signed video bitstream B is fraudulent only if all of these attempts fail.

[0092] As an overview, FIG. 11 illustrates the signal processing operations and data flow occurring during execution of the editing method shown in FIG. 6 and the verification method shown in FIG. 7, as well as functional units suitable for performing the signal processing operations. The time evolution of the signal processing flow is generally oriented from left to right. Data and signals are depicted as simple frames, while functional units are represented as frames with double vertical lines. Each functional unit may correspond to a programmable processor or a network of such processors that executes a segment of software code. Each functional unit may also be implemented as a dedicated hardware circuit, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). From another perspective, the functional units can be thought of as representing respective portions of software code (e.g., modules, routines) that execute on a common processor or processor network.

[0093] Assume initially that an input image 100 in plaintext format is provided to an encoder 1110 configured for a predictive video coding format. The encoder 1110 outputs an encoded image 110A. The encoded image 110A may be formatted like the video bitstream described with reference to FIG. 5 to include, among other things, data units corresponding to macroblocks linked by inter-frame or intra-frame prediction references. To ensure accurate predictive coding, the encoder 1110 includes a reference decoder 1111 configured to reconstruct image data from data units as they are generated, e.g., quasi-sequentially, by the encoding process within the encoder 1110. The reconstructed image data, which may be considered to form a decoded reference image 100C, is temporarily stored in a reference buffer 1112 associated with the encoder 1110. More precisely, to represent image data in a first macroblock relative to image data in a second macroblock, the encoder 1110 calculates an increment from the second macroblock in the decoded reference image 100C, rather than from the original second macroblock (i.e., as provided in the input image 100). The hash function 1113 computes fingerprints h1, h2, and h3 from the macroblocks of the decoded reference image 100C and collects (e.g., concatenates) them into a hash list H1. The hash list H1 may be a string of bits. As described above, the hash list H1 may be digitally signed, and the resulting digital signature s(H1) may be carried in a signature unit within the video bitstream.

[0094] Assume that a request to mask an image region is received from a user. To execute the user's request, the coded image 100A is input to an editing tool 1120, which includes a decoder 1121, an optional verifier 1122, and a masker 1123. The decoder 1121 is configured to reconstruct image data, particularly macroblocks, from the coded image 100A. The image reconstructed from the coded image 100A is nominally identical to the decoded reference image 100C. The presence of the verifier 1122 in the editing tool 1120 can be justified, particularly if the coded image 100A is transferred over an untrusted connection; in that case, verification can be performed before masking is performed. The expected result of the verification is that the coded image 100A is authentic, in which case it makes sense to perform masking. The masker 1123 is configured to replace a pre-specified color or pattern in a region of the image. The substitution generates a masked decoded image 100B, which is forwarded (via a trusted connection) to the second encoder 1130. The masked decoded image 100B has at least the edited portions (edited macroblocks) in plaintext form. It may be advantageous to refrain from decoding the unedited portions to the extent feasible. The second encoder 1130 outputs a further encoded image 100F, which may be made available to a recipient. Similar to the first encoder 1110, the second encoder 1130 includes a reference decoder 1131 configured to reconstruct image data from data units created by the encoding process in the second encoder 1130. The reconstructed image data, i.e., the decoded reference image 100D, is temporarily stored in a reference buffer 1132 associated with the second encoder 1130.

[0095] To ensure that the edited and then encoded image can be verified, a hash list is generated from the masked decoded image 100B using a hash function 1124. TIFF0007789738000014.tif5170 is calculated. (This corresponds to optional step 620 of the editing method 600 described above.) Note that the edited macroblock (here While the fingerprint for TIFF0007789738000015.tif5170 likely changed as a result of the masking, the remaining fingerprints (here h1 and h3) may match the fingerprints calculated from the original decoded reference image 100C. The new hash list includes the digital signature TIFF0007789738000016.tif5170 may be given. Alternatively, a hash list TIFF0007789738000017.tif5170 is calculated from the decoded reference image 100D in the reference buffer 1132 associated with the encoder 1130.

[0096] Note that the encoder 1110 may be controlled by a different entity (e.g., an author) than the editing tool 1120 and the second decoder 1130. Because the masked decoded image 100B may be tampered with before reaching the encoder 1130, the editing tool 1120 and the second decoder 1130 are preferably co-located or linked by a reliable data connection.

[0097] At the receiving end, a decoder 1140 is provided which computes a decoding process adapted to reconstruct a decoded reference image 100E from the further coded image 100F. From the decoded reference image 100E, a hash list 1142 is generated using a further hash function 1142. The verifier 1143 associated with the decoder 1140 can calculate the hash list TIFF0007789738000018.tif5170. TIFF0007789738000019.tif5170 is a hash list calculated from the masked decoded image 100B or the decoded reference image 100D. TIFF0007789738000020.tif5170. As explained above, this can be done directly (the "document approach"), as in Figure 10A, or indirectly, as in Figure 10B. If the verifier 1143 concludes that the two hash lists do in fact match, then the further encoded image 100F (or equivalently the decoded image 100E) may be released for reproduction or further processing.

[0098] It should be noted that verification of the further encoded image 100F is based on a trust relationship between the further encoder 1130 and the decoder 1140, or between persons or entities controlling these devices. Additionally or alternatively, if it is desired to verify the further encoded image 100F based on a trust relationship between the first encoder 1110 and the decoder 1140, or between persons or entities controlling these devices (e.g., between the author and the consumer), the further encoded image 100F may be endowed with an archived object containing fingerprints of the edited macroblocks. It should be noted that, due to the editing, the entire further encoded image 100F, only its unedited portions, cannot be verified based on this latter trust relationship.

[0099] It will be understood that the hash functions 1113, 1124, and 1142 appearing in FIG. 11 are equivalent in the sense that they provide equal outputs for equal inputs.

[0100] Aspects of the present disclosure have been described above primarily with reference to certain embodiments. However, as will be readily understood by those skilled in the art, other embodiments beyond those disclosed above, as defined by the appended claims, are equally possible within the scope of the inventive concept. In particular, it should be noted that the above description of various embodiments focuses on predictively coded video. This is because aspects of the present disclosure are expected to have particular advantages in prediction-based coding, where both the encoder and decoder have access to the same frame in the form of a reference frame in the encoder's and decoder's reference buffers. Therefore, in prediction-based coding, no additional steps are required to obtain the reconstructed or decoded frame to be signed and verified. However, the same approach can be used for any video, not just predictively coded video, as long as the entity signing the video and the entity verifying the video have access to the reconstructed or decoded frame of the video. In the case of prediction-based coding, it proves practically convenient to utilize fingerprints of groups of pixels that are also used as macroblocks in the coding. However, in general, fingerprints may be calculated from groups of pixels grouped in other ways. Once a frame is decoded, it does not matter how the pixels were divided for coding. For example, depending on the type of editing expected, it may be useful to divide the decoded image into smaller or larger groups of pixels than those used for encoding. For example, if the masking is always done in rectangular form, a coarser division of pixels than used for encoding may be sufficient for the signature process. On the other hand, if it is assumed that the masking can be done to more closely follow the contours of the object to be masked, a finer division may be useful for the signature process.

Claims

1. 1. A method for editing a signed video bitstream obtained by predictive coding of a video sequence, comprising: the signed video bitstream includes a plurality of data units and a plurality of associated signature units; each data unit representing one macroblock in a video frame of the video sequence that has been predictively coded, the data unit being obtained by applying an encoder to a corresponding macroblock; each signature unit comprising a digital signature of a bit string derived from a plurality of fingerprints; each fingerprint is calculated from macroblocks reconstructed from one data unit associated with said signature unit; The method comprises: receiving a request to replace a region of at least one video frame; reconstructing a first set of macroblocks in which the region is included and a second set of macroblocks that directly or indirectly reference macroblocks in the first set, each reconstructed macroblock being obtained by applying a decoder to a corresponding data unit; adding an archival object to the signed video bitstream, the archival object including a fingerprint calculated from the first set of reconstructed macroblocks and indicating the location of the macroblock represented by the data unit to which the included fingerprint relates; editing the first set of macroblocks in accordance with the request to replace the region of the at least one video frame and encoding the edited first set of macroblocks as a first set of new data units; re-encoding the second set of macroblocks as a second set of new data units; adding the first and second sets of new data units to the signed video bitstream; A method comprising:

2. adding a fingerprint calculated from the first set of edited macroblocks to the signed video bitstream; a signature unit associated with a data unit encoding the first set of macroblocks is replaced in the signed video bitstream with a substitute signature unit comprising a digital signature of a bit string derived from at least the calculated fingerprint; or The method of claim 1 , wherein at least one new signature unit is added to the signed video bitstream, the new signature unit comprising a bit string derived from a digital signature of the calculated fingerprint.

3. The method of claim 1 , wherein the archive object further includes the locations of the first set of macroblocks.

4. The method of claim 1 , wherein the second set of macroblocks is losslessly re-encoded.

5. the second set of macroblocks are re-encoded using reduced data compression; The method of claim 1 , wherein the fingerprints of the second set of the macroblocks in the signed video bitstream include robust hashes.

6. The method of claim 1 , wherein the second set of macroblocks are non-predictively re-encoded.

7. The method of claim 1 , wherein the second set of macroblocks are predictively re-encoded with reference to the edited first set of macroblocks.

8. editing and encoding the first set of macroblocks; encoding the first set of edited macroblocks losslessly and / or using reduced data compression and / or non-predictively; The method of claim 1 , comprising:

9. first providing a signature unit, reconstructing a plurality of macroblocks from respective data units associated with the signature unit; calculating a plurality of fingerprints from each of the reconstructed macroblocks; deriving a bit string from the calculated fingerprint; obtaining a digital signature of said bit string; First provide a signature unit, including The method of claim 1 further comprising:

10. The method of claim 9 , wherein the bit string is a combination of the fingerprints or a fingerprint of the combination.

11. The method of claim 1 , wherein calculating the fingerprint comprises obtaining the reconstructed macroblock from a reference decoder buffer.

12. 1. A method for verifying a signed media bitstream obtained by predictive coding of a video sequence, comprising: the signed video bitstream includes a plurality of data units, a plurality of associated signature units, and an archive object; each data unit representing one macroblock in a video frame of the video sequence that has been predictively coded, the data unit being obtained by applying an encoder to a corresponding macroblock; each signature unit comprising a digital signature of a bit string derived from a plurality of fingerprints; each fingerprint is calculated from macroblocks reconstructed from one data unit associated with said signature unit; the archive object includes at least one archived fingerprint and indicates the location of a macroblock represented by a data unit to which the archived fingerprint relates; The method comprises: reconstructing the macroblock by applying a decoder to the data units associated with the signature unit; calculating respective fingerprints from at least some of said reconstructed macroblocks; obtaining at least one archived fingerprint from the archive object; deriving a bit string from the calculated fingerprint and the obtained fingerprint; verifying the data unit associated with the signature unit using the digital signature in the signature unit; A method comprising:

13. Deriving the bit string includes determining, for each data unit associated with the signature unit, whether to calculate a fingerprint from a reconstructed macroblock or to obtain a corresponding archived fingerprint from the archive object based on the position according to the archive object. The method of claim 12.

14. A device comprising processing circuitry arranged to carry out the method of any one of claims 1 to 13.

15. A computer program comprising instructions which, when executed by a computer, cause the computer to perform a method according to any one of claims 1 to 13.

Images