Modified Video Encoding

Abstract

To provide a method and a device for encoding a video which is modified and captured.SOLUTION: A method comprises: receiving (S1010 and S1020) first / second sets of input image frames comprising image data originating from first / second sources of input image frames; encoding the first / second sets of input image frames into first / second sets of encoded image frames, and forming (S1040) a combined video stream: forming the combined video stream; and using first / second keys being keys where each key relates to the respective sources, and signing (S1050 and S1060) first / second signing groups comprising one or more encoded image frames of the first / second sets of encoded image frames.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method and device for encoding modified video. [Background technology]

[0002] Today, surveillance video cameras are widely used in many applications related to security, safety, and evidence collection. Typically, the video streams of these cameras are modified, for example, by the application of graphical overlays. Such modifications may include, for example, overlays indicating the time of capture and / or the camera's identifier. These modifications may further include bounding boxes and / or event markers. These modifications may thus be applied to aid an operator viewing the video stream in interpreting or understanding its content. Modifications may also be applied to comply with legal requirements to protect personal information, with well-known examples of such modifications, such as privacy masks.

[0003] Believably modified and / or computer-generated videos, e.g., so-called deepfakes, pose a significant security challenge because they may, for example, undermine the evidentiary value of captured video and render surveillance cameras unhelpful by reducing confidence in the captured video. This is especially true in the case of surveillance camera videos that, as described above, have been modified for non-malicious purposes. In contrast, modified and unmodified videos may be encoded and processed independently of each other.

[0004] Given this, there is therefore a need for improvements in the art. Summary of the Invention

[0005] SUMMARY OF THE PRESENT EMBODIMENT It is an object of the present invention to alleviate at least some of the above-mentioned problems in the prior art and to provide an improved solution for encoding modified captured video.

[0006] According to a first aspect, there is provided a method of processing a video stream, comprising: receiving a first set of input image frames comprising image data originating from a first source of input image frames; receiving a second set of input image frames comprising image data originating from a second source of input image frames; encoding the first set of input image frames and the second set of input image frames into a first set of encoded image frames and a second set of encoded image frames, respectively, to form a combined video stream, wherein an encoded image frame of the first set of encoded image frames exclusively references one or more other encoded image frames of the first set of encoded image frames and a set of encoded image frames of the second set of encoded image frames. forming a combined video stream in which an encoded image frame of the first set of encoded image frames references both an encoded image frame of the first set of encoded image frames and another encoded image frame of the second set of encoded image frames; signing a first signature group including one or more encoded image frames of the first set of encoded image frames using a first key, the first key associated with a source of the first input image frames; and signing a second signature group including one or more encoded image frames of the second set of encoded image frames using a second key, the second key associated with a source of the second input image frames.

[0007] "Encoding" should be understood as known in the field of video coding, which may include, for example, representing input image frames in a bitstream, potentially in a compressed form.

[0008] "Referring" should be understood as commonly known in the field of video coding, where each part of a frame as encoded may potentially refer to each other part of the same frame, and / or each encoded image frame may potentially refer to each other image frame in the encoded video stream. Typically, encoding a first frame or a part thereof may be based on encoding differences relative to a second frame or differences from a predicted evolution from the second frame, whereby the first frame as encoded refers to the second frame. This includes the case of what is called interframe coding.

[0009] "Signing" should be understood as generating a representation, such as a collection of bits or bytes, a string of characters, or the like, that indicates the identity of the key used in signing, as is commonly known in the field of computer cryptography. The first and second keys may thus be understood as cryptographic type keys that allow the identification of the source with which they are associated.

[0010] A "key" should be understood as a proxy, such as a collection of bits or bytes, a string of characters, or the like, used during signing, as is commonly known in the field of computer cryptography. In the case of public key cryptography, as known per se, the key may be a private key, which is used in signing as asserted above, and has a corresponding public key, which may later be used to verify the signature generated during signing.

[0011] The first set of input image frames may be, for example, image frames that include only original pixel data, as recorded by an image sensor of an imaging device. The second set of input image frames may be understood as image frames that include modifications or alterations, for example, compared to the original pixel data. The source of the first input image frames may be associated with the imaging device or its image sensor. Alternatively or additionally, the source of the first image frames may be associated with stored video data, such as video data stored in a video store. The source of the second input image frames may be associated with a video processing unit, i.e. an entity capable of processing and applying modifications to the pixel data of the image frames.

[0012] In general, a source may be hardware-based, such as an image sensor or encoder, and / or software-based, such as instructions stored on a non-transitory computer-readable storage medium.

[0013] The signing may be done, for example, in a separate signing unit, an encoder, or whatever.

[0014] Encoding the combined video stream such that the frames of the first set of encoded image frames exclusively reference one of the frames of the set, while the encoded frames of the second set of encoded image frames reference both the frames of the first set and the frames of the second set, allows for providing complete information from both sources in a single video stream without permanently or irrevocably altering the original pixel or image data from the source of the first input image frames, which may therefore be effectively included in the output combined video stream without any modification to those pixel data. Furthermore, using this encoding method, the original image data in the first set of encoded image frames may be used as the basis for encoding the second set of image frames in the combined video stream, which avoids duplication of data. This allows for avoiding excessive encoding, transmission and storage resources for two complete video streams, one unmodified and one modified.

[0015] Furthermore, through separately signing a first signature group including one or more image frames of the first set of encoded image frames and a second signature group including one or more image frames of the second set of image frames, each with a key associated with a respective source of the image frames, the integrity of the image from the first source may be verified through these signatures despite the presence of an image, such as an overlay, from the second source. Further, the integrity of the image of the second source may be verified separately.

[0016] In general, data from a first source of input image frames may be associated with a relatively high level of security, and data from a second source of input image frames may be associated with a relatively low level of security. For example, in the case of a first source of input image frames representing input image frames coming directly from an image sensor and a second source of input image frames associated with an overlay that may have been added to a camera downstream of the video sensor, the data directly from the sensor may have less opportunity for tampering compared to the overlay, and is therefore associated with a higher level of trust. Furthermore, the first set of input image frames may be guaranteed not to have been modified or tampered with since leaving the sensor, while the second set of input image frames can only be guaranteed not to have been tampered with since leaving the camera.

[0017] In another example, separately signing may be particularly beneficial if, for example, the source of the second input image frame, including the modification of the pixel data, is associated with or originates from another entity not necessarily related to the provider of the encoder or imaging device, in which case the corresponding data may be associated with a lower level of trust because such data from another entity may not be under the control of the camera manufacturer or camera owner and may have been modified or tampered with prior to or during the addition.

[0018] Through separate signing, the higher level of trust associated with the source of the first image frame may be retained even in the presence of data from the source of the second image frame, such as in the form of an overlay, thus improving security while retaining all image information in one combined video stream.

[0019] In summary, therefore, this method may advantageously provide a secure and efficient way of encoding modified video.

[0020] The second signature group may further include one or more encoded image frames of the first set of encoded image frames that are referenced by an encoded image frame of the second set of encoded image frames.

[0021] This allows for the combination of an image from a first source, including modifications from a second source, to be verified, improving security and flexibility.

[0022] The method may further include receiving a third set of input image frames including image data originating from a source of the third input image frames, where encoding further includes encoding the third set of input image frames into a third set of encoded image frames to form part of the combined video stream, where an encoded image frame of the third set of encoded image frames references both an encoded image frame of the first set of encoded image frames and another encoded image frame of the third set of encoded image frames. The method further includes signing a third signature group including one or more image frames of the third set of encoded image frames using a third key, the third key associated with the source of the third input image frames.

[0023] The third set of input image frames may, for example, include image frames that include modifications or alterations compared to the original pixel data from the first source, similar to the second set of input image frames. As an example, a second source may be understood as a source of pixel data modifications that is directly associated with an encoder or imaging device and is overseen or authorized by a provider of that encoder or imaging device. A third source may instead be understood as a source of pixel data modifications that is associated with another, not necessarily related to the provider of that encoder or imaging device.

[0024] A third set of input image frames may allow a further level of encoded modified image frames of independent and separate pixel data to be added to the same combined video stream without permanently and unalterably changing the original pixel data. In principle, there is nothing to prevent further sets of image frames, a fourth, fifth, etc., being correspondingly received and encoded into the same combined video stream.

[0025] The method may further include designating the first set of encoded image frames as not to be displayed image frames, such designation may take the form of, for example, a No Display (ND) flag applied to each of the frames of the first set of encoded image frames.

[0026] This allows for a convenient and efficient way of identifying image frames included in the first set of encoded image frames as being signed by the first key identified in the combined video stream. It should be noted that such designation in this disclosure may be used to identify the first set of encoded image frames in a combined video stream even if the combined video stream is not being displayed on an actual display.

[0027] Furthermore, the combined video stream may be encoded such that when a standard video decoder decodes the combined video stream, it outputs the decoded video stream as modified by the source of the second input image frames, e.g., including overlays resulting from the source of the second input image frames, for example, for display, storage, and / or analysis. On the other hand, such a decoder ignores the encoded image frames of the first set of encoded image frames as designated as image frames that are not displayed. Thus, the combined video stream, when decoded using such a standard video decoder without modification, is usable and allows easy playback, including overlays. This may be a common use of the combined video stream. On the other hand, all information from the source of the first input image frames is included in the combined video stream without modification. It may be extractable therefrom using a specially adapted decoder. Thus, the encoded image frames designated as image frames that are not displayed may be extracted from the combined video stream and output those image frames as decoded in the decoded video stream. So in summary, this makes it possible to encode a combined video stream in a way that is easy to use yet still contains all the information from both sources as signed by the first and second keys.

[0028] The source of the first input image frames may be an image sensor of a video imaging device, which may result in the first set of input image frames being encoded as if they had been captured without any modifications being applied.

[0029] The source of the second input image frames may be a sensor other than the image sensor of the video imaging device, such as an infrared sensor, a radar sensor, a laser / lidar sensor, etc. As such, graphical overlays based on other sensor data, such as radar data, may be incorporated into the combined video stream without affecting the first set of input image frames.

[0030] The source of the second input image frames may be a video processing unit. Such a video processing unit may be understood as any device, method, system, or software capable of related processing of video, such as an encoder. Other devices, methods, or systems capable of modifying image frames or video may also be considered as video processing units. The video processing unit may be computer-implemented. For example, the video processing unit may be implemented using an application programming interface (API) associated with the first source of image data. The video processing unit may be a video processing pipeline.

[0031] The video processing unit may include a graphical overlay adding functionality, which should be understood as any device, method or system capable of adding any kind of graphical overlay to an image frame.

[0032] The first key may be a hash function key and / or the second key may be a hash function key, which may be understood as the first and / or second keys being associated with a cryptographic hash function. The hash key provides increased security by reducing the likelihood or feasibility of an unauthorized or malicious signature occurring without the signature being detected as such.

[0033] The encoding may include inter-encoding the first set of input image frames and the second set of input image frames into a first set of encoded image frames and a second set of encoded image frames, respectively.

[0034] The encoding may include predictively encoding the first set of input image frames and the second set of input image frames into the first set of encoded image frames and the second set of encoded image frames, respectively. Predictive encoding may be understood as being a type of inter-frame encoding, where a previous image frame is used as a reference in encoding any particular image frame.

[0035] The method may further include associating at least one of the signatures associated with signing using the first key and the signatures associated with signing using the second key with the combined video stream. For example, one or more signatures from the signing may be included in the combined video stream and thus are included therein. Alternatively, the one or more signatures may be separate from the combined video stream but associated therewith, for example, in the same directory, database, file system, storage facility, or the like, or by being transmitted over the same data stream, channel, connection, or the like.

[0036] A signature should be understood as a collection of bits or bytes, a string of characters, or the like, that results from signing with each of the keys.

[0037] The method may further include decoding the combined video stream in a decoder. Such a video decoder may be understood as any device, method, or system capable of decoding video. The decoder may be computer-implemented.

[0038] The method may further include checking a signature included in the combined video stream and associating data related to a result of the checking with the decoded video stream.

[0039] This allows for a particularly technically efficient way of presenting the results of the signature check to the user.

[0040] The method may further include determining an origin of an image frame of the combined video stream and associating data with the image frame based on determining the origin.

[0041] This allows for a particularly technically efficient way of presenting the results of the origin determination to the user.

[0042] According to a second aspect, there is provided a non-transitory computer readable storage medium having stored thereon instructions which, when executed on a device having processing capability, perform a method according to the first aspect.

[0043] The second embodiment may provide similar advantages as the first embodiment due to their inherent relationships.

[0044] According to a third aspect, there is provided a video processing system comprising a processor, the processor being adapted to receive a first set of encoded image frames comprising image data originating from a first source of input image frames, and a second set of encoded image frames comprising image data originating from a second source of input image frames, and to encode the first set of input image frames and the second set of input image frames into a first set of encoded image frames and a second set of encoded image frames, respectively, to form a combined video stream, wherein the encoded image frames of the first set of encoded image frames exclusively reference one or more image frames of the first set of encoded image frames, and the encoded image frames of the second set of encoded image frames exclusively reference one or more image frames of the first set of encoded image frames. The encoded image frames of the set of image frames reference both an encoded image frame of the first set of encoded image frames and another encoded image frame of the second set of encoded image frames, and are configured to sign a first signature group including one or more encoded image frames of the first set of encoded image frames using a first key, the first key associated with a source of the first input image frames, and to sign a second signature group including one or more encoded image frames of the second set of encoded image frames using a second key, the second key associated with a source of the second input image frames.

[0045] The third aspect may provide similar advantages as the first aspect due to their inherent relationships.

[0046] The video processing system may further include a decoder configured to decode the combined video stream.

[0047] Further scope of applicability of the present invention will become apparent from the following detailed description. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the present invention, are given for purposes of illustration only, since various changes and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description.

[0048] Accordingly, the invention is not limited to the particular components of the devices described, or to the particular operations of the methods described, as such devices and methods may vary within the scope of the appended claims, it being understood that the invention is not limited to the particular components of the devices described, or to the particular operations of the methods described, as such devices and methods may vary within the scope of the appended claims.

[0049] These and other aspects of the invention will now be described in more detail with reference to the accompanying drawings, which should not be considered limiting, but instead should be considered for purposes of illustration and understanding.

[0050] As shown in these figures, the sizes of each layer and region may be exaggerated for illustrative purposes and are therefore provided to show the general structure, and like reference symbols refer to like elements throughout the figures. [Brief description of the drawings]

[0051] [Figure 1] 1 shows a flow chart of an example method for processing a video stream. [Diagram 2] 3 shows a first and a second set of encoded image frames that form a combined video stream. [Diagram 3] 1 shows a first, second, and third set of encoded image frames that form a combined video stream. [Figure 4] 4 shows a flow chart of an example method for processing a video stream including further steps. [Diagram 5] 1 illustrates a video processing system and its components. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0052] The present invention will now be described in more detail with reference to the accompanying drawings, in which presently preferred embodiments of the invention are shown, the invention may, however, be embodied in many different forms within the scope of the claims and should not be understood as being limited to the embodiments shown.

[0053] Fig. 1 illustrates an example method for processing a video stream. In parallel, reference is made to Fig. 5, which illustrates a video processing system 400 configured to carry out this example method.

[0054] Optional additional steps to this example method are illustrated as boxes with dotted borders in the flowcharts of this disclosure, including in Figure 1. The arrows between the boxes indicate an example order in which the steps may be performed, but should not be construed as excluding other alternatives for ordering these steps.

[0055] At S1010, a first set of input image frames 110a is received. The first set of input image frames 110a includes image data originating from a first input image frame source 401 (see FIG. 5), which may be received at an encoder 410 (see FIG. 5).

[0056] At S1020, a second set of input image frames 120a is received. The second set of input image frames 120a (see FIG. 2) includes image data originating from a second input image frame source 402 (see FIG. 5), which may also be received at the encoder 410 (see FIG. 5).

[0057] At S1040, in the encoder 410, the first set of input image frames 110a and the second set of input image frames 110b are encoded into a first set of encoded image frames 110b and a second set of encoded image frames 120b, respectively, to form a combined video stream 100 (see FIG. 5). The encoding may be performed according to a video coding format and / or standard such as H.261, H.262, H.263, H.264 / AVC, H.265 / HEVC, VP8, VP9, ​​or AV1 into a bitstream.

[0058] At S1050, a first signature group including one or more encoded image frames of the first set of encoded image frames 110b is signed using a first key associated with the source 401 of the first input image frame. The signing may be performed, for example, in the encoder 410 or in any separate signing unit 411 (see FIG. 5).

[0059] At S1060, a second signature group including one or more encoded image frames of the second set of encoded image frames 110b is signed using a second key, which is associated with the source 401 of the second input image frames. The signing may be performed, for example, in the encoder 410 or optionally in a signing unit 411 (see FIG. 5).

[0060] 2 shows how a first set of encoded image frames 110b and a second set of encoded image frames 120b may be arranged in a combined video stream 100 and how each of the frames references each other. In the figure, the references are indicated by arrows. The figure should be understood as corresponding to a chronological sequence of image frames, with the leftmost image frame being the first image frame and the rightmost image frame being the last image frame.

[0061] One frame referencing another frame may for example correspond to a predictive type of interframe encoding, as known per se in the field of video encoding.

[0062] With further reference to FIG. 2, as can be seen therein, an image frame of the first encoded image frame set 110b, in this example each of the image frames, exclusively references another image frame in the first encoded image frame set 110b.

[0063] Moreover, in this example, each image frame of the first encoded set of image frames exclusively references a preceding image frame included in the first encoded set of image frames 110b, but other configurations are equally possible, such as an image frame or each of a plurality of image frames of the first encoded set of image frames exclusively references a plurality of previous image frames of the first encoded set of image frames 110b, or a frame of the first encoded set of image frames 110b exclusively references one or more other image frames of the first encoded set of image frames 110b, both of which are temporally earlier and later.

[0064] FIG. 2 further shows how an image frame of the second encoded set of image frames 120b references both an image frame of the first encoded set of image frames 110b and another image frame in the second encoded set of image frames 120b.

[0065] Thus, encoding S1040 may include predictively encoding the first set of encoded image frames 110b and the second set of encoded image frames 120b to form the combined video stream 100.

[0066] For example, the first set of input image frames 110a may include original image data or original pixel data as captured by a video imaging device 403 (see FIG. 5).

[0067] 1 and 5, the source of the first input image frame 401 may be the image sensor 404. The source of the first input image frame 401 may be the image sensor 404 of the video imaging device 403. The source of the first input image frame 401 may be part of the video imaging device 403 or may be associated therewith in other ways as well. The source of the first input image frame 401 may be a source of original image data or original pixel data as captured by the video imaging device 403. The source of the first input image frame 401 may be associated directly with the image sensor 404 or the video imaging device 403.

[0068] The video imaging device 403 may include an image sensor 404. The video imaging device 403 may be configured and / or arranged to be in electrical and / or wireless connection with the image sensor 404. The video imaging device 403 may be adapted or configured to capture video, i.e., a sequence of subsequent image frames. The video imaging device 403 may alternatively be referred to or understood as a video camera. The video imaging device 403 may be a digital type video imaging device. The video imaging device 403 may be a monitoring and / or surveillance type video imaging device. The video imaging device 403 may be a surveillance type video imaging device. The video imaging device 403 may be a pan / tilt / zoom type video imaging device. The video imaging device 403 may be a stationary or fixed state video imaging device. The video imaging device 403 may be a portable or wearable video imaging device.

[0069] The video imaging device 403 may include a medium for storing the captured image data and / or image frames. The video imaging device 403 may include a transmitter positioned and / or configured to transmit the captured image data and / or image frames.

[0070] The video imaging device 403 may be a network-connected type video imaging device. The video imaging device 403 may be optically, electrically, or wirelessly connected to the network. The network may be, for example, a local area network (LAN) type network, a wide area network (WAN) type network, or a wireless local area network (WLAN) type network.

[0071] The image sensor 404 may be part of the video imaging device 403. The image sensor 404 may be a complementary metal oxide semiconductor (CMOS) type image sensor. The image sensor 404 may be a charge coupled device (CCD) type image sensor. The image sensor 404 may be an active pixel sensor (APS) type image sensor. The image sensor 404 may be a silicon-based image sensor.

[0072] As an example, the first set of input image frames 110a may include image data that has not been modified or edited, such as has not been modified, edited, and / or tampered with since being captured.

[0073] Alternatively, the first set of input image frames 110a may additionally include additions to the image data as captured, such as overlays added in the surveillance camera and / or from the same source as the captured image data, so that in this case the captured image data, with any possible additions, forming the first set of input image frames 110a may be seen as originating from the source of the first input image frames.

[0074] The second set of input image frames 120a may include edited or modified image data or pixel data. The second set of input image frames 120a may include, for example, graphical overlay elements such as bounding boxes, privacy masks, indicators, messages, or other types of overlay information in the form of text, numbers, symbols, etc. For example, such overlay information may relate to sensor data, such as from another sensor 405, as further illustrated below. The second set of input image frames 120a may be based on the first set of input image frames 110a. The second set of input image frames 120a may include, for example, image frames of the first set of input image frames 110a with graphical overlay elements added to the image frames. The second set of input image frames 120a may thus be understood as including modifications, edits, and / or changes to image data from a source of the first input image frames, such as captured by the video imaging device 403.

[0075] The modification, editing, or alteration of image data contained in the second set of input image frames 120a may be based on an analysis or evaluation of the first set of input image frames 110a compared to the first set of input image frames 110a. For example, an image analysis algorithm may be used to generate a privacy mask to obscure certain portions of the image frames of the first set of input image frames 110a. This privacy mask may then be applied to otherwise empty image frames or copies of image frames from the first set of input image frames 110a. The image frames with the privacy mask may then correspond to the second set of input image frames 120a.

[0076] The source 402 of the second input image frames may be a separate sensor 405 from the image sensor 404 of the video imaging device 403. The source 402 of the second input image frames, e.g., the separate sensor 405, may be part of the video imaging device 403 and / or may be contained within the same housing as the image sensor 404. The source 402 of the second input image frames may alternatively be physically separate and / or located at least some distance from the source 401 of the first input image frames or the video imaging device 403.

[0077] The other sensor 405 may be another image sensor. The other sensor 405 may be an infrared (IR) type sensor, for example an IR image sensor of an IR camera or an IR video imaging device. The other sensor 405 may be a light sensor. The other sensor 405 may be a radio detection and ranging (radar) type sensor. The other sensor 405 may be a light detection and ranging (lidar) type sensor. The other sensor 405 may be a motion sensor. The other sensor 405 may be a three-dimensional imaging sensor, a thermal camera, or a spectrometer adapted to generate a three-dimensional representation of the imaged scene.

[0078] The first key may be associated with, related to, or otherwise related to the source of the first input image frame 401. The first key may be an individual key associated with a single source of the first input image frame 401, such as a unique video imaging device 403. The first key may alternatively be a shared key for multiple sources of the first image frame 401.

[0079] The first key may belong to or be associated with the source of the first input image frame 401, e.g., an entity that develops, manufactures, provides, and / or maintains the video imaging device 403. The first key may be securely and / or exclusively accessible to the same entity. The first key may belong to or be associated with a judicial or governmental entity that oversees or regulates how captured image data is used or handled within the jurisdiction of interest. The source of the first input image frame 401 may be associated with the same entity.

[0080] The first key may be a first private key paired with a first public key according to the principles of public key cryptography, as known per se, where the first private key and the first public key may be generated or may have been generated at the source 401 of the first input image frame.

[0081] As an example, signing the first signature group with the first key S1050 may include computing a hash of image data of one or more encoded image frames of the first set of encoded image frames 110b included in the first signature group and encrypting the resulting hash using the first private key.

[0082] The hash function may be a cryptographic hash function. The hash function may be a Secure Hash Algorithm (SHA) or Digital Signature Algorithm (DSA) type hash function. The hash function may be a SHA-0, SHA-1, SHA-2, SHA-3, etc. type hash function.

[0083] Signing with the first key S1050 may result in a signature, which may be a collection of bits or bytes, a string of characters, or the like, resulting from signing with the first key. The signature associated with signing with the first key S1050 may thereby be associated with the combined video stream 100. Optionally, a public key may also be associated with the combined video stream 100 as well. For example, one or more signatures and / or public keys from signing may be included in the combined video stream and thus included therein. Alternatively, for example, the signatures and / or public keys may be separate from the combined video stream 100 but associated therewith, for example, in the same directory, database, file system, or storage facility, or by being transmitted over the same data stream, channel, connection, or the like.

[0084] The second key may be associated with, related to, or otherwise related to the source of the second input image frame 402. The second key may be an individual key associated with the source of the second input image frame 402. The second key may alternatively be a shared key for multiple sources of the second image frame 402.

[0085] The second key may belong to or be associated with an entity that develops, manufactures, provides and / or maintains the first source of input image frames 401 and / or an entity that does the same for the second source of input image frames 402. Typically, these two entities are distinct, but may also be the same.

[0086] The second key may belong to or be associated with a third party entity other than the entity that developed, manufactured, provided, and / or maintained the source of the first or second image frames 401, 402. The second key may belong to or be associated with a judicial or governmental entity that oversees or regulates how captured image data is used or handled within the jurisdiction of interest. The second key may be securely and / or exclusively accessible to one of the above entities.

[0087] The second key, like the first key, may be a second private key paired with a second public key, as known per se in public key cryptography. Like signing with the first key S1050, signing with the second key S1060 may result in a signature. The signature associated with signing with the second key S1050 may thereby be associated with the combined video stream 100, for example, through including the signature in the combined video stream. Optionally, the second public key may be associated with the combined video stream 100 as well. Alternatively, the signature and / or the second public key may be associated with the combined video stream 100 in some other manner, as exemplified above in conjunction with signing with the first key S1050.

[0088] The source 402 of the second input image frame may be a video processing unit 406 (see FIG. 5). The video processing unit 406 may generate image frames that include, or image data related to, graphical overlay elements such as bounding boxes, privacy masks, indicators, messages, or other overlay information in the form of text, numbers, symbols, etc.

[0089] The video processing unit 406 may be adapted or configured to process the images captured by the source 401 of the first input image frames into some form of image data.

[0090] The video processing unit 406 may include a graphical overlay addition functionality that may be configured or adapted to add graphical overlay elements to image frames of, for example, the first set of input image frames 110a.

[0091] The encoder 410 (see FIG. 5) may be a physical hardware-based encoder that relies on its hardware to perform the encoding. The encoder 410 may be a software-based encoder that relies on its programming or code to perform the encoding. The encoder 410 may also be a combination of the two. The encoder 410 may be integrated with the video imaging device 403. The encoder 410 may be located, for example, in the same physical housing as the video imaging device 403. The encoder 410 may be configured and / or arranged to be in electrical and / or wireless connection with the video imaging device 403. The encoder 410 may be a standard encoder.

[0092] Still referring to FIG. 1, the example method may further include receiving S2030, at the encoder 410, a set of third input image frames 130a including image data originating from a source of the third input image frames.

[0093] The example method may further include signing S2070 the third set of encoded image frames 130 with a third key, either in a separate unit such as signing unit 411, or in the encoder 410 or any of the above. The third key may be related to the source of the third input image frames.

[0094] The signing S1050, S1060, S2070 is performed on a signing group that includes one or more image frames. Typically, a signing group may correspond to each image frame of a set of encoded image frames in a group of pictures (GOP) in the combined video stream, a concept known per se in the field of video encoding. Alternatively, signing may be performed on a frame-by-frame basis, in which case the signing group corresponds to the frame being signed.

[0095] The second signature group may exclusively include encoded image frames of the second set of image frames, or alternatively, the second signature group may include image frames from both the first set of encoded image frames and the second set of image frames, where one or more encoded image frames of the first set of encoded image frames in the second signature group are referenced by encoded image frames of the second set of encoded image frames in the second signature group.

[0096] Signing S1050, S1060, S2070 may occur after or in parallel with encoding S1040. This disclosure should not be understood to exclude any alternatives.

[0097] The signing may be performed by the signing unit 411 or by a signing function of the encoder 410 .

[0098] 3 illustrates how encoding S1040 of the first set of input image frames 110a and the second set of input image frames 120a may further include encoding a third set of input image frames 130a into a third encoded set of image frames 130b to form the combined video stream 100. This diagram illustrates how an image frame of the third encoded set of image frames 130 may reference both an image frame of the first set of encoded image frames 110b and another image frame of the third encoded set of image frames 130.

[0099] In Figure 3, references are indicated by arrows, as in Figure 2. The figure should be understood as corresponding to a chronological sequence of image frames, with the leftmost image frame being the first image frame and the rightmost image frame being the last image frame.

[0100] 3 illustrates how the first, second, and third encoded image frame sets 110b, 120b, 130b may be arranged in the combined video stream 100 and how each of the image frames references each other. The references in the first encoded image frame set 110b and the second encoded image frame set 120b are as described above in conjunction with FIG. 2. Additionally, as shown for all frames in the third encoded image frame set 130b, an encoded image frame may reference both an encoded image frame in the first encoded image frame set 110b and another encoded image frame in the third encoded image frame set 130b.

[0101] This may be understood as encoding S1040 including inter-encoding the first set of input image frames 110a, the second set of input image frames 120a, and optionally the third set of input image frames 130a into first, second, and third encoded sets of image frames 110b, 120b, 130b, respectively, in the combined video stream 100.

[0102] Encoding S1040 may include predictively encoding the first set of input image frames 110a, the second set of input image frames 120a, and optionally, the third set of input image frames 130a into first, second, and third encoded sets of image frames 110b, 120b, 130b, respectively, in the combined video stream 100.

[0103] The third set of input image frames 130a may be understood to be similar to the second set of input image frames 120a, and may, for example, include additional image data or image data that is different from the image data present in the first set of input image frames 110a, i.e., the original image data or image data present in the second set of input image frames 120a.

[0104] The third source of input image frames may be understood as an additional source of input image frames or a further source of input image frames that is similar or identical in content or use to the first and second image frame sources 401, 402.

[0105] The third key may be understood to be the same as the second key. The third key may be related to, associated with, or otherwise related to the source of the third input image frame. The third key may be an individual key related to the source of the third input image frame. The second key may alternatively be a key shared across multiple sources of third image frames 402. The third key, like the first and second keys, may be a hash function key.

[0106] Like signing with the first key S1050 and signing with the second key S1060, signing with the third key S2070 may result in a signature, which may be, for example, a hash. The signature associated with signing with the third key S2070 may then be associated with the combined video stream 100, for example, through including the signature in the combined video stream. Alternatively, the signature may be associated with the combined video stream 100 in some other manner, as exemplified above.

[0107] Further sets of image frames from further image frame sources with corresponding further keys are entirely possible following the same principles as the three sets of image frames, image frame sources, and keys disclosed herein.

[0108] One example may further include, in the encoder 401, designating the first set of encoded image frames 110b as non-displayed image frames S2042.

[0109] Additionally, this example may further include, in the encoder 401, designating the second set of encoded image frames 120b and / or the third set of encoded image frames 130b as the image frames to be displayed.

[0110] As a result, any image frame in these encoded image frame sets 110b, 120b, 130b may be designated, marked, or tagged, for example, as not to be displayed, i.e., hidden, according to the video coding format used.

[0111] Such designation of image frames as not being displayed may take the form of a No Display (ND) flag in each of the frames, for example, as defined in video coding standards such as H.265, H.266 / VVC, EVC, and AV1.

[0112] Any image frame of these sets of image frames 110b, 120b, 130b may alternatively be designated, marked, or tagged as being for display, i.e., for display, in accordance with the video coding format used. The designation, marking, or tagging may be performed on the encoded image frames.

[0113] For example, tagging the first set of encoded image frames with an ND flag provides the possibility of including the signed first set of encoded image frames 110b in the combined video stream 100 in a way that prevents them from being included in the output video stream when decoded using a conventional decoder. Rather, by reference hereafter, decoded image frames that include the combined image data of the first and second sets of image frames are included in the output video stream from the decoder. This also applies in situations where the third set of encoded image frames 103b is also included in the combined video stream 100.

[0114] In practice, a standard decoder will not forward or transmit to the output video stream, e.g., for display, analysis, or storage, image frames that are designated, marked, or tagged as non-display image frames. Conversely, a standard decoder will forward or transmit to the output video stream, e.g., for display, analysis, or storage, image frames that are designated, marked, or tagged as display image frames.

[0115] FIG. 4 illustrates an example method for processing a video stream that includes additional optional steps in addition to those disclosed above in conjunction with FIG.

[0116] This example may further include decoding S3080 the combined video stream 100 in a decoder 420 (see FIG. 5).

[0117] For example, decoding may result in a decoded video stream including a set of decoded image frames 230 (see FIG. 5). The set of decoded image frames 230 may correspond to the first set of input image frames 110a as modified by the second set of input image frames 120a. This may be the case, for example, for the decoder 420 operating as a standard decoder.

[0118] Additionally or alternatively, the decoding may result in another decoded video stream including another set of decoded image frames 110c (see FIG. 5) that directly corresponds to the first set of input image frames 110a, for example through a decoder 420 extracting frames tagged as not displayed from the combined video stream, as described above.

[0119] The decoder 420 may be a physical hardware-based decoder that relies on its hardware to perform the decoding. The decoder 420 may be a software-based decoder that relies on its programming or code to perform the decoding. The decoder 420 may also be a combination of the two. The decoder 420 may be incorporated with the video imaging device 403, the storage unit, the display unit 430, or the display device. The decoder 420 may be located, for example, in the same physical housing as the video imaging device 403, the storage unit, the display unit 430, or the display device. The decoder 420 may be configured and / or arranged to be in electrical and / or wireless connection with the video imaging device 403, the storage unit, the display unit 430, and / or the display device. The decoder 420 may be a standard decoder.

[0120] Checking the signature of the image frame S3090 or signing may occur before, in parallel with, or after decoding S3080.

[0121] Checking the signature of the image frame S3090 may use a public key that corresponds to the private key used in signing. The public key may be retrieved from the combined image stream 100. Alternatively, the public key may be stored with the corresponding image source identifier, such as a camera identifier, so that it can be readily used when the transmitted video data requires verification.

[0122] Checking S3090 may include decrypting the hash that includes the signature using the public key. This makes the result of the hashing available again. A further hash may be calculated based on the received image data using the same hashing algorithm used during signing. This resulting further hash may be compared to the decrypted hash to verify both the identity of the sender and that the video data has not been modified.

[0123] The checking may be performed per signature group, such as per GOP or per frame. The checking may be performed by the signature checker unit 431. The checking may issue a signature result. This result may for example be in the form of an ok or not ok result. Based on these results, such as ok or not ok, the signature checker unit 431 or the decoder 420 may place S3092 a corresponding marker 421, 422 chosen according to and in relation to the ok or not ok status, respectively, in the decoded image frame, for example as part of the set of decoded image frames 230 or another set of decoded image frames 110c. The nature of the marker thus constitutes data related to the result of the check, which data is associated with the decoded video stream of which the image frame forms part.

[0124] The decoder 420 or the storage device, or the display device or the video playback device may include a signature checker unit 431 (see FIG. 5). The signature checker unit 431 may be connected to the decoder 420, for example for communication purposes, by an electrical or wireless connection. The signature checker unit 431 may be physically connected to the decoder 420 or may be integrated therein.

[0125] Determining the origin or authenticity of the image frame S3100 may occur before, in parallel with, or after decoding S3080. This check may be performed on a frame-by-frame basis, for example, based on whether the frame is designated as a non-displayed image or is not designated as a non-displayed image. Alternatively, the origin check may be based on a signature on the combined video stream.

[0126] The origin check may be performed by origin checker unit 432 (see FIG. 5) or in decoder 420. If the frame was designated as a non-displayed image frame, the origin check may determine that the image frame originated from a first input image frame source. If the frame was not designated as a non-displayed image frame, the origin check may determine that at least a portion of the image frame originated from a second input image frame source.

[0127] Based on the origin checker unit 432 determining whether an image frame originated from the first image frame source of the second image frame source, the origin checker unit 432 or the decoder 420 may place a marker 421 associated with the source of the first image frame or a marker 422 associated with the source of the second image frame in that image frame S3012.

[0128] Thus, the markers 421, 422 constitute data based on the result of determining S3010. That data is associated with the image frame in the form of the markers 421, 422.

[0129] The decoder 420, or the display device or video playback device, may include an origin checker unit 432. The origin checker unit 432 may be connected to the decoder 420, for example by an electrical or wireless connection for communication purposes. The origin checker unit 432 may be physically connected to the decoder 420 or may be integrated therein.

[0130] Alternatively or in addition to the markers 421, 422, data related to the results of checking the signature and / or data based on determining the origin may be added to the headers of the image frames of these decoded image frames sets 110c, 230. This data may be added to the image frame logs of these decoded image frames sets 110c. This data may be added to the metadata of the image frames of these decoded image frames sets 110c, 230.

[0131] The markers 421, 422 may be added directly to the image frames of these decoded image frame sets 110c, 230 as modified pixel values. The markers 421, 422 may include, for example, graphical overlay elements such as bounding boxes, privacy masks, indicators, messages, or other types of overlay information in the form of text, numbers, symbols, etc. The markers 421, 422 may be added near the corners and / or near the edges of the image frames of these decoded image frame sets 110c, 230.

[0132] The data may be configured to allow the decoder 420, or a display device or unit playing the decoded combined video stream, to provide predetermined markings by modifying pixels of these image frames of the set of decoded image frames 110c, 230. The marker 421 may be configured to indicate the origin of the image frames. The markers 421, 422 may be configured to indicate whether the source 401 of the first input image frame is a genuine origin of these set of decoded image frames 110c, 230 or not.

[0133] The markers 421, 422 may include icons, images, and / or logotypes associated with the source of the first input image frames 401. The markers 421, 422 may include color coding to indicate whether the source of the first image frames 401 is or is not a genuine source of these decoded image frames set 110c, 230. For example, a green colored marker 421, 422 may indicate that the source of the first image frames 401 is a genuine source of these decoded image frames set 110c, 230, and a red colored added marker 421 is not.

[0134] The data, e.g., markers 421, 422, may be adapted or configured to indicate to an operator viewing the video stream whether the image frame of interest has been modified, edited, or altered since capture, or not.

[0135] In some embodiments, if the source of the first image frame 401 is determined to be the true origin of this set of input image frames 110a, then no data is added. In this case, data may be added only if the source of the first image frame 401 is not determined to be the true origin of this set of input image frames 110a.

[0136] The above teachings may be implemented by a computer. To this end, a non-transitory computer-readable storage medium may be provided that includes instructions stored thereon that, when executed on a device having processing capabilities, implements the above example method.

[0137] The non-transitory computer-readable storage medium and / or the device having processing capability may be part of the encoder 410 and / or the decoder 420 or may be connected thereto, for example for communication, by an electrical and / or wireless connection. The non-transitory computer-readable storage medium and / or the device having processing capability may be physically connected to and / or integrated with the encoder 410 and / or the decoder 420.

[0138] Additionally, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. receiving a set of first input image frames including image data originating from a source of the first input image frames; receiving a second set of input image frames including image data originating from a second input image frame source, the second set of input image frames including modifications or alterations to the first set of input image frames; encoding the first set of input image frames and the second set of input image frames into a first set of encoded image frames and a second set of encoded image frames, respectively, to form a combined video stream, wherein an encoded image frame of the first set of encoded image frames exclusively references one or more other encoded image frames of the first set of encoded image frames and an encoded image frame of the second set of encoded image frames references both an encoded image frame of the first set of encoded image frames and another encoded image frame of the second set of encoded image frames; designating the first set of encoded image frames as image frames that will not be displayed or designating the second set of encoded image frames as image frames that will be displayed; signing a first signature group including one or more encoded image frames of the first set of encoded image frames using a first key associated with a source of the first input image frame; signing a second signature group including one or more encoded image frames of the second set of encoded image frames using a second key associated with a source of the second input image frame; A method for processing a video stream, including:

2. 2. The method of claim 1 , wherein the second signature group further includes one or more encoded image frames of the first set of encoded image frames that are referenced by an encoded image frame of the second set of encoded image frames.

3. receiving a set of third input image frames including image data originating from a source of the third input image frames; Further comprising: The encoding further includes encoding the third set of input image frames into a third set of encoded image frames to form part of the combined video stream, wherein an encoded image frame of the third set of encoded image frames references both an encoded image frame of the first set of encoded image frames and another encoded image frame of the third set of encoded image frames, the method comprising: signing a third signature group including one or more image frames of the third set of encoded image frames using a third key associated with a source of the third input image frame; The method of claim 1 further comprising:

4. The method of claim 1 , wherein the source of the first input image frame is an image sensor of a video imaging device.

5. The method of claim 4 , wherein the source of the second input image frame is a sensor separate from the image sensor of the video imaging device.

6. The method of claim 1 , wherein the source of the second input image frame is a video processing unit.

7. The method of claim 6 , wherein the video processing unit includes a graphical overlay adding function.

8. The method of claim 1 , wherein the first key is a hash function key and / or the second key is a hash function key.

9. 2. The method of claim 1 , wherein the encoding comprises inter-encoding the first set of input image frames and the second set of input image frames into the first set of encoded image frames and the second set of encoded image frames, respectively.

10. said combined video stream, a signature associated with said signing using said first key; a signature associated with said signing using said second key; and The method of claim 1 , further comprising associating the at least one of:

11. decoding, at a decoder, the combined video stream; checking a signature included in the combined video stream; associating data relating to a result of said checking with said decoded video stream; The method of claim 1 further comprising:

12. decoding, at a decoder, the combined video stream; determining an origin of image frames of the combined video stream; associating data based on said determining said origin with said image frame; The method of claim 1 further comprising:

13. A non-transitory computer readable storage medium having stored thereon instructions which, when executed on a device having processing capabilities, perform the method of any one of claims 1 to 12.

14. receiving a set of first input image frames including image data originating from a source of the first input image frames; receiving a second set of input image frames comprising image data originating from a second input image frame source, the second set of input image frames comprising modifications or alterations to the first set of input image frames; encoding the first set of input image frames and the second set of input image frames into a first set of encoded image frames and a second set of encoded image frames, respectively, to form a combined video stream, wherein an encoded image frame of the first set of encoded image frames exclusively references one or more image frames of the first set of encoded image frames and an encoded image frame of the second set of encoded image frames references both an encoded image frame of the first set of encoded image frames and another encoded image frame of the second set of encoded image frames; designating the first set of encoded image frames as image frames that are not to be displayed or designating the second set of encoded image frames as image frames that are to be displayed; signing a first signature group including one or more encoded image frames of the first set of encoded image frames using a first key associated with a source of the first input image frame; signing a second signature group including one or more encoded image frames of the second set of encoded image frames using a second key, the second key being associated with a source of the second input image frame; 16. A video processing system comprising: a processor configured to:

Images