Method, graphical encoder and system for embedding information in a video signal

Abstract

A method for embedding information in a video signal is described. The method comprises receiving a message comprising information; dividing the message into a first message part and a second message part; obtaining a first video frame and a second video frame from the video signal, wherein the second video frame is temporally subsequent to the first video frame and the video frames each comprise a predetermined number of pixels; and determining a motion map associated with the second video frame, wherein the motion map indicates a movement of individual pixels of the second video frame relative to the first video frame. The method further comprises embedding the first message part in the pixels of the second video frame, including weighting the first message part for individual pixels of the second video frame based on the motion map; and embedding the second message part in the pixels of the second video frame, including weighting the second message part for individual pixels of the second video frame based on an inverse of the motion map. Furthermore, a graphics encoder and a system configured to perform such a method are described.

Description

Technical Field

[0001] This invention relates to methods and systems for embedding information in video signals. Specifically, the invention relates to methods for embedding information in video signals while taking into account motion graphs associated with currently processed video frames, and also to a graphics encoder configured to perform the method, and a system including a camera and such a graphics encoder. Background Technology

[0002] Some applications and services involve the processing of live images (e.g., real-time video) captured by cameras. For example, in vehicles, certain driver assistance systems and / or autonomous driving controls involve cameras screening street activity near the vehicle. These systems and control processes sometimes include several processing units or software modules for different tasks, all of which rely on camera signals, and some of them also rely on object detection in the vicinity of the vehicle. Object detection includes observing and tracking objects, such as stationary objects as well as moving objects (e.g., people, another vehicle, etc.). In some cases, the camera images are also displayed at the time of capture, i.e., without any labels or highlights of one or more detected objects and / or information.

[0003] In the case of object detection, object recognition and / or classification can be completed at a different time than image processing used to display camera signals. Therefore, timestamps, etc., are used to allow for the synchronous processing and / or output of different data signals. However, this incurs processing overhead because synchronization information must be revealed from different data signals or data streams. Summary of the Invention

[0004] Therefore, the object of the present invention is to provide a method and system for improving the processing and storage of time-sensitive information and video information.

[0005] According to a first aspect of the invention, a method for embedding information in a video signal includes: receiving a message including information; dividing the message into a first message portion and a second message portion; and acquiring a first video frame and a second video frame from the video signal, wherein each video frame includes a predetermined number of pixels, and wherein the second video frame is temporally subsequent to the first video frame. The method further includes: determining a motion graph associated with the second video frame, wherein the motion graph indicates the movement of individual pixels of the second video frame relative to the first video frame.

[0006] A first message part and a second message part can be derived by dividing the message into two parts, such that the two message parts together form the original message. This division can be based on the bits representing the message, the symbols representing the message, and / or context-based parts of the message. For example, in the case where the message originates from object detection in a video signal, the message can be divided into two parts with the same number of bits or symbols, or it can be divided into two parts, each including information about a predetermined number of objects.

[0007] The first and second video frames are sequential in time, meaning the second video frame is captured at a later time than the first. While the second video frame can be captured directly after the first, the time interval between the two frames can be more than one frame. Therefore, temporal succession can also be based on the frame rate of the video signal.

[0008] The method further includes: embedding a first message portion into pixels of a second video frame, which includes weighting the first message portion for each pixel of the second video frame based on a motion map; and embedding a second message portion into pixels of the second video frame, which includes weighting the second message portion for each pixel of the second video frame based on the inverse of the motion map. In other words, the image data of the second video frame is modified and / or replaced and / or supplemented with data from the message. Therefore, the message can be stored (embedded) in the second video frame, wherein the message can represent information specific to that second video frame. This allows for frame-synchronized storage and availability of information, avoiding the use of timestamps and the corresponding processing of different data signals that require buffering and selecting synchronized data from different signals or data streams.

[0009] Furthermore, the segmented messages are embedded into pixels in a weighted manner, where the weights are defined by the motion map and the inverse of the motion map for each message portion. Specifically, the motion map represents the movement of an object or region in the second video frame, which has a different position within the second video frame compared to the first video frame. In other words, pixels representing the same object in the captured image in the second video frame are the same or very similar to pixels representing the same object in the first video frame, but are arranged at different pixel positions within the frame. Therefore, weighting the first message portion based on the motion map results in a higher influence on pixels in the second video frame that have motion, while weighting the second message portion based on the inverse of the motion map results in a higher influence on pixels in the second video frame that have less motion or are relatively static. In other words, static pixels (i.e., motion maps containing pixels with small motion amplitudes) are less affected by the embedded messages compared to dynamic pixels.

[0010] Because the message is embedded in the pixels (e.g., pixel data), it alters the original video data. This can cause artifacts or other visible modifications to the pixel data when displayed on a monitor. However, in areas with video frames that have significant movement, such artifacts or modifications are less detectable by the human visual system, as it is less sensitive to moving areas than to static areas. Therefore, if a modified second video frame containing the embedded message is displayed on a monitor, the changes to the data are not visible to the human user, or can only be slightly derived from the presence of artifacts, etc.

[0011] Embedding information into pixel data also allows for video signal compression, such as lossy compression. Therefore, messages can be stored or transmitted in frame-synchronized manner while maintaining traditional image processing. The eventual loss of some information from the message due to video signal compression is negligible, as not all information will be lost, and because object information also exists in earlier and / or later video frames. In short, sufficient information about the object can be obtained from the video stream, including the second and subsequent video frames. It should be understood that conventional error detection and correction methods (e.g., using Hamming codes) can be applied to increase the reliability of the message received after decoding.

[0012] In one implementation variation, the first message portion may include at least 50% of the message, and the second message portion may include the remainder of the message. Thus, 50% or more of the message is stored (embedded) in the second video frame, focusing on a moving area where the human visual system will barely detect the modification or may not detect it at all. The remaining portion of the message (which may be 50% or less) is stored (embedded) in the second video frame, focusing on a less moving or static area. Therefore, depending on the size of the message, information can be embedded in the second video frame without being noticed by the human visual system.

[0013] Dividing a message into a first message portion and a second message portion may include determining the total size of the message and calculating the resulting data size for each pixel (e.g., bits per pixel). Based on the data size per pixel, the message division can be selected or set with a division ratio, such as 40% and 60%, 30% and 70%, or even 10% and 90%, where, according to the motion map, the larger message portion will be embedded in the pixels of the second video frame if it is concentrated in the moving area.

[0014] In another implementation variation, embedding the first message portion may include providing a first copy of the data representing the second video frame, concatenating the first message portion to the first copy, and weighting the concatenated data based on a motion graph. Embedding the second message portion may include providing a second copy of the data representing the second video frame, concatenating the second message portion to the second copy, and inversely weighting the concatenated data based on a motion graph. Therefore, embedding the first and second message portions can be performed simultaneously on the respective data copies. Moreover, message information can be supplemented to the video data through concatenation.

[0015] For example, the second video frame can be preprocessed, such as by performing convolution, batch normalization, and / or modified linear unit processing. The resulting data representing the second video frame can be referred to as a feature map, etc. This data is then copied into the first and second copies.

[0016] The concatenation of the first message portion to the first copy and the concatenation of the second message portion to the second copy are respectively supplemented with feature maps using message bits. Alternatively, message bits (actually the bits of each message portion) can be preprocessed.

[0017] The cascaded data can be further processed, for example by performing another convolution, batch normalization, and / or correcting the linear unit processing.

[0018] In another implementation variation, the method may further include forming a stegframe by merging pixels (or data representing the second video frame) of a second video frame in which a first message portion is embedded with pixels (or data representing the second video frame) of a second video frame in which a second message portion is embedded. The stegframe corresponds to a video frame modified by embedding a message into video data (e.g., the aforementioned pixel data of the second video frame). Specifically, when a first copy and a second copy of the data representing the second video frame are provided, pixel merging can be performed quickly and easily based on parallel data or data streams output by the associated processing module. Merging pixels or data representing the second video frame in which the corresponding message portion is embedded includes considering a motion graph. Specifically, pixels with embedded data or data streams are weighted based on the motion graph and its inverse, respectively.

[0019] Optionally, the formation of the steganographic frame may also include merging the aforementioned merged data with the pixel or pixel data of the unchanged second video frame (i.e., excluding the portion of the message). For example, the embedding of the message results in modified data, which can be considered as a residual added to the original unchanged second video frame. In other words, the embedded data is added as a residual to the overlay frame.

[0020] Alternatively, if the size of the message does not require modifying all pixels in the second video frame to store (embed) the message, then specific pixels or regions of the second video frame can remain unchanged.

[0021] In another implementation variation, message partitioning may include generating a first 3D tensor with message bits for a first message portion. Specifically, the individual bits of the first message portion are copied across the width and height of one channel of the tensor, where the width and height correspond to the width and height of the pixels of the second video frame (i.e., the pixel resolution of the second video frame). Message partitioning may also include generating a second 3D tensor with message bits for a second message portion, where the individual bits are copied across the width and height of one channel of the tensor, and where the width and height correspond to the width and height of the pixels of the second video frame. For example, the number of channels in the first and second 3D tensors may correspond to the number of bits in the first and second message portions, respectively. Thus, each channel has a width and height corresponding to the width and height of the pixels of the second video frame and is filled with the same bit values ​​(0 or 1). This generation of the first and second 3D tensors facilitates concatenating messages to the video signal / data of the second video frame, for example, representing a first and second copy of the data of the second video frame to be concatenated with the corresponding message portions.

[0022] In another implementation variation, determining the motion map may include calculating individual values ​​for the motion map for each pixel as values ​​between zero (0) and 1 and having a predetermined average value of the second video frame. For example, the predetermined average value of the second video frame may be between 0.1 and 0.49, preferably between 0.1 and 0.3, and most preferably 0.25. More specifically, the motion map is determined or calculated by setting the corresponding motion map values ​​(e.g., individual motion map values ​​corresponding to pixels or groups of pixels) to 0 for static pixels or regions, 1 for (maximally moved) dynamic pixels or regions, or values ​​in between.

[0023] In another implementation variation, determining the motion map includes calculating the optical flow between the first and second video frames. Optical flow represents the motion distribution over a region of the video frame (or, as the smallest possible region of pixels). For example, optical flow can represent the motion distribution of a particular video frame region relative to a static region (value 0), relative to the minimum of all motion, or relative to the maximum of all motion (value 1). Therefore, a map with a size of (pixel resolution) of the second video frame can be provided, representing the motion distribution over the second video frame; that is, a motion map of the second video frame can be provided.

[0024] Furthermore, optical flow can output motion vectors for each pixel, each with a horizontal component u and a vertical component v. The magnitude of these motion vectors can then be calculated, for example, as sqrt(u... 2+v 2 For each frame, normalize it to between zero (0) and one (1).

[0025] Then, the average value of the normalized optical flow amplitude (normalized motion vector) is made to reach a predetermined average value of the second video frame, for example, as described above, between 0.1 and 0.49, preferably between 0.1 and 0.3, and most preferably 0.25. For example, an offset can be added to each pixel to achieve the predetermined average value mentioned above. The offset can be calculated individually for each frame, but the offset is constant within a frame. This helps to embed the message portion based on the motion graph (i.e., the message portion corresponding to higher motion values) even in video frames with fewer or no moving objects in the corresponding video signal, which would result in an amplitude close to zero without offset. Similarly, in the case of excessive motion between the first and second video frames, a negative offset can reduce otherwise higher values ​​of the motion graph.

[0026] Furthermore, setting the corresponding motion map values ​​in such a way that the average value is less than 0.5 (e.g., preferably 0.25) has the following effect: a first message portion, which may include at least 50% of the message, is hidden (embedded) in the 25% of the image content of the second video frame with the most movement, while a second message portion, which may include 50% or less of the message, is hidden (embedded) in the 75% of the second video frame with the least movement. In other words, embedding the second message portion in 75% of the second video frame allows for more area per bit (of the message), resulting in better video quality (i.e., showing fewer artifacts or other visible modifications to the video data). For example, the first feature map (i.e., the concatenated data of the second video frame in which the first message portion is embedded) is multiplied pixel-by-pixel by the corresponding motion map value, and the second feature map (i.e., the concatenated data of the second video frame in which the second message portion is embedded) is multiplied pixel-by-pixel by the inverse of the corresponding motion map value (i.e., multiplied by 1 - "motion map value"). By merging two multiplied feature maps, for example by adding the multiplied feature maps together, the aforementioned distribution of the first message portion and the second message portion on the second video frame according to dynamic and relatively static motion is achieved.

[0027] In another implementation variation, the method further includes training to embed the first and second message portions into pixels of a second video frame. Training may include building a convolutional neural network (CNN) or encoder network configured to embed the first and second message portions into pixels of the second video frame. Training the CNN may include sending a stegated frame (i.e., the second video frame in which the message is embedded) to another neural network (referred to as the "opposite network") configured to detect whether the stegated frame includes the embedded information. In other words, the opposite network, which may be a conventional network or module, is trained to evaluate whether the video signal contains either the "original" frame data or the hidden data, also known as "steganography." This stegography network can respond with a probability that the second video frame includes the hidden message. The CNN is now trained based on the results of the stegography. Specifically, the CNN is trained to embed the message in such a way that the stegography always responds with a probability of 50% or close to 50%. Therefore, the opposite network "detects" the hidden message with the same probability as the "original" video frame; that is, no clear detection is achieved.

[0028] For example, a CNN can be implemented in the preprocessing and / or postprocessing of data representing the second video frame before and after cascading, as described above.

[0029] Furthermore, according to a variation of the implementation, the method may also include sending the trained neural network (model) to the decoder. Thus, the decoder can decode the steganalyte frame and retrieve the embedded message based on the trained encoder network. Alternatively or additionally, the decoder network can be trained together with the encoder network, such that they each have corresponding models for decoding and encoding, respectively.

[0030] According to a second aspect of the invention for better understanding, a computer-readable medium is configured to store executable instructions that, when executed by a processor, cause the processor to perform a method varying according to the first aspect or at least one implementation thereof. For example, the computer-readable medium may be a volatile or non-volatile memory, such as a CD, DVD, USB stick, RAM, ROM, etc.

[0031] According to a third aspect of the invention for a better understanding, a graphics encoder is configured to perform a method varying according to the first aspect or at least one implementation thereof. This graphics encoder can be implemented in hardware, software, or a combination thereof. The graphics encoder can also be used in any graphics system that processes video image data and requires the storage of additional information in a frame-synchronized manner.

[0032] According to a fourth aspect of the invention for better understanding, a system includes: a camera configured to capture video signals; and a graphics encoder configured to perform a method variant of the first aspect or at least one implementation thereof.

[0033] The system can be implemented in a vehicle, such as a car, truck, motorcycle, bicycle, boat, or airplane. Alternatively, it can be implemented in a mobile device, such as a mobile phone, tablet, or laptop.

[0034] In one implementation variation, the system may include an object detector configured to identify and classify at least one object in a second video frame, wherein the object detector outputs a message, and wherein the information included in the message includes at least identification and / or classification information of at least one object. Such an object detector can be employed in a system to identify and classify objects around the system. For example, for a vehicle, such an object detector may be a system component of the vehicle that processes video signals captured by a camera to identify and classify objects around the vehicle, such as other vehicles, pedestrians, bicycles, trees, houses, street signs, etc.

[0035] The graphics encoder can then be configured to embed a message containing information associated with the second video frame and output by the object detector into the second video frame. Thus, the data signal can be implemented in the form of a video signal supplemented with information about objects related to a particular video frame. Therefore, object information can be stored with the video data in a frame-synchronized manner; that is, identification and / or classification information of at least one object present in the second video frame is stored with the second video frame.

[0036] In another implementation variation, the system may further include a data interface configured to output a second video frame in which a first message portion and a second message portion are embedded. This modified second video frame is also referred to as a steganography frame (or simply "stego frame") because the message is hidden within the second video frame when it can be displayed on the screen with no significant visibility of the message. Therefore, the data signal output by the data interface corresponds to the "normal" video signal and can be processed accordingly.

[0037] In another implementation variation, the data interface can be configured to compress a second video frame in which a first message portion and a second message portion are embedded. Therefore, the storage and / or transmission of compressed video data saves storage space and / or bandwidth, as is the case in conventional video image processing systems. Alternatively, a video compression module can be used before the compressed video data is passed to the data interface.

[0038] In another implementation variation, the system, particularly when installed in or used as a vehicle, may also include a driver assistance system configured to receive a second video frame from a data interface and decode messages from the second video frame. Thus, the driver assistance system can derive information about the second video frame, such as at least one object present near the vehicle. For example, if the message includes information output by an object detector, the driver assistance system can decode the identification and / or classification of one or more objects from the second video frame. Based on this information, the driver assistance system can assist the driver of the vehicle using the information about one or more objects, such as warnings, speed limit signs, pedestrians, cyclists, or other vehicle features highlighted in the second video frame. The driver assistance system may also include a display, such as a head-up display, where information about one or more objects can be displayed to the driver of the vehicle.

[0039] In another implementation variation, the system may also include a display configured to receive and display the second video frame from the data interface. Such a display can be used to simply display the video signal derived from the graphics encoder. For example, the video could be displayed on a display used to show the background of a vehicle while it is moving in reverse, as if by a virtual rearview mirror. Although messages may be embedded in the second video frame (and subsequent video frames), the resulting video signal (steganographic frame) can be displayed on the display as any other video signal.

[0040] In another implementation variation, the graph encoder may further include a neural network configured to embed a first message portion and a second message portion into pixels of a second video frame. The neural network may be a convolutional neural network (CNN) or an encoder network. Training the CNN may involve sending a stegated frame (i.e., a second video frame in which the message is embedded) to another neural network (referred to as the "opposite network") configured to detect whether the stegated frame contains the embedded information. In other words, the opposite network, which may be a conventional network or module, is trained to evaluate whether the video signal contains either the "original" frame data or the hidden data, also known as "steganography." This stegography network may respond with a probability that the second video frame contains the hidden message. The CNN is trained based on the results of the stegography. Specifically, the CNN is trained to embed the message in such a way that the stegography always responds with a probability of 50% or close to 50%. Therefore, the opposite network "detects" the hidden message with the same probability as the "original" video frame; that is, no clear detection is achieved.

[0041] Furthermore, depending on an implementation variation, the system may also include a decoder, for example, formed by or including a CNN. The decoder network can be configured to receive a trained neural network (model) from the encoder, or to load or otherwise implement a trained model. Thus, the decoder can decode steganalytes based on the trained encoder network and retrieve the embedded messages. Alternatively or additionally, the encoder network can be trained together with the decoder network, such that both have corresponding models for encoding and decoding, respectively.

[0042] According to a fifth aspect of the invention for better understanding, a vehicle includes: a camera; and a graphic encoder configured to perform a method according to the first aspect or one or more associated implementations thereof. The graphic encoder may be a dedicated encoder processor or may be part of a vehicle computer system, such as an engine control unit (ECU).

[0043] Furthermore, the vehicle may also include any components that are modified according to the fourth aspect or one of its implementations.

[0044] This invention is not limited to the aspects and variations in the form and order described. Specifically, the description of aspects and variations should not be construed as a specific limiting grouping of features. It should be understood that this invention also covers combinations of aspects and variations not explicitly described. Therefore, various variations or optional features can be combined with any other aspect, variation, optional feature, or even combination thereof. Attached Figure Description

[0045] Preferred embodiments of the invention will now be explained in more detail with reference to the schematic accompanying drawings, in which:

[0046] Figure 1 An encoder is illustrated schematically;

[0047] Figure 2 The encoder is illustrated in more detail and schematically.

[0048] Figure 3 A flowchart illustrating a method for embedding information in a video signal is shown schematically.

[0049] Figure 4 The system is illustrated schematically in the form of a vehicle;

[0050] Figure 5 Electronic devices are illustrated schematically; and

[0051] Figure 6 The training components are illustrated schematically. Detailed Implementation

[0052] Figure 1An encoder 100 configured to perform a method for embedding information in a video signal is schematically illustrated. Specifically, the encoder 100 receives a message 30 including information to be embedded in the video signal, and also receives a video frame 10 (also referred to as a "cover frame," which is used interchangeably in this invention) from the video signal. For example, the video signal includes a first video frame 9 obtained from the video signal ( Figure 2 The second video frame 10 is located after the first video frame 9 in time.

[0053] Encoder 100 divides message 30 into a first message portion 132 and a second message portion 134, for example, by partitioning the message 30 based on bits, symbols, and / or graphic objects. The first message portion 132 may include at least 50% of message 30, and the second message portion 134 may include the remaining portion of message 30. This partitioning of message 30 allows information from the corresponding message portions 132, 134 to be embedded in the second video frame 10 in different ways.

[0054] For example, encoder 100 can copy data representing overlay frame 10 (second video frame 10) to provide a first copy 112 and a second copy 114. A first message portion 132 can then be embedded into pixels of the first copy 112 of overlay frame 10, and a second message portion 134 can be embedded into pixels of the second copy 114. For example, embedding can be achieved by concatenating the first message portion 132 and the second message portion 134 to the first copy 112 and the second copy 114 of the data representing overlay frame 10, respectively.

[0055] This embedding takes into account the motion of regions and / or objects present in the overlay frame 10, because pixels altered due to the embedded message portions 132, 134 can cause artifacts or other variations in the video signal that are less visible to the human visual system in moving regions / objects. To this end, a first video frame 9 and a second video frame 10 are acquired from the video signal, and a motion map 122 associated with the second video frame 10 is determined. The motion map 122 indicates the movement of individual pixels in the second video frame 10 relative to the first video frame 9. This one or more regions can be a single pixel or a group of pixels. As an example only, an object captured by a camera outputting the video signal will be represented by a group of pixels in the second video frame 10. If an object moves relative to the camera from the first video frame 9 to the second video frame 10, the group of pixels in the second video frame 10 will have a specific motion / movement relative to the first video frame 9 (specifically, the pixel positions representing the objects in the first video frame 9 and the second video frame 10). Motion graph 122 is a representation of the entire second video frame 10 (i.e., all pixels in the width and height directions of the second video frame 10), indicating the motion / movement of individual pixels (or pixel regions) relative to the first video frame 9.

[0056] Then, embedding of the first message portion 132 and the second message portion 134 with the pixels of the second video frame 10 is performed based on motion graph 122. For example, the first message portion 132 is embedded in the pixels of the second video frame 10 (its first copy 112), which includes weighting the first message portion 132 for each pixel of the second video frame 10 based on motion graph 122. Similarly, the second message portion 134 is embedded in the pixels of the second video frame 10 (its second copy 114), which includes weighting the second message portion 134 for each pixel of the second video frame 10 based on the inverse of motion graph 122. For example, weighting may include multiplying the data including the first message portion pixel-by-pixel with the value of the motion graph, and multiplying the data including the second message portion pixel-by-pixel with the inverse of the motion graph. This embedding based on motion graph 122 and its inverse can be performed by a submodule 140 of encoder 100 (e.g., a specially trained convolutional neural network). The result is scaled frames 144, 148 including the embedded message 30.

[0057] like Figure 1As shown, by way of example only, the average movement of motion graph 122 is set to a predetermined average value, here 0.25. In other words, the individual values ​​of motion graph 122 are calculated to be values ​​between zero (static) and 1 (dynamic), and are set in such a way that the predetermined average value of the second video frame 10 is between 0.1 and 0.49, preferably between 0.1 and 0.3, and most preferably (as illustrated) 0.25. Therefore, the first message portion 132 is embedded in the pixels of the second video frame 10 with a focus on pixels (regions) with higher motion, while the second message portion 134 is embedded in the pixels of the second video frame 10 with a focus on pixels (regions) with lower motion. Thus, a portion of message 30 (i.e., the first message portion 132, for example, 50% or more of message 30) is embedded (hidden) in 25% of the relatively dynamic second video frame 10, while the remaining portion (50% or less) of message 30 is embedded in 75% of the relatively static second video frame 10.

[0058] The encoder 100 can receive or calculate motion graph 122 from optical flow 20 provided by a conventional optical flow module.

[0059] After embedding the first message portion 132 and the second message portion 134 into copies 112 and 114 of the second video frame data, a steganalytic frame 50 is formed by merging the pixels of scaled frames 144 and 148.

[0060] Figure 2 The encoder 100 is illustrated in more detail and schematically. (Already mentioned...) Figure 1 The elements and components of the encoder 100 are indicated by the same reference numerals, and their explanations will be omitted to avoid redundant disclosure.

[0061] For example, optical flow 20 can be calculated based on video signals by a neural network (e.g., a PWC-network (pyramid, twist, cost volume network)). Specifically, optical flow 20 can be calculated based on the difference between the first coverage frame 9 and a temporally subsequent coverage frame 10. Optical flow 20 is then provided to motion graph calculation 122.

[0062] The bits of message 30 are extended at position 31 to form a 3D tensor. The 3D tensor includes channels for each bit of message 30, and each bit is copied in the height and width (h×w) of each corresponding channel. Thus, each bit is spatially copied. The extended message 31 is then segmented or divided into a first message portion 132 and a second message portion 134.

[0063] In 142 and 146, the first message portion 132 and the second message portion 134 are concatenated with the overlay frame 10 or copies 112 and 114 representing the data of the overlay frame 10. As an example only, in each of the concatenated modules 142 and 146, the 3D tensors of the respective message portions 132 and 134 are added to the feature map derived from module 11. In convolutional modules 11, 143, and 145, 2D convolutional layers are performed, which transform the overlay frame 10 into a feature map. Such modules 11, 143, and 145 may include several filter kernels. For example, as an example only, 64 filter kernels with a 3×3 kernel size can be implemented for module 11, while 3 filter kernels with a 1×1 kernel size can be implemented for modules 143 and 145.

[0064] As an example only, modules 11, 143, and 145 can be implemented in the form of a CNN.

[0065] Then, at modules 144 and 148, the outputs of convolution modules 143 and 145 are multiplied pixel-by-pixel by the motion map value of motion map 122 and its inverse (1-motion map value). In other words, the first message portion 132 and the second message portion 134 are weighted based on motion map 122 and its inverse, respectively. Weighting is not performed on the message bits themselves, but rather on the data representing the second video frame (overlay frame) 10 (e.g., the feature map of frame 10).

[0066] The outputs of the scaling modules 144 and 148 are then summed to form data corresponding to the overlay frame 10 in which message 30 is embedded. Optionally, at scaling module 150, the summed data from scaling modules 144 and 148 can be further scaled by a constant, for example, to increase or decrease the data value after frame-by-frame, pixel-by-pixel multiplication. For example, in cases where there is (almost) no motion or very high motion in a frame 10, such exceptions can be balanced to ensure that the output data still includes the entire message 30.

[0067] Finally, at 151, the outputs of scaling modules 144, 148, and / or scaling module 150 can be added to the original, unchanged overlay frame 10. Therefore, message 30 is embedded (hidden) within the overlay frame 10.

[0068] Figure 3A flowchart illustrating a method for embedding information in a video signal is shown schematically. The method begins at step 305, receiving a message 30 containing information to be hidden in video frame 10. In step 310, message 30 is divided into a first message portion 132 and a second message portion 134. In step 320, a first video frame 9 and a second video frame 10 are obtained from the video signal, the second video frame 10 being temporally subsequent to the first video frame 9.

[0069] In step 330, a motion graph 122 associated with the second video frame 10 is determined. The motion graph 122 indicates the movement of each pixel of the second video frame 10 relative to the first video frame 9.

[0070] In step 350, the encoder convolutional neural network (CNN) 140 can be trained once or continuously. The encoder CNN 140 can be trained to embed the first message portion 132 and the second message portion 134 into the pixels of the second video frame 10, thereby training an adversarial network 420 to evaluate whether the video signal contains hidden data. Figure 6 The message 30 embedded in the second video frame 10 is detected with the same probability as determining that the second video frame 10 has not been altered. Therefore, the adversarial network 420 cannot actually distinguish between altered and unaltered video frames 10.

[0071] Based on the motion graph 122, the first message portion 132 and the second message portion 134 are embedded into the pixels of the second video frame 10 (steps 360 and 365). Specifically, the values ​​in the motion graph 122 are multiplied pixel-by-pixel by the data representing the second video frame 10 in which the first message portion 132 is embedded, and the values ​​in the motion graph 122 are inversely multiplied pixel-by-pixel by the data representing the second video frame 10 in which the second message portion 134 is embedded. For example, a trained CNN 140 can be used to perform the embedding in steps 360 and 365.

[0072] Subsequently, in step 370, a stegframe 50 is formed by merging data 144 representing the second video frame 10 in which the first message portion 132 is embedded with data 148 representing the second video frame 10 in which the second message portion 134 is embedded. Then, in step 380, the stegframe 50 can be output to, for example, a display for displaying the video frame 50 or to a module for decoding the stegframe 50 and extracting the message 30 therefrom.

[0073] Figure 4 System 1 is schematically illustrated, in the form of a vehicle, and is merely one example of System 1. System 1 may include a camera 210 configured to capture video signals, such as video signals including a first video frame 9 and a second video frame 10. Figure 1 and Figure 2 Camera 210 can capture video signals representing the scene near (e.g., in front of) System 1.

[0074] In addition, system 1 may include a graphic encoder 100, for example Figure 1 and Figure 2 The encoder 100. The graphic encoder 100 can form an electronic device 200 ( Figure 5 The electronic device 200 is part of or is installed in or forms part of system 1. For example, electronic device 200 may be a mobile phone, tablet computer, laptop computer, PC, ECU of vehicle 1, etc.

[0075] System 1 and / or electronic device 200 may further include object detector 250, configured to identify and classify at least one object in the video signal captured by camera 210, particularly at least one object in its second video frame 10. Object detector 250 may output a message 30 including information at least the identification and / or classification information of at least one object. This identification and / or classification information (forming message 30) will be stored together with the second video frame 10, enabling frame synchronization availability of message 30. This is achieved by embedding message 30 into the second video frame 10 by graphics encoder 100, as referenced. Figures 1 to 3 Explanation.

[0076] System 1 and / or electronic device 200 may further include a data interface 260 configured to output a second video frame 50 in which a first message portion 132 and a second message portion 134 are embedded. Optionally, compression of the video signal may be performed at the data interface 260 or at a dedicated compression module (not illustrated).

[0077] The resulting stegframe 50, output from data interface 260, can be displayed on display 225 of system 1 and / or electronic device 200. Due to the embedded message 30, stegframe 50 may cause one or more artifacts or other ultimately visible changes to the video signal. However, because the message 30 is embedded based on motion graph 122, a user (e.g., driver 5 of a vehicle) may not perceive such changes to the video signal.

[0078] The stegframe 50 output from data interface 260 can also be received by another component 270 of system 1 and / or electronic device 200, such as driver assistance system 270, which is an example of such another component 270. This component 270 is configured to decode message 30 from stegframe 50. During decoding, component 270 can obtain video information of the second video frame 10 and message 30 in a frame-synchronized manner. For example, if message 30 includes information about an object identified by object detector 250, driver assistance system 270 can provide the driver 5 with the object information alone or together with the second video frame 50, such as a warning, highlighting of an object in the video signal, etc.

[0079] Figure 6 Training components are illustrated schematically, such as those used to train encoder 100. See also the following: Figure 1 and Figure 2 , Figure 1 and Figure 2 Some components of the encoder also exist Figure 6 The components are omitted from the table to avoid redundancy. Encoder 100 may include convolutional neural network 140, which is capable of embedding message 30 into overlay frame 10 to produce stegframe 50.

[0080] To achieve good results with few or no visible effects (such as artifacts or other pixel alterations), stegframe 50 can be fed to an adversarial network 420 trained to evaluate whether stegframe 50 contains hidden data (e.g., message 30). The output of adversarial network 420 is the probability that the received frame 50 is the "original" overlay frame 10 or a modified stegframe 50 (i.e., a video frame in which message 30 is embedded). Adversarial network 420 can be a conventional network providing steganalysis of the video signal.

[0081] CNN 140 is trained in the following manner: the adversarial network 420 detects that message 30 is embedded in the second video frame 50 with the same probability as determining that the second video frame 10 has not been altered. Therefore, it is not actually possible to detect whether message 30 is embedded in video frames 10 / 50.

[0082] Once trained, encoder 100 and CNN 140 can operate without the opposing network 420 and output one or more stegframes 50. The stegframes 50 or the resulting video signal can be compressed in compression module 405 to store and / or transmit the video signal with reduced bandwidth. Decoder 410 can receive and decode the compressed video signal, i.e., decompress the video signal and retrieve message 30 from the included stegframes 50. To enable decoder 410 to operate properly and correctly "decrypt" message 30, decoder 410 may also include a neural network (CNN). This CNN of decoder 410 can be trained together with the encoder network (as described above), or the model of the encoder network can be sent from encoder 100 to decoder 410 once encoder 100 is fully trained.

[0083] The above description of the accompanying drawings should be understood as providing only exemplary embodiments of the invention, and should not limit the invention to these specific embodiments.

Claims

1. A method for embedding information in a video signal, the method comprising: receiving a message (30) comprising the information; dividing the message (30) into a first message portion (132) and a second message portion (134); obtaining a first video frame (9) and a second video frame (10) from the video signal, the first video frame (9) and the second video frame (10) each comprising a predetermined number of pixels, wherein the second video frame (10) is temporally subsequent to the first video frame (9); determining a motion map (122) associated with the second video frame (10), wherein the motion map (122) indicates movement of individual pixels in the second video frame (10) relative to the first video frame (9); embedding the first message portion (132) in the pixels of the second video frame (10), including weighting the first message portion (132) for individual pixels of the second video frame (10) based on the motion map (122); and embedding the second message portion (134) in the pixels of the second video frame (10), including weighting the second message portion (134) for individual pixels of the second video frame (10) based on an inverse of the motion map (122).

2. The method of claim 1, wherein, The first message portion (132) comprises at least 50% of the message (30), and the second message portion (134) comprises the remaining portion of the message (30).

3. The method according to claim 1 or 2, characterized in that, Embedding the first message portion (132) comprises providing a first copy (112) of data representing the second video frame (10), concatenating the first message portion (132) to the first copy (112), and weighting the concatenated data based on the motion map (122); and Embedding the second message portion (134) comprises providing a second copy (114) of data representing the second video frame (10), concatenating the second message portion (134) to the second copy (114), and weighting the concatenated data based on an inverse of the motion map (122).

4. The method of claim 3, wherein, Further comprising: forming a stego frame (50) by merging the pixels of the second video frame (10) in which the first message portion (132) is embedded with the pixels of the second video frame (10) in which the second message portion (134) is embedded.

5. The method of claim 4, wherein, Further comprising: forming the stego frame (50) by further merging the pixels of the second video frame with the pixels of the second video frame (10) obtained from the video signal.

6. The method of claim 3, wherein, partitioning the message (30) comprises generating a first 3D tensor having message bits of the first message portion (132) as individual bits copied over a width and a height of one channel of the tensor, wherein the width and the height correspond to a width and a height of the pixels of the second video frame (10), and generating a second 3D tensor having message bits of the second message portion (134) as individual bits copied over a width and a height of one channel of the tensor, wherein the width and the height correspond to a width and a height of the pixels of the second video frame (10).

7. The method of claim 1, wherein, determining the motion map (122) comprises calculating individual values of the motion map (122) for each of the pixels as values between 0 and 1 and having a predetermined average value of the second video frame (10) between 0.1 and 0.

49.

8. The method of claim 7, wherein, the predetermined average value of the second video frame (10) is between 0.1 and 0.

3.

9. The method of claim 8, wherein, the predetermined average value of the second video frame (10) is 0.

25.

10. The method of claim 1, wherein, determining the motion map (122) comprises calculating an optical flow between the first video frame (9) and the second video frame (10).

11. The method of claim 1, wherein, Further comprising: training a convolutional neural network or encoder network (140) configured to embed the first message portion (132) and the second message portion (134) into the pixels of the second video frame (10) such that a counter network (420) trained to assess whether a video signal contains hidden data detects the message (30) embedded in the second video frame (10) with the same probability as determining that the second video frame (10) is not changed.

12. A system (1) for embedding information in a video signal, comprising: a camera (210) configured to capture a video signal; and a graphical encoder (100) configured to perform the method according to any one of claims 1 to 11.

13. The system (1) according to claim 12, characterized in that, Further comprising: an object detector (250) configured to identify and classify at least one object in the second video frame (10), wherein the object detector (250) outputs the message (30), and wherein the information comprised in the message (30) comprises at least identification and / or classification information of the at least one object, wherein the graphical encoder (100) is configured to embed the message (30) containing the information associated with the second video frame (10) and output by the object detector (250) into the second video frame (10).

14. The system (1) according to claim 12 or 13, characterized in that, Further comprising: a data interface (260) configured to output a stego frame (50) having the first message portion (132) and the second message portion (134) embedded therein.

15. The system (1) according to claim 14, characterized in that, The data interface (260) is further configured to compress the stego frame (50) having the first message portion (132) and the second message portion (134) embedded therein.

16. The system (1) according to claim 14, characterized in that, Further comprising: a driver assistance system (270) configured to receive the stego frame (50) from the data interface (260) and decode the message (30) from the stego frame (50).

17. The system (1) according to claim 14, characterized in that, Further comprising: a display (225) configured to receive the stego frame (50) from the data interface (260) and display the stego frame (50).

18. The system (1) according to claim 12, characterized in that, Further comprising: an encoder convolutional neural network (140) configured to embed the first message portion (132) and the second message portion (134) into pixels of the second video frame (10); and an adversarial network (420) trained to assess whether a video signal contains hidden data, wherein the encoder convolutional neural network (140) is trained such that the adversarial network (420) detects the message (30) embedded in the second video frame (10) with the same probability as determining that the second video frame (10) has not been changed.

Images