A system and method for image coding using a dual-image model.

Abstract

To encode different image regions by using different image models depending on objects or events recognized in an image.SOLUTION: A method includes: establishing whether objects 119 constituting predetermined object types or performing predetermined event types are visible in an image 110 or not; in response to establishing that the objects are visible, encoding a region-of-interest 111 of the image using a non-generative image model 160, to obtain first image data 121; and encoding arbitrary remainder 112 of the image using a generative image model 170, to obtain second image data 122. The use of the non-generative image model enables decoding of the first image data without relying on information derived from images other than the encoded image or, if the image is a frame in a video sequence, enables decoding of the first image data without relying on information derived from images outside the video sequence.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the field of image coding including video coding. The present invention proposes a coding method for coding different image regions using different image models according to objects or events recognized within an image.

Background Art

[0002] An AI-based image codec in which a trained neural network performs important coding and decoding steps has made progress in many fields of applications and promises high-quality video at surprisingly low bitrates. In this technology, broadly speaking, the plaintext representation of the original image is given to an encoder network that outputs a binary file that can be imagined as a list of the most important visual features of the image, a feature vector. To restore the image on the receiver side, the feature vector is supplied to a decoder network trained to generate a restored image that is as practical as the original image. A common approach is to train the encoder and decoder networks tandemly based on training data representing images expected to be encountered in the intended use case. It has been repeatedly proven that an AI-based image codec can achieve excellent similarity with the original image if the two neural networks are systematically trained with appropriate data.

[0003] At times, the evidence for AI-encoded images is criticized for being low, meaning that, unlike conventional image coding, feature vectors cannot be transparently traced back to the pixels of the original image. Even when faced with large-scale proof that a particular AI-based codec successfully reconstructs an image without defects visible to the human viewer, skeptics argue that the high quality of the reconstructed image is deceptive because it may be contaminated with elements from the training data or randomized information sampled from a probability distribution. Concerns of this nature are rarely mentioned for lossy conventional image codecs that are perceived as more reliable. Here, the intentional discarding of information on the encoding side necessary to compress the image results in a corresponding visual degradation of the reconstructed image or introduces artifacts, but these imperfections can be accurately recognized as imperfections and are less likely to mislead the viewer. If skepticism towards AI-based image codecs becomes widely accepted in the community, it is likely to limit the usefulness of these codecs in collecting images for law enforcement purposes, such as surveillance and monitoring video systems.

[0004] It is desirable to obtain image data that provides irrefutable evidence while also taking advantage of the benefits of AI-based image codecs, particularly their high compression ratios. [Overview of the Initiative]

[0005] One object of this disclosure is to make available an image coding method that identifies different image regions, encodes different image regions using different image models, and from there enables a verifiable chain of encoding and decoding by a single image model. In particular, this single image model can enable decoding without relying on information derived from images other than the encoded image (or, if the image is a frame in a video sequence, without relying on information from outside the video sequence). Alternatively, the single image model should enable decoding without inserting information derived from images other than the encoded image (or outside its own video sequence). Alternatively, the single image model should enable decoding without processing the encoded image data by a function that relies on information derived from images other than the encoded image (or outside its own video sequence). Alternatively, the single model can enable non-probabilistic decoding. A further object is to use the single image model to encode image regions in which objects of high forensic relevance can be seen. A further object is to make available an image decoding method that also uses appropriate hardware required for the image coding method.

[0006] At least some of these objectives are achieved by the present invention as defined by the independent claims. The dependent claims relate to advantageous embodiments of the present invention.

[0007] A first aspect of the present invention provides a method for encoding an image. The method includes the steps of determining whether objects constituting one or more predetermined object types or performing one or more predetermined event types are visible in the image, and if such objects are determined to be visible, encoding at least one region of interest (ROI) of the image using a non-generated image model to obtain first image data representing the ROI. The rest of the image (i.e., the ROI complement) is then encoded using a generated image model to obtain second image data representing the rest. The use of a non-generated image model enables decoding the first image data without relying on information derived from images other than the encoded image. Alternatively, if the image is a frame in a video sequence, the use of a non-generated image model enables decoding the first image data without relying on information derived from images outside the video sequence.

[0008] In this disclosure, “image model” includes functional definitions (e.g., functional specifications, pseudocode, equations) and decoder functional definitions configured to convert images to and from a common image data format. Image data may be digital and may relate to pixel values, conversion coefficients, feature vectors, or similar types of information. An image model that includes at least one generating component (e.g., a generating decoder) is called a generating image model, otherwise it is classified as non-generating. An encoder may use a particular image model without being composed of a complete image model; for example, when a decoder is not required in any case, an implementer may choose to have only an encoding device having an encoder compliant with a particular image model.

[0009] Examples of generative image models include one or more of the following: machine learning-based (or AI-based) encoders, artificial neural network encoders, autoencoders, variable autoencoders, generative adversarial network (GAN) encoders, convolutional neural network (CNN) encoders, recurrent neural network (RNN) encoders, autoregressive models, and flow models. Examples of non-generative image models include encoding by transformative coding, combinations of transformative coding (e.g., interframe time predictive coding) and transformative coding (so-called hybrid coding), and one or more of the following: ITU H.26x, in particular H.264, H.265, and H.266, AOMedia Video 1 (AV1), and versions of JPEG such as JPEG2000. At least H.26x and AV1 can be described as image models using hybrid coding.

[0010] The implementation of codecs using non-generating image models also leads to the widespread use of neural networks as function approximators (or predictors or regressors). For example, the use of such function approximators is disclosed in Chinese Patent No. 110493596 and U.S. Patent Application Publication No. 20060251330 to perform conventional entropy coding and in-frame predictive coding. However, in such embodiments, the function approximator performs computational tasks that can be equally delegated to conventional processing circuits despite higher computational costs, and / or constitute isolated steps in non-generating coding or decoding algorithms. Function approximators are generally not configured for input or output in the form of feature vectors or other latent space representations. Furthermore, neural networks used as function approximators are generally not trained with information derived from images, but with function values. Therefore, for the purposes of this disclosure, the fact that non-generating image models are implemented, or can be implemented, by one or more neural networks functioning as function approximators does not change the classification of image models as non-generating.

[0011] Furthermore, it is understood that “objects” that constitute a given object type or perform a given event type can refer to material objects, people, animals, non-material objects (e.g., optical phenomena), or objects that bring about a given event (e.g., a given movement pattern that does not depend on the type of movement of the object).

[0012] The first embodiment of the method uses a non-generating image model to encode the ROI where an object is detected, so the resulting first image data can be decoded without the risk of contamination by irrelevant image data. Irrelevant image data refers to images other than the encoded image, or, if the image is a frame within a video sequence, images outside the video sequence. Unwanted contamination can be an indirect effect of training. For example, if a user believes that the decoder network absorbs graphic information from the training data and allows this information to be moved to the reconstructed image without moderation, the decoder network may be perceived as unsuitable for providing evidence for law enforcement purposes. In contrast, the encoding method of the first embodiment deals with the ROI in the context of a verifiable chain of encoding and decoding. Since the rest of the image is encoded using a generating image model (the rest is the entire image if no object is visible), the image can be encoded with a preferred overall compression ratio, thereby saving storage and transmission capacity.

[0013] In some embodiments, the use of a non-generated image model allows for the decoding of the first image data without inserting information derived from irrelevant image data. In contrast, when a generated image model is used, the decoding process can use a codebook constructed from historical graphic data, and conceptually, the encoded image data acts as instructions that combine elements of the codebook into the reconstructed image.

[0014] In some embodiments, the use of a non-generative image model enables the decoding of the first image data without processing the first image data with a function that relies on information derived from images other than the encoded image. In contrast, when a generative image model is used, the decoding process can utilize a decoding neural network (which can be abstractly understood as constructing a function) trained using historical image data, e.g., raw images or images preprocessed for the purpose of training the neural network. In addition to this type of dependency, which may be considered an offline dependency, other generative image models can utilize a decoding network that undergoes continuous updates or improvements (retraining) based on newer image data, even after the decoding network has entered commercial operation.

[0015] In some embodiments, non-generative image models enable non-stochastic decoding. In contrast, decoders for some available generative image models generate reconstructed images by stochastic sampling from a probability distribution. For example, Glow (商標) An image model with such properties can be defined using the model (https: / / openai.com / blog / glow / ). The probability distribution itself may also be defined based on information other than the image being encoded, and the indirect influence of such information may be considered to introduce an additional level of uncertainty into the decoding process. In these embodiments, a non-generative image model is used by its non-probabilistic decoder, thus reducing or eliminating the risk that the recovered image will be contaminated with randomness and / or training data.

[0016] In some embodiments, the generative image model includes an artificial neural network with trainable weights, and snapshots of the trainable weights are stored when it is determined that objects constituting one or more predetermined object types or performing one or more predetermined event types are visible in the image. By accessing the snapshots of the neural network weights, it is possible to repeat the encoding operation as many times as desired for verification purposes. These embodiments, in which the ROI is encoded using a non-generative image model and the remainder is encoded by a repeatable encoding process, thus enable complete verification of the encoded image. If the image is part of larger material (e.g., a frame in a video sequence), storing snapshots is not required for every new image in which the objects are visible; rather, storage may be omitted if one snapshot has already been stored and it is certain that the neural network weights have not been updated since. These conditions may be met when encoding is performed by a commercial software product in which the vendor distributes upgrades periodically or on an event-triggered basis. Further developments of this method ensure additional data integrity by adding a further step of generating a digital signature for a dataset containing snapshots of trainable weights, first image data, and second image data.

[0017] In some embodiments, the ROI is a suitable subset of the image, with the remainder being non-empty. In other embodiments, the ROI is the entire image, with the remainder being empty. In other words, as soon as an object constituting one or more of a given object type, or performing one or more of a given event type, is found anywhere in the image, it is decided to encode the entire image using a non-generative image model; otherwise, a generative image model is used throughout.

[0018] In different embodiments, different methods exist for determining whether at least one object is visible that constitutes one or more predetermined object types or performs one or more predetermined event types. For this purpose, a visual object recognition or event recognition process can be performed, and in particular, an automated or computer-implemented process may be used. The recognition process is visual in the sense that it is based on an encoded image. Alternatively or additionally, the presence of an object that constitutes one or more predetermined object types or performs one or more predetermined event types can be determined based on data from a detector configured to monitor a scene in an image, where the scene may correspond to the field of view of a camera recording the image, or to the vicinity of the scene. The detector may be configured to capture acoustic, vibration, chemical, mechanical, or electromagnetic excitations, including capturing alternative representations of the scene (e.g., in infrared light) that facilitate the detection of the object in question. An operator input constitutes a further method for determining whether the object in question is visible in the image, where the input may be a Boolean signal indicating that the operator has noticed that an object is visible, or an indicator image region where the operator views the object.

[0019] In the embodiments described above, when the object visibility determination step obtains the location of the recognized object (for example, when an object detection or event detection process is used), it is assumed that the ROI is defined based on this location. It is understood that the object location can be expressed in or converted to image coordinates. The ROI can be defined with a non-zero margin δ, meaning that the ROI extends at least δ units outside the recognized object. The margin δ can be applied isotropically or in only a few directions, e.g., in the direction of instantaneous motion. The limits of the recognized object may be a bounding box or an approximate contour (mask). The size of the margin δ can be determined by considering the maximum motion vector size used by the encoder, or the precision of the location and / or bounding box of the recognized object. If the ROI encoded using a non-generated image model extends only by a non-zero margin δ, the probability that the content of the ROI can be predictively encoded over a larger range increases (see also below). This helps in controlling the encoding bitrate. In particular, it may become possible to rely on motion video encoding techniques. Conversely, in embodiments where the ROI covers only recognized objects, visual objects may be entering and leaving the ROI while the non-generated image model is being used, which may result in a somewhat higher I-block / P-block ratio (or I-frame / P-frame ratio).

[0020] In some embodiments, the object or event recognition process can be restricted to a predetermined image area, so that objects outside this area do not affect the selection between the generated and non-generated image models. In some embodiments, the object or event recognition process is configured to recognize objects with high forensic relevance.

[0021] In some embodiments, the first and / or second image data indicate the location of the ROI. This indication may be relied upon on the decoding side, more precisely, in the step of combining the image regions reconstructed using the generative image model with the image regions reconstructed using the non-generative image model. The location may be implicitly or explicitly indicated.

[0022] In some embodiments, not only the remaining portion (i.e., the complement of the ROI) but also a portion of the ROI is encoded using the generative image model. The data obtained from encoding this portion can be called a third image data. This portion may be a boundary zone of the ROI from which the continuity between the ROI and the remaining portion can be verified. In particular, encoding of the entire image using the generative image model can be carried out in parallel with encoding of the ROI using a non-generative image model. In this way, two encoded copies of the ROI are available, which can be used to verify the reliability of the generative encoding model. Furthermore, especially in encoding video sequences, encoding using the generative image model can function better or more economically if image areas of a constant size and position are consistently supplied. For example, there may be more frequent and immediate opportunities to predictively represent image features of consecutive video frames, such as incremental or block movement.

[0023] In some embodiments, the image is a frame in a video sequence, and the encoding of the ROI, or the rest of the image, or both, applies an inter-frame prediction method. Inter-frame prediction may include predicting and / or interpolating the appearance of frames based on an assumed temporal correlation between consecutive frames, for example, since the frames depict the same scene. This is the case, for example, in ITU H.264 encoding (non-generative model) and can be similarly applied as a pre-processing or post-processing step in encoding with a generative image model. In these embodiments, it may be possible to determine whether an object is visible by applying prediction and / or interpolation regarding the presence and / or location of objects based on an assumed temporal correlation between consecutive frames. For example, the object or event recognition process may be performed every n frames (n≧2 is an integer) to obtain the object location, which is then processed by an interpolation operation to provide the object location in each subsequence interposed by n-1 frames. In this regard, it may be appropriate to apply interpolation if a particular recognized object reappears after n frames. In these embodiments, there is a further option to use visual motion tracking and / or camera orientation parameters and / or camera field of view parameters (of the camera recording or recording the video sequence) to track a specific recognized object through subsequent frames of the same video sequence.

[0024] In some embodiments of the first aspect of the present invention, images are associated with audio, and the audio is also selectively encoded using a non-generating audio model. For example, if a video sequence is considered, for any frame in which the object is determined to be visible, the audio associated with the frame (e.g., an audio segment recorded concurrently with the frame) is encoded using a non-generating audio model, and for other such frames in which the object is not determined to be visible, the associated audio is encoded using an arbitrary audio model. The use of a non-generating audio model enables the decoding of the audio without relying on information derived from audio data other than the associated audio. The arbitrary audio model may be a generating audio model that can provide advantageous data compression.

[0025] In one embodiment, a method is provided for encoding an image that is a frame in a video sequence. The method includes the steps of determining whether objects constituting one or more predetermined object types or performing one or more predetermined event types are visible in the image, and if such objects are determined to be visible, encoding at least one region of interest (ROI) of the image using a non-generated image model to obtain first image data representing the ROI. The rest of the image (i.e., the ROI complement) is then encoded using a generated image model to obtain second image data representing the rest. The use of a non-generated image model allows the decoding of the first image data without relying on information derived from images that are not frames in the video sequence.

[0026] In a second aspect of the present invention, a method for decoding image data representing an image is provided, the image data including first image data obtained by encoding using a non-generative image model and second image data obtained by encoding using a generative image model, the method comprising decoding the first image data using the non-generative image model without relying on information derived from an image other than the encoded image or, if the image is a frame within a video sequence, without relying on information derived from an image outside the video sequence, thereby obtaining a first region of the image; decoding the second image data using the generative image model, thereby obtaining a second region of the image; and combining the first region and the second region of the image.

[0027] Here, since the first image data has been obtained by encoding using a non-generative image model, it is decoded without the risk of contamination by irrelevant image data. Thereby, the evidence value of the restored first image region is confirmed. The decoding method is part of a verifiable chain of encoding and decoding. Since the second image data has been obtained by encoding using a generative image model, the total size of the first and second image data can be advantageously limited.

[0028] In some embodiments of the second aspect, as described above, the combining can be assisted by an indication of the position of the first region of the image (e.g., relative to the second region), which indication can be derived from the first or second image data.

[0029] In a further aspect of the present invention, there are provided an encoding device and a decoding device each configured to perform the methods of the first and second aspects. The encoding device comprises processing circuitry selectively operable according to a non-generative image model and a generative image model, the use of the non-generative image model enabling the decoding of image data representing an encoded image without relying on information derived from an image other than the encoded image, or, if the encoded image is a frame within a video sequence, enabling the decoding of the image data without relying on information derived from an image outside the video sequence. The decoding device also comprises processing circuitry having this dual functionality. The processing circuitry may be a single processor (or a single processor network) alternately executing software code corresponding to the generative image model and the non-generative image model, or the processing circuitry may be divided into a plurality of hardware parts each having one of these respective functions.

[0030] The present invention further relates to a computer program comprising instructions for causing a computer, particularly an encoding device or a decoding device, to perform the above method. The computer program can be stored or distributed on a data carrier. As used herein, a "data carrier" can be a transient data carrier such as a modulated electromagnetic or optical wave, or a non-transient data carrier. The non-transient data carrier includes volatile and non-volatile memories such as magnetic, optical, or solid-state type permanent and non-permanent storage media. Still within the scope of "data carrier", such a memory may be fixedly attached or portable.

[0031] In general, all terms used in the claims should be interpreted according to their ordinary meanings in the art unless specifically defined herein. All references to “a / an / the” element, apparatus, component, means, step, etc. should be interpreted broadly as referring to at least one example of an element, apparatus, component, means, step, etc. unless specifically stated otherwise. The steps of any method disclosed herein do not need to be performed in the exact order described unless expressly stated otherwise. [Brief explanation of the drawing]

[0032] Here, the aspects and embodiments will be described as examples with reference to the attached drawings. [Figure 1] The encoding and decoding devices according to embodiments of this specification are shown. [Figure 2] This is a flowchart of the encoding method according to the embodiments of this specification. [Figure 3] This describes the steps for defining a region of interest (ROI) based on objects recognized and localized within an image. [Figure 4] This demonstrates spatially restricted, event-based trigger conditions that induce coding using a non-generative image model. [Figure 5] This demonstrates the case where image regions encoded using a generative image model and a non-generative image model overlap. [Modes for carrying out the invention]

[0033] Herein, aspects of the present disclosure will be described more fully below with reference to the accompanying drawings illustrating specific embodiments of the present invention. However, these embodiments may be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided as examples to make the present disclosure thorough and complete and to fully convey the scope of all aspects of the present invention to those skilled in the art. Throughout the description, similar numbers refer to similar elements.

[0034] The upper part of Figure 1 shows a video sequence 130 containing multiple images 110 that constitute a video frame. In the current video frame, an object 119 is visible within an ROI 111. According to some embodiments of the present invention, the ROI 111 should be encoded using a non-generating image model, and the rest of the video frame 112 should be encoded using a generating image model.

[0035] Immediately below the video sequence 130 in Figure 1, an encoding device 150 is shown that receives the image 110 in a plaintext representation, such as a set of pixel intensity values ​​represented by plaintext or lossless coding (e.g., Huffman coding). In the encoding device 150, a discriminator 151 separates the image 110 into an ROI 111 and the rest 112, and feeds these to an encoder 160 configured to use a non-generating image model and an encoder 170 configured to use a generating image model, respectively. The encoder 170 of the generating image model can utilize a neural network (encoder network) 171 with weights trained according to images other than the image 110 being encoded. (In particular, encoder 170 may utilize a neural network 171 with weights trained according to images outside the video sequence 130.) The discriminator 151 may include an interface for receiving and analyzing sensor signals (e.g., signals from detectors monitoring the scene or vicinity of the scene in image 110) to supplement the visual data in image 110 for the common purpose of determining whether an object exists and where it is located. Encoder 160 for a non-generating image model outputs a first image data 121, and encoder 170 for a generating image model outputs a second image data 122 (and, in some embodiments, a third image data encoding the overlap between ROI 111 and the rest of the image 112). An optional signing function 152, on which the private key of an encryption key pair is deposited, can generate a digital signature 123 that allows a recipient to verify the authenticity of the first and second image data 121,122. More precisely, according to a method known to itself, the signature function 152 can calculate the hashes of the first and second image data 121,122 and generate a digital signature 123 based on the hashes.

[0036] The lower part of Figure 1 shows the decoding device 180. Since one of the purposes of image coding is to enable the transfer and storage of images, the coding and decoding devices 150,180 do not need to be located in the same place or controlled by the same entity. Similarly, image data may be retrieved from memory and used to reconstruct the original image at a much later time. The decoding device 180 implements a decoder 160' using the non-generating image model described above and a decoder 170' using the generative image model described above. Decoder 170' may preferably include a neural network (decoder network) 171' with weights acquired by training on irrelevant image data, in conjunction with encoder network 171. Furthermore, decoder 170' may also be configured to perform probabilistic sampling from a probability distribution. The two decoders 160',170' are responsible for reconstructing the ROI and the rest of image 110, respectively, and these are then combined and output. The decryption device 180 includes an optional signature verification function 182 to which the public key of the aforementioned cryptographic key pair is deposited, configured to use a digital signature 123 to evaluate whether the first and second image data 121,122 are authentic.

[0037] Figure 1 is a purely functional example of the internal workings of the encoding and decoding devices 150,180 and should be understood that it does not necessarily reflect the structure of the processing hardware in which it is implemented. In fact, a single functional block (box) may correspond to a section of computer executable code running on a processor or processor network, a dedicated component (e.g., ASIC, FPGA), or a combination thereof. Cloud implementations in which the functions of the encoding and decoding devices 150,180 are performed by network processing resources are also envisioned and are fully included in the scope of this disclosure.

[0038] For a general description of training encoder and decoder networks (synthetic networks), refer to the disclosure in U.S. Patent Application Publication No. 20210049468, in which such networks are trained according to a GAN objective function to generate image feature vectors (appearance vectors) and reconstruct the image. Paragraphs 0199, 0201, 0202, and 0206, as well as corresponding Figures 7A and 7B, are of primary relevance.

[0039] Here, with reference to the encoding method 200 in Figure 2, a more detailed description of the functionality of the encoding device 150 according to several embodiments is presented. However, since the encoding method 200 is not itself linked to the illustrated structure of the encoding device 150, it can also be implemented on a general-purpose programmable computer.

[0040] In the first step 210 of Method 200, it is determined whether an object constituting one of a predetermined object type or an object performing one of a predetermined event type is visible in the image. As stated above, the term “object” also extends to objects (of any type) that perform a predetermined event type. Step 210 may include performing a visual object recognition process, performing a visual event recognition process, obtaining data from detectors configured to monitor the scene in the image, or obtaining operator input. Alternatively or additionally, step 210 may include receiving and analyzing sensor signals that complement the visual data in image 110, such as signals from a camera or motion detector pointed at the scene in or near image 110, or another detector signal that can infer an increased probability that an object is likely to appear in image 110. For example, a sensor signal from a detector indicating the opening of a door or gate leading to a space visible in image 110 may suggest that a person or object is about to enter this space and become visible in image 110. Such sensor signals may be used as a basis for determining whether an object matching a predetermined object type or event type is visible in the image 110 (210).

[0041] In embodiments where a visual recognition process is utilized, the operator may pre-configure the process with desired object types and / or event types. In other words, the object types and event types are “predetermined” in relation to the entities performing Method 200, even if the operator is permitted to reconfigure or adjust such settings at runtime. The operator may configure the object types and / or event types so that objects of high forensic relevance are recognized. Thus, object types may include vehicles and people, more specifically, burglary tools, locksmiths, spray paint cans, knives, firearms, explosives, camouflage, uniforms, tattooed body parts, emergency vehicles, goods, packaging and cameras, and event types may include patterns such as cutting or sawing actions, spraying actions, flames, explosions, intrusions, climbing, unusually long dwellings, physical abuse and specific vehicle operations.

[0042] Along with the object type and event type, the operator may have the option to configure a condition (trigger condition) that dictates that the object in question is visible in a given image area. This is shown in Figure 4, where only objects visible within a given area 113 around the building entrance can result in a positive outcome in step 210. In contrast, the presence of an object (person) 119 outside the given area 113 is insufficient to result in a positive outcome in step 210.

[0043] In some embodiments, where image 110 is a frame in video sequence 130, step 210 may apply prediction and / or interpolation to the presence and / or position of an object. Such prediction and interpolation reflect the assumption that consecutive frames are temporally correlated, for example, because consecutive frames are captured by a camera pointed at the same scene. Similarly, a recognized object can be tracked between consecutive frames based on visual motion tracking and / or camera orientation parameters and / or camera field of view parameters. In particular, to maintain the most recent image coordinates of a recognized object that is stationary relative to the scene, it makes sense to detect and approximate invert changes in the camera's pan-tilt-zoom (PTZ) settings.

[0044] If the first step 210 has a negative result, the execution flow proceeds to step 220. If the result is positive, and an object matching a given object type or event type is found, the conditional right-hand branch of the flowchart is as follows.

[0045] The branch on the right begins with an optional step 212 to define ROI 111. ROI 111 (or more ROIs) can be defined based on the position of object 119 recognized in the first step 210. The ROI may optionally be further defined with a margin around it based on the detected extent of the recognized object 119. Such an addition of a margin is shown in Figure 3, and ROI 111 extends in units of δ outside the bounding box of the recognized object 119. The size of the margin δ may be determined by considering the maximum motion vector size used by the encoder, or by considering the position and / or the precision of the bounding box of the recognized object 119. In further refinements of these teachings, the size of the margin δ may also be determined based on the maximum motion vector size used in the most recent N frames, where N is a configurable integer. Alternatively or additionally, the size of the margin δ may be determined based on the size of the image blocks (macroblocks, I-blocks, P-blocks) used in the inter-frame predictive coding process.

[0046] As described above, step 212 is optional. For example, in such embodiments of method 200 in which a spatially restricted state is evaluated in the first step 210, it can be omitted. In this case, ROI 111 is set to be equal to a predetermined area 113 (see Figure 4). In other embodiments, the range of ROI 111 may be defined according to operator input. In further embodiments, the range of ROI 111 may be predetermined and constant with respect to image coordinates.

[0047] In the next step 214, ROI 111 is encoded using a non-generated image model 160. The output of the encoding is a first image data from which ROI 111 can be reconstructed. As described above, the non-generated image model 160 can include, to name a few, transform coding, hybrid coding (a combination of predictive and transform coding), and coding by ITU H.26x, AV1, or JPEG. If ROI 111 is encoded by transform coding, the first image data represents the transform coefficients that describe ROI 111. ROI 111 can be reconstructed by applying the transform coefficients to a basis function (e.g., cosine in the case of discrete cosine transform). In accordance with the aforementioned properties of the non-generated image model, the use of a basis function, as developed in a previous section of this disclosure, achieves the decoding of the first image data to ROI 111 without relying on information derived from images other than the encoded image. In some embodiments, where image 110 is a frame in a video sequence 130, step 214 includes applying an encoding method by inter-frame prediction. In that case, decoding utilizes previously decoded frames within video sequence 130, but does not rely on information derived from images outside of video sequence 130.

[0048] The conditional branch in the flowchart of Figure 2 is executed only when it is confirmed that an object is visible in image 110 that constitutes one of the given object types or performs one of the given event types, and includes two further optional steps.

[0049] The first optional step 216a deals with any audio content associated with image 110. Image 110, being a frame in video sequence 130, often contains associated audio content, but still images can also contain audio content. In embodiments where optional step 216a forms part of a conditional branch in the flowchart, encoder 150 includes an audio encoder portion (not shown) that is free to have a non-generating audio model and further audio models. The use of a non-generating audio model allows decoding of the audio content without relying on information derived from audio data other than the associated audio, while further audio models may be generating audio models, which have good data compression performance but in some cases have lower recognized evidence values. In such embodiments, the audio is encoded using a non-generating audio model (step 216a) or using further audio models (step 216b, mandatory branch in the flowchart). It is worth noting that the audio content, unlike image 110, is processed uniformly within each frame, i.e., without being divided into portions similar to ROI 111 and the rest of image 110 112.

[0050] In the second optional step 218, a snapshot of the neural network weights is stored, which will be used to encode the remaining portion 112 of image 110 using the generated image model. This storage allows access to the snapshot of the neural network weights to repeat the encoding operation for verification purposes as needed, thereby strengthening the evidence for the encoded version of the remaining portion 112 (second image data).

[0051] After all steps within the conditional branch (one, two, three, or four steps in different embodiments) have been performed, the execution flow proceeds to step 220, where it joins the mandatory branch of method 200.

[0052] In step 220, the remaining portion 112 of image 110 is encoded using a generative image model 170. The generative image model 170 may include one or more of the following: machine learning-based encoders, artificial neural network encoders, autoencoders, variable autoencoders, generative adversarial network (GAN) encoders, convolutional neural network (CNN) encoders, recurrent neural network (RNN) encoders, autoregressive models, and flow models. In some embodiments where image 110 is a frame in video sequence 130, step 220 includes an encoding method by inter-frame prediction.

[0053] The presence of the optional step 222 indicates that, in some embodiments, it is not only the remaining portion 112 that is encoded using the generated image model. Hereinafter, as shown in Figure 5, portion 111.1 ("overlap") of ROI 111 is also encoded using the generated image model. This generates a third image data from which the overlap 111.1 can be reconstructed using the generated image model. The overlap 111.1 may, for example, be a boundary zone (fringe, edge) of ROI 111. Complement 111.2 of the overlap 111.1 within ROI 111 is encoded using only the non-generated image model. It should be noted that steps 220 and 222 can be combined, i.e., the remaining portion 112 and the overlap 111.1 can be encoded in a common process. Similarly, it is not essential that the first and third image data are separable, for example, in the sense that ROI 111 can be reconstructed without the overlap 111.1 being reconstructed, or vice versa.

[0054] Referring further to step 222, in a particular embodiment, the encoding of the entire image 110 using the generated image model continues in parallel with the conditional encoding of the ROI 111 using the non-generated image model (step 214). This particular embodiment differs from the configuration shown in Figure 5, in which the complement 111.2 is not empty.

[0055] The audio-related step 216b has already been explained. This is executed only if step 216a in the conditional branch has not been executed.

[0056] Method 200 ends in a final step of generating a digital signature 123 that can verify the authenticity of the first and / or second image data in some embodiments. Optionally, the digital signature 123 is generated for a larger dataset containing the neural network weights similarly stored in step 218.

[0057] The decoding device 180 may be configured to perform a decoding method that takes as input image data including a first image data obtained by encoding using a non-generating image model and a second image data obtained by encoding using a generating image model. The method includes the steps of decoding the first image data using a non-generating image model to obtain a first region 111 of the image, without relying on information derived from images other than the encoded image, or, if the image is a frame in a video sequence, without relying on information derived from images outside the video sequence; decoding the second image data using a generating image model to obtain a second region 112 of the image; and combining the first and second regions of the image. The combining step can be assisted by an indication of the position of the first region of the image (for example, relative to the second region), which can be derived from the first or second image data.

[0058] The aspects of this disclosure have been described above primarily with reference to several embodiments. However, as will be readily apparent to those skilled in the art, other embodiments not disclosed above are equally possible within the scope of the invention, as defined by the appended claims.

Claims

1. A method for encoding an image, Determining whether one or more objects that constitute one or more predetermined object types or that execute one or more predetermined event types are visible in the image, In response to the determination that the aforementioned object is visible, a first image data is obtained by encoding at least one region of interest of the image using a non-generating image model, wherein the one or more objects are visible within the region of interest, and the encoding using the non-generating image model includes transformative coding or coding by a combination of predictive coding and transformative coding. By using only the encoder neural network of the generated image model, any remaining portion of the image is encoded, thereby obtaining a second image data. Includes, The generative image model includes an encoder neural network and a decoder neural network, at least one of the encoder neural network and the decoder neural network is generative, and the encoder neural network and the decoder neural network are trained in relation to each other according to images other than the image to be encoded. A method for enabling the decoding of the first image data without relying on information derived from images other than encoded images, or, if the image is a frame in a video sequence, without relying on information derived from images outside the video sequence.

2. The use of the aforementioned non-generated image model, Without inserting information derived from images other than the encoded image, or, if the image is a frame in a video sequence, without inserting information derived from images outside the video sequence, Without processing the first image data with a function that depends on information derived from an image other than the encoded image, or, if the image is a frame in a video sequence, without processing the first image data with a function that depends on information derived from an image outside the video sequence, The method according to claim 1, which enables decoding of the first image data.

3. The method according to claim 1, wherein the use of the non-generated image model enables non-stochastic decoding of the first image data.

4. The method according to claim 1, wherein the generated image model is adapted to be decoded by an artificial neural network trained using information derived from images other than the encoded image, or, if the image is a frame in a video sequence, using information derived from images outside the video sequence.

5. The method according to claim 1, wherein the generated image model is adapted to be decoded by probabilistic sampling from a probability distribution.

6. The generated image model includes an artificial neural network having trainable weights, and the method is The method according to claim 1, further comprising storing a snapshot of the trainable weights in response to the determination that the object is visible.

7. The method according to claim 6, further comprising generating a digital signature for a dataset comprising the snapshot of the trainable weights, the first image data, and the second image data.

8. Determining whether the aforementioned object is visible is Performing a visual object recognition process, To perform a visual event recognition process, Acquiring data from a detector configured to monitor the scene of the aforementioned image or the vicinity of the aforementioned scene, and Obtaining operator input The method according to claim 1, comprising one or more of the above.

9. The aforementioned determination provides the location of the recognized object. The method according to claim 8, further comprising defining the at least one region of interest based on the location of the recognized object.

10. The method according to claim 9, wherein the defined at least one region of interest extends outside the recognized object by a non-zero margin.

11. The method according to claim 8, wherein the visual object recognition process or the visual event recognition process is configured to evaluate the conditions under which an object that constitutes one or more of the predetermined object types or performs one or more of the predetermined event types is visible within a predetermined image area.

12. The aforementioned image is a frame in a video sequence, Encoding the at least one region of interest and one or more of the remaining portions includes applying an interframe prediction method. The method according to claim 1.

13. In any frame in which the aforementioned object is determined to be visible, the audio associated with the video sequence is encoded using a non-generated audio model. Encode the associated audio using an arbitrary audio model in at least several frames in which the object is not determined to be visible. It further includes, The method according to claim 1, wherein the use of the non-generated audio model enables decoding of the audio without relying on information derived from audio data other than the associated audio.

14. A device comprising a processing circuit capable of selectively operating according to a non-generating image model and a generating image model, The use of the non-generated image model enables decoding of image data representing the encoded image without relying on information derived from images other than the encoded image, or, if the encoded image is a frame in a video sequence, enables decoding of the image data without relying on information derived from images outside the video sequence. A device wherein the device is configured to perform the method according to any one of claims 1 to 13.

15. A non-temporary computer-readable storage medium storing instructions for performing steps of any one of claims 1 to 13 when executed on a device having processing capabilities.

Images