Enhancement video coding for video monitoring application

Abstract

To provide a method of enhancement video coding for video monitoring applications, and a storage medium.SOLUTION: A method of encoding input video including a sequence of video frames as a hybrid video stream comprises: downsampling the input video from an original spatial resolution to a reduced spatial resolution and an intermediate spatial resolution; providing the input video at the reduced spatial resolution to a base encoder to obtain a base encoded stream; providing a first enhancement stream based on first residuals at the intermediate spatial resolution; providing a second enhancement stream based on second residuals based at the original spatial resolution, which is at least partially encoded using temporal prediction; detecting at least one non-motion region in a video frame; and causing the set of first residuals but not the set of second residuals to vanish throughout the non-motion region.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to the field of video coding, and in particular to an enhanced video coding implementation suitable for video surveillance applications. [Background technology]

[0002] Enhancement video coding refers to techniques for adding one or more enhancement layers to a base video encoded with a base codec such that when the enhancement layers are combined with the reconstructed base video, an enhanced video stream is produced. The enhancement layers provide improved features to an existing codec, such as increased compression capabilities, lower encoding / decoding complexity, improved resolution and improved quality of the reconstructed video. The combination of the base video and one or more enhancement layers is sometimes referred to as a hybrid video stream.

[0003] Among such techniques, the Low Complexity Enhanced Video Coding (LCEVC) specification, or MPEG-5, is a recent standard approved by ISO / IEC JTC1 / SC29 / WG04 (MPEG) Video Coding. It works on top of other coding schemes, resulting in a multi-layer video coding technique, and also adds an enhancement layer (or layers) independent of the base video. The LCEVC technique takes as input a decoded video at a lower resolution and adds up to two enhancement sublayers of residuals coded with special low-complexity coding tools, such as simple temporal prediction, frequency transformation, quantization, and entropy coding, based on a comparison with the input video at the original quality. A presentation of the main features of the LCEVC standard can be found in any of the following references: [1] S. Battista et al., "Overview of the Low Complexity Enhancement Video Coding (LCEVC) Standard", IEEE Transactions on Circuits and Systems for Video Technology, vol.32, no.11, pages 7983-7995 (DOI:10.1109 / TCSVT.2022.3182793), 2022 [2] "White paper on Low Complexity Enhancement Video Coding (LCEVC)", ISO / IEC JTC1 / SC29 / AG3 N0058, January 2022 [3] WO2020188273A1 The specifications of the LCEVC standard are published as follows: [4] ISO / IEC 23094-2-Information Technology-General Video Coding-Part 2:Low Complexity Enhancement Video Coding, Standard ISO / IEC 23094-2:2021, November 2021 [5] ISO / IEC 23094-3-Information Technology-General Video Coding-Part 3:Conformance and Reference Software for Low Complexity Enhancement Video Coding, Standard ISO / IEC 23094-3:2021, 2022

[0004] The design of LCEVC foresees up to two sublayers of enhancements to the compressed video representation of the base layer. The first layer (sublayer 1) is optional and can be disabled by corresponding signaling in the LCEVC bitstream, while the second layer (sublayer 2) is mandatory. Unlike the first layer, the second layer includes a temporal prediction stage that attempts to predict each block of residual based on buffered values, or possibly to code the block without temporal prediction. For a given block, the decision of whether to use temporal prediction or not may be different for different video frames. Thus, when LCEVC is operated with two sublayers, a significant portion of the enhancement data will be coded in sublayer 1 without temporal prediction. Experience appears to have confirmed that the coding efficiency of two-layer LCEVC is relatively poor for video data with strong local temporal correlation, which is characteristic of data acquired in video surveillance applications. It would be desirable to improve data compression in cases where video data generally has strong temporal correlation, or where strong temporal correlation can be observed when regions of each frame are considered. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] WO2020188273A1 [Non-patent literature]

[0006] [Non-Patent Document 1] S. Battista et al., "Overview of the Low Complexity Enhancement Video Coding (LCEVC) Standard", IEEE Transactions on Circuits and Systems for Video Technology, vol.32, no.11, pages 7983-7995 (DOI:10.1109 / TCSVT.2022.3182793), 2022 [Non-Patent Document 2] "White paper on Low Complexity Enhancement Video Coding (LCEVC)", ISO / IEC JTC1 / SC29 / AG3 N0058, January 2022 [Non-Patent Document 3] ISO / IEC 23094-2-Information Technology-General Video Coding-Part 2:Low Complexity Enhancement Video Coding, Standard ISO / IEC 23094-2:2021, November 2021 [Non-Patent Document 4] ISO / IEC 23094-3-Information Technology-General Video Coding-Part 3:Conformance and Reference Software for Low Complexity Enhancement Video Coding, Standard ISO / IEC 23094-3:2021, 2022 Summary of the Invention

[0007] One objective of the present disclosure is to propose an enhanced video coding technique with the ability to identify cases where video data has strong temporal correlation, and to exploit the temporal correlation to improve various performance aspects, such as coding efficiency, data compression efficiency, or any of the quality metrics described in [1]. The better the coding efficiency, the lower the bit rate is required to reach a certain video quality level. Another objective is to propose an enhanced video coding technique that can exploit temporal correlations that are limited to regions of each frame of the input video (local temporal correlation). A further objective is to improve the performance of two-layer LCEVC for video data with strong local temporal correlation. A further objective is to adapt LCEVC specifically for video surveillance applications. Yet a further objective is to propose such an adaptation that minimally interferes with existing LCEVC designs.

[0008] At least some of these objects are achieved by the present invention as defined by the independent claims. The dependent claims relate to advantageous embodiments.

[0009] According to a first aspect of the present disclosure, a method is provided for encoding an input video including a sequence of video frames as a hybrid video stream. The method includes downsampling the input video from an original spatial resolution to a reduced spatial resolution and an intermediate spatial resolution, providing the input video at the reduced spatial resolution to a base encoder to obtain a base encoded stream, providing a first enhanced stream by generating a first set of residuals based on differences between the input video and a reconstructed video at the intermediate spatial resolution (e.g., the reconstructed video may have been obtained by decoding the base encoded stream and upsampling the output), quantizing the first set of residuals, and forming the first enhanced stream from the quantized first set of residuals. providing a second enhancement stream by generating a second set of residuals based on a difference between the input video and the reconstructed video at the original spatial resolution (e.g., starting from the reconstructed video at the intermediate spatial resolution, the reconstructed video at the original spatial resolution may have been obtained by adding the reconstruction of the first residuals and upsampling the output), quantizing the second set of residuals, and forming a second enhancement stream from the quantized second set of residuals; and forming a hybrid video stream from the base coding stream, the first enhancement stream, and the second enhancement stream. The second enhancement stream is at least partially coded using temporal prediction (i.e., at least some blocks, some frames, or some time segments are coded using temporal prediction), and further includes temporal signaling indicating whether temporal prediction is used. According to a first aspect, the method further includes detecting at least one non-motion region in the video frame, and causing the first set of residuals to vanish over the entire non-motion region. Preferably, the second set of residuals is not caused to disappear in non-motion regions.

[0010] An advantage associated with the first aspect of the present disclosure is that the first enhancement stream will be substantially devoid of data regarding non-motion regions. More precisely, the inventors have realized that the poor coding efficiency of two-layer LCEVC when applied to video data with strong local temporal correlation is mostly due to the first enhancement layer. The first enhancement stream is coded without temporal prediction and is therefore unlikely to be an optimal coding vehicle for input videos with strong temporal correlation. Instead, substantially all of the enhancement coding of non-motion regions will be done by the second enhancement stream (sub-layer 2 in the LCEVC standard) where temporal prediction is available. A further advantage associated with the first aspect of the present disclosure is that no modification is required on the decoding side. The decoder can properly decode the hybrid video stream without knowing that the hybrid video stream was prepared using the teachings disclosed herein.

[0011] In the terminology of this disclosure, the first set of residuals is said here to be "vanished" throughout the non-motion region if their values ​​are zero or approximately equal to zero. The allowable deviation from strict zero may correspond to coding artifacts related to the base encoder, upsampling / downsampling artifacts, signal noise, and similar contributions that are usually outside the influence of the entity performing the method. Several different measures that can be taken to achieve such vanishing are presented below. However, it should be understood that the implementation of the method will usually have a finite granularity, such as a 2x2 or 4x4 pixel block structure, which means that a block of residuals cannot generally be caused to vanish unless it is completely in the non-motion region. Thus, within the scope of this disclosure, it is not necessary that a block of residuals that only partially overlaps with a non-motion region vanish completely. It is further appreciated that for implementations in which the first residuals are transform coefficients (e.g., blocks of residuals are generated by applying a transform kernel to blocks of pixel-by-pixel differences between the input video and the reconstructed video), coefficient blocks generally cannot be lost by action of the measures disclosed herein unless the underlying pixel blocks are entirely in the non-motion regions. In each of these examples, even incomplete loss of the first set of residuals will achieve the desired effect that substantially all of the enhancement coding of the non-motion regions will be performed by the second enhancement stream.

[0012] In a first group of embodiments, the set of first residuals vanishes throughout the non-motion regions as a result of masking applied to the set of quantized first residuals, which may include replacing those quantized first residuals relating to the non-motion regions with zero or neutral values.

[0013] In a second group of embodiments, the first set of residuals disappears throughout the non-motion regions as a result of replacing the input video at the intermediate spatial resolution with a replacement video upsampled from the input video at the reduced spatial resolution (available from the step of downsampling the input video). This replacement is limited to the non-motion regions, while the input video remains substantially intact elsewhere. The input video that has undergone local replacement with downsampled-upsampled video data, rather than the input video downsampled once, shall be used to generate the first set of residuals, i.e., to calculate the difference to the reconstructed video at the intermediate spatial resolution. Due to the downsampled-upsampling operation, the input video should usually have a significantly better match with the reconstructed video in the non-motion regions so that the first set of residuals disappears, and although the first residuals may include a quality enhancement component to compensate for data compression in the base encoder, the first residuals should usually be free of resolution enhancement. In alternative terms, the downsampling-substitution with upsampled video data reduces the information content in the non-motion regions of the input video (while the spatial resolution is nominally kept equal to the intermediate spatial resolution), so that it can no longer trigger enhancements to the reconstructed video. Instead, enhancements of the reconstructed video in the non-motion regions are effectively deferred to the second layer.

[0014] In a third group of embodiments, the first set of residuals vanishes throughout the non-motion regions as a result of applying masking to the difference between the input video and the reconstructed video at the intermediate spatial resolution, or applying masking to the first set of residuals prior to quantizing the first set of residuals. In particular, masking may be applied to such first residuals that are transform coefficients. Again, masking may apply to replacing those first residuals that relate to the non-motion regions with zero or neutral values.

[0015] In a fourth group of embodiments, the first set of residuals vanishes throughout the non-motion regions as a result of subtracting a predicted difference between the input video and the reconstructed video at the intermediate spatial resolution from the input video prior to generating the first set of residuals. This subtraction is restricted to the non-motion regions, while the input video remains substantially intact elsewhere. It is assumed that the input video that has undergone a local subtraction with the predicted difference, rather than the input video downsampled once, is used to generate the first set of residuals, i.e. to calculate the difference to the reconstructed video at the intermediate spatial resolution.

[0016] In a second aspect of the present disclosure, a device and a computer program are provided for carrying out the method of the first aspect. The computer program can be stored or distributed on a data carrier. A "data carrier" as used herein can be a temporary data carrier, such as a modulated electromagnetic wave or light wave, or a non-transient data carrier. A non-transient data carrier includes volatile and non-volatile memory, such as magnetic, optical or solid-state type permanent and non-permanent storage media. Still within the scope of "data carrier", such memory can be fixedly attached or portable.

[0017] In general, all terms used in the claims should be interpreted according to their customary meaning in the art, unless expressly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, step, etc." should be openly interpreted to refer to at least one instance of that element, apparatus, component, means, step, etc., unless expressly stated otherwise. The steps of any method described herein do not have to be performed in the exact order described, unless expressly stated otherwise.

[0018] Aspects and embodiments will now be described, by way of example only, with reference to the accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1A] FIG. 2 illustrates the architecture of an LCEVC encoder in terms of functional blocks. [Figure 1B] FIG. 2 is a diagram of an example of the internal functional structure of a temporal prediction block of an LCEVC encoder. [Figure 1C] FIG. 2 is a diagram of an example of the internal functional structure of a temporal prediction block of an LCEVC encoder. [Diagram 2] A diagram showing the architecture of an LCEVC decoder. [Diagram 3] FIG. 2 illustrates an LCEVC encoder adapted according to a first group of embodiments of the present specification. [Figure 4] FIG. 2 illustrates an LCEVC encoder adapted according to a second group of embodiments of the present specification. [Figure 5A] FIG. 2 illustrates an LCEVC encoder adapted according to a third group of embodiments of the present specification. [Figure 5B] FIG. 1 illustrates an LCEVC encoder adapted according to a fourth group of embodiments of the present specification. [Figure 6] 1 is a flowchart of a method for encoding an input video as a hybrid video stream according to embodiments herein. [Figure 7] 1 is a diagram of an example video frame captured by a video surveillance camera in an urban environment. [Figure 8] FIG. 7 illustrates a codec device suitable for implementing the method of FIG. 6. [Figure 9] FIG. 1 illustrates multiple codec devices connected by a local area network and / or a wide area network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which several embodiments of the present invention are shown. However, these aspects may be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the scope of all aspects of the present invention to those skilled in the art. Like numbers refer to like elements throughout the description.

[0021] Overall system picture Those skilled in the art can gain a background understanding of the general category of enhanced coding techniques that this disclosure seeks to improve and accommodate from the initially cited references [1], [2] and [3] related to the LCEVC standard.

[0022] 1A shows, in block diagram form, an LCEVC encoder 100 configured to process a video input sequence 170 into a hybrid video stream including a base bitstream 180 and an enhanced bitstream 190. The hybrid video stream may be used to facilitate digital transmission of the input sequence 170 to a receiver or for storage thereof. The base bitstream 180 is a coded representation provided by an optional base encoder 150 of a twice downsampled version of the input sequence 170, which is output from the downscalers 130, 140. The base encoder 150, which is not specified in LCEVC, may be configured for Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Essential Video Coding (EVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), etc.

[0023] The extended bitstream 190 includes, on the one hand, an L-1 (sublayer 1) coefficient layer 191 and, on the other hand, an L-2 (sublayer 2) coefficient layer 192 and an associated temporal layer 193. Furthermore, the extended bitstream 190 may include a header 194 from which a receiver of the hybrid bitstream may obtain, for correct decoding, information about the encoder configuration 171 that was in effect when the hybrid bitstream was prepared. The encoder configuration 171 may affect any of the components of the encoder 100. In the standardized LCEVC encoder 100, sublayer 1 is optional and sublayer 2 is mandatory. The serial upscalers 110, 120 upsample the reconstructed version of the base bitstream 180. The reconstructed version of the base bitstream 180 may be obtained by decoding the output of the base encoder 150 (using the base decoder 250 in FIG. 2). Alternatively, if the base encoder 150 is configured for inter-frame predictive coding, at least a portion of the reconstructed version may be taken from a reference buffer of the base encoder 150. The upscalers 110, 120 are active regardless of whether sublayer 1 is active or not.

[0024] When sublayer 1 is active, subtractor 111, transform block 112, quantization block 113, and entropy coding block 114 operate to provide an L-1 coefficient layer 191. Subtractor 111 calculates the difference between an input video with an intermediate spatial resolution (after downsampling) and a reconstructed video with an intermediate spatial resolution (after upsampling). Furthermore, an inverse quantization 115 block, an inverse transform block 116, an L-1 filter (e.g., a deblocking filter) 117, and an adder 118 are active to prepare the processing in sublayer 2, i.e., by mimicking the actions of the first enhancement layer at the decoding side. The overall action of these blocks 115, 116, 117, 118 is to add the reconstruction of the first residual to the reconstructed video at the intermediate spatial resolution.

[0025] Within sublayer 2, subtractor 121, transform block 123, quantization block 124, and entropy coding block 125 operate to provide an L-2 coefficient layer 192. Subtractor 121 calculates the difference between the input video at the original spatial resolution and a reconstructed video at the original spatial resolution obtained by adding a reconstruction of the first residual to a reconstructed video at the intermediate spatial resolution and upsampling the sum to the original spatial resolution. Quantization block 124 may apply a quantization level equal to or different from that of quantization block 113.

[0026] Further within sublayer 2, a temporal prediction block 122 is provided which outputs data to a transform block 123 and outputs temporal signaling to an entropy coding block 126. The entropy coding block 126 is configured for entropy coding said temporal signaling as a temporal layer 193. Alternatively, the entropy coding blocks 125, 126 may be implemented as a single block (not shown). The single block may perform two parallel entropy coding processes, one on the output of the quantization block 124 and one on the temporal signaling, or a single entropy coding process operating on a multiplexed stream of the output of the quantization block 124 and the temporal signaling. Within sublayer 1, there is no temporal prediction, but each video frame of the first enhancement stream is decodable without reference to other video frames of the first enhancement stream.

[0027] The downsampling action of each of the first downscaler 130 and the second downscaler 140 can be selected independently. In conventional implementations of LCEVC, the action of the first downscaler 130 is the inverse of the action of the second upscaler 120, and the action of the second downscaler 140 is the inverse of the action of the first upscaler 110.

[0028] In an LCEVC implementation, the transform blocks 112, 123 operate on blocks of 2x2 or 4x4 pixels in the respective spatial resolution. An exemplary transform kernel DT suitable for application by the transform blocks 112, 123 is given by Equation 8 and FIG. 8 in [1]. The transform kernel DT will be referred to as the LCEVC transform kernel in this disclosure. The temporal prediction block 122 is configured to determine whether to use temporal prediction on a transform block basis, i.e., on 2x2 or 4x4 pixels in the respective spatial resolution. Furthermore, all blocks (tiles) of the 32x32 residual may be signaled to be coded without temporal prediction, and thus the contribution of the temporal signaling 193 to the total signaling overhead for a fast subsequence of a video frame may be slightly reduced.

[0029] In LCEVC and some of its further developments, temporal prediction may work at the following levels: a) The difference between the input video and a reconstructed video that has been processed (e.g., upsampled) to have the original spatial resolution; b) Coefficients obtained by applying a transformation kernel to the above difference; c) the quantized difference, or d) The quantized coefficients. For example, under option a), it is determined, with a suitable temporal and spatial granularity, for example for each predefined pixel / coefficient block in each video frame, whether the difference should be explicitly coded or whether the difference should be coded by temporal prediction, i.e., whether the difference should be explicitly coded or represented as a copy of the corresponding difference in another video frame (or possibly represented as a linear combination of the corresponding differences in one or more other video frames).

[0030] FIG. 1B shows the inner workings of a temporal prediction 122 block of an LCEVC encoder 100, which is suitable in one example to implement option b). For the moment, attention is directed to a particular coefficient block (e.g. corresponding to a particular coefficient block index, or to a particular group of pixels) in a sequence of video frames. The temporal prediction 122 block comprises a memory 122.2 for storing the value of a particular coefficient block for a most recent video frame in which the particular coefficient block was explicitly coded. It should be noted that the most recent video frame in this sense may be some previous frame in the sequence of frames of the input video, and that if a non-motion region remains for a long time, a significant number of consecutive frames may be predicted temporally. In a further development, the number of consecutive frames allowed to be predicted temporally may be limited by a preconfigured threshold. In yet a further development, the temporal prediction may be performed by reference to not only one frame, but also to multiple frames.

[0031] A comparator 122.1 performs a comparison with the contents of memory 122.2 to determine whether a particular coefficient block shall be explicitly coded in the new video frame. If the particular coefficient block in the new video frame differs from the contents of memory 122.2 by more than a threshold, it is determined that the particular coefficient block in the new video frame is to be explicitly coded. This can be achieved by closing switch 122.3, whereby the particular coefficient block in the new video frame replaces the contents of memory 122.2 and is provided to quantization block 124. If the particular coefficient block in the new video frame differs from the contents of memory 122.2 by less than a threshold, the particular coefficient block in the new video frame is coded by reference to one or more other frames, i.e., by temporal prediction. The signal from comparator 122.1 is used to control switch 122.3 and is also output as a temporal signaling that serves as documentation of the temporal prediction decision. The sequence of temporal signaling can be subjected to entropy coding (block 126) before being included in the enhanced bitstream 190. The difference between a particular coefficient block in the new video frame and the contents of memory 122.2 is l for some p≧1. p They can be compared with respect to a norm.

[0032] 1C shows a temporal prediction block 122 in another example, where option d) is implemented, in which a quantization block 124 is placed immediately downstream of the transformation block 123. As a result, the main output of the temporal prediction block 122 is fed to an entropy coding block 125.

[0033] To implement option a), a modified version of the temporal prediction 122 block shown in Fig. 1B may be used. The modification consists in moving the transform block 123 to a point slightly upstream of the quantization block 124. In an embodiment in which the second residual is equal to a pixel value difference (rather than being a transform coefficient), the transform block 123 may be omitted entirely from the encoder 100.

[0034] Conceptually, the temporal prediction block 122 has one copy of the comparator 122.1, memory 122.2, and switch 122.3 for each particular pixel / coefficient block of a video frame to enable independent decision-making regarding temporal prediction for each of these pixel / coefficient blocks. It is recalled that the components shown in Figures 1B and 1C do not necessarily reflect how a hardware structure of an encoder implementation may be used to perform the necessary operations, but rather how software instructions may be used.

[0035] FIG. 2 shows an LCEVC decoder 200 configured to process a hybrid video stream including a base bitstream 180 and an enhancement bitstream 190 into a video output sequence 270. The LCEVC decoder 200 includes a base decoding stage and corrections in the enhancement sublayer 1 and enhancement sublayer 2. A base decoder 250 is responsible for decoding the base bitstream 180. An entropy decoding block 214, an inverse quantization block 213, an inverse transform block 212, an L-1 filter 217, and an adder 211 operate to perform sublayer 1 processing. An entropy decoding block 225, 226, an inverse quantization block 224, an inverse transform block 223, a temporal prediction (reconstruction) block 222, and an adder 221 belong to sublayer 2. These blocks are implemented with corresponding or inverse tools as described with reference to FIG. 1A and / or in reference [1]. In particular, an exemplary inverse transform kernel IT suitable for application by the inverse transform blocks 212, 223 is given by Equation 10 and Figure 9 in [1]. From the receiver's perspective, at least a part of the applicable decoder configuration 271 may be derived from the header 194 in the enhanced bitstream 190. It is worth noting that no downscaler is required in the decoder 200.

[0036] Thanks to the improvements enabled by the enhanced bitstream 190, the output sequence 270 can be expected to match the input video sequence 170 (FIG. 1A) more accurately and / or be of higher perceptual quality than if the base reconstruction from the base decoder 250 were used as is.

[0037] In color input video, each pixel has multiple channels that refer to a color space, including spaces based on primary colors (e.g., RGB) or lightness and chroma (e.g., YCbCr). The extension coding schemes including LCEVC described in this subsection may be applied to grayscale input video as well as color input video. For color input video, each channel may be extension coded separately, or the three channels may be extension coded together in a joint fashion. Whether the three color channels should be coded separately or jointly may be equivalent to the design choice used in the base encoder 150, or it may be the opposite. Similarly, the improvements to existing extension coding schemes including LCEVC that will be described in the following subsections are applicable regardless of whether the baseline extension coding scheme processes the color channels separately or jointly.

[0038] In Figures 1 and 2, the functional blocks do not necessarily reflect the hardware or software structure of one or more devices implementing the LCEVC encoder 100 and decoder 200, respectively. In an implementation, one functional block may be implemented by multiple identifiable hardware components, and multiple functional blocks may be implemented in a common hardware component. Similarly, when software instructions (one or more computer programs) are used to implement the LCEVC encoder 100 or decoder 200, the software code may be structured into functions, routines, libraries, and the like in a manner deemed suitable by the implementer.

[0039] FIG. 8 illustrates a codec device 800 suitable for implementing an LCEVC encoder 100 or decoder 200 or for performing an encoding method 600 to be described with reference to FIG. 6. As illustrated, the codec device 800 includes a processing circuit 810, a memory 820, and an external interface 830. The processing circuit 810 may include one or more processing cores, which are collocated or spatially distributed across several sites. The processing circuit 810 may include one or more networked (or "cloud") processing resources. The memory 820 may be suitable for storing a computer program 821 with software instructions. The external interface 830 may be a communication interface that allows the codec device 800 to communicate with a similar device (not shown) held by a receiver of the hybrid video stream, or the external interface 830 may enable read and write operations in an external memory 890 suitable for storing the hybrid video stream. Codec device 800 may be included in or associated with a digital video camera (not shown), which may be adapted for video surveillance applications, including video surveillance.

[0040] Fig. 9 shows a deployment that allows a hybrid video stream to be exchanged between a group of codec devices 800. It should be noted that the device implementing the encoding method 600 can be connected to a receiving device over a local area network (connecting lines in the lower half of Fig. 9) or over a wide area network 990. In a video surveillance application, a remote operator terminal can be the recipient of the hybrid video stream prepared by the codec device 800. The hybrid video stream can be protected from unauthorized tampering and / or eavesdropping by encryption and signature techniques known per se. This can be particularly relevant when the video stream is to be transferred over a network.

[0041] First Group of Embodiments 6 illustrates, in flow chart form, a method 600 for encoding an input video as a hybrid video stream according to embodiments in the first, second, third, and fourth groups. This subsection primarily relates to the first group of embodiments and may be read in conjunction with FIG.

[0042] It should be noted that the encoding method 600 is not limited to the LCEVC context outlined in the previous subsection and may be implemented without full compliance with the LCEVC specification. For example, the first residual based on the difference (at pixel value level) between the input video and the reconstructed video at the intermediate spatial resolution may be equal to this difference in some embodiments. This means that the sub-layer 1 transform block 112 in FIG. 3 is absent or inactive. In other embodiments, the first residual is a transform coefficient obtained by applying a transform kernel to said difference. Full compliance with the LCEVC specification requires, in particular, that the first residual is a transform coefficient and that the applied transform kernel is equivalent to the one described in [1].

[0043] Similarly, without departing from the scope of this disclosure, the encoding method 600 may be generalized to provide a number of enhancement layers other than simply two. For example, a hybrid video stream output by the encoding method 600 may include a third, fourth, etc. enhancement stream. Each of the additional enhancement streams may be generated by components or operations similar to those used for the first or second enhancement stream, and decoding may proceed along the lines described above.

[0044] In a first step 610 of the method 600, at least one non-motion region is detected in a video frame (block 301 in FIG. 3). To illustrate such detection, FIG. 7 shows an exemplary video frame 700 from a street scene with pedestrians and cyclists captured by video surveillance equipment in a built environment. In the upper part of the video frame 700, a detected irregularly shaped non-motion region 701 is indicated by a dashed border and by hashing. Everyday experience would confirm that the non-motion region 701 is correctly detected, since it is out of reach of pedestrians and cyclists and thus indicates an area with the appearance of being visited very rarely by moving objects.

[0045] The non-motion regions 701 may be detected based on configuration data input by an operator, or the non-motion regions 701 may be detected automatically. An auto-detection algorithm deployed for this purpose may have a spatial granularity of at least 16x16 pixels, and values ​​of such pixel blocks are compared across successive video frames to determine whether motion is absent (pixel values ​​are approximately constant) or present (pixel values ​​fluctuate). The auto-detection algorithm may include calculation of pixel value differences. Alternatively, the auto-detection algorithm may use a finer granularity, down to individual pixels. In an embodiment in which the first residual is generated by applying a transformation kernel of a certain size, it is preferred to perform the auto-detection with a granularity equal to the kernel size or a coarser granularity. Furthermore, the auto-detection algorithm may have a temporal granularity corresponding to the duration of one video frame, or the duration of ten video frames, or the duration of several tens of video frames. Using a coarser granularity usually means that the detection algorithm consumes less processing resources; in video surveillance applications, non-motion periods may have durations of minutes or even hours, and therefore it may be sufficient to refresh the detection of non-motion regions with a corresponding granularity, i.e., on the order of hundreds or thousands of video frames.

[0046] Another automatic detection algorithm may rely on a machine learning model trained to recognize regions suitable for exclusion from sub-layer 1 coding based on local statistics for the input video, such as image statistics, motion statistics, image content signatures, etc. The image regions predicted by the trained machine learning model may be utilized as non-motion regions 701 in the present coding method 600. In some implementations, step 610 may be performed by an algorithm with a different purpose than the detection of non-motion regions 701, such as a noise filter or an image stabilization filter built into a video camera. Information indicative of the presence of a non-motion region may be derived from internal variables in any of these filters, from a suitable output signal of the filters, or by comparing input frames to the filters with the corresponding output frames. Furthermore, step 610 may be performed by an algorithm related to inter-frame predictive coding, i.e., an algorithm that determines at a block level whether it is economical to predictively code a block, and if the algorithm assesses that it will be economical to predictively code the block, the block may be treated as a non-motion region.

[0047] The detection of non-motion regions 701 may be applied to video frames of the input video at the original spatial resolution. Alternatively, the detection of non-motion regions 701 may be applied to video frames of the input video at an intermediate spatial resolution. In that case, the execution of step 610 may not begin before the subsequent step 620.

[0048] The sensitivity of the auto-detection algorithm (e.g., the tolerance within which pixel values ​​are considered to be approximately unchanged between video frames) can be set by optimizing the total bit rate of the encoding method 600 for a representative test video while varying the detection sensitivity. A moderate frequency of so-called false positives is not a problem in itself, since if a region is wrongly classified as a non-moving region in step 610, it will be excluded from the sub-layer 1 correction since the first residual will disappear, but will eventually be corrected in sub-layer 2 (possibly at a higher coding cost). Specifically, if a detected non-motion region of a video frame contains pixel value fluctuations (e.g., representing moving objects or illumination fluctuations), the temporal prediction block 122 will decide not to use temporal prediction for that region, which will instead be explicitly coded and the necessary enhancements will be realized by sub-layer 2.

[0049] Execution flow of method 600 proceeds to step 620, where the input video is downsampled from the original spatial resolution to a reduced spatial resolution and an intermediate spatial resolution. The input video at the reduced spatial resolution may be provided by downsampling the input video at the original spatial resolution twice, for example using a series of downscalers 130, 140 as in FIG. 1A, or directly from the input video at the original spatial resolution.

[0050] Within the scope of the present disclosure, each of the downscalers 130, 140 may be adapted for 2:2 downsampling of the input video (i.e., the width resolution is halved and the height resolution is halved), 2:1 downsampling of the input video (i.e., the resolution in the width direction of the video frame is halved and the resolution in the height direction of the video frame is maintained), 1:2 downsampling of the input video (i.e., the width resolution is maintained and the height resolution is halved), or 1:1 downsampling (i.e., the width resolution is maintained and the height resolution is maintained). The downsampling action of each of the first downscaler 130 and the second downscaler 140 may be selected independently. In conventional implementations of LCEVC, the action of the first downscaler 130 is the inverse of the action of the second upscaler 120, and the action of the second downscaler 140 is the inverse of the action of the first upscaler 110. When the second downscaler 140 is configured as a pass-through block (for trivial downscaling 1:1), the spatial resolution of the base encoder 150 (corresponding to the "reduced spatial resolution" in the claims) and the spatial resolution of the sub-layer 1 (corresponding to the "intermediate spatial resolution" in the claims) will be equal. In this configuration, the sub-layer 1 may help to improve the quality and / or fidelity of the output video with respect to the input video, but the sub-layer 1 does not change the spatial resolution. The upsampling performed by the upscalers 110, 120 is described in Section III of [1].

[0051] In a next step 630, the input video at the reduced spatial resolution is provided to the base encoder 150 to obtain the base encoded stream 180. It is emphasized that the base encoder 150 operates independently of the enhancement layers. Rather, the encoder 100 can be successfully implemented without the need to check or modify settings and internal variables of the base encoder 150.

[0052] In the next step 640, a first enhanced stream is provided. This includes a step 641 of generating a first set of residuals based on the difference between the input video and the reconstructed video at the intermediate spatial resolution. Said difference may be calculated by a subtractor 111 operating at the level of a single pixel. In this example, the reconstructed video is obtained by decoding the base encoding stream and upsampling (or trivially upsampling) the output. The first residuals may be said difference, to which no further processing is applied, or the first residuals may be transform coefficients obtained by applying a transform kernel to said difference. The size of the transform kernel may be adapted for 2x2 or 4x4 pixel blocks, and the output may be an equal-sized coefficient block ("first set of residuals").

[0053] Step 640 further includes step 642 of quantizing the first set of residuals (block 113 in FIG. 3 ) and step 644 of forming a first enhancement stream from the quantized set of first residuals. Optionally, step 644 may include subjecting the quantized set of first residuals to entropy coding. Suitable quantization and entropy coding algorithms are described in Sections VI and VII of [1].

[0054] According to a first group of embodiments, step 640 further comprises a step 643 (block 302 in FIG. 3) of applying a masking to the set of quantized first residuals. The masking may comprise replacing those quantized first residuals that relate to non-motion regions with zero values ​​(or equivalently with neutral values ​​representing the absence of image content). Conceptually, the "mask" corresponds to the detected non-motion region. This will cause the set of first residuals to disappear over the entire non-motion region, but leave the first residuals outside the non-motion region intact. The replacement may be performed with a granularity of one set of first residuals, for example, one transform block if a transform is used, or with the same granularity as that used in step 610 for the detection of non-motion regions.

[0055] In the next step 650, a second enhancement stream is provided. This includes a step 651 of generating a second set of residuals based on the difference between the input video and the reconstructed video in the original spatial resolution. Here, the reconstructed video in the original spatial resolution is obtained from the reconstructed video in the intermediate spatial resolution, i.e. by adding the reconstruction of the first residual and upsampling (or trivially upsampling) the output. The second residual may be this difference to which no further processing is applied, or the second residual may be a transform coefficient obtained by applying a transform kernel to said difference. As in the case of the first residual, the second residual is subjected to quantization (step 652, block 124) before being included in the second enhancement stream (step 654, which may optionally include entropy coding, block 125). The quantization level to be used in step 652 may be configured in view of the expected noise level of the input video, for example, the quantization level (quantization step) may be set large enough so that a significant portion of noise artifacts in a nominally 0-valued signal are rounded to 0. The quantization level used in step 652 may be configured independently of the quantization level used in step 642; in this regard, the two sub-layers of the enhancement encoder 100 are independent.

[0056] Common to all embodiments disclosed herein, the second enhanced stream is at least partially encoded using temporal prediction. The temporal predictive encoding is partial in the sense that at least some blocks, some video frames, or some subsequences of video frames in the input video 170 are encoded in this manner. The second enhanced stream includes a temporal signaling 193, which indicates with a suitable temporal and spatial granularity (e.g., for each predefined pixel / coefficient block in each video frame) whether the second residual is encoded by temporal prediction, i.e., whether the second residual is represented by referring to one or more other video frames, or whether the second residual is explicitly encoded. The temporal signaling 193 may be entropy coded before it is included in the enhanced bitstream 190 as a temporal layer. The decision to explicitly code or to use temporal prediction (block 653 in FIG. 6) may be based on the similarity with the one or more other video frames, e.g., whether the error introduced by the use of temporal prediction has an acceptable magnitude. Temporal prediction can operate at the following levels: a) The difference between the input video and the reconstructed video at the original spatial resolution; b) Coefficients obtained by applying a transformation kernel to the above difference; c) the quantized difference, or d) The quantized coefficients. Option a) has been chosen for some LCEVC implementations; see [2] and section IV in [1]. Option b) is described in [3]. Options c) and d) are covered by the flowchart in Figure 6.

[0057] After step 650 is completed for a sequence of input video, in the next step 660, a corresponding segment of the hybrid video stream may be formed. Execution of the encoding method 600 may either end here or resume with step 610 of detecting non-motion regions. The hybrid video stream may be decoded by a general decoder that is unmodified in light of the teachings herein, including the standardized LCEVC decoder 200 in FIG. 2.

[0058] The above description is summarized by Fig. 3, which shows an LCEVC encoder 100 adapted according to a first group of embodiments, in which the first set of residuals vanishes throughout the non-motion regions as a result of the masking applied to the quantized first set of residuals. Here, block 301 represents an algorithm operable to detect non-motion regions and corresponds to step 610. Block 302 represents the masking operation in step 643. Alternatively, the detection algorithm 301 may take its input from a point between the first downscaler 130 and the second downscaler 140, so that the detection is performed at the same spatial resolution as the masking.

[0059] It should be noted that this first group of embodiments has been described in a relatively complete and detailed manner, including possible variations and alternatives, whereas subsequent groups of embodiments will be described more concisely to avoid pointless repetition. It should be appreciated that the technical features of the first group of embodiments, except those related to the masking 642 of the quantized first set of residuals, can be understood in this context and utilized in embodiments other than the first group.

[0060] Second Group of Embodiments FIG. 4 shows an LCEVC encoder 100 adapted according to a second group of embodiments, in which a first set of residuals vanishes throughout non-motion regions as a result of replacing the input video at the intermediate spatial resolution with a replacement video upsampled from the input video at the reduced spatial resolution.

[0061] In a first step 610 of the method 600 shown in Figure 6, at least one non-motion region is detected in a video frame (block 401 in Figure 4). The detection may be performed in one of several ways described above in the context of the first group of embodiments.

[0062] Then, in step 620, the input video is downsampled from the original spatial resolution to the reduced spatial resolution and to the intermediate spatial resolution. In a second group of embodiments, prior to generating 641 the first set of residuals, the input video at the intermediate spatial resolution is replaced with a replacement video upsampled from the input video at the reduced spatial resolution throughout the non-motion regions of the video frame. This can be considered to correspond to providing a dual resolution video frame (substep 620.1) in which the non-motion regions have a reduced spatial resolution (but are represented in the intermediate spatial resolution to enable processing, e.g., by the subtractor 111) and the remainder of the video frame has an intermediate spatial resolution. Thus, in a simple implementation in which the upsampling operation does not involve smooth interpolation, the dual resolution video frame formally has an intermediate spatial resolution throughout, but the pixel values ​​in the non-motion regions vary with a granularity corresponding to the reduced spatial resolution, e.g., in blocks of 2x2 pixels.

[0063] 4 shows an implementation in which each frame of the input video is divided, prior to downsampling, into at least one non-motion region on the one hand and into a complement of the non-motion region on the other hand (block 402). The division can be guided by the output of a non-motion region detection algorithm, symbolized by block 401. The complement undergoes downsampling to an intermediate spatial resolution (downsampler 130). The at least one non-motion region undergoes downsampling to a reduced spatial resolution, followed by upsampling to the intermediate resolution (block 404). The upsampling operation is preferably the same or equivalent to the upsampling operation applied to the reconstructed video (block 110). The at least one non-motion region and their complement are then combined into a video frame at the intermediate resolution (block 403). As a result of the operation in block 404, it can be expected that the information density in the at least one non-motion region is lower than in the remainder of the video frame.

[0064] In an alternative implementation, the downsampling-upsampling block 404 is replaced by a block (not shown) that takes the output of the second downscaler 140, upsamples it, and replaces the upsampled data into at least one non-motion region of the video frame.

[0065] In a next step 630 , the input video at the reduced spatial resolution is provided to the base encoder 150 to obtain the base encoded stream 180 .

[0066] In the next step 640, a first enhancement stream is provided, i.e. by generating a first set of residuals (step 641), quantizing the first set of residuals (step 642), and forming a first enhancement stream from the quantized first set of residuals (step 644). In step 641, since dual resolution video frames are used, a first set of residuals for non-motion regions will be generated based on the difference between the input video at the reduced spatial resolution (albeit nominally upsampled to the intermediate spatial resolution) on the one hand and the reconstructed video at the intermediate spatial resolution on the other hand. Outside the non-motion regions, the first set of residuals is generated based on the difference between the input video and the reconstructed video at the intermediate spatial resolution. In this way, the first set of residuals is zero or nearly zero (i.e. vanishes) throughout the non-motion regions and the correction is deferred to the next sub-layer of the enhancement encoder 100.

[0067] In the next step 650, a second enhancement stream is provided, i.e. by generating a second set of residuals based on differences between the input video and the reconstructed video at the original spatial resolution (step 651), quantizing said second residuals (step 652), and including said second residuals in a second enhancement stream (step 654). The second enhancement stream is at least partially encoded using temporal prediction, as determined in step 653.

[0068] After step 650 is completed for a sequence of input video, a corresponding segment of the hybrid video stream may be formed in a next step 660. Execution of the encoding method 600 may either end or resume from step 610 of detecting non-motion regions.

[0069] Third Group of Embodiments 5A illustrates an LCEVC encoder 100 adapted according to a third group of embodiments, in which a first set of residuals vanishes throughout non-motion regions as a result of applying masking to the differences between the input video and the reconstructed video at the intermediate spatial resolution. In some embodiments, the first set of residuals constitutes the differences, while in other embodiments, the first set of residuals are transform coefficients calculated from the differences. The third group further includes embodiments in which masking is applied to the first set of residuals that are transform coefficients, and similar to the embodiment illustrated in FIG. 5A, the masking is applied downstream of block 112, prior to quantizing the first set of residuals.

[0070] In a first step 610 of the method 600 shown in Figure 6, at least one non-motion region is detected in a video frame (block 501 in Figure 5A). The detection may be performed in one of several ways described above in the context of the first group of embodiments.

[0071] Then, in step 620, the input video is downsampled from the original spatial resolution to the reduced spatial resolution and the intermediate spatial resolution.

[0072] In a next step 630 , the input video at the reduced spatial resolution is provided to the base encoder 150 to obtain the base encoded stream 180 .

[0073] In the next step 640, a first enhancement stream is provided, i.e. by generating a first set of residuals (step 641), quantizing the first set of residuals (step 642) and forming the first enhancement stream from the quantized set of first residuals (step 644).

[0074] According to some embodiments in the third group, step 641 includes a sub-step 641.1 (block 502) of applying a masking to the differences between the input video and the reconstructed video at the intermediate spatial resolution. The masking may include replacing those values ​​of said differences that relate to non-motion regions with zero values ​​(or equivalently with neutral values ​​representing the absence of image content). Conceptually, the "mask" corresponds to the non-motion region. According to other embodiments in the third group, in which the first residuals are transform coefficients, sub-step 641.1 includes applying such a masking to the set of first residuals prior to quantization (step 642, block 113). In this case, which may correspond to placing block 502 between blocks 112 and 113, the masking may be applied to all transform blocks derived entirely from pixels in the non-motion region. The masking may optionally be applied to all transform blocks derived entirely or partially from pixels in the non-motion region. In either case, the compensation of the non-motion region(s) will be deferred to the second sub-layer of the enhancement encoder 100.

[0075] In the next step 650, a second enhancement stream is provided, i.e. by generating a second set of residuals based on differences between the input video and the reconstructed video at the original spatial resolution (step 651), quantizing said second residuals (step 652), and including said second residuals in a second enhancement stream (step 654). The second enhancement stream is at least partially encoded using temporal prediction, as determined in step 653.

[0076] After step 650 is completed for a sequence of input video, a corresponding segment of the hybrid video stream may be formed in a next step 660. Execution of the encoding method 600 may either end or resume from step 610 of detecting non-motion regions.

[0077] Fourth Group of Embodiments 5 shows an LCEVC encoder adapted according to a fourth group of embodiments, in which the first set of residuals vanishes throughout the non-motion regions as a result of subtracting from the input video the predicted difference between the input video and the reconstructed video at the intermediate spatial resolution prior to generating the first set of residuals. Outside the non-motion regions, the first set of residuals is generated in the usual way, i.e., based on the difference between the input video and the reconstructed video at the intermediate spatial resolution.

[0078] In a first step 610 of the method 600 shown in Figure 6, at least one non-motion region is detected in a video frame (block 501 in Figure 5B). The detection may be performed in one of several ways described above in the context of the first group of embodiments.

[0079] Then, in step 620, the input video is downsampled from the original spatial resolution to the reduced spatial resolution and the intermediate spatial resolution.

[0080] In a next step 630 , the input video at the reduced spatial resolution is provided to the base encoder 150 to obtain the base encoded stream 180 .

[0081] In the next step 640, a first enhancement stream is provided, i.e. by generating a first set of residuals (step 641), quantizing the first set of residuals (step 642) and forming the first enhancement stream from the quantized set of first residuals (step 644).

[0082] According to an embodiment in the fourth group, in step 635, the difference between the input video and the reconstructed video at the intermediate spatial resolution is predicted and subtracted from the non-motion area(s) of each video frame of the input video. The predicted difference can be considered as a prediction of the output of the subtractor 111. The subtraction is performed before the first residual is generated (step 641). Figure 5B shows one possible implementation to achieve this subtraction. Here, the subtractor 505 subtracts the output of the first upscaler 110 (i.e., the reconstructed video at the intermediate spatial resolution) from the output of the first downscaler 130 (i.e., the input video downsampled to the intermediate spatial resolution). Block 503 separates the non-motion region(s) from the remainder of the video frame, block 506 subtracts from this the output of subtractor 505, and then block 504 combines the non-motion region(s) with the remainder and provides its output to subtractor 111. As a result, the difference, which is the output of subtractor 111, is zero or nearly zero (i.e., vanishes) in the non-motion regions. ((input video at intermediate resolution)-(predicted difference))-(reconstructed video at intermediate resolution) = ((input video at medium resolution) - ((input video at medium resolution) - (reconstructed video at medium resolution)) - (reconstructed video at medium resolution) =0. In this way, the compensation of the non-motion regions will be deferred to the second sub-layer of the enhancement encoder 100. The remainder of the video frame will be processed as usual, i.e., in both sub-layer 1 and sub-layer 2.

[0083] In the next step 650, a second enhancement stream is provided, i.e. by generating a second set of residuals based on differences between the input video and the reconstructed video at the original spatial resolution (step 651), quantizing said second residuals (step 652), and including said second residuals in a second enhancement stream (step 654). The second enhancement stream is at least partially encoded using temporal prediction, as determined in step 653.

[0084] After step 650 is completed for a sequence of input video, a corresponding segment of the hybrid video stream may be formed in a next step 660. Execution of the encoding method 600 may either end or resume from step 610 of detecting non-motion regions.

[0085] Conclusion Aspects of the present disclosure have been primarily described above with reference to a few embodiments, however, as will be readily appreciated by those skilled in the art, embodiments other than those disclosed above are equally possible within the scope of the present invention as defined by the appended claims.

Claims

1. 1. A method for encoding an input video comprising a sequence of video frames as a hybrid video stream, the method comprising: downsampling the input video from an original spatial resolution to a reduced spatial resolution and an intermediate spatial resolution; providing the input video at the reduced spatial resolution to a base encoder to obtain a base encoded stream; The first enhancement stream is generating a first set of residuals based on differences between the input video and a reconstructed video at the intermediate spatial resolution; quantizing the first set of residuals; forming the first enhancement stream from the set of quantized first residuals; By providing The second enhancement stream is generating a second set of residuals based on differences between the input video and a reconstructed video at the original spatial resolution; quantizing the second set of residuals; and forming the second enhancement stream from the set of quantized second residuals; and By providing the second enhancement stream is at least partially encoded using temporal prediction and further comprises temporal signaling indicating whether temporal prediction is used. providing the second enhancement stream; forming the hybrid video stream from the base encoded stream, the first enhanced stream, and the second enhanced stream; Including, The method further comprising: Detecting at least one non-motion region in a video frame; causing the first set of residuals but not the second set of residuals to vanish across the non-motion region; and The method of claim 1, further comprising:

2. The method of claim 1 , wherein the set of first residuals is caused to vanish across the non-motion regions by applying masking to the set of quantized first residuals.

3. The first set of residuals is replacing the input video at the intermediate spatial resolution with a replacement video upsampled from the input video at the reduced spatial resolution prior to generating the set of first residuals in the non-motion regions of the video frames. The method of claim 1 , wherein the motion is caused to vanish over the entire non-motion region by

4. downsampling the input video, and providing a dual resolution video frame having the reduced spatial resolution in the non-motion areas and the intermediate spatial resolution elsewhere. The method of claim 3 , comprising:

5. 2. The method of claim 1 , wherein the first set of residuals is caused to vanish across the non-motion regions by applying masking to the difference between the input video and the reconstructed video at the intermediate spatial resolution, or by applying masking to the first set of residuals prior to the quantizing.

6. The first set of residuals is subtracting from the input video a predicted difference between the input video and the reconstructed video at the intermediate spatial resolution prior to generating the first set of residuals in the non-motion regions of the video frames. The method of claim 1 , wherein the motion is caused to vanish over the entire non-motion region by

7. The method of claim 1 , wherein each video frame of the first enhanced stream is decodable without reference to other video frames of the first enhanced stream.

8. 2. The method of claim 1 , wherein providing the second enhancement stream further comprises: for each set of second residuals or quantized second residuals in a video frame, determining whether temporal prediction should be used with reference to one or more other video frames; and indicating by the temporal signaling whether temporal prediction is used in the video frame.

9. The method of claim 1 , wherein the at least one non-motion region is detected in video frames of the input video at the original spatial resolution or in video frames of the input video at the intermediate spatial resolution.

10. The method of claim 1 , wherein the intermediate spatial resolution is finer than the reduced spatial resolution or the intermediate spatial resolution and the reduced spatial resolution are equal.

11. 2. The method of claim 1 , wherein the first residual and / or the second residual is generated by applying a transformation kernel of size 2x2 pixels or 4x4 pixels to the difference between the input video and the reconstructed video.

12. The method of claim 11 , wherein the transform kernel is a Low Complexity Enhanced Video Coding (LCEVC) transform kernel.

13. The method of claim 1 , wherein the first set of residuals and the second set of residuals are quantized using different quantization levels.

14. A device comprising processing circuitry configured to perform the method of claim 1.

15. 13. A non-transitory computer readable storage medium having stored thereon a computer program comprising instructions, the instructions, when executed by a processing circuit, causing the processing circuit to perform the method of claim 1.

Images