Enhanced video coding for video surveillance applications
By making the residual of the first enhancement stream zero in the non-motion region of the video frame and using time prediction to encode the second enhancement stream, the problem of low coding efficiency of LCEVC in strongly temporally correlated video data is solved, thus improving the coding efficiency and compression performance of video surveillance applications.
Patent Information
- Application Number
- CN202410624511.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-25
- Filing Date
- 2024-05-20
- Publication Date
- 2024-11-26
AI Technical Summary
Existing Low Complexity Enhanced Video Coding (LCEVC) technology has poor coding efficiency when processing video data with strong temporal correlation, especially in video surveillance applications, where it is difficult to effectively compress video data with strong local temporal correlation.
By detecting non-motion regions in video frames and making the residual of the first enhancement stream zero in these regions, a second enhancement stream is encoded using time prediction. The enhancement encoding of non-motion regions is deferred to the second layer, independent of existing LCEVC designs, and maintaining decoder compatibility.
It improves the encoding efficiency and compression performance of video data in video surveillance applications, reduces the bit rate required to achieve a specific video quality level, and maintains compatibility with existing decoders.
Smart Images

Figure CN119031132A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of video coding, and in particular, to the implementation of enhanced video coding suitable for video surveillance applications. Background Art
[0002] Enhanced video coding refers to a technique that adds one or more enhancement layers to a base video encoded with a base codec so that when the enhancement layers are combined with the reconstructed base video, an enhanced video stream is produced. The enhancement layers provide improved features such as compression capability extension, lower encoding / decoding complexity, increased resolution, and increased quality of the reconstructed video to the existing codec. The combination of the base video and one or more enhancement layers may be referred to as a hybrid video stream.
[0003] Among these technologies, the Low Complexity Enhancement Video Coding (LCEVC) specification or MPEG-5 is the latest standard approved by ISO / IEC JTC1 / SC29 / WG04 (MPEG) Video Coding. It works on top of other coding schemes, resulting in a multi-layer video coding technology and adds enhancement layers independently of the base video. The LCEVC technology takes a lower resolution decoded video as input and, based on a comparison with the input video of the original quality, adds a residual of up to two enhancement sub-layers, which are encoded with specialized low-complexity coding tools such as simple temporal prediction, frequency transform, quantization, and entropy coding. An introduction to the main features of the LCEVC standard can be found in any of the following references:
[0004] [1] S. Battista et al., “Overview of the Low Complexity Enhancement Video Coding (LCEVC) Standard,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, No. 11, pp. 7983-7995 (DOI: 10.1109 / TCSVT.2022.3182793), 2022;
[0005] [2] “Low Complexity Enhancement Video Coding (LCEVC) White Paper”, ISO / IEC JTC1 / SC29 / AG3 N0058, January 2022;
[0006] [3] WO2020188273A1.
[0007] The LCEVC standard specifications are published as:
[0008] [4] ISO / IEC 23094-2—Information technology—Generic video coding—Part 2: Low complexity enhancement video coding, standard ISO / IEC 23094-2:2021, November 2021;
[0009] [5] ISO / IEC 23094-3—Information technology—Generic video coding—Part 3: Conformance and reference software for low complexity enhancement video coding, standard ISO / IEC 23094-3:2021, 2022.
[0010] The design of LCEVC foresees up to two sub-layers of enhancements to the base layer compressed video representation. The first layer (sub-layer 1) is optional and can be disabled by corresponding signaling in the LCEVC bitstream, while the second layer (sub-layer 2) is mandatory. Unlike the first layer, the second layer includes a temporal prediction stage that attempts to predict each residual block based on buffered values or to encode the block without temporal prediction. For a given block, the decision whether to use temporal prediction may be different for different video frames. Therefore, when operating LCEVC with two sub-layers, a large part of the enhancement data will be encoded in sub-layer 1 without temporal prediction. Experience seems to confirm that the coding efficiency of two-layer LCEVC is relatively poor for video data that has strong temporal correlations locally, which is a characteristic of data obtained in video surveillance applications. In cases where such video data generally have strong temporal correlations or where strong temporal correlations can be observed when considering areas of each frame, it would be desirable to improve data compression. Summary of the invention
[0011] One object of the present disclosure is to propose an enhanced video coding technique having the ability to identify segments of video data having strong temporal correlations, and to exploit the temporal correlations to improve various performance aspects such as coding efficiency, data compression efficiency, or any of the quality metrics discussed in [1]. The better the coding efficiency, the lower the bit rate required to achieve a particular video quality level. Another object is to propose an enhanced video coding technique capable of exploiting temporal correlations (local temporal correlations) confined to regions of each frame of the input video. A further object is to improve the performance of two-layer LCEVC for video data having strong local temporal correlations. A further object is to make LCEVC particularly suitable for video surveillance applications. An even further object is to propose such an adaptation with minimal interference to existing LCEVC designs.
[0012] At least some of these objects are achieved by the present disclosure as defined by the independent claims.The dependent claims relate to advantageous embodiments.
[0013] According to a first aspect of the present disclosure, a method for encoding an input video including a sequence of video frames into a hybrid video stream is provided. The method includes: downsampling the input video from an original spatial resolution to a reduced spatial resolution and an intermediate spatial resolution; providing the input video at the reduced spatial resolution to a base encoder to obtain a base coded stream; providing a first enhanced stream by generating a set of first residuals based on the difference between the input video and a reconstructed video at an intermediate spatial resolution (for example, the reconstructed video may have been obtained by decoding the base coded stream and upsampling the output), quantizing the set of first residuals, and forming a first enhanced stream from the set of quantized first residuals; providing a second enhanced stream by generating a set of second residuals based on the difference between the input video and the reconstructed video at the original spatial resolution (for example, starting from the reconstructed video at the intermediate spatial resolution, the reconstructed video at the original spatial resolution may have been obtained by adding a reconstruction of the first residual and upsampling the output), quantizing the set of second residuals, and forming a second enhanced stream from the set of quantized second residuals; and forming a hybrid video stream from the base coded stream, the first enhanced stream, and the second enhanced stream. The second enhancement stream is at least partially encoded using temporal prediction (i.e., at least some blocks, some frames, or some time periods are encoded using temporal prediction), and further comprises temporal signaling indicating whether temporal prediction is used. According to the first aspect, the method further comprises: detecting at least one non-motion region in the video frame; and causing the set of first residuals to become zero in the entire non-motion region. Preferably, the set of second residuals is not caused to become zero in the non-motion region.
[0014] An advantage associated with the first aspect of the present disclosure is that the first enhancement stream will contain substantially no data associated with non-motion areas. More precisely, the inventors have recognized that the low coding efficiency of the two-layer LCEVC when applied to video data with strong local temporal correlations is primarily due to the first enhancement layer. The first enhancement stream is encoded without temporal prediction and is therefore unlikely to be the optimal encoding carrier for input video with strong temporal correlations. Instead, enhanced encoding of substantially all non-motion areas will be implemented via a second enhancement stream (sublayer 2 in the LCEVC standard) in which temporal prediction is available. A further advantage of the first aspect of the present disclosure is that no modification is required on the decoding side. The decoder is able to correctly decode the mixed video stream without knowing that the mixed video stream was prepared using the teachings disclosed herein.
[0015] In the terminology of the present disclosure, if the value of the set of first residuals is zero or approximately equal to zero in the entire non-motion area, the set of first residuals is said to be "zeroed" here. Acceptable deviations from exact zero may correspond to coding artifacts associated with the basic encoder, upsampling / downsampling artifacts, signal noise, and similar contributions that are generally outside the influence of the entity performing the method. A variety of different measures that can be taken to achieve this zeroing will be described below. However, it should be understood that the implementation of the method will generally have a limited granularity such as a 2×2 or 4×4 pixel block structure, which means that the residual block cannot generally be zeroed unless the residual block is completely located in the non-motion area. Therefore, within the scope of the present disclosure, it is not necessary for the residual block that only partially overlaps the non-motion area to be completely zeroed. With respect to the implementation where the first residual is a transform coefficient (e.g., a residual block is generated by applying a transform kernel to a pixel-by-pixel difference block between the input video and the reconstructed video), it should be further understood that unless the underlying pixel block is completely located in the non-motion area, the coefficient block cannot generally be zeroed due to the action of the measures disclosed herein. In each of these examples, even an incomplete zeroing of the set of first residuals will achieve the target effect, ie enhancement coding of substantially all non-motion regions will be performed by means of the second enhancement stream.
[0016] In a first set of embodiments, the set of first residuals is zeroed throughout the non-motion region as a result of a mask applied to the set of quantized first residuals.The mask may comprise replacing those quantized first residuals associated with the non-motion region with zero or a neutral value.
[0017] In a second set of embodiments, the set of first residuals becomes zero throughout the non-motion regions as a result of replacing the input video at the intermediate spatial resolution with a replacement video that has been upsampled from the input video at the reduced spatial resolution (obtained from the step of downsampling the input video). This replacement is limited to the non-motion regions, and the input video is substantially intact elsewhere. The input video that has been locally replaced with the downsampled-upsampled video data, rather than the downsampled input video once, is used to generate the set of first residuals, i.e., to calculate the difference relative to the reconstructed video at the intermediate spatial resolution. Due to the downsample-upsample operation, the input video should generally have significantly better consistency with the reconstructed video in the non-motion regions, so that the set of first residuals becomes zero; the first residuals may contain quality enhancement components to compensate for data compression in the base encoder, but they should generally have no resolution enhancement. In other words, the replacement of the downsampled-upsampled video data reduces the information content in the non-motion regions of the input video (while nominally keeping the spatial resolution equal to the intermediate spatial resolution), so that it can no longer cause any enhancement to the reconstructed video. Instead, the enhancement of the reconstructed video in the non-motion regions is substantially deferred to the second layer.
[0018] In a third set of embodiments, as a result of applying a mask to the difference between the input video and the reconstructed video at the intermediate spatial resolution or applying a mask to the set of first residuals before quantizing the set of first residuals, the set of first residuals is zeroed throughout the non-motion region. Specifically, the mask can be applied to such first residuals that are transform coefficients. In addition, the mask can be applied to replace those first residuals associated with the non-motion region with zero or neutral values.
[0019] In a fourth set of embodiments, the set of first residuals is zeroed throughout non-motion regions as a result of subtracting predicted differences between the input video and the reconstructed video at the intermediate spatial resolution from the input video before generating the set of first residuals. This subtraction is limited to non-motion regions, while the input video is substantially intact elsewhere. The input video that has undergone local subtraction with predicted differences, rather than the once downsampled input video, is used to generate the set of first residuals, i.e., to calculate differences relative to the reconstructed video at the intermediate spatial resolution.
[0020] In a second aspect of the present disclosure, an apparatus and a computer program for implementing the method of the first aspect are provided. The computer program may be stored or distributed on a data carrier. As used herein, a "data carrier" may be a temporary data carrier such as a modulated electromagnetic wave or light wave or a non-temporary data carrier. Non-temporary data carriers include volatile and non-volatile memories such as permanent and non-permanent storage media of magnetic, optical or solid-state type. Still within the scope of a "data carrier", such memory may be fixedly mounted or portable.
[0021] In general, unless explicitly defined herein, all terms used in the claims should be interpreted according to their ordinary meaning in the art. Unless explicitly stated otherwise, all references to "an / an / the element, device, component, means, step, etc." should be openly interpreted as referring to at least one instance of the element, device, component, means, step, etc. Unless explicitly stated otherwise, the steps of any method disclosed herein do not have to be performed in the exact order described. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Aspects and embodiments will now be described by way of example with reference to the accompanying drawings, in which:
[0023] Figure 1A The architecture of the LCEVC encoder is shown in terms of functional blocks;
[0024] Figure 1B and Figure 1C are two examples of the internal functional structure of the temporal prediction block of the LCEVC encoder;
[0025] Figure 2Shows the architecture of the LCEVC decoder;
[0026] Figure 3 The diagram shows an LCEVC encoder adapted according to the first set of embodiments herein;
[0027] Figure 4 The diagram shows an LCEVC encoder adapted according to the second set of embodiments herein;
[0028] Figure 5A The diagram shows an LCEVC encoder adjusted according to the third group of embodiments herein;
[0029] Figure 5B The diagram shows an LCEVC encoder adjusted according to the fourth group of embodiments herein;
[0030] Figure 6 is a flowchart of a method for encoding an input video into a hybrid video stream according to an embodiment herein;
[0031] Figure 7 is a sample video frame acquired by a video surveillance camera in an urban environment;
[0032] Figure 8 Shown suitable for execution Figure 6 A codec device according to the method; and
[0033] Fig. 9 Multiple codec devices are shown connected via a local area network and / or a wide area network. DETAILED DESCRIPTION
[0034] Hereinafter, aspects of the present disclosure will be more fully described with reference to the accompanying drawings on which certain embodiments of the present invention are shown. However, these aspects can be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that the present disclosure will be thorough and complete and fully convey the scope of all aspects of the present invention to those skilled in the art. Throughout the specification, the same numbers refer to the same elements.
[0035] System Overview
[0036] Those skilled in the art can gain a background understanding of the general class of enhancement coding techniques that the present disclosure seeks to improve and adapt from the references [1], [2], and [3] originally cited in connection with the LCEVC standard.
[0037] Figure 1AAn LCEVC encoder 100 configured to process a video input sequence 170 into a hybrid video stream including a base bitstream 180 and an enhancement bitstream 190 is shown in block diagram form. The hybrid video stream can be used to facilitate digital transmission of the input sequence 170 to a recipient or for storage thereof. The base bitstream 180 is an encoded representation of a twice downsampled version of the input sequence 170 provided by any base encoder 150, the output from the reducers 130, 140. The base encoder 150 not specified in LCEVC can be configured for Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Base Video Coding (EVC), Versatile Video Coding (VVC), or AOMedia Video 1 (AV1), among others.
[0038] The enhancement bitstream 190 contains, on the one hand, the L-1 (sub-layer 1) coefficient layer 191 and, on the other hand, the L-2 (sub-layer 2) coefficient layer 192 and the associated temporal layer 193. In addition, the enhancement bitstream 190 may contain a header 194 from which the recipient of the mixed bitstream can obtain information about the encoder configuration 171 that was in effect when the mixed bitstream was prepared, for correct decoding. The encoder configuration 171 may affect any of the components of the encoder 100. In the standardized LCEVC encoder 100, sub-layer 1 is optional and sub-layer 2 is mandatory. The serial upscalers 110, 120 upsample the reconstructed version of the base bitstream 180. The reconstructed version of the base bitstream 180 may be upsampled (using Figure 2 The reconstructed version may be obtained by decoding the output of the base encoder 150 with the base decoder 250 in FIG. Alternatively, if the base encoder 150 is configured for inter-frame prediction coding, at least part of the reconstructed version may be taken from the reference buffer of the base encoder 150. The upscaling devices 110 and 120 are effective regardless of whether sub-layer 1 is effective.
[0039] When sub-layer 1 is active, the subtractor 111, the transform block 112, the quantization block 113 and the entropy coding block 114 operate to provide the L-1 coefficient layer 191. The subtractor 111 calculates the difference between the input video with the intermediate spatial resolution (after downsampling) and the reconstructed video with the intermediate spatial resolution (after upsampling). In addition, the inverse quantization block 115, the inverse transform block 116, the L-1 filter (e.g., deblocking filter) 117 and the adder 118 are active to prepare for the processing in sub-layer 2, namely by emulating the actions of the first enhancement layer on the decoding side. The overall action of these blocks 115, 116, 117, 118 is to add the reconstruction of the first residual to the reconstructed video at the intermediate spatial resolution.
[0040] Within sublayer 2, a subtractor 121, a transform block 123, a quantization block 124, and an entropy coding block 125 operate to provide an L-2 coefficient layer 192. The subtractor 121 calculates the difference between the input video at the original spatial resolution and the reconstructed video at the original spatial resolution obtained by adding the reconstruction of the first residual to the reconstructed video at the intermediate spatial resolution and upsampling the sum to the original spatial resolution. The quantization block 124 may apply the same level of quantization as the quantization block 113 or a different level of quantization.
[0041] Still within sub-layer 2, a temporal prediction block 122 is provided which outputs data to a transform block 123 and temporal signaling to an entropy coding block 126. The entropy coding block 126 is configured to entropy encode the temporal signaling into a temporal layer 193. Alternatively, the entropy coding blocks 125, 126 may be implemented as a single block (not shown). The single block may perform two parallel entropy coding processes, one on the output of the quantization block 124 and one on the temporal signaling, or may perform a single entropy coding process operating on a multiplexed stream of the output of the quantization block 124 and the temporal signaling. In sub-layer 1, there is no temporal prediction, but each video frame of the first enhancement stream is decodable without reference to any other video frame of the first enhancement stream.
[0042] The respective downsampling actions of the first downsizer 130 and the second downsizer 140 can be independently selected. In conventional implementations of LCEVC, the actions of the first downsizer 130 are opposite to those of the second upsizer 120, and the actions of the second downsizer 140 are opposite to those of the first upsizer 110.
[0043] In the LCEVC implementation, the transform blocks 112, 123 operate on blocks of 2x2 pixels or 4x4 pixels at their respective spatial resolutions. An example transform kernel DT suitable for application by the transform blocks 112, 123 is given by Equation 8 in [1] and Figure 8 is given. In the present disclosure, the transform kernel DT will be referred to as the LCEVC transform kernel. The temporal prediction block 122 is configured to decide whether to use temporal prediction on the basis of the transform block (i.e., 2×2 pixels or 4×4 pixels at the corresponding spatial resolution). In addition, the entire block (tile) of 32×32 residuals can be signaled to be encoded without temporal prediction, so that the contribution of the temporal signaling 193 to the total signaling overhead of the fast-moving subsequence of the video frame can be slightly reduced.
[0044] In LCEVC and some of its further developments, temporal prediction can work at the following levels:
[0045] a) the difference between the input video and the reconstructed video that has been processed (e.g., upsampled) to the original spatial resolution;
[0046] b) coefficients obtained by applying the transformation kernel to the difference;
[0047] c) quantify the difference, or
[0048] d) Quantized coefficients.
[0049] For example, under option a), a decision is made at an appropriate temporal and spatial granularity (e.g., for each predefined pixel block / coefficient block in each video frame) whether the differences are to be explicitly coded or coded by temporal prediction. That is, a decision is made whether the differences are to be explicitly coded or expressed as a copy of the corresponding differences in another video frame (or possibly expressed as a linear combination of the corresponding differences in one or more other video frames).
[0050] In one example, Figure 1B The diagram illustrates the internal operating mechanism of the temporal prediction block 122 of an LCEVC encoder 100 suitable for implementing option b). Now focus on a specific coefficient block in the sequence of video frames (e.g. corresponding to a specific coefficient block index or a specific pixel group). The temporal prediction block 122 includes a memory 122.2 for storing the value of the specific coefficient block of the latest video frame in which the specific coefficient block was explicitly encoded. Note that the latest video frame in this sense may be several frames earlier in the frame sequence of the input video; if non-motion areas remain for a long time, a considerable number of consecutive frames can be temporally predicted. In a further development, the number of consecutive frames allowed to be temporally predicted can be limited by a preconfigured threshold. In a further development, temporal prediction can occur not only by referring to one frame, but also by referring to multiple frames.
[0051] In order to decide whether a particular coefficient block should be explicitly encoded in the new video frame, a comparator 122.1 performs a comparison with the contents of memory 122.2. If the particular coefficient block in the new video frame differs from the contents of memory 122.2 by more than a threshold value, a decision is made to explicitly encode the particular coefficient block in the new video frame. This can be achieved by closing switch 122.3, whereby the particular coefficient block in the new video frame replaces the contents of memory 122.2 and is fed to quantization block 124. If the particular coefficient block in the new video frame differs from the contents of memory 122.2 by less than a threshold value, the particular coefficient block in the new video frame is encoded by reference to one or more other frames, i.e., by temporal prediction. The signal from comparator 122.1 is used to control switch 122.3 and is also output as a temporal signaling of a file that serves as a temporal prediction decision. Before being included in the enhancement bitstream 190, the sequence of temporal signaling can be subjected to entropy encoding (block 126). According to l p The norm, for some p ≥ 1, can compare the difference between a particular block of coefficients in the new video frame and the contents of memory 122.2.
[0052] Figure 1C Refer to the temporal prediction block 122 in another example of implementation option d). Here, the quantization block 124 is arranged immediately downstream of the transform block 123. Therefore, the main output of the temporal prediction block 122 is fed to the entropy coding block 125.
[0053] To implement option a), you can use Figure 1B 1. The modification consists in moving the transform block 123 to a point just upstream of the quantization block 124. In an embodiment where the second residual is equal to the pixel value difference (rather than the transform coefficients), the transform block 123 can be omitted from the encoder 100 entirely.
[0054] Conceptually, the temporal prediction block 122 has one copy of the comparator 122.1, memory 122.2 and switch 122.3 for each specific pixel block / coefficient block of the video frame to allow independent decisions to be made regarding the temporal prediction of each of these pixel blocks / coefficient blocks. Recall that Figure 1B and Figure 1C The components shown in the figure do not necessarily reflect how the hardware structure of the encoder is implemented; rather, software instructions can be used to implement the necessary operations.
[0055] Figure 2 An LCEVC decoder 200 is illustrated that is configured to process a mixed video stream including a base bitstream 180 and an enhancement bitstream 190 into a video output sequence 270. The LCEVC decoder 200 includes a base decoding stage and corrections in enhancement sub-layers 1 and 2. The base decoder 250 is responsible for decoding the base bitstream 180. An entropy decoding block 214, an inverse quantization block 213, an inverse transform block 212, an L-1 filter 217, and an adder 211 operate to perform sub-layer 1 processing. Entropy decoding blocks 225, 226, an inverse quantization block 224, an inverse transform block 223, a temporal prediction (reconstruction) block 222, and an adder 221 belong to sub-layer 2. These blocks are referred to as Figure 1A and / or the corresponding or inverse tools described in reference [1]. Specifically, an example inverse transform kernel IT suitable for application by the inverse transform blocks 212, 223 is given by equation 10 in [1] and Fig. 9 From the receiver's perspective, at least part of the applicable decoder configuration 271 can be derived from the header 194 in the enhancement bitstream 190. It is worth noting that no reducer is required in the decoder 200.
[0056] Due to the improvements made possible by the enhancement bitstream 190, it can be expected that the output sequence 270 is similar to the input video sequence 170 ( Figure 1A) more accurately matches and / or has a higher perceived quality than if the unchanged base reconstruction from base decoder 250 had been used.
[0057] In color input video, each pixel has multiple channels related to a color space including a space based on primary colors (e.g., RGB) or luminance and chrominance (e.g., YCbCr). The enhancement coding scheme including LCEVC described in this subsection can be applied to grayscale input video as well as color input video. In the case of color input video, each channel can be enhancement encoded separately, or the three channels can be enhancement encoded together in a joint manner. Whether to encode the three color channels separately or jointly can be the same design choice as used in the basic encoder 150, or it can be the opposite. Similarly, regardless of whether the baseline enhancement coding scheme processes the color channels separately or jointly, the improvements to the existing enhancement coding scheme including LCEVC described in the following subsections are applicable.
[0058] exist Figures 1A-1C and Figure 2 In the present invention, the functional blocks do not necessarily reflect the hardware structure or software structure of one or more devices that implement the LCEVC encoder 100 and decoder 200, respectively. In implementation, a functional block can be implemented by multiple identifiable hardware components, and multiple functional blocks can be implemented in common hardware components. Similarly, if software instructions (one or more computer programs) are used to implement the LCEVC encoder 100 or decoder 200, the software code can be structured into functions, routines, libraries, etc. in any manner deemed appropriate by the implementer.
[0059] Figure 8 The diagram shows an embodiment of an LCEVC encoder 100 or a decoder 200 suitable for implementing the LCEVC encoder 100 or a decoder 200 suitable for performing ... Figure 6 A codec device 800 of the encoding method 600 described herein. As shown, the codec device 800 includes a processing circuit 810, a memory 820, and an external interface 830. The processing circuit 810 may include one or more processing cores located at the same location or spatially distributed in several locations. The processing circuit 810 may include one or more networked (or "cloud") processing resources. The memory 820 may be suitable for storing a computer program 821 with software instructions. The external interface 830 may be a communication interface that allows the codec device 800 to communicate with an analog device (not shown) held by a recipient of the mixed video stream, or it may allow read operations and write operations in an external memory 890 suitable for storing the mixed video stream. The codec device 800 may be included in or associated with a digital camera (not shown). The digital camera may be suitable for video surveillance applications including video surveillance.
[0060] Fig. 9 The diagram shows an arrangement that allows the exchange of mixed video streams between a set of codec devices 800. Note that the devices performing the encoding method 600 may be connected via a local area network ( Fig. 9 The mixed video stream can be protected from unauthorized tampering and / or eavesdropping by encryption and signing techniques known per se. This may be particularly relevant when the video stream is to be transmitted over a network.
[0061] The first set of embodiments
[0062] Figure 6 A method 600 of encoding an input video into a mixed video stream according to the first, second, third and fourth groups of embodiments is illustrated in flowchart form. This subsection will mainly relate to the first group of embodiments and may be combined with Figure 3 Come read.
[0063] Note that the encoding method 600 is not limited to the LCEVC environment outlined in the previous subsection, but can be implemented without full compliance with the LCEVC specification. For example, in some embodiments, the first residual based on the difference (at the pixel value level) between the input video and the reconstructed video at the intermediate spatial resolution may be equal to the difference. This means that Figure 3 The sublayer 1 transform block 112 in does not exist or is inactive. In other embodiments, the first residual is a transform coefficient that has been obtained by applying a transform kernel to the difference. Full compliance with the LCEVC specification requires that the first residual is a transform coefficient and the transform kernel applied is the same as the transform kernel described in [1], etc.
[0064] Likewise, without departing from the scope of the present disclosure, the encoding method 600 can be generalized to provide a different number of enhancement layers, not just two enhancement layers. For example, the hybrid video stream output by the encoding method 600 may include a third enhancement stream, a fourth enhancement stream, etc. Each of the additional enhancement streams may be generated by similar components or operations as for the first enhancement stream or the second enhancement stream, and decoding may proceed along the above lines.
[0065] In a first step 610 of method 600, at least one non-motion region ( Figure 3 To illustrate this detection, Figure 7An example video frame 700 from a street scene with pedestrians and cyclists captured by a video surveillance device in a building environment is shown. In the upper portion of the video frame 700, a detected irregularly shaped non-motion region 701 is indicated by a dashed boundary line and hashing. Since the non-motion region 701 depicts an area of the facade that is inaccessible to pedestrians and cyclists and is therefore rarely visited by moving objects, everyday experience seems to confirm that the non-motion region 701 has been correctly detected.
[0066] The non-motion area 701 can be detected based on configuration data entered by an operator, or it can be detected automatically. The automatic detection algorithm deployed for this purpose can have a spatial granularity of at least 16×16 pixels, where the values of such pixel blocks are compared across consecutive video frames to determine whether motion is absent (pixel values are approximately constant) or present (pixel values change). The automatic detection algorithm may include the calculation of the variance of pixel values. Alternatively, the automatic detection algorithm can use a finer granularity, down to a single pixel. In an embodiment in which the first residual is generated by applying a transformation kernel of a specific size, it is preferred to perform automatic detection at a granularity equal to the kernel size or coarser. In addition, the automatic detection algorithm can have a temporal granularity corresponding to the duration of one video frame or the duration of ten video frames or the duration of dozens of video frames. Using a coarser granularity generally means that the detection algorithm consumes less processing resources; in video surveillance applications, non-motion periods may have a duration of several minutes or even hours, and therefore refreshing the detection of non-motion areas at a corresponding granularity (i.e., on the order of hundreds or thousands of video frames) may be sufficient.
[0067] Another automatic detection algorithm may rely on a machine learning model that has been trained to identify areas suitable for exclusion from sub-layer 1 encoding based on local statistics of the input video (e.g., image statistics, motion statistics, image content signatures, etc.). For the purposes of the present encoding method 600, image areas predicted by the trained machine learning model can be used as non-motion areas 701. In some embodiments, step 610 may be implemented by an algorithm having a different purpose than detecting non-motion areas 701 (e.g., a noise filter or an image stabilization filter integrated in a camera). Information indicating the presence of non-motion areas may be derived from internal variables of any of these filters, from appropriate output signals of the filters, or by comparing the input frame of the filter with the corresponding output frame. In addition, step 610 may be implemented by an algorithm related to inter-frame prediction coding (i.e., an algorithm that determines whether it is economical to predictively encode a block at a block level); if the algorithm assesses that predictively encoding a block is economical, the block may be considered a non-motion area.
[0068] The detection of the non-motion region 701 may be applied to video frames of the input video at the original spatial resolution. Alternatively, the detection of the non-motion region 701 may be applied to video frames of the input video at the intermediate spatial resolution. In this case, the execution of step 610 may not start earlier than the subsequent step 620.
[0069] The sensitivity of the automatic detection algorithm (e.g., the tolerance within which pixel values are considered to be approximately constant between video frames) can be set by optimizing the overall bit rate of the encoding method 600 for a representative test video while varying the detection sensitivity. Because if a region is incorrectly classified as a non-motion region in step 610, the region will be excluded from sub-layer 1 correction because the first residual becomes zero, but will eventually be corrected in sub-layer 2 (possibly at a higher encoding cost), so moderate frequency of so-called false positives are not significant in themselves. Specifically, if a detected non-motion region of a video frame contains pixel value changes (e.g., indicating a moving object or lighting fluctuations), the temporal prediction block 122 will decide not to use temporal prediction for the region, the region will instead be explicitly encoded, and the necessary enhancement will be implemented by sub-layer 2.
[0070] The execution flow of method 600 proceeds to step 620 of downsampling the input video from the original spatial resolution to the reduced spatial resolution and the intermediate spatial resolution. The input video at the reduced spatial resolution can be obtained by, for example, using a method similar to Figure 1A The video is provided by a series of downscaling devices 130, 140 in the image processing unit or directly from the input video at the original spatial resolution, by downsampling the input video at the original spatial resolution twice.
[0071] Within the scope of the present disclosure, each of the downscalers 130, 140 can be adapted for 2:2 downsampling of the input video (i.e., the width resolution is halved and the height resolution is halved), 2:1 downsampling of the input video (i.e., the resolution in the width direction of the video frame is halved and the resolution in the height direction of the video frame is maintained), 1:2 downsampling of the input video (i.e., the width resolution is maintained and the height resolution is halved) or 1:1 downsampling (i.e., the width resolution is maintained and the height resolution is maintained). The respective downsampling actions of the first downscaler 130 and the second downscaler 140 can be independently selected. In a conventional implementation of LCEVC, the action of the first downscaler 130 is opposite to the action of the second upscaler 120, and the action of the second downscaler 140 is opposite to the action of the first upscaler 110. When the second downscaler 140 is configured as a pass-through block (for ordinary 1:1 downscaling), the spatial resolution of the base encoder 150 (corresponding to the "reduced spatial resolution" in the claims) and the spatial resolution of the sub-layer 1 (corresponding to the "intermediate spatial resolution" in the claims) will be equal. With this configuration, sub-layer 1 can help improve the quality of the output video and / or the fidelity of the output video relative to the input video, but it does not change the spatial resolution. The upsampling performed by the upscalers 110, 120 is described in Section III of [1].
[0072] In the next step 630, the input video at the reduced spatial resolution is provided to the base encoder 150 to obtain the base coded stream 180. It is emphasized that the base encoder 150 operates independently of the enhancement layer. More specifically, the encoder 100 can be successfully implemented without any need to check or modify the settings and internal variables of the base encoder 150.
[0073] In a next step 640, a first enhancement stream is provided. This includes a step 641 of generating a set of first residuals based on the difference between the input video and the reconstructed video at the intermediate spatial resolution. The difference can be calculated by a subtractor 111 operating at the level of a single pixel. In this example, the reconstructed video is obtained by decoding the base coded stream and upsampling (or ordinary upsampling) the output. The first residuals can be the differences without any further processing applied, or the first residuals can be transform coefficients obtained by applying a transform kernel to the differences. The size of the transform kernel can be suitable for 2×2 or 4×4 pixel blocks, and the output can be a block of coefficients of equal size ("a set of first residuals").
[0074] Step 640 further includes a step 642 of quantizing the set of first residuals ( Figure 3113 in block 114) and step 644 of forming a first enhancement stream from the quantized set of first residuals. Optionally, step 644 may include subjecting the set of first quantized residuals to entropy coding. Suitable quantization algorithms and entropy coding algorithms are described in Sections VI and VII of [1].
[0075] According to a first group of embodiments, step 640 further comprises a step 643 of applying a mask to the set of quantized first residuals ( Figure 3 302 in ). The masking may include replacing those quantized first residuals associated with non-motion regions with zero values (or equivalently, with neutral values representing a lack of image content). Conceptually, the "mask" corresponds to the detected non-motion regions. This will cause the set of first residuals to become zero throughout the non-motion region, but leave the first residuals outside the non-motion region intact. The replacement may be performed at the granularity of a set of first residuals (e.g., at the granularity of a transform block if a transform is used), or at the same granularity as that used to detect the non-motion regions in step 610.
[0076] In a next step 650, a second enhancement stream is provided. This includes a step 651 of generating a set of second residuals based on the difference between the input video and the reconstructed video at the original spatial resolution. Here, the reconstructed video at the original spatial resolution is obtained from the reconstructed video at the intermediate spatial resolution, i.e., by adding a reconstruction of the first residual and upsampling the output (or ordinary upsampling). The second residual may be this difference without any further processing applied, or the second residual may be a transform coefficient obtained by applying a transform kernel to the difference. Similar to the first residual, the second residual is subjected to quantization (step 652, box 124) before being included in the second enhancement stream (step 654, which may optionally include entropy encoding, box 125). The quantization level to be used in step 652 can be configured in view of the expected noise level of the input video; for example, the quantization level (quantization step size) can be set large enough so that most noise artifacts in a nominally zero-valued signal are rounded to zero. The quantization level used in step 652 can be configured independently of the quantization level used in step 642; in this regard, the two sub-layers of the enhancement encoder 100 are independent.
[0077] Common to all embodiments disclosed herein, the second enhancement stream is encoded at least in part using temporal prediction. The temporal prediction encoding is partial in the sense that at least some blocks, some video frames, or some subsequences of video frames in the input video 170 are encoded using temporal prediction encoding. The second enhancement stream includes temporal signaling 193 indicating, at a suitable temporal and spatial granularity (e.g., for each predefined pixel / coefficient block in each video frame), whether the second residuals are encoded by temporal prediction (i.e., whether they are expressed by reference to one or more other video frames) or whether the second residuals are explicitly encoded. The temporal signaling 193 can be entropy encoded before it is included as a temporal layer in the enhancement bitstream 190. The decision whether to explicitly encode or use temporal prediction ( Figure 6 Block 653 in ( ) may be based on the degree of similarity to the one or more other video frames, such as whether the error difference introduced by using temporal prediction is of acceptable size. Temporal prediction may operate at the following levels:
[0078] a) The difference between the input video and the reconstructed video at the original spatial resolution;
[0079] b) coefficients obtained by applying the transformation kernel to the difference;
[0080] c) quantify the difference, or
[0081] d) Quantized coefficients.
[0082] Option a) has been chosen for some LCEVC implementations; see [2] and Section 4 of [1]. Option b) is discussed in [3]. Figure 6 The flowchart in covers options c) and d).
[0083] In the next step 660, after completing step 650 for the sequence of input videos, corresponding segments of the mixed video stream can be formed. Execution of the encoding method 600 can end here, or resume from the step 610 of detecting non-motion areas. The mixed video stream can be decoded by a general decoder that has not been modified in view of the teachings herein; this includes Figure 2 The standardized LCEVC decoder 200 in.
[0084] By illustrating the LCEVC encoder 100 adjusted according to the first set of embodiments Figure 3The above description is summarized, wherein, due to the application of the mask to the set of quantized first residuals, the set of first residuals becomes zero in the entire non-motion area. Here, corresponding to step 610, block 301 represents an algorithm that can be operated to detect the non-motion area. Box 302 represents the mask operation in step 643. Alternatively, the detection algorithm 301 can obtain its input from a point between the first reducer 130 and the second reducer 140, thereby performing detection at the same spatial resolution as the mask.
[0085] Note that the present first group of embodiments has been described in a relatively complete and detailed manner, including possible variations and alternatives, and the embodiments of the subsequent groups will be discussed more concisely to avoid meaningless repetition. It should be understood that the technical features of the first group of embodiments (except for the technical features related to the mask 642 of the first residual of the group quantization) can be obtained from this context and put into use in embodiments outside the first group.
[0086] Second Group of Embodiments
[0087] Figure 4 The diagram illustrates an LCEVC encoder 100 adapted according to a second set of embodiments, wherein the set of first residuals becomes zero throughout non-motion regions due to replacement of the input video at the intermediate spatial resolution with a substitute video that has been upsampled from the input video at the reduced spatial resolution.
[0088] exist Figure 6 In the first step 610 of the method 600 illustrated in FIG. 6 , at least one non-motion region ( Figure 4 Detection may be implemented in one of several ways described above in the context of the first set of embodiments.
[0089] Then, in step 620, the input video is downsampled from the original spatial resolution to the reduced spatial resolution and the intermediate spatial resolution. In a second set of embodiments, before generating 641 the set of first residuals, the input video at the intermediate spatial resolution is replaced throughout the non-motion region of the video frame with an alternative video upsampled from the input video at the reduced spatial resolution. This can be considered to be equivalent to providing (sub-step 620.1) a double-resolution video frame in which the non-motion region has the reduced spatial resolution (although represented at the intermediate spatial resolution to allow, for example, processing by the subtractor 111), and the remainder of the video frame has the intermediate spatial resolution. Thus, in a simple implementation in which the upsampling operation does not include smooth interpolation, the double-resolution video frame formally always has the intermediate spatial resolution, but the pixel values in the non-motion region vary at a granularity corresponding to the reduced spatial resolution (e.g., by 2×2 pixel blocks).
[0090] Figure 4An embodiment is shown in which, before any downsampling, each frame of the input video is segmented (block 402) into at least one non-motion region on the one hand, and into the complement of the non-motion region on the other hand. The segmentation may be guided by the output of the non-motion region detection algorithm symbolized by block 401. The complement undergoes downsampling to an intermediate spatial resolution (downsampler 130). At least one non-motion region undergoes downsampling to a reduced spatial resolution and is subsequently upsampled to the intermediate resolution (block 404). The upsampling operation is preferably the same or equivalent to the upsampling operation (block 110) applied to the reconstructed video. The at least one non-motion region and its complement are then combined (block 403) into a video frame at the intermediate resolution. As a result of the operation in block 404, it may be expected that the information density in the at least one non-motion region is lower than the information density in the remainder of the video frame.
[0091] In an alternative embodiment, the downsample-upsample block 404 is replaced by a block (not shown) that takes the output of the second downsizer 140, upsamples it, and replaces the upsampled data into at least one non-motion region of the video frame.
[0092] In a next step 630 , the input video at the reduced spatial resolution is provided to the base encoder 150 to obtain a base coded stream 180 .
[0093] In a next step 640, a first enhancement stream is provided, i.e., a first enhancement stream is provided by generating (step 641) a set of first residuals, quantizing (step 642) the set of first residuals, and forming (step 644) the first enhancement stream from the set of quantized first residuals. Because a dual-resolution video frame is used in step 641, the set of first residuals for non-motion areas will be generated based on the difference between the input video at a reduced spatial resolution (although nominally upsampled to an intermediate spatial resolution) on the one hand and the reconstructed video at the intermediate spatial resolution on the other hand. Outside the non-motion areas, the set of first residuals is generated based on the difference between the input video and the reconstructed video at the intermediate spatial resolution. In this way, the set of first residuals is zero or approximately zero (i.e., becomes zero) throughout the non-motion areas, and correction is postponed to the next sublayer of the enhancement encoder 100.
[0094] In a next step 650, a second enhancement stream is provided by generating (step 651) a set of second residuals based on the difference between the input video and the reconstructed video at the original spatial resolution, quantizing said second residuals (step 652) and including them in the second enhancement stream (step 654). As determined in step 653, the second enhancement stream is at least partially encoded using temporal prediction.
[0095] In a next step 660, after completing step 650 for the sequence of input videos, corresponding segments of the hybrid video stream can be formed. The execution of the encoding method 600 can end or resume from the step 610 of detecting non-motion areas.
[0096] The third group of embodiments
[0097] Figure 5A The LCEVC encoder 100 is shown adapted according to a third group of embodiments, wherein the set of first residuals is zeroed throughout non-motion regions due to the application of a mask to the difference between the input video and the reconstructed video at the intermediate spatial resolution. In some embodiments, the set of first residuals constitutes the difference, while in other embodiments, the set of first residuals are transform coefficients calculated from the difference. The third group further includes embodiments where a mask is applied to the set of first residuals that are transform coefficients; similar to Figure 5A In the embodiment depicted in , downstream of block 112 , a mask is applied before quantizing the set of first residuals.
[0098] exist Figure 6 In the first step 610 of the method 600 illustrated in FIG. 6 , at least one non-motion region ( Figure 5A Detection may be performed in one of several ways described above in the context of the first set of embodiments.
[0099] Then, in step 620, the input video is downsampled from the original spatial resolution to a reduced spatial resolution and an intermediate spatial resolution.
[0100] In a next step 630 , the input video at the reduced spatial resolution is provided to the base encoder 150 to obtain a base coded stream 180 .
[0101] In a next step 640, a first enhancement stream is provided by generating (step 641) a set of first residuals, quantizing (step 642) the set of first residuals and forming (step 644) the first enhancement stream from the set of quantized first residuals.
[0102] According to some embodiments in the third group, step 641 includes a sub-step 641.1 (box 502) of applying a mask to the difference between the input video and the reconstructed video at the intermediate spatial resolution. The mask may include replacing those values of the difference associated with non-motion areas with zero values (or equivalently, with neutral values representing the lack of image content). Conceptually, the "mask" corresponds to the detected non-motion areas. According to other embodiments in the third group, where the first residuals are transform coefficients, sub-step 641.1 includes applying such a mask to the set of first residuals before quantization (step 642, box 113). In this case, this may correspond to placing box 502 between boxes 112 and 113, and the mask may be applied to all transform blocks derived entirely from pixels in non-motion areas. The mask may optionally be applied to all transform blocks derived in whole or in part from pixels in non-motion areas. In either case, correction of non-motion areas will be postponed to the second sublayer of the enhancement encoder 100.
[0103] In a next step 650, a second enhancement stream is provided by generating (step 651) a set of second residuals based on the difference between the input video and the reconstructed video at the original spatial resolution, quantizing said second residuals (step 652) and including them in the second enhancement stream (step 654). As determined in step 653, the second enhancement stream is at least partially encoded using temporal prediction.
[0104] In a next step 660, after completing step 650 for the sequence of input videos, corresponding segments of the hybrid video stream can be formed. The execution of the encoding method 600 can end or resume from the step 610 of detecting non-motion areas.
[0105] Fourth Group of Embodiments
[0106] Figure 5B The diagram shows an LCEVC encoder 100 adapted according to a fourth set of embodiments, wherein, before generating the set of first residuals, the set of first residuals is zeroed throughout the non-motion region by subtracting the predicted difference between the input video and the reconstructed video at the intermediate spatial resolution from the input video. Outside the non-motion region, the set of first residuals is generated in a conventional manner, i.e., based on the difference between the input video and the reconstructed video at the intermediate spatial resolution.
[0107] exist Figure 6 In the first step 610 of the method 600 illustrated in FIG. 6 , at least one non-motion region ( Figure 5B Detection may be performed in one of several ways described above in the context of the first set of embodiments.
[0108] Then, in step 620, the input video is downsampled from the original spatial resolution to a reduced spatial resolution and an intermediate spatial resolution.
[0109] In a next step 630 , the input video at the reduced spatial resolution is provided to the base encoder 150 to obtain a base coded stream 180 .
[0110] In a next step 640, a first enhancement stream is provided by generating (step 641) a set of first residuals, quantizing (step 642) the set of first residuals and forming (step 644) the first enhancement stream from the set of quantized first residuals.
[0111] According to the embodiments in the fourth group, in step 635, the difference between the input video and the reconstructed video at the intermediate spatial resolution is predicted and subtracted from the non-motion region of each video frame of the input video. The predicted difference can be considered as a prediction of the output of the subtractor 111. The subtraction is performed before generating the first residual (step 641). Figure 5B One possible implementation of this subtraction is shown. Here, the output of the first upscaling device 110 (i.e., the reconstructed video at the intermediate spatial resolution) is subtracted by a subtractor 505 from the output of the first downscaling device 130 (i.e., the input video downsampled to the intermediate spatial resolution). Block 503 separates the non-motion region from the remainder of the video frame and block 506 subtracts the output of the subtractor 505 therefrom, at which point block 504 combines the non-motion region with the remainder and feeds the output into the subtractor 111. As a result, the difference in the output of the subtractor 111 will be zero or approximately zero (i.e., become zero) in the non-motion region:
[0112] ((input video at intermediate resolution) - (predicted difference)) - (reconstructed video at intermediate resolution)
[0113] =((input video at intermediate resolution)-((input video at intermediate resolution)-(reconstructed video at intermediate resolution)))-(reconstructed video at intermediate resolution)
[0114] =0.
[0115] In this way, the correction of non-motion areas will be postponed to the second sub-layer of the enhancement encoder 100. The rest of the video frame will be processed normally, ie in both sub-layer 1 and sub-layer 2.
[0116] In a next step 650, a second enhancement stream is provided by generating (step 651) a set of second residuals based on the difference between the input video and the reconstructed video at the original spatial resolution, quantizing said second residuals (step 652) and including them in the second enhancement stream (step 654). As determined in step 653, the second enhancement stream is at least partially encoded using temporal prediction.
[0117] In a next step 660, after completing step 650 for the sequence of input videos, corresponding segments of the hybrid video stream can be formed. The execution of the encoding method 600 can end or resume from the step 610 of detecting non-motion areas.
[0118] Conclusion
[0119] Aspects of the disclosure have mainly been described above with reference to a few embodiments. However, as readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.
Claims
1. A method for encoding an input video comprising a sequence of video frames into a hybrid video stream, in, The method comprises: downsampling the input video from an original spatial resolution to a reduced spatial resolution and an intermediate spatial resolution; providing the input video at the reduced spatial resolution to a basic encoder to obtain a basic coded stream; The first enhancement stream is provided by the following steps: generating a set of first residuals based on differences between the input video and the reconstructed video at the intermediate spatial resolution; quantizing the set of first residuals; and forming the first enhancement stream from the set of quantized first residuals; The second enhancement stream is provided by the following steps: generating a set of second residuals based on differences between the input video and the reconstructed video at the original spatial resolution; quantizing the set of second residuals; and forming the second enhancement stream from the set of quantized second residuals, wherein the second enhancement stream is encoded at least in part using temporal prediction, and further comprises temporal signaling indicating whether temporal prediction is used; forming the mixed video stream from the base coded stream, the first enhancement stream and the second enhancement stream, Characterized in that the method further comprises: detecting at least one non-motion region in a video frame; and The set of first residuals, but not the set of second residuals, is made zero throughout the non-motion region.
2. The method according to claim 1, in, The set of first residuals is made zero in the entire non-motion region by applying a mask to the set of quantized first residuals.
3. The method according to claim 1, in, The first set of residuals is made zero in the entire non-motion region by the following steps: Prior to generating the set of first residuals, the input video at the intermediate spatial resolution is replaced in the non-motion regions of the video frames with a substitute video upsampled from the input video at the reduced spatial resolution.
4. The method according to claim 3, in, Downsampling the input video includes: A dual-resolution video frame is provided having the reduced spatial resolution in the non-motion area and the intermediate spatial resolution elsewhere.
5. The method according to claim 1, in, The set of first residuals are driven to zero throughout the non-motion region by applying a mask to the difference between the input video and the reconstructed video at the intermediate spatial resolution or by applying a mask to the set of first residuals prior to the quantization.
6. The method according to claim 1, in, The first set of residuals is made zero in the entire non-motion region by the following steps: In the non-motion region of the video frame, a predicted difference between the input video and the reconstructed video at the intermediate spatial resolution is subtracted from the input video before generating the set of first residuals.
7. The method according to claim 1, in, Each video frame of the first enhancement stream is decodable without reference to any other video frame of the first enhancement stream.
8. The method according to claim 1, in, Providing the second enhancement stream further comprises: for each set of second residuals or each set of quantized second residuals in a video frame, determining whether to use temporal prediction with reference to one or more other video frames, and indicating by the temporal signaling whether to use temporal prediction in the video frame.
9. The method according to claim 1, in, The at least one non-motion region is detected in a video frame of the input video at the original spatial resolution or in a video frame of the input video at the intermediate spatial resolution.
10. The method according to claim 1, in, The intermediate spatial resolution is finer than the reduced spatial resolution, or the intermediate spatial resolution and the reduced spatial resolution are equal.
11. The method according to claim 1, in, The first residual and / or the second residual is generated by applying a transform kernel of size 2×2 pixels or 4×4 pixels to a difference between the input video and the reconstructed video.
12. The method according to claim 11, in, The transform kernel is a Low Complexity Enhancement Video Coding LCEVC transform kernel.
13. The method according to claim 1, in, The set of first residuals and the set of second residuals are quantized using different quantization levels.
14. An apparatus comprising processing circuitry arranged to perform the method of claim 1.
15. A non-transitory computer-readable storage medium having stored thereon a computer program including instructions which, when executed by a processing circuit, cause the processing circuit to implement the method of claim 1.
Citation Information
Patent Citations
Low complexity enhancement video coding
WO2020188273A1
Inter-layer reference picture enhancement for multiple layer video coding
US20160065976A1
Processing of residulas in video coding
US20220191521A1
Processing of residuals in video coding
WO2020188229A1
Temporal signalling for video coding technology
WO2020188271A1
