Encoder and method for encoding a frame sequence

Abstract

When encoding a sequence of frames (113), a linear transformation (101), for example a wavelet transformation (104), is applied to the sequence of frames frame by frame. At least a portion of the linearly transformed frames thus obtained is encoded in an inter-frame mode based on a differential representation of the linearly transformed frames. The differential representation corresponds to the difference between a representation of a portion of the linearly transformed frame on the one hand and a corresponding portion of at least one other linearly transformed frame on the other hand. Data compression (106, 107) is then applied to the differential representation of said portion of the linearly transformed frames.

Description

Technical Field

[0001] One aspect of the present invention relates to an encoder suitable for encoding a sequence of frames. For example, the encoder can be used to encode a video stream for efficient transmission or storage, or both. Other aspects of the present invention relate to methods for encoding a sequence of frames, computer programs for encoders, decoders, and methods for decoding an encoded sequence of frames. Background Art

[0002] US Patent 9332258 mentions that compression of digital images is usually achieved through 3 steps: decorrelation transform, entropy coding and rate allocation. Decorrelation transform reduces entropy: the transformed image may have lower entropy because the probability of occurrence is concentrated on a small subset of coefficient values. Decorrelation transforms commonly used in image compression are color transforms, inter / intra prediction, DCT or wavelet transforms. Entropy coding reduces the amount of transformed image data: a smaller amount of data can represent the transformed image. Finally, rate allocation selects the data that will become part of the compressed image output stream to achieve the desired compression ratio.

[0003] U.S. Patent 9332258 describes a technique for compressing an input data stream into an output data stream having acceptable compression efficiency while minimizing the complexity of the encoder and decoder. A sequence of m-bit words in the input data stream is grouped into n groups of m-bit words, n being greater than or equal to 2. For each group, the value of the maximum code line index (GCLI) is detected. The GCLI is the index of the highest weighted non-zero bit among the bits of the word in the group, excluding any sign bit. The output data stream then comprises a group of n GCLI bit words corresponding to the n m-bit words in the associated group in the input stream. The GCLI bit of each word is the lowest weighted GCLI bit of the corresponding word in the input stream, together with the value of the GCLI. Summary of the invention

[0004] An improved solution is needed that allows encoding a sequence of frames that better meets at least one of the following criteria: high compression rate, good image quality and moderate cost.

[0005] One aspect of the present invention provides an encoder suitable for encoding a frame sequence, the encoder comprising:

[0006] a linear transformation component adapted to apply a linear transformation to the sequence of frames frame by frame to obtain a sequence of linearly transformed frames having an entropy lower than that of the frames in the sequence of frames to be encoded, and

[0007] an encoding component adapted to encode the sequence of linearly transformed frames to obtain a sequence of coded frames containing an amount of data less than the linearly transformed frames,

[0008] wherein the encoding component is adapted to encode at least a portion of the linearly transformed frame in an inter-frame mode, wherein the encoding component provides a differential representation of at least a portion of the linearly transformed frame encoded in the inter-frame mode, the differential representation corresponding to the difference between a representation of at least a portion of the linearly transformed frame on the one hand and a corresponding at least a portion of at least one other linearly transformed frame on the other hand, and wherein the encoding component applies data compression to the differential representation of at least a portion of the linearly transformed frame so as to obtain at least a portion of an encoded frame.

[0009] Other aspects of the present invention provide a method for encoding a frame sequence, a computer program for an encoder and a decoder, respectively.

[0010] The main difference from conventional coding schemes is that in inter-mode, a differential representation of at least a portion of a linearly transformed frame is processed instead of a differential representation of a frame or a portion thereof in its original form. This allows an implementation with lower complexity in terms of hardware or software or both, which in turn allows an implementation with low cost and low power consumption. Furthermore, applying inter-mode coding allows high compression to be achieved without significantly affecting the image quality.

[0011] The fact that the differences between linearly transformed frames are encoded instead of the differences between the original frames does not in principle have an adverse effect on the image quality. This is because the subtraction operation after the linear transformation is equivalent to the subtraction operation before the linear transformation. Applying the inverse of the linear transformation to the differential representation obtained by the subtraction operation after the linear transformation will produce the differential representation obtained by the subtraction operation before the linear transformation. Conversely, applying the linear transformation to the differential representation obtained by the subtraction operation before the linear transformation will produce the differential representation obtained by the subtraction operation after the linear transformation.

[0012] It should be noted that in an encoder as defined above, the encoding component does not need to perform any further transformations, which may be decorrelated. Therefore, in such an encoder, the linearly transformed frames do not need to undergo further transformations that may reduce entropy. In contrast, the data compression applied to the differential representation as described above may include applying entropy coding directly to the differential representation so that the differential representation is represented with less data.

[0013] For illustrative purposes, some embodiments of the present invention are described in detail with reference to the accompanying drawings. In this description, additional features will appear and advantages will be apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0015] Figure 1 is a block diagram of a video encoder.

[0016] Figure 2 is a frequency diagram illustrating the subbands defined by the video encoder.

[0017] Figure 3 is a schematic diagram of the linear transformation frame produced in the video encoder.

[0018] Figure 4 is a block diagram of a coding mode selection module in a video encoder.

[0019] Figure 5 is a conceptual diagram of extracting GCLI values ​​from a set of four samples in the encoding mode selection module.

[0020] Figure 6 is a schematic diagram of a linearly transformed frame showing a portion coded in intra-frame mode and other portions coded in inter-frame mode.

[0021] Figure 7 is a block diagram of a reference frame buffer component in a video encoder.

[0022] Figure 8 is a block diagram of a video decoder. DETAILED DESCRIPTION

[0023] Figure 1 A video encoder 100 is schematically shown. Figure 1 A block diagram of a video encoder 100 is provided. The video encoder 100 may be included, for example, in a communication device that can capture video or store video or both, such as a smart phone. The video encoder 100 may enable a communication device to efficiently transmit video to another communication device via a communication channel that may be provided by a network.

[0024] The video encoder 100 includes a linear transformation component 101 and an encoding component 102. The linear transformation component 101 may include a reversible color transformation module 103 and a wavelet transformation module 104. The encoding component 102 may include an encoding mode selection module 105, a quantization module 106, an entropy coding module 107, a data packing module 108, a rate allocation module 109, a dequantization module 110, a reconstruction module 111, and a reference frame buffer component 112.

[0025] The video encoder 100 operates basically as follows: The video encoder 100 receives a video stream 113 comprising a sequence of frames. In response, the video encoder 100 outputs an encoded video stream 114, which may have a much lower data rate than the received video stream 113.

[0026] The frames in the received video stream 113 can be viewed as a matrix of pixels, including rows and columns of pixels. A pixel can include three components that combine to convey brightness information and color information related to the pixel. These three components are represented according to a color coordinate system, for example, a color coordinate system called RGB.

[0027] The reversible color transform module 103 transforms the three components of the pixel into three components of another color coordinate system. This reversible color transform can reduce entropy. Therefore, the reversible color transform module provides a reversible color transformed frame, which may have less entropy than the frame in its original form. This helps image compression.

[0028] For convenience and simplicity, it is assumed that the reversible color transform frame is a matrix of pixels whereby a pixel has only one component, for example only the luminance component (Y). That is, it is assumed that the reversible color transform frame is a single sample matrix representing only a single component, whereas in reality, the reversible color transform frame can be viewed as a set of three sample matrices, each representing a particular component. For example, the reversible color transform frame may actually include a sample matrix representing the luminance (Y) component, another sample matrix representing the first color component (Cr), and yet another sample matrix representing the second color component (Cb). These sample matrices may be processed independently as described below. The samples may be in the form of binary words representing the values ​​of the associated components.

[0029] The wavelet transform module 104 applies a wavelet transform to the reversible color transform frame. The wavelet transform involves filtering and subsampling operations. These operations produce a set of corresponding subband frames, which together constitute a linear transform frame. The subband frames represent the spectral content of the reversible color transform frame in a specific spectral band, which is called a subband. The wavelet transform performed by the wavelet transform module 104 can be similar to the wavelet transform in existing wavelet-based image coding schemes, such as the known image coding scheme JPEG-XS.

[0030] Figure 2 The sub-bands 201-207 defined by the wavelet transform module 104 are conceptually shown. The sub-bands 201-207 are represented by a two-dimensional frequency diagram having a horizontal direction. The horizontal axis represents the frequency and the vertical axis represents The vertical axis of frequency on the wavelet transform module 104 is shown. The wavelet transform module 104 thus provides corresponding subband frames for these corresponding subbands 201-207. The subband frames can be viewed as subband coefficient matrices. The size of the subband coefficient matrix relative to the reversible color transform frame depends on the subsampling factor applied by the wavelet transform module 104 to obtain the relevant subband frames.

[0031] Figure 3Schematically showing the individual sub-band frames 301-307 that together constitute a linear transformation frame. The individual sub-band frames 301-307 are shown in schematic diagrams. Figure 2 is associated with the specific sub-band shown in . Figure 3 The particular subband with which a subband frame is associated is indicated by the reference number associated with that particular subband.

[0032] The linear transformation frame includes four relatively small low frequency sub-band frames 301-304 and three relatively large high frequency sub-band frames 305-307. For the three high frequency sub-band frames 305-307, the sub-sampling factor is 2. For the four low frequency sub-band frames 301-304, the sub-sampling factor is 4. Among the four low frequency sub-band frames 301-304, there is a lowest sub-band frame 301, which corresponds to a low pass filtered and scaled version of the reversible color transformation frame. In this case, the scale is equivalent to subsampling.

[0033] from Figure 3 It can be understood that the subband frame includes spectrum and spatial information. Figure 1 Utilized in the illustrated video encoder 100 and described in greater detail. That is, certain portions of a subband frame may be associated with a reversible color transform frame and similarly located portions of a frame to be encoded.

[0034] Reference again Figure 1 , the encoding component 102 can independently encode each sub-band frame 301-307, except for the rate allocation and data packaging of the combined compressed sub-band frame. That is, the encoding mode selection module 105, the quantization module 106, the dequantization module 110 and the entropy coding module 107 can independently process each sub-band frame 301-307 in sequence. In an alternative embodiment, the encoding component 102 can operate according to a parallel processing scheme. In such an embodiment, the encoding component 102 may include multiple modules of the following types: encoding mode selection, quantization, dequantization and entropy coding. That is, there can be one such module for each sub-band frame included in the linear transformation frame. For simplicity and convenience, the encoding of a single sub-band frame is described below. Other sub-band frames are encoded in a similar manner.

[0035] For encoding a sub-band frame, the reference frame buffer component 112 includes a representation of a corresponding sub-band frame associated with another frame that has been previously encoded in the video stream 113. The term "corresponding" means that both sub-band frames are associated with the same sub-band. For convenience and brevity, the representation of a corresponding sub-band frame associated with a previously encoded frame will be referred to as a corresponding previous sub-band frame in the following. More generally, for encoding a linearly transformed frame, the reference frame buffer component 112 may include a representation of another linearly transformed frame that has been previously encoded. Both linearly transformed frames include corresponding sub-band frames 301-307, such as Figure 3 shown.

[0036] The encoding component 102 may encode the subband frame in portions. A portion of the subband frame to be encoded may include a group of 32 samples on the same row. Depending on the subsampling factor (which may be 2 or 4 in this embodiment), this portion of the subband frame may correspond to a block of 64×2 samples or a block of 128×4 samples in the frame to be encoded, respectively. For convenience and brevity, a portion of the subband frame to be encoded is referred to as a subband frame portion hereinafter.

[0037] To encode the sub-band frame portion, the encoding mode selection module 105 retrieves the corresponding portion in the corresponding previous sub-band frame from the reference frame buffer component 112. In this embodiment, the position of the corresponding portion in the corresponding previous sub-band frame is similar to the position of the sub-band frame portion to be encoded. For convenience and brevity, the corresponding portion in the corresponding previous sub-band frame is referred to as the corresponding previous sub-band frame portion below.

[0038] Figure 4 The encoding mode selection module 105 is schematically shown in more detail. Figure 4 A block diagram of the coding mode selection module 105 is provided. The coding mode selection module 105 includes a comparator 401, two maximum coding line index extractors 402, 403, a data buffer 404, a coding mode decision unit 405, and a multiplexer 406. For convenience, one of the two maximum coding line index extractors will be referred to as the inter-mode GCLI extractor 402 hereinafter. The other maximum coding line index extractor will be referred to as the intra-mode GCLI extractor 403 hereinafter.

[0039] The coding mode selection module 105 operates substantially as follows. A comparator 401 provides a differential representation of a sub-band frame portion to be coded. The differential representation corresponds to the difference between the sub-band frame portion to be coded and a corresponding previous sub-band frame portion retrieved from a frame buffer component. A multiplexer 406 receives the differential representation of the sub-band frame portion to be coded at one input and the differential representation of the sub-band frame portion itself at another input.

[0040] The coding mode selection module 105 causes the coding component 102 to encode the sub-band frame portion in intra-frame mode or inter-frame mode. In intra-frame mode, the multiplexer 406 passes the sub-band frame portion present in the linearly transformed frame to the Figure 1The quantization module 106 shown in FIG. 107 is used to encode the sub-band frame portion present in the linearly transformed frame. Thus, in the intra-frame mode, the sub-band frame portion present in the linearly transformed frame is encoded, which involves data compression performed by the quantization module 106 and the entropy coding module 107. In the inter-frame mode, the multiplexer 406 passes the differential representation of the sub-band frame portion to the quantizer. Thus, in the inter-frame mode, the differential representation of the sub-band frame portion is encoded, which also involves data compression performed by the quantization module 106 and the entropy coding module 107.

[0041] The main difference from conventional coding schemes is that in inter-frame mode, a differential representation of at least a portion of a linearly transformed frame is processed, rather than a differential representation of a frame or a portion thereof in its original form. Therefore, the coding scheme according to the present disclosure does not require the application of an inverse linear transform to at least a portion of a linearly transformed frame, which is required in conventional coding schemes. Therefore, the coding scheme according to the present disclosure allows for an implementation with lower complexity in hardware or software or both, which in turn allows for a low-cost and low-power implementation.

[0042] The fact that the differences between transformed frames are encoded instead of the differences between the original frames does not in principle have a negative impact on the image quality. This is because the reversible color transform and the wavelet transform mentioned above are both linear. This means that a subtraction operation after these transforms is equivalent to a subtraction operation before these transforms. Applying the inverse of these transforms to the differential representation obtained by subtraction after these transforms will produce the differential representation obtained by subtraction before these transforms. Conversely, applying the relevant transform to the differential representation obtained by subtraction before these transforms will produce the differential representation obtained by subtraction after these transforms.

[0043] The coding mode selection module 105 decides whether to encode the sub-band frame portion in intra-frame mode or inter-frame mode in the following manner. The inter-frame mode GCLI extractor 402 provides a set of eight GCLI values ​​related to eight groups of four samples included in the differential representation of the sub-band frame portion. For convenience, these GCLI values ​​will be referred to as inter-frame mode GCLI values ​​hereinafter. The intra-frame mode GCLI extractor 403 also provides a set of eight GCLI values, which are related to eight groups of four samples included in the sub-band frame portion to be encoded. For convenience, these GCLI values ​​will be referred to as intra-frame mode GCLI values ​​hereinafter. The inter-frame mode GCLI extractor 402 and the intra-frame mode GCLI extractor 403 can each operate in the manner described in U.S. Patent 9332258. The operation includes GCLI extraction, which will be briefly summarized below.

[0044] Figure 5 The extraction of GCLI values ​​from a set of four samples 501-504 is schematically shown. Figure 5A conceptual diagram of this extraction is provided, in which four samples 501-504 in the form of binary words are shown. In this example, the binary words each include a number of bits that define a value. The binary words may also each include a sign bit, which is not shown in the example for simplicity. Figure 5 Not indicated.

[0045] In the hierarchical plurality of bits, each bit has a specific hierarchical rank, denoted by M, ... 5, 4, 3, 2, 1. M is an integer value corresponding to the number of bits included in the hierarchical plurality of bits. The bit with the highest hierarchical rank M is usually referred to as the most significant bit. The bit with the lowest hierarchical rank 1 is usually referred to as the least significant bit.

[0046] In a group of four samples, bits with the same binning rank constitute a so-called bit plane. Thus, a group of four samples includes as many bit planes as there are bits in the binning multiple bits. Each bit plane has a binning rank corresponding to the binning rank of the bits contained in the bit plane. The most significant bits constitute the most significant bit plane, i.e. the highest binning rank M, while the least significant bits constitute the least significant bit plane, i.e. the lowest binning rank 1.

[0047] The extraction of the GCLI value consists in determining which bit planes include at least one non-zero bit, starting from the most significant bit plane and moving towards the least significant bit plane. The GCLI value represents the hierarchical rank of the bit plane. The GCLI value may correspond to, for example, an integer value representing the hierarchical rank, where an integer value of 1 indicates the least significant bit plane and an integer value of M indicates the most significant bit plane. Figure 5 In the example shown, the bit planes from M to 5 each include only zero bits. For convenience, such bit planes including only zero bits will be referred to as zero bit planes hereinafter. Figure 5 In the example shown, bit plane 4 is the highest ranking bit plane that includes at least one non-zero bit. In this example, the GCLI value may therefore be 4.

[0048] The buffer temporarily stores eight GCLI values ​​provided by the inter-mode GCLI extractor 402 based on the differential representation of the sub-band frame portion to be encoded. The buffer also temporarily stores eight GCLI values ​​provided by the intra-mode GCLI extractor based on the sub-band frame portion to be encoded.

[0049] The coding mode decision unit 405 calculates the sum of the eight GCLI values ​​that have been provided by the inter-mode GCLI extractor 402. This sum will be referred to as the inter-mode GCLI sum value hereinafter. The coding mode decision unit 405 also calculates the sum of the eight GCLI values ​​that have been provided by the intra-mode GCLI extractor 403. This sum will be referred to as the intra-mode GCLI sum value hereinafter.

[0050] The intra mode GCLI sum value indicates the data compression factor that can be achieved when the sub-band frame portion is encoded in the intra mode. The inter mode GCLI sum value indicates the data compression factor that can be achieved when the sub-band frame portion is encoded in the inter mode. In fact, the above GCLI sum values ​​all represent estimates of the data compression factor.

[0051] In the case that the intra mode GCLI sum value is less than or equal to the inter mode GCLI sum value, the coding mode selection module 105 encodes the sub-band frame portion in the intra mode. To this end, the coding mode decision unit 405 controls the multiplexer 406 so that the multiplexer 406 passes the sub-band frame portion to be encoded to the output end of the coding mode selection module 105. Therefore, in the intra mode, the sub-band frame portion is further processed by the quantization module 106, the entropy coding module 107 and the data packing module 108.

[0052] In the case where the inter-mode GCLI sum value is less than the intra-mode GCLI sum value, the coding mode selection module 105 may encode the sub-band frame portion in the intra-mode or the inter-mode. If the coding mode decision unit 405 determines that a predetermined number K of corresponding previous sub-band frame portions have been encoded in the inter-mode, the intra-mode is selected, K representing an integer value. In that case, the sub-band frame portion is forced to be encoded in the intra-mode, whereas otherwise it would be encoded in the inter-mode. By doing so, the coding mode decision unit 405 ensures that at least one sub-band frame portion in the sequence of K+1 corresponding sub-band frame portions is encoded in the intra-mode. Therefore, 1 divided by K+1 can be regarded as representing the minimum frequency that the corresponding sub-band frame portion is encoded in the intra-mode.

[0053] In case the inter-mode GCLI sum is less than the intra-mode GCLI sum and less than K corresponding previous sub-band frames have been encoded in the inter-mode, the coding mode selection module 105 encodes the sub-band frame portion in the inter-mode. In this case, the coding mode decision unit 405 controls the multiplexer 406 so that the multiplexer 406 passes the differential representation of the sub-band frame portion to be encoded to the output of the coding mode selection module 105. Therefore, in the inter-mode, the differential representation is further processed by the quantization module 106, the entropy coding module 107 and the data packing module 108.

[0054] The coding mode selection module 105 therefore evaluates whether the data compression factor indicated by the inter mode GCLI sum value is higher than the data compression factor indicated by the intra mode GCLI sum value. In case the above conditions apply, the sub-band frame portion is in principle encoded in the inter mode. Nevertheless, the coding mode selection module 105 may decide to force encoding of the sub-band frame portion in the intra mode to ensure a predefined minimum frequency of applying the intra mode encoding. In case the above conditions do not apply, the sub-band frame portion is encoded in the intra mode.

[0055] The coding mode selection module 105 thus provides several types of output data for the sub-band frame portion to be encoded. The coding mode selection module 105 provides the sub-band frame data to be compressed. The sub-band frame data is the sub-band frame portion present in a linearly transformed frame or a differential representation of the sub-band frame portion, depending on whether the intra mode or the inter mode is selected, respectively. Furthermore, the coding mode selection module 105 provides a coding mode flag, which indicates whether the sub-band frame portion is encoded in intra mode according to the inter mode. The coding mode selection module 105 further provides a GCLI encoding value, which is the intra mode GCLI value or the inter mode GCLI value described above, depending on whether the sub-band frame portion is encoded in intra mode or in inter mode, respectively.

[0056] Figure 6 Schematically shows the Figure 3 The same linearly transformed frame shown in FIG. 1 is shown, where sub-band frame portions encoded in intra-frame mode and other sub-band frame portions encoded in inter-frame mode are indicated. Sub-band frame portions encoded in intra-frame mode are represented by relatively dark-toned rectangles. Sub-band frame portions encoded in inter-frame mode are represented by relatively light-toned rectangles.

[0057] Figure 6 It is clearly stated that the subband frames are independently coded. For example, the first subband frame part in the lowest subband frame 301 can be coded in inter-frame mode, while the first subband frame part in another low-frequency subband frame 302, 303, 304 can be coded in intra-frame mode.

[0058] As mentioned above, the coding mode decision unit 405 ensures that the corresponding sub-band frame part is periodically encoded in the intra-frame mode in the linear transformation frame sequence. This can be regarded as a refresh mechanism, which can quickly obtain better image quality at the decoding end. It can be said that the video decoder can quickly lock Figure 1 The video encoder 100 shown provides an encoded video stream 114. More importantly, this also allows for more robust video transmission. In the event that a portion of the encoded video stream 114 is lost due to, for example, a transmission error, the affected area of ​​the frame can be restored relatively quickly.

[0059] In all corresponding subband frame sequences associated with the same subband, the position of a portion of a subband frame that is forcibly coded in intra-frame mode may be different relative to a portion of a previous subband frame that is forcibly coded in intra-frame mode. Figure 6, assuming that the first subband frame portion in the lowest subband frame 301 has been forced to be encoded in intra-frame mode. In this case, in the subsequent lowest subband frame, the second subband frame portion located after the first subband frame portion can be forced to be encoded in intra-frame mode. In a further subsequent lowest subband frame, the third subband frame portion located after the second subband frame portion can be forced to be encoded in intra-frame mode, and so on. This can be regarded as a shifting or dephasing scheme, according to which the subband frame portions are periodically encoded in intra-frame mode. This scheme helps in efficient rate allocation and helps the video decoder to quickly lock the encoded video stream 114.

[0060] The minimum frequency of forcibly encoding sub-band frame parts in intra mode may depend on the sub-band frames to which these sub-band frame parts belong. Figure 3 and Figure 6 , the minimum frequency of the low frequency sub-band frames 301-304 may be higher than the minimum frequency of the high frequency sub-band frames 305-307. That is, the above number K+1, which represents the inverse of the minimum frequency, may be higher for the high frequency sub-band frames 305-307 than for the low frequency sub-band frames 301-304. This allows a low resolution video representation of acceptable quality to be obtained quickly at the decoding end, and a high resolution video representation to be obtained later, without significantly affecting the compression efficiency. For example, in one embodiment, a low resolution representation may be obtained after 4 frames, and a high resolution representation may be obtained after 32 frames.

[0061] Reference again Figure 1 , the quantization module 106 quantizes the sub-band frame data provided by the encoding mode selection module 105. Therefore, the quantization module 106 provides quantized sub-band frame data. The quantization module 106 applies a quantization factor when quantizing the sub-band frame data. In the case where the quantization factor is relatively high, the sub-band frame data is coarsely quantized to provide a relatively high data compression. Conversely, in the case where the quantization factor is relatively low, the sub-band frame data is finely quantized, providing a relatively low data compression. The quantization factor can be controlled by the rate allocation module 109 so that the encoded video stream 114 has a desired data rate while optimizing image quality.

[0062] The entropy coding module 107 applies an entropy coding operation to the quantized sub-band frame data. The entropy coding operation may include, for example, the operation described in the U.S. Patent 9332258 already mentioned above. In that case, the entropy coding module 107 receives the GCLI coded value from the coding mode selection module 105 described above. The GCLI coded value represents the zero bit plane in the quantized sub-band frame data. The entropy coding module 107 can then remove these zero bit planes. In fact, the zero bit planes are replaced by the GCLI coded value, which indicates the position of these bit planes in the sub-band frame data. The entropy coding module 107 can also apply entropy coding to the GCLI coded value, so as to express these values ​​with as little data as possible. U.S. Patent 9332258 describes some entropy coding techniques that can be used for this effect.

[0063] In addition to removing zero bit planes, the entropy coding module 107 can also truncate samples in the quantized sub-band frame data. This truncation can involve removing one or more least significant bit planes. This is also described in U.S. Pat. No. 9,332,258. The rate allocation module 109 can control the truncation, deciding whether one or more least significant bit planes should be removed, and if so, how much to remove. The rate allocation module 109 can control the truncation so as to achieve a desired data rate for the encoded video stream 114 while optimizing image quality.

[0064] The data packing module 108 packs several types of data into the encoded video stream 114. This includes the entropy coded sub-band frame data provided by the entropy coding module 107. This may further include, for example, the entropy coded GCLI value, and the coding mode flag and quantization factor that the quantization module 106 has applied.

[0065] The rate allocation module 109 may control the quantization module 106 and the entropy coding module 107 based on the GCLI coded value provided by the coding mode selection module 105. The GCLI coded value provides an estimate of a data compression factor that may be achieved by removing only zero bit planes. The rate allocation module 109 may compare the data compression factor with an expected data compression factor representing an expected data rate for the encoded video stream 114. For example, if there is a relatively large gap between the aforementioned data compression factors, the rate allocation module 109 may decide that a relatively high quantization factor should be applied, or that a relatively strict truncation should be applied, or both.

[0066] As mentioned above, the reference frame buffer component 112 may include a representation of a previously encoded linearly transformed frame. This representation constitutes a reference for the inter-mode encoding of the sub-band frame portion of the current linearly transformed frame to be encoded. The representation of the previous linearly transformed frame may be obtained as follows.

[0067] The dequantization module 110 applies dequantization to the quantized sub-band frame data provided by the quantization module 106. Thus, the dequantization module 110 provides dequantized sub-band frame data, which can be considered as a coarser representation of the sub-band frame data that has been generated by the coding mode selection module 105. In the case of intra-mode coding, the dequantized sub-band frame data is a coarser representation of the sub-band frame portion present in the linearly transformed frame. In the case of inter-mode coding, the dequantized sub-band frame data is a coarser representation of the differential representation of the sub-band frame portion.

[0068] The reconstruction module 111 processes the dequantized subband frame data in the case of inter-frame mode coding. The reconstruction module 111 retrieves the aforementioned corresponding previous subband frame portion from the reference frame buffer component 112. That is, the corresponding previous subband frame portion has been used to generate a differential representation of the subband frame portion, based on which the dequantized subband frame data has been generated. The reconstruction module 111 then adds the corresponding previous subband frame portion to the dequantized subband frame data. Thus, a reconstructed version of the subband frame portion that has currently been encoded is obtained. The reconstruction module 111 applies the reconstructed version of the subband frame portion to the reference frame buffer component 112.

[0069] In case of intra mode coding, the reconstruction module 111 passes the dequantized sub-band frame data directly to the reference frame buffer component 112. That is, in this case, the dequantized sub-band frame data constitutes a reconstructed version of the sub-band frame portion.

[0070] Thus, during the encoding process, the reference frame buffer component 112 continuously receives reconstructed subband frame portions that may collectively form a reconstructed subband frame. More generally, the reference frame buffer component 112 continuously receives reconstructed subband frames that collectively form a reconstructed version of a linearly transformed frame. Ideally, the reconstructed version of the linearly transformed frame corresponds to a decoded version of the linearly transformed frame in a video decoder, an embodiment of which will be presented below. That is, in general, the reference used to encode data should ideally correspond to the reference used to decode the same data. Thus, if the reference used to decode data consists of previously decoded data, the reference used to encode the data should be a decoded version of the previously encoded data, rather than the data in its original form.

[0071] Figure 7 The reference frame buffer component 112 is schematically shown.

[0072] Figure 7 A block diagram of the reference frame buffer component 112 is provided. The reference frame buffer component 112 includes a basic entropy encoder 701, a buffer memory 702, and a basic entropy decoder 703.

[0073] The reference frame buffer component 112 operates substantially as follows. The basic entropy encoder 701 applies GCLI extraction to the reconstructed version of the subband frame portion. The GCLI extraction may be similar to that described above with reference to Figure 5 The basic entropy encoder 701 removes zero bit planes in each sample group. The basic entropy encoder 701 may further truncate samples by removing one or more least significant bit planes. Thus, reduced samples are obtained, which may lack one or more most significant bits that are zero, and may lack one or more least significant bits that have been truncated.

[0074] The basic entropy encoder 701 generates a compressed reconstructed version of the subband frame portion, which includes the reduced samples. The compressed reconstructed version may further include the GCLI value that has been obtained by GCLI extraction, as described above with reference to Figure 4 and Figure 5 The compressed reconstructed version may include the GCLI value in entropy encoded form.

[0075] The buffer memory 702 stores compressed reconstructed versions of the sub-band frame portions. Thus, during the encoding process, the buffer memory 702 sequentially stores compressed reconstructed sub-band frame portions, which may together form a compressed reconstructed sub-band frame. More generally, the reference frame buffer component 112 continuously stores compressed reconstructed sub-band frames, which together form a compressed reconstructed version of a linearly transformed frame.

[0076] The basic entropy decoder 703 decompresses the compressed reconstructed sub-band frame portion, which may be required for inter-mode coding of the corresponding subsequent sub-band frame portion. In this way, the aforementioned representation corresponding to the previous sub-band frame portion is obtained.

[0077] Figure 1 The video encoder 100 shown in therefore applies frame buffer compression. Therefore, a relatively small amount of memory is sufficient to store the sub-band frames, and more generally, to store the linearly transformed frames required for encoding in the inter-frame mode. In addition, the frame buffer compression relaxes the bandwidth requirements related to the transmission of data to and from the buffer memory 702, which involves the reconstructed sub-band frame portions. In embodiments where the buffer memory 702 takes the form of an external memory circuit, and other entities of the video encoder 100 are incorporated into the signal processing circuit, this may be a significant advantage. In such embodiments, as well as in other embodiments, the relaxed bandwidth requirements generally contribute to achieving relatively low power consumption.

[0078] However, frame buffer compression may affect image quality to some extent. This is because the representation of the corresponding previous sub-band frame portion that can be used for inter-mode coding is a coarser version of the corresponding previous sub-band frame portion itself. Nevertheless, in many embodiments, frame buffer compression can be configured so that the loss of image quality is barely visually noticeable, or even not visually noticeable at all. That is, frame buffer compression can be visually lossless. Moreover, in principle, frame buffer compression does not affect the delay between the start of decoding and the achievement of satisfactory image quality.

[0079] Figure 8 The video decoder represented by the block diagram is schematically illustrated. The video decoder can receive and decode the video data transmitted by the above reference Figure 1-7 The video encoder 100 described above generates a coded video stream 114. The video decoder comprises a depacketization module 801, an entropy decoding and dequantization module 802, a decoding and reconstruction module 803, a decoding buffer memory 804, and an inverse wavelet and color transformation module 805.

[0080] The basic operation of the video decoder is as follows: The unpacking module 801 performs operations opposite to those performed by the above-mentioned packing module. Thus, the unpacking module 801 retrieves various types of data from the encoded video stream 114, such as entropy-encoded sub-band frame data; entropy-encoded GCLI values, and coding mode flags.

[0081] The entropy decoding and dequantization module 802 performs operations opposite to those performed by the entropy encoding module 107 described above. For example, the entropy decoding and dequantization module 802 may retrieve GCLI encoding values ​​from the entropy encoded GCLI values. The entropy decoding and dequantization module 802 may then restore the removed zero bit planes based on these GCLI encoding values. The entropy decoding and dequantization module 802 may further add dummy bits to replace the least significant bits removed due to truncation. Thus, the quantized sub-band frame data is recreated and then dequantized. The dequantization operation is based on the quantization factors that have been packaged into the encoded video stream 114. Thus, the entropy decoding and dequantization module 106 provides decoded sub-band frame data.

[0082] The decoding and reconstruction module 803 processes the decoded sub-band frame data based on the coding mode flag. In the case where the coding mode flag indicates intra-frame mode coding, the decoded sub-band frame data is a decoded version of the sub-band frame portion, which can be directly used to reconstruct the linear transformation frame without further processing. In this case, the decoding and reconstruction module 803 applies the decoded version of the sub-band frame portion to the decoding buffer memory 804 for temporary storage. In addition, the decoded version of the sub-band frame portion is applied to the inverse wavelet and color transformation module 805.

[0083] In case the coding mode flag indicates inter-mode coding, the decoded sub-band frame data is a decoded version of the differential representation of the sub-band frame portion. In this case, the decoding reconstruction module 803 retrieves the corresponding previously decoded sub-band frame portion from the decoding buffer memory 804. The decoding reconstruction module 803 then adds the corresponding previously decoded sub-band frame portion to the decoded version of the differential representation of the sub-band frame portion. Thus, a decoded version of the sub-band frame portion is obtained, which is then applied to the decoding buffer memory 804 and the inverse wavelet and color transformation module 805.

[0084] Therefore, compared to conventional video decoders, the decoding buffer memory 804 temporarily stores sub-band frames as part of a linearly transformed frame, rather than a fully decoded frame in the spatial domain. Figure 8 In the illustrated video decoder 800, linearly transformed frames are used as references for inter-mode decoding rather than fully decoded frames.

[0085] The inverse wavelet and color transform module 805 applies the inverse of the wavelet transform and the inverse of the reversible color transform to the decoded version of the sub-band frame portion. That is, an inverse linear transform is applied to the decoded version of the sub-band frame portion. By applying the inverse linear transform to the consecutive decoded versions of the sub-band frame portion, the inverse linear transform is applied to the decoded version of the sub-band frame constituting the decoded version of the linearly transformed frame. Then the decoded version of the sub-band frame is obtained. Figure 1 A decoded version of a frame encoded by video encoder 100 is shown.

[0086] Notice

[0087] The embodiments described above in conjunction with the accompanying drawings are presented by way of illustration. The present invention can be implemented in many different ways. To illustrate this, some alternatives are briefly pointed out.

[0088] The present invention can be applied to various types of products or methods related to video compression or video decompression or both.For example, the present invention can be applied in a multifunctional integrated circuit capable of performing at least one other function besides video compression.

[0089] There are many different ways to implement the video encoder and video decoder according to the present invention. Any module in the presented embodiments may be implemented by a circuit that may be dedicated or programmable, or by a suitably programmed processor, or a combination thereof. A computer program may define one or more operations performed by a video encoder, which has been referred to herein. Figure 1-7 Similarly, a computer program may define one or more operations to be performed by a video decoder, which has been described with reference to Figure 8 In this regard, Figure 1 , 4, 7 and 8 can also be viewed, at least in part, as flow charts representing such computer programs, and methods that can be performed by a processor when executing the computer programs. Figure 1 The reversible color transformation module shown can be considered to represent a color transformation step. Similarly, the other modules can be considered to represent steps of the method.

[0090] In the products or methods according to the invention, there are many different ways to implement a linear transformation. In the presented embodiment, the linear transformation comprises a reversible color transformation. In other embodiments, this transformation can be omitted. In the presented embodiment, the linear transformation comprises a wavelet transformation. In other embodiments, the linear transformation can be, for example, a discrete cosine based transformation.

[0091] In the product or method according to the present invention, there are many different ways to implement the encoding of compressed data. In the embodiment presented, this encoding involves GCLI extraction, zero plane removal and truncation. In other embodiments, other entropy coding schemes can be used.

[0092] In the product or method according to the present invention, there are many different ways to implement the reference frame buffer component. In the presented embodiment, the reference frame buffer component applies frame buffer compression. In other embodiments, frame buffer compression can be omitted so that the reference frame buffer component does not need to include a data compression encoder and a data compression decoder.

[0093] For the purpose of inter-mode coding, there are many different ways to provide a representation of a corresponding portion of another linearly transformed frame. In the presented embodiment, the corresponding portion is the portion that has the same position in the previous linearly transformed frame. That is, motion estimation or motion compensation is not used. In other embodiments, motion estimation can be used to identify the corresponding portion. In addition, the corresponding portion used for inter-mode coding does not have to be based on a single previous linearly transformed frame. For example, the corresponding portion can be based on a combination of portions in two or more previous linearly transformed frames.

[0094] Usually, there are many different ways to implement the present invention, and different implementations may have different topologies. In any given topology, a single entity may perform multiple functions, or multiple entities may jointly perform a single function. In this respect, the accompanying drawings are very schematic.

[0095] The above description shows that the embodiments described with reference to the accompanying drawings illustrate the invention, rather than limiting it. The invention can be implemented in a variety of alternative ways within the scope of the appended claims. All changes within the meaning and scope of equivalents of the claims should be included within their scope. Any reference signs in the claims should not be interpreted as limiting the claims. The verb "comprise" in a claim does not exclude the presence of other elements or other steps listed in the claim. The same applies to similar verbs such as "comprise" and "contain". Mentioning an element in the singular in a claim relating to a product does not exclude that the product may include a plurality of such elements. Similarly, mentioning a step in the singular in a claim relating to a method does not exclude that the method may include a plurality of such steps. The fact that the various dependent claims define various additional features does not exclude combinations of additional features other than those reflected in the claims.

Claims

1. An encoder (100), adapted to encode a frame sequence, the encoder comprising: a linear transformation component (101) adapted to apply a linear transformation to the frame sequence frame by frame to obtain a linearly transformed frame sequence having an entropy lower than that of the frames in the frame sequence to be encoded, and An encoding component (102) adapted to encode the sequence of linearly transformed frames to obtain a sequence of encoded frames containing less data than the linearly transformed frames, wherein the encoding component is adapted to encode at least a portion of the linearly transformed frame in inter-frame mode and to encode at least another portion of the same linearly transformed frame in intra-frame mode to obtain an encoded frame, wherein the at least a portion and the at least another portion have different spatial positions within the same linearly transformed frame, Thus, in inter-frame mode, the encoding component provides a differential representation of a portion of a linearly transformed frame, the differential representation corresponding to the difference between a representation of a portion of the linearly transformed frame on the one hand and a corresponding portion of at least one other linearly transformed frame on the other hand, and wherein the encoding component applies data compression to the differential representation of the portion of the linearly transformed frame, Thus, in intra-frame mode, the encoding component applies the data compression directly to the other part of the same linearly transformed frame.

2. The encoder according to claim 1, wherein the encoding component (102) is adapted to evaluate whether the following condition applies to a portion of the linearly transformed frame: when the portion of the linearly transformed frame is encoded in inter-frame mode, an encoded version of the portion of the linearly transformed frame is obtained, which comprises a smaller amount of data than an encoded version obtained by encoding the portion of the linearly transformed frame in intra-frame mode, and wherein The encoding component is adapted to encode the portion of the linearly transformed frame in an inter-mode if the condition applies, and to encode the portion of the linearly transformed frame in an intra-mode if the condition does not apply.

3. Encoder according to claim 1, wherein the encoding component (102) is suitable for obtaining a representation of a corresponding portion of the at least one other linearly transformed frame by applying an inverse operation of the data compression to a corresponding portion of at least one encoded frame, the corresponding portion of the at least one encoded frame being obtained by encoding the at least one other linearly transformed frame.

4. Encoder according to claim 1, wherein the encoding component (102) is adapted to periodically encode individual linearly transformed frame parts in intra-frame mode.

5. Encoder according to claim 4, wherein the encoding component (102) is adapted to periodically encode in intra-mode with a minimum frequency respective linearly transformed frame portions having a similar spatial position, and to periodically encode in intra-mode with the same minimum frequency respective linearly transformed frame portions having another similar spatial position, but dephased with respect to the intra-mode encoding of the first mentioned respective linearly transformed frame portions.

6. Encoder according to claim 4, wherein the encoding component (102) is adapted to periodically encode individual linearly transformed frame portions belonging to a group of linearly transformed frame portions in intra-frame mode with a minimum frequency, and to periodically encode individual linearly transformed frame portions belonging to another group of linearly transformed frame portions in intra-frame mode with a different minimum frequency.

7. Encoder according to claim 6, wherein the linear transformation component (101) is adapted to define respective subbands, wherein the set of linearly transformed frame portions and the further set of linearly transformed frame portions are associated with different subbands.

8. The encoder according to any one of claims 1 to 7, wherein the data compression applied by the encoding component (102) comprises a maximum encoding row index encoding operation.

9. Encoder according to any one of claims 1 to 7, wherein the data compression applied by the encoding component (102) comprises a quantization operation.

10. An encoder according to any one of claims 1 to 7, wherein the encoding component (102) is adapted to decode an encoded frame to obtain a decoded linearly transformed frame, wherein the encoding component is adapted to apply a data storage compression to the decoded linearly transformed frame to obtain a compressed version of the decoded linearly transformed frame, wherein the encoding component is adapted to store the compressed version of the decoded linearly transformed frame in a frame buffer memory (702), and wherein the encoding component is adapted to apply an inverse algorithm of the data storage compression to the compressed version of the decoded linearly transformed frame to obtain a decompressed version of the decoded linearly transformed frame, and obtain a representation of at least a part of the linearly transformed frame from the decompressed version, the representation being used to provide a differential representation of at least a corresponding version of the linearly transformed frame to be encoded in inter-frame mode.

11. The encoder of claim 10, wherein the data storage compression applied by the encoding component (102) to the decoded linear transformed frame comprises a maximum encoded row index encoding operation.

12. A method for encoding a frame sequence, the method comprising: a linear transformation step in which a linear transformation is applied to the frame sequence frame by frame to obtain a linearly transformed frame sequence having an entropy lower than the entropy of the frames in the frame sequence to be encoded, and an encoding step, wherein the linearly transformed frame sequence is encoded to obtain an encoded frame sequence containing less data than the linearly transformed frame, wherein, in the encoding step, at least a portion of the linearly transformed frame is encoded in an inter-frame mode, and at least another portion of the same linearly transformed frame is encoded in an intra-frame mode to obtain an encoded frame, wherein the at least a portion and the at least another portion have different spatial positions within the same linearly transformed frame, Thereby, in inter-mode, a differential representation of at least a portion of said linearly transformed frame encoded in inter-mode is provided, said differential representation corresponding to the difference between a representation of a portion of said linearly transformed frame on the one hand and a corresponding portion of at least one other linearly transformed frame on the other hand, and wherein data compression is applied to the differential representation of a portion of said linearly transformed frame, Thereby, in intra-mode, said data compression is directly applied to said further portion of said same linearly transformed frame.

13. A computer program product for an encoder (100), the computer program product comprising a set of instructions enabling the encoder to perform the method according to claim 12.

14. A decoder (800) adapted to decode a sequence of coded frames provided by an encoder according to any one of claims 1 to 11, the decoder being adapted to decode at least a portion of a coded frame in inter-frame mode to obtain at least one decoded portion of a linearly transformed frame, and to decode at least another portion of the same coded frame in intra-frame mode to obtain at least another decoded portion of the same linearly transformed frame.

Images