Encoder and method for encoding a sequence of frames - Patents.com

Abstract

When encoding a series of frames (113), a linear transform (101), such as a wavelet transform (104), is applied to the series of frames frame by frame. At least one portion of the linearly transformed frame is inter-mode coded based on a differential representation of the linearly transformed frame. The differential representation corresponds to the difference between the linearly transformed portion of the frame on the one hand and a representation of a corresponding portion of at least one other linearly transformed frame on the other hand. Data compression (106, 107) is then applied to the differential representation of the linearly transformed portion of the frame.

Description

[Technical Field]

[0001] An aspect of the invention relates to an encoder adapted to encode a sequence of frames. The encoder can be used, for example, to encode a video stream for efficient transmission or storage or both. Further aspects of the invention relate to a method for encoding a sequence of frames, a computer program for the encoder, a decoder, and a method for decoding the encoded sequence of frames. [Background technology]

[0002] U.S. Patent No. 9,332,258 states that digital image compression is typically performed in three steps: de-correlating transform, entropy coding, and rate allocation. The de-correlating transform reduces entropy, i.e., the entropy of the transformed image can be low because the occurrence probability of a small subset of coefficient values ​​increases. Commonly used de-correlating transforms in image compression are color transform, inter / intra prediction, DCT transform, or wavelet transform. Entropy coding reduces the data size of the transformed image, allowing the transformed image to be displayed with less data. Finally, rate allocation selects the data that will be included in the output stream of the compressed image to achieve a desired compression ratio.

[0003] U.S. Patent No. 9,332,258 describes a technique for compressing an input data stream into an output data stream with acceptable compression efficiency while minimizing the complexity of the encoder and decoder. A sequence of m-bit words in the input data stream is grouped into groups of n m-bit words, where n is greater than or equal to 2. For each group, a maximum coding line index (GCLI) value is found. The GCLI is the index of the most significant non-zero bit among the bits of the words in that group, excluding any sign bits. The output data stream contains groups of n GCLI bits corresponding to the n m-bit words in that group in the input stream. The GCLI bit of each word is the least significant GCLI bit of the corresponding word in the input stream and the GCLI value. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,332,258 Summary of the Invention [Problem to be solved by the invention]

[0005] What is needed is an improved solution that can encode a sequence of frames that better meets at least one of the following criteria: high compression, high image quality, and moderate cost. [Means for solving the problem]

[0006] According to an aspect of the present invention as defined in claim 1, there is provided an encoder adapted to encode a sequence of frames, said encoder comprising: a linear transformation assembly adapted to apply a linear transformation frame-by-frame to the sequence of frames to obtain a sequence of linearly transformed frames having less entropy than frames in the sequence of frames to be encoded; a coding assembly adapted to code the sequence of linearly transformed frames to obtain a sequence of coded frames having a smaller amount of data than the linearly transformed frames; Equipped with The encoding assembly is adapted to encode at least one portion of the linearly transformed frame in an inter-mode, where the encoding assembly provides a differential representation of the at least one portion of the linearly transformed frame encoded in the inter-mode, the differential representation corresponding to a difference between, on the one hand, the at least one portion of the linearly transformed frame and, on the other hand, a representation of a corresponding at least one portion of at least one other linearly transformed frame, and the encoding assembly is provided with an encoder that applies data compression to the differential representation of the at least one portion of the linearly transformed frame so as to obtain the at least one portion of the encoded frame.

[0007] According to further aspects of the invention as set forth in claims 12, 13 and 14, there are provided a method for encoding a sequence of frames, a computer program for an encoder and a decoder, respectively.

[0008] The main difference with conventional coding schemes is that in inter mode, a differential representation of at least one portion of a linearly transformed frame is processed in its original form, rather than a differential representation of the frame or its portion. This allows for a less complex implementation in terms of hardware and / or software, resulting in a lower cost and power consumption implementation. Furthermore, by applying inter mode coding, high compression can be achieved without significant degradation of image quality.

[0009] In principle, encoding the difference between linearly transformed frames rather than the difference between the original frames does not adversely affect image quality. This is because the difference operation after the linear transformation is equivalent to the difference operation before the linear transformation. Applying the inverse of the linear transformation to the difference representation obtained by the difference operation after the linear transformation will result in the difference representation obtained by the difference operation before the linear transformation. Conversely, applying a linear transformation to the difference representation obtained by the difference operation before the linear transformation will result in the difference representation obtained by the difference operation after the linear transformation.

[0010] It should be noted that in the above-described encoder, the encoding assembly does not need to perform any further transformations that may be decorrelated. Thus, in such an encoder, the linearly transformed frames do not need to undergo any further transformations that may reduce entropy. Rather, the data compression applied to the differential representation as described above consists in applying entropy coding directly to the differential representation in order to represent the differential representation with less data.

[0011] For illustrative purposes, certain embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which additional features, some of which will become apparent as advantages become apparent, are presented. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of a video encoder. [Figure 2] FIG. 2 is a frequency diagram illustrating the subbands defined by the video encoder. [Figure 3] FIG. 3 is a diagram of an image of a linearly transformed frame generated in a video encoder. [Figure 4] FIG. 4 is a block diagram of a coding mode selection module in a video encoder. [Figure 5] FIG. 5 is a conceptual diagram illustrating GCLI value extraction from a group of four samples in the coding mode selection module. [Figure 6]FIG. 6 is a diagram of an image of a linearly transformed frame showing parts coded in intra mode and other parts coded in inter mode. [Figure 7] FIG. 7 is a block diagram of a reference frame buffer assembly in a video encoder. [Figure 8] FIG. 8 is a block diagram of a video decoder. DETAILED DESCRIPTION OF THE INVENTION

[0013] Figure 1 illustrates a schematic diagram of a video encoder 100. Figure 1 provides a block diagram of the video encoder 100. The video encoder 100 may be included in, for example, a communication device, such as a smartphone, that can capture video and / or store video. The video encoder 100 enables a communication device to efficiently transmit video to another communication device over a communication path that may be provided by a network.

[0014] The video encoder 100 includes a linear transform assembly 101 and an encoding assembly 102. The linear transform assembly 101 may include a reversible color transform module 103 and a wavelet transform module 104. The encoding assembly 102 may include a coding mode selection module 105, a quantization module 106, an entropy coding module 107, a data packaging module 108, a rate allocation module 109, a dequantization module 110, a reconstruction module 111, and a reference frame buffer assembly 112.

[0015] Video encoder 100 basically operates as follows: video encoder 100 receives a video stream 113, which includes a series of frames. In response, video encoder 100 outputs an encoded video stream 114, which may have a significantly lower data rate than the received video stream 113.

[0016] A frame of the received video stream 113 can be considered as a pixel matrix, which includes rows of pixels and columns of pixels. A pixel can include three components that, when combined, convey brightness and color information about the pixel. The three components are represented according to a color coordinate system, such as the color coordinate system known as RGB.

[0017] The reversible color transformation module 103 transforms the three components of a pixel into three components of another color coordinate system. Such a reversible color transformation can reduce entropy. Accordingly, the reversible color transformation module provides a reversibly color-transformed frame, which may have less entropy than the original frame. This is useful for image compression.

[0018] For convenience and simplicity, a reversible color transformed frame is considered to be a pixel matrix, whereby a pixel has only one component, e.g., only the luminance component (Y). That is, a reversible color transformed frame is a single matrix of samples representing only a single component, whereas in practice a reversible color transformed frame can be considered as a set of three matrices of samples, each of which represents a particular component. For example, a reversible color transformed frame may in fact include a matrix of samples representing the luminance (Y) component, another matrix of samples representing a first color component (Cr), and yet another matrix of samples representing a second color component (Cb). These matrices of samples can be processed individually, as described below. The samples can be in the form of binary words representing the values ​​of that component.

[0019] The wavelet transform module 104 applies a wavelet transform to the reversibly color-transformed frame. The wavelet transform requires filtering and subsampling processes, resulting in a set of subband frames that collectively constitute the linearly transformed frame. The subband frames represent the spectral content of the reversibly color-transformed frame in specific spectral bands, referred to as subbands. The wavelet transform performed by the wavelet transform module 104 can be similar to those in existing wavelet-based image coding schemes, such as the image coding scheme known as JPEG-XS.

[0020] 2 shows a conceptual diagram of each of the subbands 201 to 207 defined by the wavelet transform module 104. Each of the subbands 201 to 207 is shown in a two-dimensional frequency diagram, which is divided into horizontal frequencies F H The horizontal axis indicates the frequency F V and a vertical axis representing the subband width. The wavelet transform module 104 thus provides each of these subbands 201-207 with a respective subband frame. A subband frame can be viewed as a matrix of subband coefficients. The size of the matrix of subband coefficients for a reversibly color transformed frame depends on the subsampling factor applied by the wavelet transform module 104 to obtain that subband frame.

[0021] Figure 3 illustrates a schematic representation of each of the subband frames 301-307 that together comprise a linearly transformed frame. Each of the subband frames 301-307 is shown as a diagram of an image. A subband frame is associated with a particular subband as illustrated in Figure 2. Figure 3 illustrates the particular subband with which the subband frame is associated by associating a reference sign with this particular subband.

[0022] The linearly transformed frame includes four relatively small, low-frequency subband frames 301-304 and three relatively large, high-frequency subband frames 305-307. For the three high-frequency subband frames 305-307, the subsampling factor is 2. For the four low-frequency subband frames 301-304, the subsampling factor is 4. Among the four low-frequency subband frames 301-304 is a lowest-frequency subband frame 301, which corresponds to a low-pass filtered, downscaled version of the reversibly color transformed frame. In this sense, downscaling is equivalent to subsampling.

[0023] Subband frames contain both spectral and spatial information, as can be seen in Figure 3. This property is exploited in the video encoder 100 shown in Figure 1 and described in more detail below: specific portions of a subband frame can be associated with similarly positioned portions of the reversibly color transformed frame and the frame to be encoded.

[0024] Referring to FIG. 1, the encoding assembly 102 can encode each subband frame 301-307 individually, except for rate allocation and data packaging when combining compressed subband frames. That is, the coding mode selection module 105, the quantization module 106, the inverse quantization module 110, and the entropy coding module 107 can process each subband frame 301-307 individually in succession. In other embodiments, the encoding assembly 102 can operate in a parallel processing manner. In such an embodiment, the encoding assembly 102 can include multiple modules of the following types: coding mode selection module, quantization module, inverse quantization module, and entropy coding module. That is, one such module for each subband frame can be included in a linearly transformed frame. The encoding of a single subband frame will be described below for simplicity and convenience. Other subband frames are encoded similarly.

[0025] To encode a subband frame, the reference frame buffer assembly 112 contains a representation of a corresponding subband frame relative to another previously encoded frame in the video stream 113. The term corresponding indicates that both such subband frames are associated with the same subband. A representation of a corresponding subband frame relative to a previously encoded frame will hereinafter be referred to as a corresponding previous subband frame for brevity and simplicity. More generally, to encode a linearly transformed frame, the reference frame buffer assembly 112 may contain a representation of another previously encoded linearly transformed frame. Both such linearly transformed frames include the respective subband frames 301-307 shown in FIG. 3.

[0026] The encoding assembly 102 can encode the subband frame by portions. A portion of the subband frame to be encoded can include a group of 32 samples on the same line. Due to the subsampling factor, which in this embodiment can be 2 or 4, such a portion of the subband frame can correspond to 64 blocks of 2 samples and 128 blocks of 4 samples in the frame to be encoded. A portion of the subband frame to be encoded will hereinafter be referred to as a subband frame portion for the sake of brevity and simplicity.

[0027] To encode a subband frame portion, the encoding mode selection module 105 retrieves the corresponding portion of the corresponding previous subband frame from the reference frame buffer assembly 112. In this embodiment, the corresponding portion has a position in the corresponding previous subband frame similar to the position of the subband frame portion to be encoded. The corresponding portion of the corresponding previous subband frame will hereinafter be referred to as the corresponding previous subband frame portion for simplicity and brevity.

[0028] Figure 4 is a more detailed schematic diagram of the coding mode selection module 105. Figure 4 provides a block diagram of the coding mode selection module 105. The coding mode selection module 105 comprises a comparator 401, two greatest coded line index extractors 402, 403, a data buffer 404, a coding mode decision unit 405, and a multiplexer 406. One of the two greatest coded line index extractors will be referred to hereinafter as the inter-mode GCLI extractor 402 for simplicity. The other of the two greatest coded line index extractors will be referred to hereinafter as the intra-mode GCLI extractor 403.

[0029] The coding mode selection module 105 basically operates as follows: a comparator 401 provides a differential representation of the subband frame portion to be coded, the differential representation corresponding to the difference between the subband frame portion to be coded and a corresponding previous subband frame portion, said difference having been retrieved from the frame buffer assembly; a multiplexer 406 receives a differential representation between the subband frame portion to be coded at one input and the subband frame portion itself at another input.

[0030] The coding mode selection module 105 causes the coding assembly 102 to code the subband frame portions in either intra-mode or inter-mode. In intra-mode, the multiplexer 406 passes the subband frame portions present in the linearly transformed frame to the quantization module 106 shown in FIG. 1. Accordingly, in intra-mode, the subband frame portions present in the linearly transformed frame are coded, which requires data compression by the quantization module 106 and the entropy coding module 107. In inter-mode, the multiplexer 406 passes the differential representation of the subband frame portions to the quantizer. Accordingly, in inter-mode, the differential representation of the subband frame portions is coded, which also requires data compression by the quantization module 106 and the entropy coding module 107.

[0031] The main difference with conventional coding schemes is that in inter mode, the differential representation of at least one portion of a linearly transformed frame is processed in its original form rather than as a differential representation of the frame or a portion thereof. Accordingly, the coding scheme according to the present disclosure does not require the application of an inverse linear transform to at least one portion of a linearly transformed frame, whereas conventional coding schemes require the application of an inverse linear transform. Thus, the coding scheme according to the present disclosure allows for implementation with reduced complexity in terms of hardware and / or software, which in turn allows for implementation with lower cost and power consumption.

[0032] In principle, encoding the differences between transformed frames rather than encoding the differences between original frames does not adversely affect image quality. This is because the reversible color transform and wavelet transform described above are both linear. This implies that the difference operations after these transforms are equivalent to the difference operations before these transforms. Applying the inverse of these transforms to the difference representations obtained by performing the difference operations after these transforms will yield the difference representations obtained by the operations before these transforms. Conversely, applying a transform related to the difference representations obtained by performing the difference operations before these transforms will yield the difference representations obtained by performing the difference operations after these transforms.

[0033] The coding mode selection module 105 determines whether to code a subband frame portion in intra mode or inter mode as follows: The inter mode GCLI extractor 402 provides a set of eight GCLI values ​​associated with eight groups of four samples included in the differential representation of the subband frame portion. These GCLI values ​​are hereinafter referred to as inter mode GCLI values ​​for simplicity. The intra mode GCLI extractor 403 also provides a set of eight GCLI values ​​associated with eight groups of four samples included in the subband frame portion to be coded. These GCLI values ​​are hereinafter referred to as intra mode GCLI values ​​for simplicity. The inter mode GCLI extractor 402 and the intra mode GCLI extractor 403 may each operate as described in U.S. Pat. No. 9,332,258. This operation consists in the extraction of GCLIs, which will be briefly outlined below.

[0034] Figure 5 illustrates a schematic of the extraction of a GCLI value from a group of four samples 501-504. Figure 5 provides a conceptual diagram of this extraction, representing the four samples 501-504 in the form of binary words. In this example, the binary words each include a hierarchical set of bits that define a value. Each binary word may further include a sign bit, which is not shown in Figure 5 for simplicity.

[0035] In the layered plurality of bits, each bit has a particular layered rank, denoted M, ... 5, 4, 3, 2, 1. M is an integer value corresponding to the number of bits included in the layered plurality of bits. The bit with the highest layered rank M is usually referred to as the most significant bit. The bit with the lowest layered rank 1 is usually referred to as the least significant bit.

[0036] In a group of four samples, bits with the same hierarchical order constitute a so-called bit plane. Thus, a group of four samples contains as many bit planes as there are bits in the layered plurality of bits. Each bit plane has a hierarchical order corresponding to the hierarchical order of the bits contained in the bit plane. The most significant bit constitutes the most significant bit plane, which has the highest hierarchical order M, while the least significant bit constitutes the least significant bit plane, which has the lowest hierarchical order 1.

[0037] The extraction of the GCLI value consists in determining whether a bitplane contains at least one non-zero bit, starting from the most significant bitplane towards the least significant bitplane. The GCLI value indicates the hierarchical order of this bitplane. The GCLI value may correspond, for example, to an integer value representing the hierarchical order, where the integer value 1 indicates the least significant bitplane and the integer value M indicates the most significant bitplane.

[0038] In the example shown in Figure 5, bit planes M through 5 each contain only zero bits. Such bit planes contain only zero bits and will be referred to hereafter for simplicity as zero bit planes. In the embodiment shown in Figure 5, bit plane 4 is the highest hierarchical bit plane that contains at least one non-zero bit. In this example, the GCLI value can therefore be 4.

[0039] The buffer temporarily stores eight GCLI values ​​provided by the inter-mode GCLI extractor 402 based on a differential representation of the sub-band frame portion to be coded, and also temporarily stores eight GCLI values ​​provided by the intra-mode GCLI extractor based on the sub-band frame portion to be coded.

[0040] The coding mode decision unit 405 calculates the sum of the eight GCLI values ​​provided by the inter-mode GCLI extractor 402. This sum is hereinafter referred to as the inter-mode GCLI sum value. The coding mode decision unit 405 also calculates the sum of the eight GCLI values ​​provided by the intra-mode GCLI extractor 403. This sum is hereinafter referred to as the intra-mode GCLI sum value.

[0041] The intra-mode GCLI sum indicates the data compression factor that can be achieved if the subband frame portion is coded in intra-mode. The inter-mode GCLI sum indicates the data compression factor that can be achieved if the subband frame portion is coded in inter-mode. In fact, the above-mentioned GCLI sums represent estimates of the data compression factor, respectively.

[0042] The coding mode selection module 105 codes the subband frame portion in intra mode if the intra mode GCLI sum value is less than or equal to the inter mode GCLI sum value. To that end, the coding mode decision unit 405 controls the multiplexer 406 so that the multiplexer 406 passes the subband frame portion to be coded to the output of the coding mode selection module 105. Accordingly, in intra mode, the subband frame portion is further processed by the quantization module 106, the entropy coding module 107 and the data packaging module 108.

[0043] The coding mode selection module 105 can code a subband frame portion in intra mode or inter mode if the inter mode GCLI sum value is smaller than the intra mode GCLI sum value. The intra mode is selected when the coding mode decision unit 405 determines that a predetermined number K of corresponding previous subband frame portions have all been coded in inter mode, where K represents an integer value. In that case, the subband frame portion is forced to be coded in intra mode, whereas otherwise it will be coded in inter mode. By doing so, the coding mode decision unit 405 ensures that at least one subband frame portion in a series of K+1 corresponding subband frame portions is coded in intra mode. Thus, K+1, which is greater than 1, can be considered to represent the minimum frequency at which the corresponding subband frame portion is coded in intra mode.

[0044] The coding mode selection module 105 codes a subband frame portion in inter mode if the inter mode GCLI sum value is smaller than the intra mode GCLI sum value and if the corresponding previous subband frame less than K is coded in inter mode. In that case, the coding mode decision unit 405 controls the multiplexer 406 to pass the differential representation of the subband frame portion to be coded to the coding mode selection module 105. Accordingly, in inter mode, the differential representation is further processed by the quantization module 106, the entropy coding module 107 and the data packaging module 108.

[0045] The coding mode selection module 105 therefore evaluates whether the data compression factor indicated by the inter-mode GCLI sum value is greater than the compression factor indicated by the intra-mode GCLI sum value. If the above condition applies, the subband frame portion is in principle coded in inter-mode. The coding mode selection module 105 can nevertheless decide to force coding of the subband frame portion in intra-mode to guarantee a predetermined minimum frequency for which coding in intra-mode is applied. If the above condition does not apply, the subband frame portion is coded in intra-mode.

[0046] The coding mode selection module 105 thus provides several types of output data for the subband frame portions to be coded. The coding mode selection module 105 provides the subband frame data to be compressed. The subband frame data is either the subband frame portion present in the linearly transformed frame or a differential representation of the subband frame portion, depending on whether an intra mode or an inter mode is selected, respectively. Furthermore, the coding mode selection module 105 provides a coding mode flag, which indicates whether the subband frame portion is coded in intra mode, giving an inter mode. The GCLI coding mode selection module 105 also provides a GCLI coding value, which is either an intra mode GCLI value or the above-mentioned inter mode GCLI value, depending on whether the subband frame portion is coded in intra mode or inter mode, respectively.

[0047] Figure 6 shows a schematic representation of the same linearly transformed frame as shown in Figure 3, where a subband frame portion is coded in intra mode and another subband frame portion is coded in inter mode. The subband frame portions coded in intra mode are shown by darker rectangles. The subband frame portions coded in inter mode are shown by lighter rectangles.

[0048] 6 clearly illustrates that the subband frames are coded individually, for example, the first subband frame portion of the lowest subband frame 301 can be coded in inter mode, while the first subband frame portions of the other lower frequency subband frames 302, 303, 304 can be coded in intra mode.

[0049] As described above, the coding mode determination unit 405 ensures that corresponding subband frame portions in a series of linearly transformed frames are periodically coded in intra-mode. This can be regarded as a refresh mechanism, which allows relatively good image quality to be quickly achieved on the decoding side. The video decoder can, so to speak, quickly keep locking onto the coded video stream 114 provided by the video encoder 100 shown in FIG. 1. This also enables more stable video transmission. Even if a portion of the coded video stream 114 is lost due to, for example, a transmission error, the affected frame area can be relatively quickly restored.

[0050] In a series of corresponding subband frames all relating to the same subband, a portion of a subband frame that is forced to be coded in intra-mode may be positioned differently from a portion of a preceding subband frame that was forced to be coded in intra-mode. For example, referring to FIG. 6, assume that a first subband frame portion 301 of a lowest subband frame is forced to be coded in intra-mode. In that case, a second subband frame portion that follows the first subband frame portion in a subsequent lowest subband frame may be forced to be coded in intra-mode. A third subband frame portion that follows the second subband frame portion in a subsequent lowest subband frame may be forced to be coded in intra-mode, and so on. This can be viewed as a shift or phase shift scheme, whereby subband frame portions are periodically coded in intra-mode. Such a scheme facilitates efficient rate allocation and allows a video decoder to keep up with the coded video stream 114.

[0051] The minimum frequency at which subband frame portions are forced to be coded in intra-mode depends on the subband frame to which these subband frame portions belong. Referring to Figures 3 and 6, the minimum frequency may be higher for the low-frequency subband frames 301-304 than for the high-frequency subband frames 305-307. That is, the aforementioned number K+1, which represents the reciprocal of the minimum frequency, may be higher for the high-frequency subband frames 305-307 than for the low-frequency subband frames 301-304. This allows for the rapid acquisition of a low-resolution video representation of acceptable quality at the decoding side and a slightly later high-resolution video representation, without significantly compromising the effectiveness of the compression. For example, in one embodiment, the low-resolution representation is obtained after four frames, while the high-resolution representation is obtained after 32 frames.

[0052] Referring again to FIG. 1 , the quantization module 106 quantizes the subband frame data provided by the coding mode selection module 105. In response, the quantization module 106 provides quantized subband frame data. The quantization module 106 applies a quantization factor when quantizing the subband frame data. When the quantization factor is relatively high, the subband frame data undergoes coarse quantization, which provides relatively high data compression. Conversely, when the quantization factor is relatively low, the subband frame data undergoes fine quantization, which provides relatively low data compression. The quantization factor can be adjusted by the rate allocation module 109 to ensure that the encoded video stream 114 has a desired data rate while optimizing image quality.

[0053] The entropy coding module 107 applies an entropy coding process to the quantized subband frame data. The entropy coding process may include, for example, the process described in U.S. Pat. No. 9,332,258, already mentioned above. In that case, the entropy coding module 107 receives GCLI coded values ​​from the coding mode selection module 105, described above. The GCLI coded values ​​indicate zero bit-planes in the quantized subband frame data. The entropy coding module 107 can then remove these zero bit-planes. In effect, the zero bit-planes are replaced with GCLI coded values ​​that indicate where these bit-planes are located in the subband frame data. The entropy coding module 107 also applies entropy coding to the GCLI coded values ​​so that these values ​​are represented with as little data as possible. U.S. Pat. No. 9,332,258 describes entropy coding techniques that can be used to that effect.

[0054] In addition to removing zero bit-planes, the entropy coding module 107 can also truncate samples in the quantized subband frame data. Such truncation may require removing one or more least significant bit-planes, as also described in U.S. Pat. No. 9,332,258. The rate allocation module 109 adjusts the truncation and determines whether one or more least significant bit-planes should be removed, and if so, how many. The rate allocation module 109 can adjust the truncation to achieve a desired data rate for the coded video stream 114 while optimizing image quality.

[0055] Data packaging module 108 packages several types of data into encoded video stream 114. This includes entropy-coded subband frame data provided by entropy coding module 107. It may also include, for example, entropy-coded GCLI values, coding mode flags, and quantization coefficients applied by quantization module 106.

[0056] The rate allocation module 109 can control the quantization module 106 and the entropy coding module 107 based on the GCLI coding value provided by the coding mode selection module 105. The GCLI coding value can estimate a data compression factor obtained by removing only zero bit-planes. The rate allocation module 109 can compare this data compression factor with a desired data compression factor that represents a desired data rate of the coded video stream 114. If, for example, the difference between the aforementioned data compression factors is relatively large, the rate allocation module 109 can determine whether to apply a relatively large quantization factor, a relatively large truncation, or both.

[0057] As mentioned above, the reference frame buffer assembly 112 may contain a representation of a previously linearly transformed frame that has already been coded. This representation constitutes a reference for inter-mode coding of the subband frame portion of the currently coded linearly transformed frame. The representation of the previously linearly transformed frame is obtained as follows:

[0058] The inverse quantization module 110 applies inverse quantization to the quantized subband frame data provided by the quantization module 106. In response, the inverse quantization module 110 provides dequantized subband frame data, which can be considered as a coarser representation of the subband frame data generated by the coding mode selection module 105. In the case of intra-mode coding, the dequantized subband frame data is a coarser representation of the subband frame portions present in the linearly transformed frame. In the case of inter-mode coding, the dequantized subband frame data is a coarser representation of a differential representation of the subband frame portions.

[0059] When encoding in inter mode, the reconstruction module 111 processes the dequantized subband frame data. The reconstruction module 111 retrieves the corresponding previous subband frame from the reference frame buffer assembly 112. That is, the corresponding previous subband frame is used to generate a differential representation of the subband frame portion based on which the dequantized subband frame data was generated. The reconstruction module 111 then adds the corresponding previous subband frame portion to the dequantized subband frame data. Accordingly, a reconstructed version of the currently encoded subband frame portion is obtained. The reconstruction module 111 applies the reconstructed version of the subband frame portion to the reference frame buffer assembly 112.

[0060] When encoding in intra mode, the reconstruction module 111 passes the dequantized subband frame data directly to the reference frame buffer assembly 112, i.e., in that case, the dequantized subband frame data constitutes a reconstructed version of the subband frame portion.

[0061] Thus, during the encoding process, the reference frame buffer assembly 112 successively receives a plurality of reconstructed subband frame portions that collectively form a reconstructed subband frame. More generally, the reference frame buffer assembly 112 successively receives reconstructed subband frames that collectively form a reconstructed version of a linearly transformed frame. Ideally, the reconstructed version of the linearly transformed frame corresponds to a decoded version of the linearly transformed frame in the video decoder embodiments presented below. That is, generally, the reference for encoding data should ideally correspond to the reference used to decode the same data. Thus, if the reference used to decode data consists of previously decoded data, the reference used to encode the data should be a decoded version of the previously encoded data rather than the data in its original form.

[0062] Figure 7 illustrates a schematic of the reference frame buffer assembly 112. Figure 7 provides a block diagram of the reference frame buffer assembly 112. The reference frame buffer assembly 112 comprises a base entropy encoder 701, a buffer memory 702, and a base entropy decoder 703.

[0063] The reference frame buffer assembly 112 basically operates as follows: A basic entropy coder 701 applies GCLI extraction to a reconstructed version of the subband frame portion. This GCLI extraction may be similar to that described above in connection with FIG. 5. The basic entropy coder 701 removes zero bit-planes in each group of samples. The basic entropy coder 701 may further truncate samples by removing one or more least significant bit-planes. Accordingly, compressed samples are obtained that may lack one or more most significant bits that are zero and one or more truncated least significant bits.

[0064] The basic entropy coder 701 generates a compressed reconstruction of the subband frame portion, which includes compressed samples. The compressed reconstruction may further include GCLI values ​​obtained by GCLI extraction as described herein above in connection with Figures 4 and 5. The compressed reconstruction may include GCLI values ​​in entropy coded form.

[0065] The buffer memory 702 stores compressed reconstructed versions of subband frame portions. Accordingly, during the encoding process, the buffer memory 702 successively stores compressed reconstructed subband frame portions that can collectively form a compressed reconstructed subband frame. More generally, the reference frame buffer assembly 112 successively stores compressed reconstructed subband frames that collectively form a compressed reconstructed version of a linearly transformed frame.

[0066] The base entropy decoder 703 recovers the compressed and reconstructed subband frame portions that may be required to encode the corresponding subsequent subband frame portions in inter mode, thus obtaining the above-mentioned representation of the corresponding previous subband frame.

[0067] The video encoder 100 illustrated in FIG. 1 therefore employs frame buffer compression. Accordingly, a relatively small amount of memory is sufficient to store subband frames, and more generally, linearly transformed frames, as required for encoding in inter mode. Furthermore, frame buffer compression reduces the bandwidth requirements associated with transmitting and receiving data relating to reconstructed subband frame portions to and from the buffer memory 702. This can be a significant advantage in embodiments where the buffer memory 702 is in the form of an external memory circuit, while other entities of the video encoder 100 are integrated into a single processing circuit. In these and other embodiments, the reduced bandwidth requirements generally contribute to relatively lower power consumption.

[0068] However, frame buffer compression may affect image quality to some extent, since the representation of the corresponding prior subband frame portion available for encoding in inter mode is a coarser version of the corresponding prior subband frame portion itself. Nevertheless, in many embodiments, it is possible to configure frame buffer compression such that the loss of image quality is barely noticeable or not noticeable at all. That is, frame buffer compression can be visually lossless. Moreover, in principle, frame buffer compression does not affect the delay between starting decoding and reaching sufficient image quality.

[0069] 8 shows an overview of a video decoder, which is shown in block diagram form. The video decoder is capable of receiving and decoding the coded video stream 114 generated by the video encoder 100 described above in relation to FIGS. 1 to 7. The video decoder includes an unpacking module 801, an entropy decoding and inverse quantization module 802, a decoding reconstruction module 803, a decoding buffer memory 804, and an inverse wavelet transform and color transform module 805.

[0070] The video decoder basically operates as follows: An unpack module 801 performs the inverse of the process performed by the pack module described above. Accordingly, the unpack module 801 retrieves various types of data from the coded video stream 114, such as entropy-coded sub-band frame data, entropy-coded GCLI values, and coding mode flags.

[0071] The entropy decoding and inverse quantization module 802 performs the inverse process of that performed by the entropy encoding module 107 described above. For example, the entropy decoding and inverse quantization module 802 can retrieve GCLI coded values ​​from entropy-coded GCLI values. The entropy decoding and inverse quantization module 802 can then restore the removed zero bit-planes based on these GCLI coded values. The entropy decoding and inverse quantization module 802 can further add dummy bits to replace the least significant bits removed as a result of the truncation. The quantized subband frame data is reconstructed accordingly and then undergoes an inverse quantization process. The inverse quantization process is based on the quantization coefficients packed in the coded video stream 114. The entropy decoding and inverse quantization module 106 accordingly provides decoded subband frame data.

[0072] The decoding and reconstruction module 803 processes the decoded subband frame data based on the coding mode flag. If the coding mode flag indicates intra-mode coding, the decoded subband frame data is a decoded version of a subband frame portion that can be directly used to reconstruct a linearly transformed frame without further processing. In that case, the decoding and reconstruction module 803 applies the decoded version of the subband frame portion to a decoding buffer memory 804 for temporary storage. Furthermore, the decoded version of the subband frame portion is applied to an inverse wavelet transform and color transform module 805.

[0073] If the coding mode flag indicates inter-mode coding, the decoded subband frame data is a decoded version of the differential representation of the subband frame portion. In that case, the decoding reconstruction module 803 retrieves the corresponding previously decoded subband frame portion from the decoding buffer memory 804. The decoding reconstruction module 803 then adds the corresponding previously decoded subband frame portion to the decoded version of the differential representation of the subband frame portion. Accordingly, a decoded version of the subband frame portion is obtained and applied to the decoding buffer memory 804 and the inverse wavelet transform and color transform module 805.

[0074] Thus, contrary to conventional video decoders, the decoding buffer memory 804 temporarily stores subband frames that are portions of linearly transformed frames rather than fully decoded frames in the spatial domain. That is, in the video decoder 800 shown in FIG. 8, the linearly transformed frames serve as references for decoding in inter-mode rather than fully decoded frames.

[0075] The inverse wavelet transform and color transform module 805 applies an inverse wavelet transform and an inverse reversible color transform to the decoded versions of the subband frame portions. That is, an inverse linear transform is applied to the decoded versions of the subband frame portions. By applying the inverse linear transform to successive decoded versions of the subband frame portions, the inverse linear transform is applied to the decoded versions of the subband frames that make up the decoded version of the linearly transformed frame. This results in a decoded version of the frame encoded by the video encoder 100 shown in FIG. 1.

[0076] Things to keep in mind The embodiments described above with reference to the drawings are presented for illustrative purposes. The invention can be embodied in many different ways. To illustrate this, some alternatives are simply shown.

[0077] The present invention is applicable to many types of products or methods related to video compression and / or video decompression, for example, in multi-function integrated circuits that can perform at least one function other than video compression.

[0078] There are numerous and varied ways of implementing a video encoder and video decoder in accordance with the present invention. Any of the modules in the presented embodiments may be implemented by dedicated or programmable electrical circuitry, or by a suitably programmed processor, or by a combination thereof. A computer program may define one or more operations to be performed by a video encoder, such as those described in conjunction with FIGS. 1-7. Similarly, a computer program may define one or more operations to be performed by a video encoder, such as those described in conjunction with FIG. 8. In this regard, the block diagrams of FIGS. 1, 4, 7, and 8 may be considered, at least in part, to represent flowchart diagrams of such computer programs and methods that a processor may perform when executing the computer programs, respectively. For example, the reversible color transformation module illustrated in FIG. 1 may be considered to represent a color transformation step. Similarly, other modules may be considered to represent method steps.

[0079] There are many different ways to implement the linear transform in a product or method according to the present invention. In the embodiment provided, the linear transform comprises a reversible color transform. In other embodiments, this transform can be dispensed with. In the embodiment provided, the linear transform comprises a wavelet transform. In other embodiments, the linear transform may be, for example, a discrete cosine-based transform.

[0080] There are many different ways to implement the encoding of compressed data in a product or method according to the present invention. In the embodiment presented, such encoding requires GCLI extraction, zero bit-plane removal, and truncation. In other embodiments, other entropy coding schemes may be used.

[0081] There are many different ways to implement a reference frame buffer assembly in a product of the present invention or to implement a method of the present invention. In the embodiment presented, the reference frame buffer assembly uses frame buffer compression. In other embodiments, frame buffer compression may be omitted, such that the reference frame buffer assembly does not need to include a video encoder for data compression and a video decoder for data compression.

[0082] There are many different ways to obtain a representation of a corresponding portion of another linearly transformed frame for encoding in inter mode. In the presented embodiment, the corresponding portion is a portion that has the same location in a previously linearly transformed frame; i.e., no motion prediction or motion compensation is used. In other embodiments, motion prediction may be used to identify the corresponding portion. Furthermore, the corresponding portion used for encoding in inter mode does not necessarily have to be based on a single previously linearly transformed frame. For example, the corresponding portion may be based on a combination of portions in two or more previously linearly transformed frames.

[0083] In general, there are many different ways of implementing the invention, and depending on the way, different implementations may have different topologies. In any given topology, a single entity may perform several functions, or several entities may jointly perform a single function. In this respect, the drawings are highly schematic.

[0084] It has been stated above that the embodiments described in connection with the drawings illustrate the invention rather than limit it. The invention can be embodied in numerous alternative ways within the scope of the appended claims. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope. Any reference signs in a claim should not be construed as limiting that claim. The verb "comprise" in a claim does not exclude the presence of other elements or steps than those listed in that claim. The same applies to similar verbs such as "include" and "contain." A reference to a singular element in a product claim does not exclude that the product may include a plurality of such elements. Similarly, a reference to a singular step in a method claim does not exclude that the method may include a plurality of such steps. The mere fact that each dependent claim defines additional features does not exclude combinations of additional features other than those reflected in that claim.

Claims

1. An encoder (100) adapted to encode a sequence of frames, the encoder comprising: a linear transformation assembly (101) adapted to apply a linear transformation to the sequence of frames, frame by frame, to obtain a sequence of linearly transformed frames having less entropy than a frame in the sequence of frames to be coded; a coding assembly (102) adapted to code the sequence of linearly transformed frames to obtain a sequence of coded frames having a smaller amount of data than the linearly transformed frames, the encoding assembly is adapted to encode at least one portion of a linearly transformed frame in an inter mode and to encode at least one other portion of the same linearly transformed frame in an intra mode to obtain an encoded frame, whereby in the inter mode the encoding assembly provides a differential representation of the portion of the linearly transformed frame, the differential representation corresponding to a difference between, on the one hand, the portion of the linearly transformed frame and, on the other hand, a representation of a corresponding portion of at least one other linearly transformed frame, and the encoding assembly applies data compression to the differential representation of the portion of the linearly transformed frame, whereby in the intra mode the encoding assembly applies data compression directly to another portion of the same linearly transformed frame, The encoding assembly is an encoder adapted to cyclically encode each linearly transformed frame portion in the intra mode.

2. 2. The encoder of claim 1, wherein the encoding assembly is adapted to evaluate, for a portion of a linearly transformed frame, whether the following condition is met: if the portion of the linearly transformed frame is coded in the inter mode, a coded version of the portion of the linearly transformed frame with less data than a coded version of the portion of the linearly transformed frame that would be obtained by coding the portion of the linearly transformed frame in the intra mode; and wherein the encoding assembly is adapted to code the portion of the linearly transformed frame in the inter mode if the condition is met, and to code the portion of the linearly transformed frame in the intra mode if the condition is not met.

3. 3. The encoder of claim 1, wherein the encoding assembly (102) is adapted to obtain a representation of the corresponding portion of the at least one other linearly transformed frame by applying an inverse of the data compression to the corresponding portion of the at least one encoded frame obtained by encoding the corresponding portion of the at least one other linearly transformed frame.

4. 4. The encoder of claim 3, wherein the encoding assembly (102) is adapted to cyclically encode in the intra-mode each linearly transformed frame portion having the same position with a minimum frequency, and to cyclically encode in the intra-mode each linearly transformed frame portion having another equivalent position with the same minimum frequency but phase-shifted relative to the intra-mode encoding of each linearly transformed frame portion.

5. 5. The encoder of claim 3, wherein the encoding assembly (102) is adapted to cyclically encode in the intra mode each linearly transformed frame portion belonging to a group of linearly transformed frame portions with a minimum frequency, and to cyclically encode in the intra mode each linearly transformed frame portion belonging to another group of linearly transformed frame portions with a different minimum frequency.

6. 6. The encoder of claim 5, wherein the linear transformation assembly (101) is adapted to define each subband, whereby one group and another group of the linearly transformed frame portions are associated with different subbands.

7. The encoder of any of claims 1 to 6, wherein the data compression applied by the encoding assembly (102) comprises a maximum encoding line index encoding process.

8. The encoder of any of claims 1 to 7, wherein the data compression applied by the encoding assembly (102) comprises a quantization process.

9. 9. The encoder of claim 1, wherein the encoding assembly is adapted to decode an encoded frame to obtain a decoded linearly transformed frame, the encoding assembly is adapted to apply data storage compression to the decoded linearly transformed frame to obtain a compressed version of the decoded linearly transformed frame, the encoding assembly is adapted to store the compressed version of the decoded linearly transformed frame in a frame buffer memory, and the encoding assembly is adapted to apply an inverse of the data storage compression to the compressed version of the decoded linearly transformed frame to obtain a decoded version of the decoded linearly transformed frame from which a representation of at least a portion of a linearly transformed frame is obtained that is used to provide a differential representation of at least a corresponding version of the linearly transformed frame to be encoded in the inter mode.

10. 10. The encoder of claim 9, wherein the data storage compression that the encoding assembly (102) applies to the decoded linear transform frames comprises a maximum encoding line index encoding process.

11. 1. A method of encoding a sequence of frames, the method comprising: a linear transformation step of applying a linear transformation to the series of frames, frame by frame, to obtain a series of linearly transformed frames having lower entropy than a frame in the series of frames to be encoded; an encoding step of encoding the series of linearly transformed frames to obtain a series of encoded frames having a smaller amount of data than the linearly transformed frames; In the encoding step, at least one portion of a linearly transformed frame is encoded in an inter mode while at least one other portion of the same linearly transformed frame is encoded in an intra mode to obtain an encoded frame, whereby in the inter mode a differential representation of the portion of the linearly transformed frame is obtained, the differential representation corresponding to a difference between the portion of the linearly transformed frame on the one hand and a representation of a corresponding portion of at least one other linearly transformed frame on the other hand, and data compression is applied to the differential representation of the portion of the linearly transformed frame, whereby in the intra mode the data compression is applied directly to another portion of the same linearly transformed frame; A coding method in which each linearly transformed frame portion is coded cyclically in said intra mode.

12. A computer program for an encoder (100) comprising a set of instructions enabling the encoder to carry out the method of claim 11.

13. An encoder (100) according to any one of claims 1 to 10, a decoder (800) adapted to decode the sequence of encoded frames obtained by the encoder, 1. A video system, comprising: a decoder adapted to decode at least one portion of a frame coded in the inter mode to obtain at least one decoded portion of a linearly transformed frame; and to decode at least one other portion of the same coded frame in the intra mode to obtain at least one other decoded portion of the same linearly transformed frame.

Images