Methods for handling transformation coefficients

Abstract

This invention relates to methods, software products, digital cameras, and other image processing systems for processing a set of transform coefficients. The method includes, for each block of transform coefficients representing an image: sorting the coefficients of the block into a sequence; encoding the sorted coefficients to produce a codeword sequence, each codeword including one or more coded coefficients; and dividing the codeword sequence into two or more subsequences.

Description

Technical Field

[0001] This invention relates to methods, systems, and software products for processing transform coefficients, such as discrete cosine transform (DCT) coefficients generated by transforming spatial domain image data to the frequency domain. More specifically, the invention relates to methods, systems, and software products for encoding transform coefficients in a manner that facilitates efficient decoding and storage, particularly of a reduced version of the original spatial domain image data. Methods and systems for decoding image data and generating images therefrom are also disclosed. Background Technology

[0002] Any reference to prior art in this specification is not an admission or implication that such prior art forms part of common general knowledge in any jurisdiction, or that such prior art could reasonably be expected to be understood, considered relevant, and / or combined with other prior art by a person skilled in the art.

[0003] Most professional and consumer digital cameras utilize a single image sensor that captures color images by optically filtering incident light using a color filter array (CFA). The CFA is configured to capture one of three colors of light (typically red, green, and blue) at each of the image sensor's photo points. Because each photo point only has information about a single color, an interpolation (or "de-mosaic") process is needed to provide two "missing" color channels to each photo point. When working with tri-color image acquisition, this process typically triples the amount of image data. Additionally, further processing (including white balance, color transformation, noise reduction, and tone mapping) is usually required before the image can be visually displayed on a monitor.

[0004] Such processing can be performed directly on the camera hardware before compressing the processed data for storage and / or later playback. Various compression techniques have been developed, including the MPEG standard, the H.26X standard, and the "Blackmagic RAW" format developed by the applicant. These techniques are based on dividing the input image into fixed-size spatial domain image data blocks (e.g., N×N pixel blocks, M×N pixel blocks) and transforming the image data of each block into a block of transform coefficients. A common approach involves using a discrete cosine transform (DCT) to generate corresponding frequency domain data blocks (e.g., blocks with N×N or M×N DCT coefficients) with the same number of coefficients as the corresponding spatial domain blocks. After the DCT transform, the data of each frequency domain block is quantized and entropy-coded to produce a series of codewords representing the compressed block's spatial frequency. The codewords of all blocks are concatenated into a codeword sequence representing the entire image, which can be stored or transmitted for subsequent decoding.

[0005] Professional cameras often also allow image data to be stored in its "raw" form (i.e., data from the image sensor that has not yet been de-mosaiced). This allows certain processing decisions (such as the specific de-mosaic algorithm to be used) to be postponed, for example, until post-production. However, even in this case, the same encoding process can be performed.

[0006] Some aspects of the post-production workflow (such as editing and color grading) do not always require images to be displayed at full resolution, as captured by the image sensor. However, because images are always stored in an encoded form (primarily for compressing image data), full-resolution decoding still needs to be performed before the image can be opened and scaled down for viewing at a lower resolution, even when only a lower-resolution image is needed. This problem has become even more pronounced with the high-resolution image capture capabilities of modern digital cameras.

[0007] One method of reduction (i.e., generating lower-resolution images) involves reducing the dimensionality of transform coefficient blocks to below the block size used during encoding. This block size reduction is applied to data that has already undergone at least some form of decompression during the decoding phase. To further decode the data into a displayable form, an inverse transform (e.g., IDCT) is performed on the reduced-size blocks to generate correspondingly reduced-size spatial data blocks. This is in... Figure 1 As shown in the figure, Figure 1 The correspondence between the size of the frequency domain blocks and the spatial data blocks obtained from them after applying the corresponding inverse DCT transform is shown respectively. Figure 1 It is shown that:

[0008] • Generate a full-scale N×N spatial block (2) from the N×N frequency domain block (3);

[0009] • Generate a half-scale (N / 2×N / 2) spatial domain block (4) from the corresponding half-scale N / 2×N / 2 frequency domain block (5); and

[0010] • Generate a quarter-scale (N / 4×N / 4) spatial domain block (6) from the corresponding N / 4×N / 4 frequency domain block (7).

[0011] In each case, scaled-down DCT blocks (5, 7) are shown relative to the full-scale DCT block (3) to show their relative scale.

[0012] U.S. Patent Application No. 20040114812 describes another reduction method in the form of a matrix transformation that directly generates pixel data of the reduced image from a set of transformation coefficients output by the inverse DCT.

[0013] The present invention aims to provide an alternative method for processing transform coefficients, including for storing a downsized version of the original image data and / or generating a downsized version of the original image data prior to DCT transform. Summary of the Invention

[0014] According to a first aspect of the present invention, a method for processing a set of transform coefficients is provided, the method comprising:

[0015] The transform coefficients are divided into multiple blocks; and

[0016] For each block:

[0017] Sort the coefficients of the block into a sequence;

[0018] The sorted coefficient sequence is encoded to produce a codeword sequence, where each codeword includes one or more encoded coefficients; and

[0019] Divide the codeword sequence into two or more subsequences.

[0020] Transform coefficients can be divided into multiple blocks by generating blocks of transform coefficients from corresponding spatial regions of the image, i.e., spatial blocks or tiles defined in the encoded image. By using a method of dividing a codeword sequence (each codeword sequence comprising one or more encoded transform coefficients) into separate subsequences (or “substreams”) of codewords, the present invention (at least in preferred embodiments) allows for the efficient generation of lower-resolution images from transform coefficients that are still in encoded form. Unlike the prior art described above, the present invention does not always require decoding the entire image to produce a lower-resolution image.

[0021] Furthermore, by dividing the codeword sequence into separate subsequences, the original image data can still be encoded in a single pass. In one form, the invention can be implemented using a single entropy encoder without adding any additional computational complexity. The invention also maintains the compression efficiency associated with encoding coefficients into single blocks, rather than encoding separate groups of subblocks.

[0022] Preferably, the number of codewords in each subsequence and the number of coding coefficients contained in each codeword are determined at least in part by the downsizing ratio. In other words, the coefficients required to generate a lower-resolution image at a specific downsizing ratio are allocated to a single subsequence or multiple subsequences. For example, for a 1 / 2 downsizing ratio, approximately 25% of the coefficients can be allocated to the first subsequence, and the remaining 75% of the coefficients can be allocated to the second subsequence, which will not need to be entropy-decoded. Therefore, the downsizing process for generating an image at a downsizing ratio of 1 / 2 or smaller (i.e., 1 / 4, 1 / 8, etc.) does not require decoding the second subsequence.

[0023] In some implementations, the sorting of the coefficient sequence of the blocks is a modified zigzag sort that includes permutation regions and zigzag sorting regions. The use of the modified zigzag sort (zigzag being the standard sorting used in entropy coding) essentially groups the coefficients in a more efficient manner. These coefficients are encoded sequentially according to the order of the zigzag pattern, and the modified zigzag pattern advantageously arranges the transform coefficients into a sequence such that coefficients defining sub-blocks (of transform coefficients) corresponding to the selected reduction ratio are encoded together. This has the advantage that coefficients not needed for generating the reduced image are less likely to be decoded during decoding. In turn, this reduces or eliminates the performance bottlenecks incurred in performing unnecessary decoding operations, such as entropy decoding and inverse DCT operations.

[0024] According to some implementations, each block has a dimension of N×N, and the zigzag sorting region includes the coefficients on the diagonal and below each block.

[0025] According to a second aspect of the present invention, a method is provided for decoding an encoded image file generated using an embodiment of the first aspect of the present invention. The method includes:

[0026] Select image resolution;

[0027] Select one or more subsequences suitable for generating images at the selected image resolution; and

[0028] Decode the selected subsequence.

[0029] According to a third aspect of the present invention, a computer software product is provided, the computer software product comprising instructions that, when executed by a processor, cause the processor to process a set of transformation coefficients by performing an embodiment of the first aspect of the present invention.

[0030] According to a fourth aspect of the present invention, a method for storing image data frames is provided, the method comprising:

[0031] The image data frame is transformed into a set of transform coefficients; the transform coefficients are processed using an embodiment of the first aspect of the invention; and the selected subsequence is stored in the subsequence.

[0032] It can store all subsequences (or fewer than all subsequences).

[0033] Preferably, the subsequences selected for storage are based on the reduction ratio. In other words, only the subsequences containing coefficients that produce a scaled-down image at the desired ratio during decoding are selected for storage. Other subsequences generated during encoding can be discarded.

[0034] The present invention also provides computer software comprising instructions that, when executed by a processor, cause the processor to perform the method according to this aspect of the invention.

[0035] In a fifth aspect of the invention, a method is provided for processing a set of transform coefficients, the transform coefficients being arranged into a plurality of blocks, each block representing a spatial region of an image, and the method comprising:

[0036] For each block:

[0037] Sort the coefficients of the block into a coefficient sequence;

[0038] The sorted coefficient sequence is encoded to produce a codeword sequence, where each codeword includes one or more encoded coefficients; and

[0039] Divide the codeword sequence into two or more codeword subsequences.

[0040] The number of coefficients in each subsequence can be determined at least in part by the reduction ratio.

[0041] Dividing a codeword sequence into two or more codeword subsequences may include: defining a first transform coefficient subblock based on a first reduction ratio; and assigning codewords to the first codeword subsequence until the first codeword subsequence contains all the codewords required to recreate the first transform coefficient subblock by decoding the first codeword subsequence.

[0042] Dividing a codeword sequence into two or more codeword subsequences may also include: defining a second transform coefficient subblock based on a second reduction ratio; and assigning codewords not in the first codeword subsequence to the second codeword subsequence until the combination of the second codeword subsequence and the first codeword subsequence contains all the codewords required to recreate the second transform coefficient subblock by decoding the codeword combination.

[0043] The method may further include defining a first transform coefficient sub-block based on a corresponding reduction ratio, and dividing the codeword sequence into first codeword sub-sequences, wherein the first codeword sub-sequences contain the codewords required for decoding the first transform coefficient sub-block. The method may further include defining a second transform coefficient sub-block based on a corresponding reduction ratio, and dividing the codeword sequence into second codeword sub-sequences, wherein the second codeword sub-sequences contain codewords that are necessary for decoding the second transform coefficient sub-block when combined with codewords contained in the first codeword sub-sequence. The method may also further include defining a third transform coefficient sub-block based on a corresponding reduction ratio; and dividing the codeword sequence into third codeword sub-sequences, wherein the third codeword sub-sequence contains codewords that are necessary for decoding the third transform coefficient sub-block when combined with codewords contained in the first and second codeword sub-sequences. In any of these embodiments, the method may include dividing the codeword sequence into a third codeword subsequence, wherein the third codeword subsequence contains codewords that are necessary for decoding the third transform coefficient subblock when combined with codewords contained in the first and second codeword subsequences. In any of the foregoing embodiments, dividing the codeword sequence into two or more codeword subsequences may further include assigning the remaining codewords corresponding to the block to at least one additional codeword subsequence.

[0044] In all the above embodiments of the fifth aspect of the invention, the transform coefficient block may include an array of M×N transform coefficients, wherein N may be equal to or not equal to M, and at least one codeword subsequence in the codeword subsequence represents a transform coefficient subblock including an array of M / D×N / D transform coefficients, wherein 1 / D is the reduction ratio of the subblock.

[0045] In some implementations, dividing a sequence of codewords into subsequences of two or more codewords includes:

[0046] Create a codeword subsequence containing enough codewords to regenerate a transform coefficient subblock containing M / D x N / D transform coefficients, where 1 / D is the reduction ratio of the subblock.

[0047] Sorting the coefficients of each block into the coefficient sequence is a modified zigzag sort that includes a permutation region and a zigzag sorting region. In this case, if the dimension of each block is NxN, the zigzag sorting region preferably includes the coefficients on the diagonal and below each block.

[0048] The method in any of the above embodiments may further include arranging corresponding subsequences of a plurality of blocks into a common subsequence. In this case, the method may include storing an image file containing one or more of the common subsequences. The method may also include storing the plurality of common subsequences in the image file such that a reduced image can be generated at at least one predetermined reduction ratio corresponding to the respective common subsequences.

[0049] In a sixth aspect of the invention, a method is provided for decoding an encoded image generated using a method according to an embodiment of the fifth aspect of the invention. This method may include:

[0050] Select image resolution;

[0051] Select one or more codeword subsequences suitable for generating an image at the selected image resolution; and

[0052] Decode the selected subsequence.

[0053] For each block representing a spatial region of an image, the method includes: decoding a subsequence fewer than all subsequences to produce a transform coefficient subblock with fewer transform coefficients than the block before encoding.

[0054] In another aspect of the invention, a method is provided for generating a reduced image from coded image data, said coded image data having been generated by a method according to an embodiment of the first or fifth aspect of the invention. The method may include:

[0055] Select image resolution;

[0056] For each block representing a spatial region of the image:

[0057] Decode a codeword subsequence that is less than all codeword subsequences to produce a transform coefficient subblock with fewer transform coefficients than the block before encoding;

[0058] The inverse transform of the transform coefficient sub-blocks is performed using an inverse transform that generates spatial domain image data corresponding to the selected image resolution, and...

[0059] The spatial domain image data of each sub-block is processed to generate a scaled-down image at the selected image resolution.

[0060] A sub-block of transform coefficients can have multiple coefficients corresponding to the selected image resolution.

[0061] In the above embodiments, the method may include reordering the transform coefficient sub-blocks before performing the inverse transform, wherein the reordered transform coefficient sub-blocks are zigzag patterns.

[0062] Another aspect of the present invention relates to a computer software product comprising instructions that, when executed by a processor, cause the processor to perform an embodiment of the methods disclosed herein.

[0063] Another aspect of the present invention relates to a method for storing image data frames, the method comprising:

[0064] Transform the image data frame into a set of transformation coefficients;

[0065] The encoding method implementing the embodiments according to any one of the first or fifth aspects described above; and

[0066] Store one or more codeword subsequences.

[0067] One or more codeword subsequences are preferably stored in a manner that facilitates the retrieval of selected codeword subsequences. However, it should be noted that in some embodiments, all subsequences may be stored to enable full-resolution decoding.

[0068] Unless the context otherwise requires, the term “comprise” and its variations such as “compring”, “comprises”, and “comprised” as used herein are not intended to exclude additional attachments, components, wholes, or steps.

[0069] Other aspects of the invention and other embodiments of the aspects described in the foregoing paragraphs will become apparent from the following description, given by way of example and with reference to the accompanying drawings. Attached Figure Description

[0070] Figure 1 This is a block diagram illustrating a prior art method for image reduction in the frequency domain (DCT domain);

[0071] Figure 2 This is a schematic diagram of a digital camera;

[0072] Figure 3 This is a flowchart of the coding phase of a method according to an embodiment of the present disclosure;

[0073] Figure 4A and Figure 4B A zigzag pattern for exemplary modifications of 8×8 DCT coefficient matrices and 12×12 DCT coefficient matrices that can be used in embodiments of this disclosure is shown.

[0074] Figure 4C and Figure 4DExemplary conventional zigzag patterns for 8×8 DCT coefficient matrices and 12×12 DCT coefficient matrices are shown, and how they can be used in embodiments of this disclosure are illustrated.

[0075] Figure 5 It is a block diagram illustrating various methods for generating subsequences;

[0076] Figure 6A and Figure 6B This is a schematic diagram illustrating the process of encoding a single spatial block and a series of spatial blocks using the methods described herein.

[0077] Figure 7 This is a flowchart of the decoding phase of the method according to an embodiment of the present invention;

[0078] Figure 8A , Figure 8B and Figure 8C These are schematic diagrams illustrating the process of decoding image data using the methods described herein to generate full-resolution, low-resolution, and medium-resolution images; and

[0079] Figure 9 This is a diagram of two transformation coefficient blocks sorted according to an embodiment of the present invention. Detailed Implementation

[0080] Various embodiments of the present invention will now be described with reference to specific examples, but the scope of the present invention should not be considered limited to these examples.

[0081] Figure 2 This is a schematic representation of a camera 10 capable of implementing the method according to this disclosure. However, those skilled in the art will recognize that the method according to this disclosure (including encoding and decoding operations) can be implemented in software running on general-purpose computing hardware and special-purpose computing hardware. For example, the method according to this disclosure can be readily integrated into video and image editing software.

[0082] Camera 10 includes an image capture system 12 configured to convert received light into image data. In this example, image capture system 12 includes an image sensor 12A (e.g., a CCD image sensor chip or a CMOS image sensor chip or the like) and an associated optical filter 12B (e.g., an IR cutoff filter or an optical low-pass filter). An optical system 14, such as a lens, is also provided to form an image on the image sensor. The image sensor typically (if the image sensor is not a monochrome camera) includes a color filter array (CFA) in the form of a spatial array of color filters, which is positioned in front of the image sensor chip to enable the capture of color images. The filters of the CFA are spatially correlated with photographic points of a single image sensor, such that each photographic point has a corresponding spectrally selective filter. In this example, the CFA used is a Bayer CFA with alternating red, green, and blue color filters; however, the techniques of the present invention are not limited to use with any particular optical system or CFA and can also be used with monochrome imaging systems.

[0083] The image data generated by the image capture system 12 includes an array of pixel values ​​corresponding to the light level received at each photographic point of the image sensor 12A. The image data is then passed to an image processing subsystem 18. The image processing subsystem 18 may include one or more data processors, such as an ASIC or FPGA, or a microprocessor with associated software, and is configured to perform a series of image processing tasks. A working memory 20 is provided to allow temporary storage of data or software during image processing and other tasks.

[0084] The image processing subsystem 18 also includes a video encoding system 22. The video encoding system 22 is typically implemented by providing software configured to enable a processor (or FPGA or ASIC) to implement one or more video codecs. This system can be used to encode and compress image data into a desired format using the methods according to this disclosure.

[0085] The image processing subsystem 18 may further include a format conversion system 24 that processes image data into a format (e.g., Y'CbCr) that is better suited for transmission on the video transmission system 26 or easier for downstream processing. The format conversion system 24 is configured to format the image into one of the video transmission formats, thereby enabling transmission before being delivered to the video transmission system 26. This may include transcoding the image data from its original format to an appropriate video transmission format (one of the appropriate video transmission formats) for the video transmission system 26.

[0086] The video transmission system is operable to transmit (and optionally receive) video output data (e.g., output data to an external video recorder or recorder / monitor) via a video interface having at least one video output port. The video interface can be bidirectional and therefore also includes a video input port. As an example, the video interface can be an SDI interface or other similar interface.

[0087] The camera also includes a data storage system in the form of a memory control subsystem 28, configured to control the persistent storage of video data (and any other data) on local non-volatile memory 30. Local memory 30 can use removable memory such as a memory card or a removable hard drive. However, in general, the memory control subsystem 28 is arranged to send control signals to and / or receive control signals from local memory 30 to control the storage and retrieval of video data on memory 30 and also to perform any processing or formatting of the stored data. Memory 30 can be a solid-state drive operating according to the Serial ATA protocol; in this case, the memory control subsystem will operate to control the operation of the SATA drive and manage the reading and writing of data to the SATA drive.

[0088] It should be noted that the various embodiments of the present invention, including aspects relating to the encoding and decoding of image data, can be implemented on a wide variety of devices, such as:

[0089] Computing systems (e.g., laptops, desktop computers, servers, or cloud computing platforms) that run software that processes or displays images or videos, including but not limited to: image / video playback; image / video editing such as non-linear editing systems; image / video color correction; computer games; computer game development; AR and VR systems; and

[0090] Video processing hardware, such as video code converters; cameras; video storage devices; video switchers, etc.

[0091] The method for processing image data according to embodiments of this disclosure is provided by Figure 3 The flowchart is shown. The method begins at step 40, where spatial domain image data is received into an image buffer, such as video coding system 22. The image data will include a series of pixel values ​​representing image frames. Each pixel value will be represented with a given bit depth. The bit depth of the pixel values ​​will determine the accuracy with which the received light level will be represented within the dynamic range of the image sensor.

[0092] Spatial domain image data can be arranged in a wide variety of formats. For example, spatial domain image data can be:

[0093] Arranged in the bitstream, for example, in the order in which the bitstream is read from the image sensor;

[0094] The pixels are separated into color planes, in which pixel values ​​corresponding to pixels with the same CFA filter color (i.e., pixels of the same color) are grouped together.

[0095] Depixelate and arrange into a full-resolution color planar image; and

[0096] The image is processed into one or more luminance and chrominance images, which may or may not conform to a color space definition, such as a member of the YCbCr color space family.

[0097] In this example, the image data arriving at input buffer 40 is in the Blackmagic RAW format described in the applicant's published U.S. Patent Application No. 2018 / 0367775. In the case of Blackmagic RAW, the image data will include image data representing a luminance image at full resolution of image sensor 12A and two lower-resolution chrominance images. Those skilled in the art will understand that the method of the present invention can be performed on image data in other formats, such as other raw formats and RGB or YUV data.

[0098] In step 50, the image data of each of the luminance and chrominance images is transformed (step 50) using Discrete Cosine Transform (DCT). This transformation involves first dividing the luminance image and each of the two chrominance images into blocks of appropriate spatial dimensions (e.g., 8×8). Most commonly, the spatial blocks will have dimensions selected from the following, but other dimensions can also be used:

[0099] 8×8 pixels; 12×12 pixels; 4×8 pixels; and 6×12 pixels.

[0100] The specific block size and shape used can be selected based on a range of factors, such as: the size of the image sensor; the expected frequency components in the image; the processing characteristics of the device for processing the image (e.g., processing and / or memory capacity); processing speed requirements; and the desired image quality.

[0101] Each block is then converted to a frequency domain representation using a suitable transform, such as a DCT. Before calculating the DCT for each block, its values ​​can be shifted to ensure a zero-centered range. The application of the DCT produces a block with an upper-left ingress (or DC coefficient), which typically has a relatively large amplitude defining the block's fundamental intensity level. The remaining coefficients in the block are called AC coefficients.

[0102] The DCT block is then quantized by referring to an appropriate quantization matrix (step 60). The quantization operation divides each DCT coefficient in the block by a specified constant value and rounds the result to the nearest integer value. The quantized coefficient block is typically smaller in magnitude than the coefficients before quantization. Often, many higher-frequency coefficients are rounded to zero (i.e., zero is the integer closest to the input coefficient after dividing the input coefficient by the specified constant value in the quantization matrix).

[0103] After quantization, the quantization coefficients of each block are arranged in a specified order (step 70). According to a preferred embodiment, a modified zigzag sort is used. Exemplary modified zigzag sorts for 8×8 blocks and 12 / 12 blocks are as follows: Figure 4A and Figure 4B As shown. In both figures, the zigzag order starts from the DC coefficient (1) in the top left corner and proceeds through the coefficients in the numerically indicated order. To aid in the visualization of the zigzag order, the path of the order is shown in... Figure 4A and Figure 4B The solid line indicates the center.

[0104] The conventional zigzag pattern (in) Figure 4C The text is incomplete and appears to be a fragment of a larger document. A direct translation isn't possible without further context or clarification. Figure 4D (As shown for a 12×12 block) It begins with the DC coefficient and then proceeds along a series of consecutive adjacent coefficients along a counter-diagonal path. The first diagonal path begins with the coefficient horizontally adjacent to the DC coefficient. When the pattern reaches the edge of the coefficient matrix, it proceeds to the nearest adjacent coefficient located on the edge of the coefficient matrix, then turns along the next diagonal of the coefficients and proceeds with consecutive coefficients.

[0105] Unlike the conventional zigzag pattern ( Figure 4C and Figure 4D The zigzag pattern is different from the modified one. Figure 4A and Figure 4B It has a discontinuity, at which one or both of the following aspects occur:

[0106] The zigzag pattern advances to the non-adjacent coefficient;

[0107] The zigzag pattern changes its direction of travel at coefficients that are not located at the edges of the coefficient matrix.

[0108] In this way, Figure 4A and Figure 4BThe modified zigzag pattern creates a series of sub-blocks. The zigzag pattern appears to traverse a series of squares with a common origin at the DC coefficients. The size of the nested squares increases from 2×2 squares to 4×4 squares, then 8×8 squares, and so on. Each sub-block in this example contains those transformation coefficients between consecutive squares, i.e., all the coefficients within the square, except for those already assigned to earlier sub-blocks. Different numbers of such nested squares can be defined. Therefore, discontinuities in the zigzag pattern can be jumps to start a new square in the zigzag pattern (by...). Figure 4A and Figure 4B The coefficients 4 to 5, 16 to 17 and Figure 4B (The dashed arrows 64 to 65 in the diagram indicate this), or skip the coefficients that appear in earlier squares of the zigzag pattern along the same diagonal path (such as...). Figure 4A and Figure 4B The discontinuity (shown by dashed arrows indicating coefficients 5 to 6, 17 to 18, 20 to 21, etc.) is shown in the figure.

[0109] Other shapes of sub-blocks are also possible, for example: with Figure 5 The sub-block corresponding to subsequence S2 in block 65; in an embodiment where the blocks used to divide the image are rectangular (i.e., the blocks have M×N dimensions, and M≠N, and the sub-blocks preferably also have the same shape).

[0110] As in Figure 4A and 4B As can be seen, the modified zigzag sort includes both the replacement region and the region with the standard zigzag sort. Figure 4A In the case of the 8×8 block shown, the diagonals of the coefficients extending between the bottom left (coefficient 36) and top right (coefficient 30) of the block, as well as the diagonal below, are in a standard zigzag arrangement, where the remainder of the block comprises a permutation region. The permutation region contains nested squares of the coefficients discussed above, and any coefficients surrounding them along the same diagonals.

[0111] In step 80, each coefficient in the ordered coefficient sequence is encoded. Huffman coding is preferred; however, other coding techniques can also be used. Run-length coding can also be applied to the coefficient sequence before Huffman coding. Run-length coding can improve the compression ratio by storing multiple entries with the same value as a count (i.e., the number of times the value repeats) and the value itself. Preferably, the use of run-length coding is limited to zero runs resulting from the quantization of high-frequency AC coefficients. Furthermore, since run-length coding can sometimes increase the amount of data that needs to be stored (i.e., in the case of long runs without repeating values), it can be appropriately omitted from the coding process.

[0112] Huffman coding of an ordered sequence of coefficients involves assigning a unique codeword to each non-zero coefficient in the input. The run of a zero-value coefficient is typically indicated by the codeword of the next non-zero coefficient (or by the codeword at the end of the block). The codewords produced by Huffman coding have variable lengths depending on the probability of each coefficient's occurrence. The resulting code is uniquely decodable and instantaneous (i.e., without a prefix).

[0113] Assigning codewords to each non-zero coefficient in an ordered sequence results in an ordered codeword stream.

[0114] Huffman coding has been found to be appropriately effective for encoding raw image data.

[0115] In step 90, the serial stream of codewords is divided into multiple individual sub-streams by a stream demultiplexer (performed on video coding system 22). The stream demultiplexing function can also be integrated into other functional units of the encoder, such as a Huffman encoder or an output buffer.

[0116] In an alternative implementation, the coefficients are divided into separate substreams by a stream demultiplexer before being encoded. Figure 3 (Steps 80 and 90 in the original text are reversed). According to this implementation, the stream demultiplexer separates coefficients into subsequences that can be processed and / or encoded individually. Such an implementation may require multiple encoders to encode the coefficients of multiple subsequences. However, if shorter subsequences can share an encoder while at least one longer subsequence has a dedicated encoder, the number of encoders may not need to match the total number of subsequences generated by the demultiplexer.

[0117] Typically, a stream demultiplexer can be operated to construct individual subsequences by referring to one or more reduction ratios. Preferably, the reduction ratio is selected to improve decoding efficiency when generating a downsized version of the image.

[0118] Figure 5 The document illustrates three examples of how to map transform coefficients in a block to different subsequences. The three examples are shown below:

[0119] Example 45 defines three subsequences for an image with 8×8 transform coefficient blocks;

[0120] Example 65 defines four subsequences for an image with 8×8 transform coefficient blocks; and

[0121] Example 55 defines four subsequences for an image with a 12×12 transform coefficient block.

[0122] However, in other embodiments of the invention, other block sizes and numbers of subsequences may be used.

[0123] First, let's turn to the case of an 8×8 block (45). In this example, the first subsequence (S0) is encoded to the first DCT coefficients (i.e., DC coefficients) and the number of AC coefficients required to produce a block size corresponding to a predetermined reduction ratio. A 1 / 4 reduction ratio for an 8×8 coefficient block means that a 2×2 coefficient sub-block must be encoded in the subsequence S0, so at least 3 AC coefficients and DC coefficients are required in the subsequence S0.

[0124] The next subsequence (S1) corresponds to a lower reduction ratio (e.g., a 1 / 2 reduction ratio) and contains codewords that encode multiple additional AC DCT coefficients required to generate sub-blocks with dimensions corresponding to the reduction ratio. To generate sub-blocks suitable for producing a 1 / 2 scaled-down image, a 4×4 coefficient sub-block is encoded as S1. In this example, the codewords in S0 encode 12 coefficients, and when combined with the codewords of S0, the required 16 coefficients can be decoded.

[0125] The remaining 48 coefficients were assigned to the third subsequence S2. Figure 5 A subsequence construction method applied to a 12×12 block (55) is also shown. In this case, subsequence S0 encodes enough coefficients to produce a block size corresponding to a 1 / 6 reduction ratio, i.e., a 2×2 DCT coefficient block. S1, when combined with S0, encodes enough coefficients to produce a block size corresponding to a 1 / 3 reduction ratio, which is a 4×4 DCT coefficient block; and S2 encodes enough coefficients combined with S0 and S1 to decode a block size corresponding to a 2 / 3 reduction ratio, i.e., an 8×8 DCT coefficient block. S3 includes all the remaining coefficients required to generate a 12×12 coefficient block so that a full-resolution image can be recreated.

[0126] Figure 5 An alternative subsequence construction method is also shown for generating an 8×8 block (65) of four subsequences. In this example, subsequences S0 and S1 are based on the example in box 45. In this case, subsequence S2 has enough coefficients assigned to it so that combining S0 and S1 can recreate at least all the DCT coefficients above the main diagonal of the 8×8 coefficient block. Therefore, subsequence S2 includes 16 coefficients. Subsequence S3 includes 32 coefficients.

[0127] This subsequence allocation method can advantageously generate subsequences with more even-numbered characters to maximize processing efficiency. This method can double the processing speed in real-time implementations because it guarantees that the largest subsequence (i.e., substream S3) does not have more than half the coefficients of each block.

[0128] Combination Figure 5 The described subsequences are based on DCT coefficients (although they are derived from any suitable transformation); however, in Figure 3 In the implementation method, the subsequence generated by step 90 actually contains a codeword sequence.

[0129] As illustrated below, the use of run-length encoding can mean that earlier subsequences will encode the transform coefficients designated to fall into subsequent subsequences. Therefore, a given subsequence (when combined with all earlier (typically lower frequency components) subsequences) will contain enough data (codewords or DCT coefficients, if encoding has not yet occurred) to recreate a block of transform coefficients of a predetermined size. Preferably, this size corresponds to a specific reduction ratio. For a given block being encoded, the subsequence can be empty if the following encoding scheme is used: the entire contents of the subsequence can be encoded into lower-numbered subsequences without the inefficiency of encoding earlier subsequences, for example, due to run-length encoding, there exists a run long enough to extend across the entire subblock of zero-value coefficients, the subblock will be effectively encoded as codewords appearing in earlier subsequences, and for that block, the subsequence will be practically empty. EOB tags can be assigned to the same subsequence including the last encoded non-zero coefficients. Therefore, empty subsequences or “missing” coefficients will not cause problems during decoding, because decoding later subsequences will always require decoding all earlier subsequences.

[0130] Back Figure 3 The subsequences generated by the stream demultiplexer are stored in the output buffer (step 95). Each subsequence in the subsequence is typically stored in a designated block of non-volatile memory so that the decoder can read and decode only the specific coded subsequences required for specific reduced-resolution decoding. Stream features such as the number of subsequences, start position, address offset, and pointers to coding coefficients, along with metadata such as the DCT and quantization matrices, block size, and other information required during the decoding phase, are also stored in the frame header of the coded block.

[0131] Depending on the combination of run-length encoding, entropy encoding, and the position of the end-of-block (EOB) marker, the illustrated encoding method is well-suited for managing a wide range of reduction ratios while maintaining compression efficiency. Alternatively, run-length encoding can be performed as if using the full block size and standard zigzag order. In this case, the run-length encoded value is almost always included in the subsequence at the start of the run; however, the length can extend into the next higher subsequence, or even span multiple subsequences.

[0132] The efficiency of the encoding method also stems from the fact that it achieves the same compression ratio as when performed on the full image, but achieves measurably faster partial decoding for the reduced resolution.

[0133] supply Figure 6A and Figure 6BTo further aid understanding Figure 3 The methods cover various aspects. First, turn to... Figure 6A This illustrates an exemplary method for operating on a single pixel block of an image. Image 900 is divided into spatial blocks (e.g., 901). It is then transformed to the frequency domain using DCT 905 to generate a block 910 comprising multiple DCT coefficients representing block 901 within image 900. Block 910 has the same size (in coefficients) as block 901 (in pixels). The DCT coefficients are as follows... Figure 3 The coefficients are quantized as shown in step 60. The coefficients are serially read into the encoder, and in step 80, the encoder performs entropy encoding. In step 700, the order in which the serial stream of coefficients is read into the encoding stage is set. Preferably, the coefficients are ordered such that the coefficients within each of the sub-blocks S0, S1, and S2 in the DCT block 900 are consecutive, and most preferably follow a specific order. Figure 4A or Figure 4B The zigzag pattern described herein. Serial encoded stream 915 includes the following codewords (ordered from earliest to latest): ES0, ES1, ES2 (It should be noted that in...) Figure 6A and Figure 6B In this context, the codeword stream is a queue that moves to the right on a first-in, first-out basis, such that the rightmost codeword ( Figure 6A (ES0 in the process is processed first). Step 90 ( Figure 3 The sequence in the stream is passed to demultiplexer 920 (starting from ES0), which splits the serial encoded stream into three subsequence streams: stream 0, stream 1, and stream 2. The codewords of each subsequence can then be buffered individually (followed by the corresponding subsequence codewords from subsequent blocks) and passed in such a way that they can be processed individually as needed for further processing or storage.

[0134] Figure 6B Extend by showing the same processing for several blocks 1000 (block 1...block N). Figure 6A Blocks 1 through N can represent different spatial blocks within the same image, or spatial blocks in different images. For example, blocks processed sequentially can represent blocks from each image in a set of Y, Cb, and Cr component images, or from component images of any other color coding scheme. The coefficients of each DCT coefficient block are arranged into sub-blocks by sorting step 70, such that block i has: coefficients S i A sub-block of 0, which includes a 2×2 block containing the DC coefficient; sub-block S i 1, which has the next 12 coefficients and is in conjunction with block S i The 0-combination completes the 4×4 array; and the remaining sub-blocks complete the 48 coefficients of the 8×8 block. Blocks 1...N are encoded to generate a serially arranged codeword sequence 1010. First, according to... Figure 6AEncode block 1, then block 2, and so on up to block N. Therefore, the serially arranged codeword sequence 1010 comprises the following encoded sub-blocks (ordered from first to last): ES 1 0, ES 1 1. ES 1 2. ES 2 0, ES 2 1. ES 2 2、……、ES i 0, ES i 1. ES i 2. ES N 0, ES N 1. ES N 2, where the superscript indicates the block number, and the last number indicates the sub-block number. As mentioned above, in Figure 6B In the diagram, the codeword sequence is shown as a first-in-first-out queue, which moves toward the multiplexer (i.e., to the right and down), such that the queue (ES) 1 The first codeword in (0) is processed first. Then, the codeword sequence 1010 is demultiplexed and formed into three subsequences (corresponding to three sub-blocks in the ordered coefficients of the block). Subsequence 0 includes the sub-block S0 representing S0. 1 0, S 2 0……S i 0……S N The codewords for each coefficient in the coefficients of 0. In addition to subsequence 0, subsequence 1 also includes the codewords representing the sub-block S1. 1 1. S 2 1……S i 1……S N The codewords necessary for each coefficient in the coefficients of 1. In addition to the subsequences 1 and 0, stream 3 also includes the codewords representing the subblock S2. 1 2. S 2 2……S i 2……S N The codewords required for each coefficient in the coefficients of 2.

[0135] Some embodiments of the present invention are applied to storing lower-resolution images (by storing only selected subsequences), thereby enabling the storage of more images or longer video sequences in the same amount of memory.

[0136] The decoding stage 110 of the method according to the embodiments of this disclosure is performed by... Figure 7 The flowchart is shown. Decoding is performed on the encoded image files generated during the encoding phase, as described herein.

[0137] In step 510, the resolution of the image to be decoded is selected. This selection can be active, whereby the user chooses the resolution of the image to be generated, or passive, in the sense that a downstream process only requires or allows the use of an image at a certain resolution—for example, decoding an image to display on a low-resolution monitor. As discussed above, the image data is encoded and stored in a manner that helps enhance decoding because only the image data necessary to generate an image at the desired resolution is decoded. For example, image data can be decoded at a 1 / 2 scaling ratio, which for an 8×8 block size involves decoding only the smaller dimensional blocks that are measured as 4×4. Similarly, decoding a 12×12 block size at a 1 / 2 scaling ratio involves decoding only the reduced 6×6 block size.

[0138] Typically, any fixed integer ratio of k / N can be used as the reduction ratio, where N is the block size used to encode the image data, and k is an integer ranging from 1 to N.

[0139] After resolution selection, the subsequences required to obtain the selected resolution are accessed (e.g., read from memory) and passed through a stream multiplexer (step 520) to generate a serial stream of codewords from which the image at the selected resolution can be decoded. For example, to decode an image 45 encoded using 8×8 blocks... Figure 5 Subsequences S0 and S1 are selected at a 1 / 2 scale ratio. As mentioned above, these subsequences collectively contain 16 coding coefficients from a 4×4 sub-block.

[0140] Similarly, in order to decode an 8×8 block at a 1 / 4 scale ratio, only the subsequence S0 (which contains four coded symbols in a 2×2 grid) needs to be selected.

[0141] In step 530, entropy decoding is performed on the retrieved subsequence. Decoding Huffman-coded data is typically done word-by-word by referring to the codebook. This is necessary because each unique codeword can only be distinguished from other words in the encoded alphabet by decoding at a separate bit level. Due to these characteristics of the encoding scheme, it is generally not possible to start decoding at an arbitrary position in the encoded bitstream. The encoded bitstream itself does not provide information about where to start decoding or when to move to the next block unless the previous block is fully decoded. Therefore, the decoding start point is typically retrieved from the image metadata, and then the image metadata is decoded sequentially.

[0142] While storing additional information about the encoded stream (in metadata or in the stream itself), such as additional decoding start points, is technically feasible, this tends to increase the size of the encoded bitstream and thus adversely affect compression efficiency.

[0143] Any run-length encoding applied to the data prior to Huffman coding is reversed during the decoding step. Entropy decoding 530 produces a set of DCT blocks with reduced resolution, each DCT block including a quantized DCT coefficient matrix.

[0144] In step 535, if a non-standard zigzag pattern is used during encoding, the quantized DCT coefficients of each block are reordered (i.e., "removed from the zigzag") back to their original order before performing step 70 above.

[0145] Then, inverse quantization is performed on each block of quantized DCT coefficients (step 540). Inverse quantization involves reversing the quantization steps of the encoding stage as much as possible (step 60). Inverse quantization is performed with reference to the same quantization matrix selected for quantization (step 60). The resulting dequantized DCT block is very (but not completely) similar to the original DCT coefficient matrix, i.e., the matrix generated during the DCT step (step 50). The DCT coefficient matrix is, in turn, a representation of the original image file in the frequency domain.

[0146] After decoding, the DCT coefficient matrix undergoes an inverse DCT (step 550). This process first performs a two-dimensional inverse DCT on the matrix, rounds the result to integer values, and shifts the integer values ​​to restore the original data range.

[0147] Following the inverse DCT, the image is reconstructed from the data generated during the inverse DCT (step 560). As described above, the decoded image has reduced resolution at the scale ratio selected in step 510. However, it should be understood that the image is generated by decoding only the required transform coefficients. In other words, coefficients in subsequences that are not needed to generate the image at the selected resolution are not decoded. Even when the image is not represented by subsequences that precisely correspond to the desired reduction ratio, the various embodiments of the present invention can still offer advantages over conventional image decoding. In conventional schemes, the entire image needs to be decoded, while for the embodiments of the present invention, only those codeword subsequences required to decode the desired number (and pattern) of transform coefficients need to be decoded.

[0148] exist Figures 8A to 8C The document provides a series of three examples demonstrating how implementation methods can be used to generate full-resolution images. Figure 8A ); Generate low-resolution images ( Figure 8B ) and generating intermediate resolution images ( Figure 8C ).

[0149] The image being generated is encoded using 8×8 pixel blocks, thus creating N pixel blocks. During encoding, the spatial domain pixel blocks are converted into frequency domain blocks using 8×8 DCT, and the resulting coefficient blocks have their own... Figure 4A The coefficients for the modified zigzag pattern sorting. As mentioned above, Figure 4A The sub-blocks are encoded and arranged into corresponding subsequences of encoded data (stream 0, stream 1, stream 2) as described above, and are shown as a right-shifted codeword queue. The three subsequences of encoded data can be stored together or separately from each other. Furthermore, each subsequence can be received via a different data transmission channel or serially received on a single channel.

[0150] First go to Figure 8A Since a full-resolution image needs to be generated (i.e., the chosen resolution is full resolution), all subsequences of the encoded data (stream 0, stream 1, stream 2) need to be processed. Therefore, method 1100 begins by acquiring (e.g., receiving via a transmission channel, accessing from memory, or reading from a buffer, etc.) all three subsequences, stream 0, stream 1, and stream 2. These codewords are passed (preferably as a queue of codewords, shown in the figure as a right-moving queue, where the rightmost codeword is processed first) to stream multiplexer 1110 to generate a serially arranged sequence 1120 of codewords. Stream multiplexer 1110 preferably outputs a serial sequence of codewords (ordered from first to last) as follows: ES 1 0, ES 1 1. ES 1 2. ES 2 0, ES 2 1. ES 2 2, ...ES i 0, ES i 1. ES i 2. ES N 0, ES N 1. ES N 2, where the superscript indicates the block number, and the last number indicates the sub-block number. Similarly, this sequence... Figures 8A to 8C The queue shown in the middle is moved away from the stream multiplexer, making ES 1 0 is processed first.

[0151] The codeword sequence is then decoded 1130 into multiple DCT coefficient blocks 1140. The coefficient blocks are sized to allow the creation of a full-resolution image, requiring the DCT blocks to match the size of their encoded pixel array. The DCT blocks 1140 contain DCT coefficients in an order selected during encoding. In this example, the DCT blocks 1140 are arranged according to the modified zigzag arrangement 45 of Figure 4. They are then arranged in an order that allows the inverse transform to be applied to them. Next, the inverse DCT 1150 is applied to each block 1140 to create a corresponding 8×8 pixel block 1155. The recreated pixel blocks 1155 can then be assembled and processed into a final full-resolution image 1160.

[0152] Now go to Figure 8B , Figure 8B It shows the relationship with Figure 8A This is a similar diagram to the one shown, but for the case where a low-resolution image is to be recreated. Since a low-resolution image is to be generated, fewer than all subsequences of encoded data need to be processed. In this example, assuming a 1 / 4 resolution image is to be generated, method 1200 begins by acquiring only the subsequence stream 0 (e.g., received via a transport channel, accessed from memory, or read from a buffer, etc.). Codewords can be passed to stream multiplexer 1110 to generate a serial arrangement sequence 1120 of codewords. However, assuming no multiplexing is actually performed in this example, stream multiplexer 1110 can be omitted. The resulting codeword stream 1220 is then decoded 1130 to create a series of DCT blocks 1240, which have... Figure 4A The S0 block is the same size, 2×2. The DCT coefficients are rearranged into a standard order if necessary, and the inverse DCT is applied to each block in block 1240 to create a corresponding 2×2 pixel block 1255. Since the DCT coefficient block 1240 is 1 / 4 the size of the full-resolution block, only 2×2 IDCT needs to be applied to generate each 2×2 pixel sub-image 1255. The recreated pixel blocks 1155 can then be assembled and processed into a final image 1260 at 1 / 4 the resolution of the original image.

[0153] Now go to Figure 8C , Figure 8C It shows the relationship with Figure 8A and Figure 8B This is a similar diagram, but for the case where a medium-resolution image is to be recreated. Since a medium-resolution image is to be generated, fewer than all subsequences of encoded data need to be processed. In this example, assuming a 1 / 2 resolution image is to be generated, method 1300 begins by acquiring only two subsequences, stream 0 and stream 1 (e.g., received via a transport channel, accessed from memory, or read from a buffer, etc.). Codewords can be passed to stream multiplexer 1110 to generate a serial arrangement sequence 1320 of codewords. Stream 2 does not need to be acquired (e.g., read or received) because it is not required for generating a 1 / 2 resolution image. The resulting codeword stream 1320 is then decoded 1130 to create a series of DCT blocks 1340, which have... Figure 4A The combined S0+S1 blocks have the same dimension, i.e., 4×4. Next, inverse DCT is applied to each block in block 1340 to create a corresponding 4×4 pixel block 1355. Since the DCT coefficient block 1340 is half the size of the full-resolution block, only 4×4 IDCT needs to be applied to generate each 4×4 pixel sub-image 1355. The recreated pixel blocks 1355 can then be assembled and processed into a final image 1360 with half the resolution of the original image.

[0154] Figure 4C and Figure 4DAn implementation using a standard zigzag pattern is shown. Figure 4C and 4D In step 70, blocks 400C and 400 are sorted using a standard zigzag pattern, as indicated by the block coefficient numbering and the path shown. These flows are in step 80 and according to... Figure 3 It is encoded and then demultiplexed into several streams by a stream demultiplexer. (Source: [Original Source Name]) Figure 4C Block 400C is divided into three subsequences, as follows:

[0155] Substream S0 represents codewords with coefficients 1 to 5;

[0156] Substream S1 represents codewords with coefficients 6 to 25; and

[0157] Substream S2- represents codewords with coefficients 26 to 64.

[0158] By selecting these subsequences for demultiplexing the encoded data stream, versions of the image can be conveniently decoded at the following resolutions:

[0159] A reduction ratio of 1 / 4 is achieved by only accessing and decoding substream S0;

[0160] A reduction ratio of 1 / 2 of the full resolution is achieved by additionally accessing and decoding substream S1;

[0161] Full resolution is achieved by additionally accessing and decoding substream S2.

[0162] From Figure 4D Block 400D is divided into the following four subsequences:

[0163] Substream S0 represents codewords with coefficients 1 to 5;

[0164] Substream S1- represents codewords with coefficients 6 to 25;

[0165] Substream S2- represents codewords with coefficients from 26 to 104; and

[0166] Substream S3- represents codewords with coefficients 105 to 114.

[0167] By selecting these subsequences for demultiplexing the encoded data stream, versions of the image can be conveniently decoded at the following resolutions:

[0168] A reduction ratio of 1 / 6 of full resolution is achieved by only accessing and decoding substream S0;

[0169] A reduction ratio of 1 / 3 of the full resolution is achieved by additionally accessing and decoding substream S1;

[0170] A 2 / 3 reduction ratio to full resolution is achieved by additionally accessing and decoding substream S2; and

[0171] Full resolution is achieved by additionally accessing and decoding substream S2.

[0172] Although this example uses conventional zigzag sorting for DCT coefficients, by dividing the data into three subsequences that can be accessed and decoded individually, this implementation provides the ability to generate a lower-resolution version of the encoded image more economically than the standard decoding process, which requires accessing and then decoding all encoded streams before downscaling.

[0173] Reference Figure 9 An example of an encoding method according to an embodiment of the present invention is shown. A first block 505 and a second block 520 are shown, each of which is 8×8 in dimension. Each of blocks 505 and 520 includes a DC coefficient 'dc0' in the upper left corner and 63 AC coefficients 'ac1'......'ac63' constituting the remainder of the block. The AC coefficients are ordered in a modified zigzag order as discussed above; that is, the elements in and below the diagonal 525 (ac28-ac35) are in standard zigzag order, and the remaining elements are permuted. Blocks 505 and 520 may represent different spatial blocks in the same image or spatial blocks in different images, such as corresponding spatial blocks in a correlated chroma image and a luminance image.

[0174] The permutation of elements for an 8×8 matrix is ​​performed by Figure 4A The following is a definition of a sorting graph as discussed in [the text].

[0175] As described above, Huffman coding is deployed to encode the coefficients, where each non-zero coefficient is assigned a binary codeword. The run of a zero-valued coefficient is typically indicated by the codeword of the next non-zero coefficient (or by the codeword at the end of the block).

[0176] In the illustrated embodiment, three subsequences are created using the 8×8 block generator 45 shown in Figure 4. The codeword for element ac1 assigned to block 505 is named Y0_ac1_cw, and the codeword for block 520 is named Y1_ac1_cw. The other elements of both blocks 505 and 520 are named in a similar manner. Zero-value padding can be inserted at appropriate points in the output, such as at the end of each block, multiple blocks, or each subsequence.

[0177] The three subsequences can be output in any order.

[0178] With all coefficients non-zero, the contents of each subsequence generated by the stream multiplexer are as follows:

[0179] Substream 0:Y0_dc0_cw,Y0_ac1_cw,Y0_ac2_cw,Y0_ac3_cw,Y1_dc0_cw,Y1_ac1_cw,Y1_ac2_cw,Y1_ac3_cw

[0180] Substream 1:Y0_ac4_cw,Y0_ac5_cw,Y0_ac6_cw,Y0_ac7_cw,Y0_ac8_cw,Y0_ac9_cw,Y0_ac10_cw,Y0_ac11_cw,Y0_ac12_cw,Y0_ac13_cw,Y0_ac14_cw,Y0_ac15_ cw,Y1_ac4_cw,Y1_ac5_cw,Y1_ac6_cw,Y1_ac7_cw,Y1_ac8_cw,Y1_ac9_cw,Y1_ac10_cw,Y1_ac11_cw,Y1_ac12_cw,Y1_ac13_cw,Y1_ac14_cw,Y1_ac15_cw

[0181] Substream 2:Y0_ac16_cw,Y0_ac17_cw,Y0_ac18_cw,…Y0_ac62_cw,Y0_ac63_cw,Y1_ac16_cw,Y1_ac17_cw,Y1_ac18_cw,…Y1_ac62_cw,Y1_ac63_cw

[0182] As mentioned above, subsequences can be output in any order, and therefore, a compressed data stream or file can, for example, include substream 0, then substream 1, then substream 2.

[0183] In this case, for example, if all coefficients except ac2, ac3, ac4, and ac5 in block Y0(505) are non-zero, then according to standard Huffman coding, these zero coefficients are not assigned codewords, but are indicated by the codeword ac6 containing zero run-length information. The data content of each subsequence is as follows:

[0184] Substream 0:Y0_dc0_cw,Y0_ac1_cw,Y0_ac6_cw,Y1_dc0_cw,Y1_ac1_cw,Y1_ac2_cw,Y1_ac3_cw

[0185] Substream 1:Y0_ac7_cw,Y0_ac8_cw,Y0_ac9_cw,Y0_ac10_cw,Y0_ac11_cw,Y0_ac12_cw,Y0_ac13_cw,Y0_ac14_cw,Y0_ac15_cw,Y1_ac4_cw,Y1 _ac5_cw,Y1_ac6_cw,Y1_ac7_cw,Y1_ac8_cw,Y1_ac9_cw,Y1_ac10_cw,Y1_ac11_cw,Y1_ac12_cw,Y1_ac13_cw,Y1_ac14_cw,Y1_ac15_cw

[0186] Substream 2:Y0_ac16_cw,Y0_ac17_cw,Y0_ac18_cw,…Y0_ac62_cw,Y0_ac63_cw,Y1_ac16_cw,Y1_ac17_cw,Y1_ac18_cw,…Y1_ac62_cw,Y1_ac63_cw

[0187] In this case, the codeword Y0_ac6_cw has been "promoted" to substream S0. This has the advantage that during decoding, the decoder can decode the first four coefficients of DCT block Y0 using only the data in substream S0.

[0188] In this case, the compressed data stream or file can be composed of substream S0, then substream S1, and then substream S2.

[0189] Various aspects and embodiments of the present invention are defined in the following clauses.

[0190] Clause 1. A method for processing a set of transform coefficients, said transform coefficients being arranged into a plurality of blocks, each block representing a spatial region of an image; and said method comprising:

[0191] For each block:

[0192] Sort the coefficients of the block into a coefficient sequence;

[0193] The sorted coefficient sequence is encoded to produce a codeword sequence, where each codeword includes one or more encoded coefficients; and

[0194] The codeword sequence is divided into two or more codeword subsequences.

[0195] Clause 2. The method according to Clause 1, wherein the number of coefficients in each subsequence is determined at least in part by the reduction ratio.

[0196] Clause 3. Dividing the codeword sequence into the two or more codeword subsequences according to the method described in Clause 1 or 2 includes:

[0197] A first transformation coefficient sub-block is defined based on a first reduction ratio; and

[0198] Codewords are assigned to the first codeword subsequence until the first codeword subsequence contains all the codewords required to recreate the first transform coefficient subblock by decoding the first codeword subsequence.

[0199] Clause 4. The method according to Clause 3, wherein dividing the codeword sequence into the two or more codeword subsequences further comprises:

[0200] The second transformation coefficient sub-block is defined based on the second reduction ratio; and

[0201] Codewords not in the first codeword subsequence are assigned to the second codeword subsequence until the combination of the second and first codeword subsequences contains all the codewords required to recreate the second transform coefficient subblock by decoding the codeword combination.

[0202] Clause 5. The method according to any one of Clauses 1 or 2, the method further comprising: defining a first transform coefficient subblock based on a corresponding reduction ratio; and dividing the codeword sequence into a first codeword subsequence, wherein the first codeword subsequence contains those codewords necessary for decoding the first transform coefficient subblock.

[0203] Clause 6. The method according to Clause 5, the method further comprising: defining a second transform coefficient sub-block based on a corresponding reduction ratio; and dividing the codeword sequence into a second codeword sub-sequence, wherein the second codeword sub-sequence contains codewords that are necessary for decoding the second transform coefficient sub-block when combined with codewords contained in the first codeword sub-sequence.

[0204] Clause 7. The method according to Clause 6, the method further comprising: defining a third transform coefficient sub-block based on a corresponding reduction ratio; and dividing the codeword sequence into a third codeword sub-sequence, wherein the third codeword sub-sequence contains codewords that are necessary for decoding the third transform coefficient sub-block when combined with codewords contained in the first codeword sub-sequence and the second codeword sub-sequence.

[0205] Clause 8. The method according to any one of Clauses 5 to 7, the method comprising: dividing the codeword sequence into a third codeword subsequence, wherein the third codeword subsequence contains codewords that are necessary for decoding the third transform coefficient subblock when combined with codewords contained in the first codeword subsequence and the second codeword subsequence.

[0206] Clause 9. The method according to any one of Clauses 3 to 8, wherein dividing the codeword sequence into the two or more codeword subsequences further comprises: assigning the remaining codewords corresponding to the block to at least one additional codeword subsequence.

[0207] Clause 10. The method according to any one of Clauses 1 to 9, wherein the transform coefficient block comprises an array of M×N transform coefficients, wherein N may be equal to or not equal to M, and at least one codeword subsequence in the codeword subsequence represents a transform coefficient subblock comprising an array of M / D×N / D transform coefficients, wherein 1 / D is the reduction ratio of the subblock.

[0208] Clause 11. The method according to Clause 10, wherein dividing the codeword sequence into the two or more codeword subsequences comprises:

[0209] Create a codeword subsequence containing enough codewords to regenerate a transform coefficient subblock containing M / D x N / D transform coefficients, where 1 / D is the reduction ratio of the subblock.

[0210] Clause 12. The method according to any of the preceding clauses, wherein sorting the coefficients of each block into the coefficient sequence is a modified zigzag sort that includes a permutation region and a zigzag sorting region.

[0211] Clause 13. The method according to Clause 3, wherein each block has a dimension of NxN, and the zigzag sorting region includes coefficients on the diagonal and below each block.

[0212] Clause 14. The method according to any of the preceding clauses further includes arranging corresponding subsequences of the plurality of blocks into a common subsequence.

[0213] Clause 15. The method according to Clause 14, the method comprising storing an image file containing one or more of the common subsequences.

[0214] Clause 16. The method according to Clause 15, wherein the method includes storing a plurality of common subsequences in the image file such that a reduced image can be generated at at least one predetermined reduction ratio corresponding to the respective common subsequences.

[0215] Clause 17. A method for decoding an encoded image generated using the method according to any one of the preceding clauses, the method comprising:

[0216] Select image resolution;

[0217] Select one or more codeword subsequences suitable for generating an image at the selected image resolution; and

[0218] Decode the selected subsequence.

[0219] Clause 18. The method according to Clause 17, wherein, for each block representing a spatial region of the image, the method comprises:

[0220] Decode a subsequence that is less than all subsequences to produce a transform coefficient subblock with fewer transform coefficients than the block before encoding.

[0221] Clause 19. A method for generating a reduced image from coded image data, said coded image data having been generated by the method according to any one of Clauses 1 to 17;

[0222] The method includes:

[0223] Select image resolution;

[0224] For each block representing a spatial region of the image:

[0225] Decode a codeword subsequence that is less than all codeword subsequences to produce a transform coefficient subblock with fewer transform coefficients than the block before encoding;

[0226] The inverse transform of the transform coefficient sub-block is performed using an inverse transform that generates spatial domain image data corresponding to the selected image resolution; and

[0227] The spatial domain image data of each sub-block is processed to generate a scaled-down image at the selected image resolution.

[0228] Clause 20. The method according to Clause 18 or 19, wherein the transform coefficient subblock has a plurality of coefficients corresponding to the selected image resolution.

[0229] Clause 21. The method according to any one of Clauses 19 or 20, the method comprising reordering the transform coefficient sub-blocks prior to performing the inverse transform, wherein the reordered transform coefficient sub-blocks are a zigzag pattern.

[0230] Clause 22. A computer software product comprising instructions that, when executed by a processor, cause the processor to process a set of transformation coefficients using a method according to any one of Clauses 1 to 16.

[0231] Clause 23. A computer software product comprising instructions that, when executed by a processor, cause the processor to process an image using a method according to any one of Clauses 17 to 21.

[0232] Clause 24. A method for storing image data frames, the method comprising:

[0233] The image data frame is transformed into a set of transformation coefficients;

[0234] The method described in any one of Clauses 1 to 16 shall be enforced; and

[0235] Store one or more codeword subsequences from the codeword subsequences.

[0236] Clause 25. The method according to Clause 24, wherein the one or more codeword subsequences are stored in a manner that facilitates retrieval of a selected codeword subsequence from the codeword subsequences.

[0237] Clause 26. A method for recording moving images, the method comprising:

[0238] A sequence of image data frames captured in a camera equipped with an image processing system and data storage media;

[0239] For multiple image data frames, the image processing system performs the method described in accordance with clause 24 or 25.

[0240] Clause 27. A camera including an image processing system configured to perform the method described in any one of Clauses 1 to 21, 24 or 25.

[0241] Clause 28. An image processing system configured to perform the method described in any one of Clauses 1 to 21, 24 or 25.

[0242] The embodiments described in this invention provide a novel type of storage of compressed image data in the form of entropy-encoded DCT coefficients by creating multiple subsequences. This method can be applied to various DCT block sizes, to different coefficient groupings depending on the desired reduction ratio, and can be combined with different entropy coding methods.

[0243] It should be understood that the invention disclosed and defined in this specification extends to all alternative combinations of two or more individual features mentioned in or becoming apparent from the text or drawings. All these different combinations constitute various alternative aspects of the invention.

Claims

1. A method for processing a set of transform coefficients, said transform coefficients being arranged into a plurality of blocks, each block representing a spatial region of an image; and said method comprising: For each block: Sort the coefficients of the block into a coefficient sequence; The sorted coefficient sequence is encoded to produce a codeword sequence, where each codeword includes one or more coded coefficients; as well as Dividing the codeword sequence into two or more codeword subsequences, wherein the number of coefficients in each subsequence is at least partially determined by a reduction ratio, and wherein dividing the codeword sequence into the two or more codeword subsequences includes: Define a first transform coefficient subblock based on a first reduction ratio; and assign codewords to a first codeword subsequence until the first codeword subsequence contains all the codewords required to recreate the first transform coefficient subblock by decoding the first codeword subsequence.

2. The method according to claim 1, wherein, Dividing the codeword sequence into the two or more codeword subsequences includes: The second transformation coefficient sub-block is defined based on the second reduction ratio; and Codewords not in the first codeword subsequence are assigned to the second codeword subsequence until the combination of the second and first codeword subsequences contains all the codewords required to recreate the second transform coefficient subblock by decoding the codeword combination.

3. The method according to claim 1 or 2, wherein, Dividing the codeword sequence into the two or more codeword subsequences further includes: assigning the remaining codewords corresponding to the block to at least one additional codeword subsequence.

4. The method according to any one of claims 1 to 2, wherein, The transform coefficient block comprises an array of M×N transform coefficients, wherein N may be equal to or not equal to M, and at least one codeword subsequence in the codeword subsequence represents a transform coefficient subblock comprising an array of M / D×N / D transform coefficients, wherein 1 / D is the reduction ratio of the subblock.

5. The method according to any one of claims 1 to 2, wherein, Sorting the coefficients of each block into the coefficient sequence is a modified zigzag sort that includes the permutation region and the zigzag sorting region.

6. The method according to any one of claims 1 to 2, further comprising arranging corresponding subsequences of the plurality of blocks into a common subsequence.

7. The method of claim 6 further comprises storing a plurality of common subsequences in an image file to enable the generation of a reduced image at at least one predetermined reduction ratio corresponding to the respective common subsequences.

8. A method for decoding an encoded image generated using the method according to any one of claims 1 to 2, the method comprising: Select image resolution; Select one or more codeword subsequences suitable for generating an image at the selected image resolution; as well as Decode the selected subsequence.

9. The method according to claim 8, wherein, For each block representing a spatial region of the image, the method includes: Decode a subsequence that is less than all subsequences to produce a transform coefficient subblock with fewer transform coefficients than the block before encoding.

10. A method for generating a reduced image from coded image data, said coded image data representing a set of transform coefficients processed using the method according to any one of claims 1 to 2, wherein, The image data includes a codeword sequence divided into two or more codeword subsequences; the method includes: Select image resolution; For each block representing a spatial region of the image: Decode a codeword subsequence that is less than all codeword subsequences to produce a transform coefficient subblock with fewer transform coefficients than the block before encoding; The inverse transform of the transform coefficient sub-block is performed using an inverse transform that generates spatial domain image data corresponding to the selected image resolution; and The spatial domain image data of each sub-block is processed to generate a scaled-down image at the selected image resolution.

11. The method according to claim 10, wherein, The transform coefficient sub-block has multiple coefficients corresponding to the selected image resolution.

12. The method of claim 10, further comprising reordering the transform coefficient sub-blocks before performing the inverse transform, wherein, The reordered transformation coefficient sub-blocks form a zigzag pattern.

13. A method for storing image data frames, the method comprising: The image data frame is transformed into a set of transformation coefficients; Perform the method according to any one of claims 1 to 2; as well as Store one or more codeword subsequences from the codeword subsequences.

14. The method according to claim 13, wherein, The one or more codeword subsequences are stored in a manner that facilitates the retrieval of a selected codeword subsequence from the codeword subsequences.

15. A method for recording moving images, the method comprising: A sequence of image data frames captured in a camera equipped with an image processing system and data storage media; For multiple image data frames, the image processing system performs the method according to claim 13 or claim 14.

16. A camera, comprising: An image capture system configured to convert received light into image data; An image processing system, the image processing system including one or more data processors configured to process the image data, the image processing system including a video coding system configured to encode and compress the image data; as well as A memory control system configured to control the storage of coded image data; as well as The image processing system is configured to perform the method according to any one of claims 1 to 2.

17. An image processing system, the image processing system comprising one or more data processors configured to process image data, the image processing system comprising a video coding system configured to encode and compress the image data, wherein, The image processing system is configured to perform the method according to any one of claims 1 to 2.

18. A computer software product comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 2.

Images