Image encoding apparatus, image decoding apparatus, image encoding method and image decoding method
Patent Information
- Application Number
- IN202448033311
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-12-17
- Filing Date
- 2024-04-26
- Publication Date
- 2026-10-22
- Estimated Expiration
- 2039-11-14
AI Technical Summary
In HEVC, large orthogonal transformations (e.g., 16 x 16 or 32 x 32 pixels) lead to increased code amounts for quantization matrices, limiting fine control over frequency component quantization, especially for components other than the DC component.
An image encoding apparatus that divides image data into basic blocks, further subdivided into sub-blocks using specific patterns, and employs a holding mechanism for multiple quantization matrices corresponding to these sub-block sizes. It encodes at least one quantization matrix by treating elements within a predetermined range around the DC component as individual values and groups others based on a preset number of elements, reducing the overall code amount.
This approach allows for efficient encoding of images while minimizing the code amount of the quantization matrix, enabling finer control over low-frequency quantization sensitive to human vision, thereby improving compression efficiency without compromising image quality.
Abstract
Description
TECHNICAL FIELD
[0001] The present invention relates to an image encoding apparatus, an imagedecoding apparatus, and control methods and programs therefor.BACKGROUND ART
[0002] As an encoding method of compression recording of a moving image,there is known an HEVC (High Efficiency Video Coding) method (to be referredto as HEVC hereinafter). In HEVC, a basic block of a size larger than aconventional macroblock (16 x 16 pixels) is adopted to improve the encodingefficiency. The basic block of the large size is called a CTU (Coding Tree Unit),and the maximum size is 64 x 64 pixels. The CTU is further divided into sub-blocks as units of prediction or transform.
[0003] In HEVC, processing of weighting, in accordance with a frequencycomponent, a coefficient (to be referred to as an orthogonal transformationcoefficient hereinafter) which is obtained after performing orthogonaltransformation and is called a quantization matrix is used. By reducing highfrequency component data whose deterioration is hardly conspicuous to humanvision, it is possible to improve the compression efficiency while maintainingimage quality. PTL 1 discloses a technique of encoding such quantizationmatrix.CITATION LISTPATENT LITERATURE
[0004] PTL 1: Japanese Patent Laid-Open No. 2013-38758SUMMARY OF INVENTIONTECHNICAL PROBLEM
[0005] In HEVC, orthogonal transformation of, for example, 16 x 16 pixels or32 x 32 pixels, which is larger than conventional orthogonal transformation of 8 x8 pixels, is used, and a quantization matrix corresponding to the large orthogonaltransformation is also used. To prevent the code amount of the quantizationmatrix from increasing, instead of encoding all 256 elements of the quantizationmatrix corresponding to orthogonal transformation of 16 x 16 pixels, only 64elements are encoded and expanded by up-sampling processing on the decodingside. Furthermore, an element at the upper left end position of the quantizationmatrix corresponding to a DC component in a frequency domain is additionallyencoded. Therefore, in HEVC, when using large orthogonal transformation,quantization processing of a frequency component other than a DC componentcannot be controlled finely.
[0006] The present invention has been made to solve the above-describedproblem, and provides a technique of encoding an image while suppressing thecode amount of a quantization matrix used for encoding.SOLUTION TO PROBLEM
[0007] To solve the problem, for example, an image encoding apparatusaccording to the present invention has the following arrangement. That is, thereis provided an image encoding apparatus for encoding image data, characterizedby comprising: first encoding means for dividing encoding target image data into basicblocks of a preset size, further dividing each basic block into at least one sub-block using one of a plurality of preset sub-block division patterns, andperforming frequency transform, quantization, and entropy encoding on a sub3block basis; holding means for holding a plurality of quantization matricescorresponding to sizes of sub-blocks included in the sub-block division patternused by the first encoding means;second encoding means for encoding at least one of the plurality ofquantization matrices held by the holding means; and combining means for combining encoded data generated by the secondencoding means and encoded data obtained by the first encoding means, wherein with respect to the quantization matrix having the same size asthe size of the basic block in one of a horizontal direction and a vertical direction,the holding means holds, as an individual value, an element at a position within apreset predetermined range from a storage position of an element correspondingto a DC component, and holds the same element on a basis of a preset number ofelements at a position outside the predetermined range.ADVANTAGEOUS EFFECTS OF INVENTION
[0008] According to the present invention, it is possible to encode an imagewhile suppressing the code amount of a quantization matrix used for encoding.Other features and advantages of the present invention will be apparent from thefollowing description taken in conjunction with the accompanying drawings.Note that the same reference numerals denote the same or like componentsthroughout the accompanying drawings.BRIEF DESCRIPTION OF DRAWINGS
[0009] The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and, togetherwith the description, serve to explain principles of the invention. Fig. 1 is a block diagram of an image encoding apparatus according tothe first embodiment; Fig. 2 is a block diagram of an image decoding apparatus according tothe second embodiment; Fig. 3 is a flowchart illustrating image encoding processing in the imageencoding apparatus according to the first embodiment; Fig. 4 is a flowchart illustrating image decoding processing in the imagedecoding apparatus according to the second embodiment; Fig. 5 is a block diagram showing an example of the hardwarearrangement of a computer applicable as an image encoding apparatus ordecoding apparatus; Fig. 6A is a view showing an example of a bit stream structure generatedaccording to the first embodiment; Fig. 6B is a view showing another example of the bit stream structuregenerated according to the first embodiment;Fig. 7A is a view showing an example of sub-block division used in theembodiment; Fig. 7B is a view showing another example of sub-block division used inthe embodiment; Fig. 7C is a view showing still another example of sub-block divisionused in the embodiment; Fig. 7D is a view showing still another example of sub-block divisionused in the embodiment; Fig. 7E is a view showing still another example of sub-block divisionused in the embodiment;Fig. 7F is a view showing still another example of sub-block divisionused in the embodiment; Fig. 8A is a view showing an example of a quantization matrix used inthe embodiment; Fig. 8B is a view showing another example of the quantization matrixused in the embodiment; Fig. 8C is a view showing still another example of the quantizationmatrix used in the embodiment; Fig. 8D is a view showing still another example of the quantizationmatrix used in the embodiment; Fig. 9A is a view showing still another example of the quantizationmatrix used in the embodiment;Fig. 9B is a view showing still another example of the quantizationmatrix used in the embodiment; Fig. 10A is a view showing a scan method of the elements of thequantization matrix used in the embodiment; Fig. 10B is a view showing another scan method of the elements of thequantization matrix used in the embodiment; Fig. 10C is a view showing still another scan method of the elements ofthe quantization matrix used in the embodiment; Fig. 10D is a view showing still another scan method of the elements ofthe quantization matrix used in the embodiment;Fig. 11A is a view showing a difference value matrix of the quantizationmatrix generated according to the embodiment; Fig. 11B is a view showing a difference value matrix of the quantizationmatrix generated according to the embodiment; Fig. 11C is a view showing a difference value matrix of the quantizationmatrix generated according to the embodiment; Fig. 11D is a view showing a difference value matrix of the quantizationmatrix generated according to the embodiment; Fig. 12A is a table showing an example of an encoding table used forencoding of the difference values of the quantization matrix; and Fig. 12B is a table showing another example of the encoding table usedfor encoding of the difference values of the quantization matrix.DESCRIPTION OF EMBODIMENTS
[0010] Embodiments of the present invention will be described in detail belowwith reference to the accompanying drawings. Note that the arrangementsaccording to the following embodiments are merely examples, and the presentinvention is not limited to the illustrated arrangements.
[0011] [First Embodiment] Fig. 1 is a block diagram showing an image encoding apparatusaccording to the first embodiment. The image encoding apparatus includes acontrol unit 150 that controls components to be described below and controls theoverall apparatus. The control unit 150 includes a CPU, a ROM storing aprogram executed by the CPU, various parameters, and the like, and a RAM usedas the work area of the CPU. Note that after loading an OS (Operating System)stored in the ROM or another nonvolatile storage device, the program, and theparameters into the RAM, the program may be executed under the control of theOS.
[0012] The image encoding apparatus further includes an input terminal 101, ablock dividing unit 102, a quantization matrix holding unit 103, a prediction unit104, a transforming / quantizing unit 105, an inverse quantizing / inversetransforming unit 106, an image regenerating unit 107, a frame memory 108, anin-loop filtering unit 109, an encoding unit 110, a combined encoding unit 111, anoutput terminal 112, and a quantization matrix encoding unit 113.
[0013] In the above-described arrangement, image data (for example, 30frames / sec) is supplied from an image generation source (not shown) to the inputterminal 101 of this apparatus. Note that any type of image generation sourcecan be used but is simply an image capturing unit, a video camera, or a hard diskor storage medium storing image data to be encoded.
[0014] The block dividing unit 102 divides, into a plurality of basic blocks, theimage data input via the input terminal 101, and supplies the basic block-basedimage data to the prediction unit 104 of the succeeding stage.
[0015] The quantization matrix holding unit 103 holds a plurality of kinds ofquantization matrices to be used in quantization processing. A method ofgenerating the quantization matrices will be described in detail later.
[0016] The prediction unit 104 inputs the basic block-based image data from theblock dividing unit 102, decides a method of dividing the image data into aplurality of smaller sub-blocks, and performs intra prediction as intra-frameprediction or inter prediction as inter-frame prediction for each sub-block, therebygenerating predicted image data. The prediction unit 104 calculates predictionerror data as the difference between the image data of the sub-block and itspredicted image data, and supplies it. The prediction unit 104 outputs, togetherwith the prediction error data, information necessary for prediction, for example,information concerning sub-block division and information of a prediction mode,a motion vector, and the like. The information necessary for prediction will bereferred to as prediction information hereinafter.
[0017] The transforming / quantizing unit 105 obtains a transform coefficient byperforming, for each sub-block, orthogonal transformation (frequency transform)for the prediction error data supplied from the prediction unit 104. Thetransforming / quantizing unit 105 then obtains a quantization coefficient byquantizing the transform coefficient of the sub-block of interest using thequantization matrix held by the quantization matrix holding unit 103.
[0018] The inverse quantizing / inverse transforming unit 106 regenerates thetransform coefficient by inversely quantizing the quantization coefficient inputfrom the transforming / quantizing unit 105 using the quantization matrix held bythe quantization matrix holding unit 103, and also regenerates the prediction errordata by performing inverse orthogonal transformation for the transformcoefficient.
[0019] The image regenerating unit 107 generates predicted image data byreferring to the frame memory 108, as needed, based on the predictioninformation input from the prediction unit 104. Then, the image regeneratingunit 107 generates regenerated image data from the generated predicted imagedata and the prediction error data from the inverse quantizing / inversetransforming unit 106, and outputs it to the frame memory 108 again.
[0020] The in-loop filtering unit 109 performs in-loop filtering processing suchas deblocking filtering or sample adaptive offsetting for the regenerated imagedata obtained from the image regenerating unit 107 and stored in the framememory 108, and stores again, in the frame memory 109, the image data havingundergone the filtering processing.
[0021] The encoding unit 110 encodes the quantization coefficient input fromthe transforming / quantizing unit 105 and the prediction information input fromthe prediction unit 104, generates code data, and outputs it.
[0022] The quantization matrix encoding unit 113 encodes the quantizationmatrix input from the quantization matrix holding unit 103, generates encodeddata of the quantization, and outputs it.
[0023] The combined encoding unit 111 generates header code data using theencoded data of the quantization matrix encoded by the quantization matrixencoding unit 113. Furthermore, the combined encoding unit 111 forms a bitstream by combining the code data of the image input from the encoding unit 110with the generated header code data, and externally outputs it via the outputterminal 112.
[0024] Note that the output destination is not particularly limited but a storagedevice such as a recording medium, a file server on the network, or the like maybe used.
[0025] An image encoding operation in the image encoding apparatus accordingto the embodiment will be described in more detail below.
[0026] This embodiment adopts the arrangement for inputting moving imagedata on a frame basis but may adopt an arrangement for inputting still image datafor one frame. This embodiment will describe only intra prediction encodingprocessing for the sake of descriptive simplicity but the present invention is notlimited to this and is also applicable to inter prediction encoding processing.Furthermore, for the sake of description, this embodiment assumes that the size ofthe basic block based on which the block dividing unit 101 divides an image ofone frame is 16 pixels in the horizontal direction x 16 pixels in the verticaldirection (to be simply referred to as 16 x 16 pixels hereinafter). However, thisis merely an example, and other sizes may be used.
[0027] Before encoding of an image, the quantization matrix is encoded.
[0028] First, the quantization matrix holding unit 103 generates a plurality ofquantization matrices in accordance with sub-blocks of a plurality of sizes usedfor encoding, and holds them. A thick frame denoted by reference numeral 800in Fig. 8A represents an example of a quantization matrix corresponding to a sizeof 8 x 8 pixels, and values in the matrix indicate elements. An element in the 0throw and 0th column at the upper left corner of the quantization matrix 800corresponds the transform coefficient of a DC component (direct currentcomponent), and the remaining elements respectively correspond to the transformcoefficients of AC component (alternating current component). A method ofdeciding each element forming the quantization matrix is not particularly limited.For example, a predetermined initial value may be used or may be setindividually. The quantization matrix may be generated in accordance with thecharacteristic of the image.
[0029] Similarly, the quantization matrix holding unit 103 also generates aquantization matrix corresponding to the size of another sub-block. Fig. 8Bshows a quantization matrix 801 corresponding to orthogonal transformation of 16x 8 pixels according to this embodiment. An individual value "4" is set as anelement in the 0th row and the 0th column corresponding to a DC component, andan individual value "5" is set as an element in the 0th row and the first columnnext to the right of that element. The element in the 0th row and the first columnof the quantization matrix 801 shown in Fig. 8B is an element corresponding tothe transform coefficient of an AC component estimated to have a lowestfrequency in the shape of the sub-block. On the other hand, with respect to theremaining elements, one value is set for two elements adjacent to each other in thehorizontal direction in order to reduce the code amount. For example, the samevalue "9" is set as each of the elements in the 0th row and the second column andthe 0th row and the third column, and the same value "10" is set as each of theelements in the first row and the second column and the first row and the thirdcolumn.
[0030] Fig. 8C shows a quantization matrix 802 corresponding to orthogonaltransformation of 8 x 16 pixels according to this embodiment. An individualvalue "4" is set as an element in the 0th row and the 0th column corresponding toa DC component, and an individual value "5" is set as an element in the first rowand the 0th column immediately under that element. The element in the first rowand the 0th column of the quantization matrix 802 shown in Fig. 8C is an elementcorresponding to the transform coefficient of an AC component estimated to havea lowest frequency in the shape of the sub-block. On the other hand, withrespect to the remaining elements, one value is set for two elements adjacent toeach other in the vertical direction in order to reduce the code amount. Forexample, the same value "9" is set as each of the elements in the second row andthe 0th column and the third row and the 0th column, and the same value "10" isset as each of the elements in the second row and the first column and the thirdrow and the first column.
[0031] Fig. 8D shows a quantization matrix 803 corresponding to orthogonaltransformation of 16 x 16 pixels according to this embodiment. An individualvalue "4" is set as an element in the 0th row and the 0th column corresponding toa DC component, and, as elements corresponding to AC components adjacent tothe DC component, an individual value "5" is set as each of an element in the firstrow and the 0th column next to the right of that element and an element in the 0throw and the first column immediately under that element, and an individual value"6" is set as a lower right element in the first row and the first column. On theother hand, with respect to the remaining elements, one value is set for 2 x 2elements, that is, four elements adjacent to each other in the horizontal andvertical directions in order to reduce the code amount.
[0032] As described above, individual values are respectively set as an elementcorresponding to a DC component in a quantization matrix and elementscorresponding to a plurality of low-frequency components around that element,and with respect to the remaining elements, a value is set on a basis of a pluralityof elements. This can readily reduce the code amount of the quantization matrix,and finely control quantization of a low-frequency portion to which human visionis sensitive.
[0033] The quantization matrix holding unit 103 holds the thus generatedquantization matrices 800 to 803. In this embodiment, the four kinds ofquantization matrices 800 to 803 shown in Figs. 8A to 8D are held in a two-dimensional shape. However, each element in each quantization matrix is notlimited to this, as a matter of course. Furthermore, it is possible to hold aplurality of quantization matrices for orthogonal transformation of the same sizedepending on a prediction method (to be described later), for example, dependingon whether to use intra prediction or inter prediction or whether an encodingtarget is a luminance block or a color difference block. In general, to implementquantization processing corresponding to the visual characteristic of the human,an element in a low-frequency portion corresponding to the upper left portion ofeach of the quantization matrices 800 to 803 has a small value and an element in ahigh-frequency portion corresponding to a lower right portion of each of thequantization matrices 800 to 803 has a large value, as shown in Figs. 8A to 8D.
[0034] The quantization matrix encoding unit 113 sequentially reads out, fromthe quantization matrix holding unit 106, the quantization matrices 800 to 803stored in the two-dimensional shape, scans each element, and calculates thedifference, thereby generating a one-dimensional matrix. In this embodiment,each of the quantization matrices 800 to 803 shown in Figs. 8A to 8D is encoded.More specifically, with respect to the quantization matrix 800 shown in Fig. 8A,the quantization matrix encoding unit 113 calculates, for each element, thedifference between adjacent elements in the scan order using a scan method alongwith arrows shown in Fig. 10A. For example, the quantization matrix 800 for 8x 8 pixels shown in Fig. 8A is scanned by a diagonal scan shown in Fig. 10A.The first element "4" located at the upper left corner is scanned, and the element"9" located immediately under the element "4" is scanned next, therebycalculating +5 as the difference. Furthermore, to encode the first element ("4" inthis embodiment) of the quantization matrix, the difference from a predeterminedinitial value (for example, "8") is calculated. The present invention, however, isnot limited to this, as a matter of course, and the difference from an arbitrary valueor the value of the first element may be used.
[0035] Similarly, with respect to the quantization matrix 801 shown in Fig. 8B,the quantization matrix encoding unit 113 calculates, for each element, thedifference from the immediately preceding element in the scan order using thescan method along arrows shown in Fig. 10B. For example, the quantizationmatrix 801 for 16 x 8 pixels shown in Fig. 8B is scanned by a scan method shownin Fig. 10B. The first element "4" located at the upper left end is scanned, andthe element "5" located immediately next to the right of the element "4" isscanned next, thereby encoding +1 as the difference. Next, the element "9"located immediately under the element "5" is scanned, thereby encoding +4 as thedifference.
[0036] The quantization matrix 802 for 8 x 16 pixels shown in Fig. 8C isscanned by a scan method shown in Fig. 10C. The first element "4" located atthe upper left end is scanned, and the element "5" located immediately under theelement "4" is scanned next, thereby encoding +1 as the difference. Then, theelement "9" located immediately under the element "5" is scanned, therebyencoding +4 as the difference.
[0037] The quantization matrix 803 for 16 x 16 pixels shown in Fig. 8D isscanned by a scan method shown in Fig. 10D. The first element "4" located atthe upper left end is scanned, and the element "5" located immediately under theelement "4" is scanned next, thereby encoding +1 as the difference. Next, theelement "5" located next to the right of the element at the upper left end isscanned, thereby encoding 0 as the difference. Then, the element "6" locatedimmediately under that element "5" is scanned, thereby encoding +1 as thedifference.
[0038] As described above, in this embodiment, with respect to the quantizationmatrices 800 to 803 shown in Figs. 8A to 8D, one-dimensional difference matrices1000 to 1003 shown in Figs. 11A to 11D are generated using the scan methodsshown in Figs. 10A to 10D, respectively. The quantization matrix encoding unit113 further generates quantization matrix code data by encoding the differencematrix. This embodiment assumes that encoding is performed using an encodingtable shown in Fig. 12A. However, the encoding table is not limited to this, andfor example, an encoding table shown in Fig. 12B may be used. The thusgenerated quantization matrix code data is supplied to the combined encoding unit111 of the succeeding stage.
[0039] Referring back to Fig. 1, the combined encoding unit 111 encodes headerinformation necessary for encoding of image data, and combines the code data ofthe quantization matrix. Subsequently, encoding of the image data will bedescribed.
[0040] The block dividing unit 102 divides, into a plurality of basic blocks,image data for one frame input from the input terminal 101, and supplies the basicblock-based image data to the prediction unit 104. As described above, in thisembodiment, the size of the basic block is 16 x 16 pixels.
[0041] The prediction unit 104 executes prediction processing for the basicblock-based image data input from the block dividing unit 102. Morespecifically, the prediction unit 104 decides a sub-block division method of furtherdividing the basic block into sub-blocks, and decides an intra prediction mode ofhorizontal or vertical prediction on a sub-block basis. Note that the sub-blockdivision method is decided from the encoded data amount of an immediatelypreceding frame and the like.
[0042] Figs. 7A to 7D each show an example of a sub-block division pattern.Reference numeral 700 denoting an outer thick frame in each of Figs. 7A to 7Drepresents a basic block which has a size of 16 x 16 pixels in this embodiment.A rectangle within the thick frame represents a sub-block. Fig. 7A shows anexample of a basic block = 1 sub-block. Fig. 7B shows an example ofconventional square sub-block division, in which the basic block of 16 x 16 pixelsis divided into four sub-blocks of a size of 8 x 8 pixels. On the other hand, Figs.7C and 7D each show an example of rectangle sub-block division. In Fig. 7C,the basic block is divided into two vertically long sub-blocks of 8 x 16 pixels. InFig. 7D, the basic block is divided into two horizontally long sub-blocks of 16 x 8pixels.
[0043] In this embodiment, it is decided whether the basic block of 16 x 16pixels is not divided (Fig. 7A), is divided into two vertically long sub-blocks (Fig.7C), is divided into two horizontally long sub-blocks (Fig. 7D), or undergoesquad-tree division (Fig. 7B). However, the sub-block division method is notlimited to them. As shown in Figs. 7E and 7F, ternary-tree division for dividingthe basic block at a ratio of 1 : 2 : 1 may be used.
[0044] Predicted image data is generated from the decided intra prediction modeand encoded pixels. Then, the prediction unit 104 generates, for each sub-bock,prediction error data as the difference between the input image data and thegenerated predicted image data, and supplies the generated prediction error data tothe transforming / quantizing unit 105. Furthermore, information of sub-blockdivision, the intra prediction mode, and the like is supplied as predictioninformation to the encoding unit 110 and the image regenerating unit 107.
[0045] The transforming / quantizing unit 105 performs orthogonaltransformation / quantization for the input prediction error data to generate aquantization coefficient. More specifically, the transforming / quantizing unit 105performs orthogonal transformation processing corresponding to the size of thesub-block to generate an orthogonal transformation coefficient. Then, thetransforming / quantizing unit 105 quantizes, on a sub-block basis, the orthogonaltransformation coefficient using the quantization matrix of the corresponding sizestored in the quantization matrix holding unit 103, thereby generating aquantization coefficient. This embodiment assumes that the quantization matrixshown in Fig. 8D is used for sub-block division of 16 x 16 pixels corresponding toFig. 7A. Similarly, the quantization matrix shown in Fig. 8C is used for sub-block division of 8 x 16 pixels corresponding to Fig. 7C, and the quantizationmatrix shown in Fig. 8B is used for sub-block division of 16 x 8 pixelscorresponding to Fig. 7D. The quantization matrix shown in Fig. 8A is used forsub-block division (quad-tree division), shown in Fig. 7B, of 1 / 2 the size of thebasic block in the vertical and horizontal directions. Thetransforming / quantizing unit 105 supplies the generated quantization coefficient(transform coefficient after quantization) to the encoding unit 110 and the inversequantizing / inverse transforming unit 106.
[0046] The inverse quantizing / inverse transforming unit 106 regenerates, on asub-block basis, the transform coefficient by inversely quantizing the inputquantization coefficient using the corresponding quantization matrix stored in thequantization matrix holding unit 103. Then, the inverse quantizing / inversetransforming unit 106 regenerates, on a sub-block basis, the prediction error databy performing inverse orthogonal transformation for the regenerated transformcoefficient, and supplies the regenerated prediction error data to the imageregenerating unit 107.
[0047] The image regenerating unit 107 regenerates predicted image data on asub-block basis based on the prediction information input from the prediction unit104 by referring to the frame memory 108, as needed. The image regeneratingunit 107 regenerates, on a sub-block basis, the image data of the sub-block byadding the regenerated predicted image data and the prediction error data inputfrom the inverse quantizing / inverse transforming unit 106, and stores theregenerated image data in the frame memory 108.
[0048] The in-loop filtering unit 109 reads out the regenerated image data fromthe frame memory 108 on a sub-block basis, and performs in-loop filteringprocessing such as deblocking filtering. Then, the in-loop filtering unit 109stores again, in the frame memory 108, the image data having undergone thefiltering processing.
[0049] The encoding unit 110 performs, on a sub-block basis, entropy encodingfor the quantization coefficient generated by the transforming / quantizing unit 105and the prediction information input from the prediction unit 104, therebygenerating code data. Any entropy encoding method can be used, and Golombcoding, arithmetic coding, Huffman coding, or the like can be used. Thegenerated code data is supplied to the combined encoding unit 111.
[0050] The combined encoding unit 111 forms a bit stream by multiplexing thecode data input from the encoding unit 110 and the like together with the above-described header code data. Finally, the bit stream is externally output from theterminal 112.
[0051] Fig. 6A shows an example of the data structure of the output bit steamaccording to the first embodiment. A sequence header includes the code datacorresponding to each of the quantization matrices 800 to 803 shown in Fig. 8A to8D, which is formed by the encoding result of each element. However, anencoding position is not limited to this, and may be in a picture header portion oranother header portion, as a matter of course. If the quantization matrix ischanged in one sequence, the quantization matrix can be updated by newlyencoding it. At this time, the entire quantization matrix may be rewritten or aportion of the quantization matrix to be rewritten may be changed by designatinga transform block size.
[0052] The processing procedure of encoding processing of one frame in theimage encoding apparatus according to the first embodiment will be describedbelow with reference to a flowchart illustrated in Fig. 3. Note that in thefollowing description, in each step, the main subject executes each process underthe control of the control unit 150.
[0053] Before encoding of an image, in step S301, the quantization matrixholding unit 103 generates a plurality of kinds of quantization matrices and holdsthem. This embodiment assumes that the quantization matrix holding unit 103generates the four kinds of quantization matrices 800 to 803 shown in Figs. 8A to8D, and holds them.
[0054] In step S302, the quantization matrix encoding unit 113 calculates thedifference of each element by scanning the quantization matrix generated and heldin step S301, and then generates a one-dimensional difference matrix. In thisembodiment, the quantization matrix encoding unit 113 obtains the differencevalue between two successive elements along the scan order shown in each ofFigs. 10A to 10D by scanning each of the quantization matrices 800 to 803 shownin Figs. 8A to 8D, thereby generating a one-dimensional matrix of the differencevalues shown in each of Figs. 11A to 11D. Then, the quantization matrixencoding unit 113 generates binary codes from the individual difference values inthe generated matrix with reference to the encoding table shown in Fig. 12A (orFig. 12B), thereby generating encoded data.
[0055] In step S303, the combined encoding unit 111 generates headerinformation necessary for encoding of the image data together with the generatedquantization matrix code data, and outputs the header information.
[0056] In step S304, the block dividing unit 102 divides the frame-based inputimage on a basic block basis. In step S305, the prediction unit 104 inputs imagedata of one basic block generated in step S304, and performs transform into oneor more sub-blocks. Then, the prediction unit 104 executes predictionprocessing for each sub-block, and generates predicted image data and predictioninformation such as sub-block division information and an intra prediction mode.Then, the prediction unit 104 calculates prediction error data from the image dataof each sub-block and the predicted image data.
[0057] In step S306, the transforming / quantizing unit 105 performs orthogonaltransformation for the prediction error data of each sub-block calculated in stepS305 to generate a transform coefficient. Furthermore, thetransforming / quantizing unit 105 quantizes the prediction error data of each sub-block using the quantization matrix generated and held in step S301 to generate aquantization coefficient. In this embodiment, the quantization matrix 803 shownin Fig. 8D is used for each sub-block obtained by sub-block divisioncorresponding to Fig. 7A. Similarly, the quantization matrix 802 shown in Fig.8C is used for each sub-block obtained by sub-block division shown in Fig. 7C.The quantization matrix 801 shown in Fig. 8B is used for each sub-block obtainedby sub-block division shown in Fig. 7D. Furthermore, the quantization matrix800 shown in Fig. 8A is used for four sub-blocks obtained by quad-tree division,as shown in Fig. 7B.
[0058] In step S307, the inverse quantizing / inverse transforming unit 106performs, using the quantization matrix generated and held in step S301, inversequantization for the quantization coefficient of each sub-block generated in stepS305, thereby regenerating the transform coefficient. The inversequantizing / inverse transforming unit 106 also performs inverse orthogonaltransformation for the transform coefficient of each sub-block, therebyregenerating the prediction error data for each sub-block.
[0059] In step S308, the image regenerating unit 107 regenerates the predictedimage data for each sub-block based on the prediction information generated instep S305. The image regenerating unit 107 regenerates the image data for eachsub-block by adding the prediction error data of the corresponding sub-blockgenerated in step S307 to the generated predicted image data of each sub-block.
[0060] In step S309, the encoding unit 110 encodes, for each sub-block, theprediction information generated in step S305 and the quantization coefficientgenerated in step S306, thereby generating code data. Furthermore, a bit streamis generated by including other code data.
[0061] In step S310, the control unit 150 determines whether encoding of the allthe basic blocks in the frame has ended. If encoding has ended, the control unit150 advances to step S311; otherwise, the control unit 150 returns the process tostep S305 to encode the next basic block.
[0062] In step S311, the in-loop filtering unit 109 performs in-loop filteringprocessing for the image data regenerated in step S308 to generate an imagehaving undergone the filtering processing, thereby ending the processing.
[0063] In the above-described processing, when encoding the second frame or asubsequent frame of the moving image, steps S304 to S311 are repeated until thefinal frame to be encoded is input.
[0064] With the above arrangement and operation, it is possible to reduce thecode amount of the quantization matrix by individually encoding, in step S302, apredetermined number of elements (an element for a DC component andelements, adjacent to the element, for relatively low-frequency components) of thequantization matrix and encoding the remaining elements on a basis of a pluralityof elements. Furthermore, it is possible to finely control quantization of a low-frequency portion to which human vision is sensitive.
[0065] Note that this embodiment adopts the arrangement in which all the fourkinds of quantization matrices 800 to 803 shown in Figs. 8A to 8D are encoded.However, an arrangement in which the quantization matrix is generated usinganother quantization matrix in a decoding side apparatus, instead of individuallyencoding the four kinds of quantization matrices, may be adopted. For example,in an arrangement for generating a quantization matrix of 16 x 8 pixels using anencoded quantization matrix of 16 x 16 pixels instead of encoding thequantization matrix of 16 x 8 pixels, the decoding side apparatus generates thequantization matrix 801 shown in Fig. 8B by thinning out some elements from thequantization matrix 803 shown in Fig. 8D. Similarly, the decoding sideapparatus can also generate the quantization matrix 802 shown in Fig. 8C bythinning out some elements from the quantization matrix 803 shown in Fig. 8D.Furthermore, by deciding, between the encoding side apparatus and the decodingside apparatus, a method of calculating the element "5" in the 0th row and the firstcolumn in Fig. 8B or the element "5" in the first row and the 0th column in Fig.8C, it is possible to generate the quantization matrix 801 shown in Fig. 8B or thequantization matrix 802 shown in Fig. 8C using the quantization matrix 800shown in Fig. 8A. The calculation method at this time is not particularly limited.Calculation may be performed by adding a specific difference value (+1 in thisembodiment) to the element "4" in the 0th row and the 0th column or by usinginterpolation with peripheral elements. Note that in Fig. 8C, +1 is added to theelement corresponding to the DC component with respect to the lower and rightelements of the element corresponding to the DC component of the transformcoefficient, and +2 is added to the element corresponding to the DC componentwith respect to the lower right element.
[0066] Furthermore, the element "5" in the 0th row and the first column in Fig.8B or the element "5" in the first row and the 0th column in Fig. 8C may beencoded together with the elements of the quantization matrix of 16 x 16 pixels.
[0067] This can encode the bit stream with the reduced code amount of thequantization matrix.
[0068] Furthermore, an arrangement for selecting, for each quantization matrix,whether to generate the quantization matrix from another quantization matrix orencode each element and encoding an identifier in a header may be adopted. Forexample, an arrangement for encoding, in a header portion, as a quantizationmatrix encoding method information code, information indicating whether togenerate each element of a quantization matrix from another quantization matrixor individually encode elements, and generating a bit stream shown in Fig. 6Bmay be adopted. This can selectively generate a bit stream which prioritizesimage quality control corresponding to the size of the sub-block or a bit streamwith a smaller code amount of the quantization matrix.
[0069] This embodiment adopts the arrangement in which the quantizationmatrices shown in Figs. 8A to 8D corresponding to sub-block division / orthogonaltransformation processes shown in Figs. 7A to 7D are used. However, theorthogonal transformation size and quantization matrix used are not limited tothem. For example, if ternary-tree division for division at a ratio of 1 : 2 : 1, asshown in Fig. 7E or 7F, is further used, a quantization matrix 900 or 901 shown inFig. 9A or 9B may further be used.
[0070] As described above, the quantization matrix holding unit 103 accordingto the embodiment individually holds an element at a position corresponding toeach transform coefficient within a preset predetermined range (a range of adistance "1" in this embodiment) from the storage position of the elementcorresponding to the DC component in the long side direction of the quantizationmatrix 801 or 802 shown in Fig. 8B or 8C whose size is the same as that of thebasic block (a size of 16 x 16 pixels in this embodiment) in only one of thehorizontal and vertical directions and is different in the other direction, and holdsthe same element on a basis of a preset number (two in this embodiment) ofelements of the transform coefficients at a position outside the predeterminedrange in the long side direction. On the other hand, in the short side direction ofthe quantization matrix shown in Fig. 8B or 8C, an element of the quantizationmatrix is arranged for each transform coefficient.
[0071] With respect to the quantization matrix whose size is the same as that ofthe basic block in both the horizontal and vertical directions, four elements (arange of a distance "1" in this embodiment) at the position of the elementcorresponding to the DC component and positions adjacent to that position on theright, lower, and lower right sides are individually held for respectivecorresponding transform coefficients, as shown in Fig. 8D. With respect to theremaining transform coefficients, one element is held for four adjacent transformcoefficients. In the case of the quantization matrix of the size of 8 x 8 pixels, asshown in Fig. 8A, an element is held for each transform coefficient.
[0072] Since assignment of elements to transform coefficients is changedadaptively in accordance with the size of the quantization matrix, it is possible tosuppress the code amount of the quantization matrix and suppress image qualitydeterioration.
[0073] Note that the "predetermined range" may be defined based on the scanorder shown in each of Figs. 10A to 10D when encoding the elements of thequantization matrix. With respect to the block size of the coefficients of 8 x 16or 16 x 8 pixels, a scan shown in Fig. 10B or 10C is performed forencoding / decoding of the elements of the quantization matrix. At this time, inthe processing of the decoding apparatus (to be described later), an elementobtained by one scan operation (a range of a distance "1") from the elementcorresponding to the DC component of the transform coefficient is assigned foreach transform coefficient, and then one element is assigned for every twotransform coefficients. That is, in the processing of the decoding apparatus, thequantization matrix 801 or 802 shown in Fig. 8B or 8C is obtained. With respectto the block size of the transform coefficients of 16 x 16 pixels, elements obtainedby three scan operations (a range of a distance "3") from the elementcorresponding to the DC component of the transform coefficient are assigned forrespective transform coefficients, and then one element is assigned for every fourtransform coefficients. That is, the quantization matrix 803 shown in Fig. 8D isobtained. With respect to the block size of the transform coefficients of 8 x 8pixels, elements obtained by scanning all the elements from the elementcorresponding to the DC component of the transform coefficient are assigned forrespective transform coefficients. That is, the quantization matrix 800 shown inFig. 8A is obtained in the processing of the decoding apparatus. As describedabove, based on the scan on the quantization matrix, assignment of elements tothe transform coefficients may be adaptively changed in accordance with the sizeof the quantization matrix. In this case, it is unnecessary to particularly considerthe positional relationship between the elements.
[0074] [Second Embodiment]The second embodiment will describe an image decoding apparatus fordecoding encoded data generated by the image encoding apparatus of the above-described first embodiment. Fig. 2 is a block diagram of the image decodingapparatus.
[0075] The image decoding apparatus includes a control unit 250 that controlscomponents to be described below and controls the overall apparatus. Thecontrol unit 250 includes a CPU, a ROM storing a program executed by the CPU,various parameters, and the like, and a RAM used as the work area of the CPU.Note that after loading an OS (Operating System) stored in the ROM or anothernonvolatile storage device, the program, and the parameters into the RAM, theprogram may be executed under the control of the OS.
[0076] The image decoding apparatus further includes an input terminal 201, aseparation decoding unit 202, a decoding unit 203, an inverse quantizing / inversetransforming unit 204, an image regenerating unit 205, a frame memory 206, anin-loop filtering unit 207, an output terminal 208, and a quantization matrixdecoding unit 209.
[0077] The components of the image decoding apparatus and their operations inthe above-described arrangement of the image decoding apparatus will bedescribed below. Note that the image decoding apparatus is configured to input,on a frame basis, a bit stream generated by the above-described image encodingapparatus but may be configured to input a still image bit stream for one frame.The second embodiment will describe only intra prediction decoding processingfor the sake of descriptive simplicity but the present invention is not limited to thisand is also applicable to inter prediction decoding processing.
[0078] The separation decoding unit 202 inputs an encoded bit stream via theinput terminal 201, separates information concerning decoding processing andcode data concerning a coefficient from the bit stream, and decodes code dataexisting in the header portion of the bit stream. The separation decoding unit202 according to this embodiment separates encoded data of a quantization matrixfrom the encoded bit stream, and supplies the encoded data to the quantizationmatrix decoding unit 209. The separation decoding unit 202 also separatesencoded data of image data from the encoded bit stream, and supplies it to thedecoding unit 203. In other words, the separation decoding unit 202 performs anoperation reverse to that of the combined encoding unit 111 shown in Fig. 1.
[0079] The quantization matrix decoding unit 209 decodes the encoded data ofthe quantization matrix supplied from the separation decoding unit 202, andregenerates and holds quantization matrices 800 to 803.
[0080] The decoding unit 203 decodes the code data of the image data inputfrom the separation decoding unit 202, and regenerates a quantization coefficientand prediction information.
[0081] Similar to the inverse quantizing / inverse transforming unit 106 shown inFig. 1, the inverse quantizing / inverse transforming unit 204 performs inversequantization for the quantization coefficient using one of the regeneratedquantization matrices 800 to 803 to obtain a transform coefficient, and furtherperforms inverse orthogonal transformation to regenerate prediction error data.
[0082] The image regenerating unit 205 generates predicted image data based onthe input prediction information by referring to the frame memory 206, as needed.Then, the image regenerating unit 205 regenerates the image data by adding theprediction error data regenerated by the inverse quantizing / inverse transformingunit 204 to the prediction image data, and stores the regenerated image data in theframe memory 206.
[0083] Similar to the in-loop filtering unit 109 shown in Fig. 1, the in-loopfiltering unit 207 performs in-loop filtering processing such as deblockingfiltering for the regenerated image data stored in the frame memory 207, andstores again, in the frame memory, the image data having undergone the filteringprocessing. Then, the output terminal 208 outputs the regenerated image data toan external device (for example, a display device).
[0084] The operation of each component in the image decoding apparatus shownin Fig. 2 will be described in more detail next.
[0085] The second embodiment adopts the arrangement for inputting, on a framebasis, the bit stream generated in the first embodiment but may adopt anarrangement for inputting a still image bit stream for one frame. Furthermore,this embodiment will describe only intra prediction decoding processing for thesake of descriptive simplicity but the present invention is not limited to this and isalso applicable to inter prediction decoding processing.
[0086] The bit stream for one frame input via the input terminal 201 is suppliedto the separation decoding unit 202. The separation decoding unit 202 separatesinformation concerning decoding processing and code data concerning acoefficient from the bit stream, and decodes code data existing in the headerportion of the bit stream. More specifically, the separation decoding unit 202regenerates the code data of the quantization matrix. The separation decodingunit 202 according to this embodiment extracts the code data of the quantizationmatrix from the sequence header of the bit stream shown in Fig. 6A, and suppliesit to the quantization matrix decoding unit 209. Subsequently, the separationdecoding unit 202 regenerates sub-block-based code data of the basic block ofpicture data, and supplies it to the decoding unit 203.
[0087] The quantization matrix decoding unit 209 decodes the input code data ofthe quantization matrix, and regenerates a plurality of one-dimensional differencematrices. In this embodiment, decoding is performed using an encoding tableshown in Fig. 12A (or Fig. 12B), similar to the first embodiment. However, theencoding table is not limited to this, and another encoding table may be used aslong as it is the same as in the first embodiment. The quantization matrixdecoding unit 209 inversely scans the plurality of regenerated one-dimensionaldifference matrices to regenerate a plurality of two-dimensional quantizationmatrices. In this example, an operation reverse to the scan of the quantizationmatrix encoding unit 113 of the first embodiment is performed. That is, thequantization matrix decoding unit 209 according to this embodiment regeneratesthe quantization matrices 800 to 803 shown in Figs. 8A to 8D from differencematrices 1000 to 1003 shown in Figs. 11A to 11D using the scan methods shownin Figs. 10A to 10D, respectively, and holds the regenerated quantization matrices800 to 803.
[0088] The practical quantization matrix decoding processing according to thisembodiment will be described by exemplifying the decoding processing of thequantization matrix 801 of 16 x 8 pixels shown in Fig. 8B. The quantizationmatrix decoding unit 209 regenerates the quantization matrix shown in Fig. 8Bfrom a difference matrix 1001 shown in Fig. 11B using the scan method shown inFig. 10B. The quantization matrix decoding unit 209 calculates "4" as the firstelement from "8" as an initial value and the first difference value "-4" in Fig. 11B,decides the value "4" of an element at a position of the 0th row and the 0thcolumn as the first scan position in Fig. 10B, and arranges the element "4" in aportion in the 0th row and the 0th column in Fig. 8B. Next, the quantizationmatrix decoding unit 209 calculates "5" as the second element from the element"4" in the 0th row and the 0th column in Fig. 8B and the second difference value"1" in Fig. 11B, and arranges "5" in a portion in the 0th row and the first columnas the second scan position in Fig. 10B. Similarly, the quantization matrixdecoding unit 209 calculates "9" as the third element from the element "5" in the0th row and the first column in Fig. 8B and the third difference value "4" in Fig.10B, and arranges "9" in a portion in the first row and the 0th column and the firstrow and the first column as the third scan position in Fig. 8B. It is possible toregenerate the quantization matrix 801 shown in Fig. 8B by repeating the similarprocessing.
[0089] As described above, it is possible to decode the bit stream, in whichquantization of a low-frequency portion to which human vision is sensitive isfinely controlled, while reducing the code amount by arranging individual valuesin a plurality of elements corresponding to the low-frequency components of thequantization matrix and arranging a value on a basis of a plurality of elementswith respect to the remaining elements.
[0090] Referring back to Fig. 2, the decoding unit 203 decodes the code data ofeach sub-block in the basic block, and regenerates the quantization coefficient andthe prediction information. The decoding unit 203 supplies the regeneratedquantization coefficient to the inverse quantizing / inverse transforming unit 204,and supplies the regenerated prediction information to the image regenerating unit205.
[0091] The inverse quantizing / inverse transforming unit 204 performs, on a sub-block basis, inverse quantization for the input quantization coefficient using oneof the quantization matrices 800 to 803 regenerated and held by the quantizationmatrix decoding unit 209, thereby generating an orthogonal transformationcoefficient. The inverse quantizing / inverse transforming unit 204 alsoregenerates prediction error data by performing inverse orthogonal transformationfor the generated orthogonal transformation coefficient on a sub-block basis, andsupplies it to the image regenerating unit 205. At this time, the quantizationmatrix to be used is determined in accordance with the size of the sub-block to bedecoded. This embodiment assumes that the quantization matrix 803 shown inFig. 8D is used for sub-block division corresponding to Fig. 7A. Similarly, thequantization matrix 802 shown in Fig. 8C is used for sub-block division shown inFig. 7C. The quantization matrix 801 shown in Fig. 8B is used for sub-blockdivision shown in Fig. 7D. Furthermore, the quantization matrix shown in Fig.8A is used for quad-tree sub-block division shown in Fig. 7B.
[0092] The image regenerating unit 205 regenerates the predicted image data ona sub-block basis based on the prediction information input from the decodingunit 203 by referring to the frame memory 206, as needed. Then, the imageregenerating unit 205 regenerates the image data of the sub-block by adding, tothe predicted image data, the prediction error data supplied from the inversequantizing / inverse transforming unit 204, and stores the regenerated image data inthe frame memory 206. The image data stored in the frame memory 206 is aprediction reference candidate when decoding another sub-block.
[0093] Similar to the in-loop filtering unit 109 shown in Fig. 1, the in-loopfiltering unit 207 reads out the regenerated image data from the frame memory206 on a sub-block basis, and performs in-loop filtering processing such asdeblocking filtering. Then, the in-loop filtering unit 207 stores again, in theframe memory 206, the image having undergone the filtering processing.
[0094] The regenerated image stored in the frame memory 206 is finally,externally output from the terminal 208.
[0095] The processing procedure of decoding processing of one frame in theimage decoding apparatus according to the second embodiment will be describedbelow with reference to a flowchart shown in Fig. 4. Note that in the followingdescription, in each step, the main subject executes each process under the controlof the control unit 250.
[0096] In step S401, the separation decoding unit 202 separates informationconcerning the decoding processing and code data concerning a coefficient from abit stream, thereby decoding code data of a header portion. More specifically,the separation decoding unit 202 regenerates code data of a quantization matrix.
[0097] In step S402, the quantization matrix decoding unit 209 decodes the codedata of the quantization matrix regenerated in step S401 to regenerate the one-dimensional difference matrices 1000 to 1003 shown in Figs. 11A to 11D.Furthermore, the quantization matrix decoding unit 209 inversely scans theregenerated one-dimensional difference matrices 1000 to 1003 to regenerate two-dimensional quantization matrices. That is, the quantization matrix decodingunit 209 inversely scans the difference matrices 1000 to 1003 shown in Figs. 11Ato 11D to regenerate the quantization matrices 800 to 803 shown in Figs. 8A to 8Dusing the scan methods shown in Fig. 10A to 10D, respectively, and holds them.Note that the decoding side apparatus already knows the scan methods shown inFigs. 10A to 10D and assignment of elements of the quantization matrices totransform coefficients. That is, each element of the quantization matrix isassigned to a transform coefficient in accordance with the distance from anelement corresponding to the DC component of the transform coefficient. That is, the quantization matrix decoding unit 209 assigns one element ata position corresponding to each transform coefficient within a presetpredetermined range (a range of a distance "1" in this embodiment) from thestorage position of the element corresponding to the DC component in the longside direction of the quantization matrix shown in Fig. 8B or 8C whose size is thesame as that of the basic block (a size of 16 x 16 pixels in this embodiment) inonly one of the horizontal and vertical directions and is different in the otherdirection, and assigns the same element on a basis of a preset number (two in thisembodiment) of elements of the transform coefficients at a position outside thepredetermined range in the long side direction. On the other hand, in the shortside direction of the quantization matrix shown in Fig. 8B or 8C, an element ofthe quantization matrix is assigned for each transform coefficient.
[0098] With respect to the quantization matrix whose size is the same as that ofthe basic block in both the horizontal and vertical directions, the quantizationmatrix decoding unit 209 individually assigns, for respective correspondingtransform coefficients, four elements (a range of a distance "1" in thisembodiment) at the position of the element corresponding to the DC componentand positions adjacent to that position on the right, lower, and lower right sides, asshown in Fig. 8D. With respect to the remaining transform coefficients, thequantization matrix decoding unit 209 assigns one element for four adjacenttransform coefficients. In the case of the quantization matrix 800 of the size of 8x 8 pixels, as shown in Fig. 8A, an element is held for each transform coefficient.
[0099] Since assignment of elements to transform coefficients is changedadaptively in accordance with the size of the quantization matrix, it is possible tosuppress the code amount of the quantization matrix and suppress image qualitydeterioration.
[0100] Note that the "predetermined range" may be defined based on the scanorder shown in each of Figs. 10A to 10D when decoding the elements of thequantization matrix. With respect to the block size of the transform coefficientsof 8 x 16 or 16 x 8 pixels, a scan shown in Fig. 10B or 10C is performed fordecoding of the elements of the quantization matrix. At this time, thequantization matrix decoding unit 209 assigns, for each transform coefficient, anelement obtained by one scan operation (a range of a distance "1") from theelement corresponding to the DC component of the transform coefficient, andthen assigns one element for every two transform coefficients. That is, thequantization matrix 801 or 802 shown in Fig. 8B or 8C is obtained. With respectto the block size of the transform coefficients of 16 x 16 pixels, the quantizationmatrix decoding unit 209 assigns, for respective transform coefficients, elementsobtained by three scan operations (a range of a distance "3") from the elementcorresponding to the DC component of the transform coefficient, and then assignsone element for every four transform coefficients. That is, the quantizationmatrix 803 shown in Fig. 8D is obtained. With respect to the block size of thetransform coefficients of 8 x 8 pixels, the quantization matrix decoding unit 209assigns, for respective transform coefficients, elements obtained by scanning allthe elements from the element corresponding to the DC component of thetransform coefficient. That is, the quantization matrix shown in Fig. 8A isobtained. As described above, based on the scan on the quantization matrix,assignment of elements to the transform coefficients may be adaptively changedin accordance with the size of the quantization matrix. In this case, it isunnecessary to particularly consider the positional relationship between theelements.
[0101] In step S403, the decoding unit 203 decodes the code data separated instep S401, and regenerates the prediction information and the quantizationcoefficient for each sub-block.
[0102] In step S404, the inverse quantizing / inverse transforming unit 204obtains, for each sub-block, the transform coefficient by performing inversequantization using one of the quantization matrices 800 to 803 regenerated in stepS402 for the quantization coefficient. The inverse quantizing / inversetransforming unit 204 also performs inverse orthogonal transformation for thetransform coefficient to regenerate prediction error data of each sub-block.
[0103] In step S405, the image regenerating unit 205 regenerate predictioninformation and predicted image data for each sub-block generated in step S403.Then, the image regenerating unit 205 regenerate the image data for each sub-block by adding the prediction error data generated in step S404 to theregenerated predicted image data.
[0104] In step S406, the control unit 250 determines whether decoding of all thebasic blocks in the frame has ended. If it is determined that decoding has ended,the control unit 250 advances to step S407; otherwise, the control unit 250 returnsthe process to step S403 to decode the next basic block.
[0105] In step S407, the in-loop filtering unit 207 performs in-loop filteringprocessing for the image data regenerated in step S405 to generate an imagehaving undergone the filtering processing, thereby ending the processing.
[0106] Note that when decoding the second frame or a subsequent frame of themoving image, the processes in steps S403 to S407 are repeated until the end ofthe encoded bit stream or until a stop instruction is received from the user via anoperation unit (not shown).
[0107] With the above-described arrangement and operation, it is possible todecode the bit stream with the reduced code amount of the quantization matrix,generated in the first embodiment, in which only first some elements of thequantization matrix are individually encoded and the remaining elements areencoded on a basis of a plurality of elements.
[0108] Note that the second embodiment adopts the arrangement for decodingall the four kinds of quantization matrices 800 to 803 shown in Figs. 8A to 8Dfrom a bit stream but may adopt an arrangement for generating anotherquantization matrix from one quantization matrix decoded from a bit stream.For example, a quantization matrix of 16 x 8 pixels may be generated using aquantization matrix of 16 x 16 pixels decoded from a bit stream, instead of beingdecoded from a bit stream. In this case, the quantization matrix shown in Fig.8B is generated by thinning out some elements from the quantization matrixshown in Fig. 8D. Similarly, the quantization matrix shown in Fig. 8C isgenerated by thinning out some elements from the quantization matrix shown inFig. D. Furthermore, by deciding a method of calculating the element "5" in the0th row and the first column in Fig. 8B or the element "5" in the first row and the0th column in Fig. 8C within the above-described "predetermined range", it ispossible to generate the quantization matrix 801 shown in Fig. 8B or thequantization matrix 802 shown in Fig. 8C using the quantization matrix 800shown in Fig. 8A. The calculation method at this time is not particularly limited.Calculation may be performed by adding a specific difference value (+1 in thisembodiment) to the element "4" in the 0th row and the 0th column or by usinginterpolation with peripheral elements. Furthermore, the element "5" in the 0throw and the first column in Fig. 8B or the element "5" in the first row and the 0thcolumn in Fig. 8C may be decoded from the bit stream together with the elementsof the quantization matrix of 16 x 16 pixels. This can decode the bit stream withthe further reduced code amount of the quantization matrix.
[0109] Furthermore, an arrangement for decoding, from a header, an identifierfor selecting whether to generate each quantization matrix from anotherquantization matrix or decode each element may be adopted. For example, anFig. 8D. arrangement for decoding, as a quantization matrix encoding method informationcode, from the header portion of the bit stream shown in Fig. 6B, informationindicating whether to generate each element of a quantization matrix from anotherquantization matrix or decode each element individually may be adopted. Thiscan decode the bit stream for which whether to prioritize image quality controlaccording to the size of the sub-block or reduce the code amount of thequantization matrix is selected.
[0110] This embodiment adopts the arrangement in which the quantizationmatrices 800 to 803 shown in Figs. 8A to 8D corresponding to the sub-blockdivision / orthogonal transformation processes shown in Figs. 7A to 7D are used.However, the orthogonal transformation size and quantization matrix used are notlimited to them. For example, if ternary-tree division for division at a ratio of 1 :2 : 1, as shown in Fig. 7E or 7F is further used, a quantization matrix 900 or 901shown in Fig. 9A or 9B may further be used.
[0111] [Third Embodiment] Each of the above-described embodiments assumes that each processingunit shown in Fig. 1 or 2 is formed by a hardware component. However,processing performed by each processing unit shown in Fig. 1 or 2 may beimplemented by a computer program.
[0112] Fig. 5 is a block diagram showing an example of the hardwarearrangement of a computer applicable to an image display device according toeach of the above-described embodiments.
[0113] A CPU 501 executes each of the processes described as those performedby an image processing apparatus according to each of the above-describedembodiments while controlling the overall computer using a computer programand data stored in a RAM 502 or a ROM 503. That is, the CPU 501 functions aseach processing unit shown in Fig. 1 or 2.
[0114] The RAM 502 includes an area for temporarily storing a computerprogram and data loaded from an external storage device 506, data externallyacquired via an I / F (interface) 507, and the like. Furthermore, the RAM 502includes a work area used by the CPU 501 to execute various kinds of processes.That is, for example, the RAM 502 can be assigned as a frame memory and canprovide other various areas appropriately.
[0115] The ROM 503 stores the setting data, boot program, and the like of thecomputer. An operation unit 504 is formed by a keyboard, a mouse, and the like,and can input various instructions to the CPU 501 when operated by the user ofthe computer. A display unit 505 displays a processing result of the CPU 501.The display unit 505 is formed by, for example, a liquid crystal display.
[0116] The external storage device 506 is a large-capacity information storagedevice represented by a hard disk drive. The external storage device 506 storesan OS (Operating System) and a computer program for causing the CPU 501 toimplement the function of each unit shown in Fig. 1 or 2. Furthermore, theexternal storage device 506 may store each image data as a processing target.
[0117] The computer program and data stored in the external storage device 506are loaded into the RAM 502, as needed, under the control of the CPU 501, andare processing targets of the CPU 501. The I / F 507 can be connected to anetwork such as a LAN or the Internet, and other devices such as a projectiondevice and a display device. The computer can acquire or send various kinds ofinformation via the I / F 507. Reference numeral 508 denotes a bus that connectsthe above-described units.
[0118] In the above arrangement, when the apparatus is powered on, the CPU501 executes the boot program in the ROM 503, loads the OS stored in theexternal storage device 506 into the RAM 502, and activates the OS. As a result,this apparatus can perform communication via the interface 507, therebyfunctioning as an information processing apparatus. Then, when, under thecontrol of the OS, the CPU 501 loads an application (corresponding to Fig. 3)concerning image encoding from the external storage device 506 into the RAM502 and executes it, the CPU 501 functions as the various processing units shownin Fig. 1, and this apparatus functions as an image encoding apparatus. On theother hand, when the CPU 501 loads an application (corresponding to Fig. 4)concerning image decoding from the external storage device 506 into the RAM502 and executes it, the CPU 501 functions as the various processing units shownin Fig. 2, and this apparatus functions as an image decoding apparatus.
[0119] (Other Embodiments)The present invention can be implemented by processing of supplying aprogram for implementing one or more functions of the above-describedembodiments to a system or apparatus via a network or storage medium, andcausing one or more processors in the computer of the system or apparatus to readout and execute the program. The present invention can also be implemented bya circuit (for example, an ASIC) for implementing one or more functions. The present invention is not limited to the above embodiments andvarious changes and modifications can be made within the spirit and scope of thepresent invention. Therefore, to apprise the public of the scope of the presentinvention, the following claims are made.
Claims
1. An image decoding apparatus for decoding an image from encoded data, the apparatus characterized by comprising: a decoding unit configured to decode data for indicating a plurality of difference values for a quantization matrix used for inverse quantizing processing which derives transform coefficients from quantized transform coefficients; a deriving unit configured to derive a plurality of values corresponding to at least a part of elements in the quantization matrix by using the data, and derive the quantization matrix based on the plurality of values; and an inverse quantizing unit configured to perform the inverse quantizing processing on an object block using the quantization matrix and a quantization parameter, wherein, in a case where a width of the object block is 8 samples and a height of the object block is 16 samples, the deriving unit derives the quantization matrix by: setting a first value among the plurality of values as a first element corresponding to DC component in the quantization matrix, wherein the first value is not set for a plurality of elements in the quantization matrix used for the inverse quantizing processing, and the first value is derived by using difference from a predetermined initial value; setting a second value among the plurality of values as a second element which is adjacent to the first element and is below the first element, wherein the second value is not set for a plurality of elements in the quantization matrix used for the inverse quantizing processing, and the second value is derived by using difference from the first value; and setting a third value among the plurality of values as a third element and a fourth element, wherein the third element is adjacent to the second element and is below the second element, the fourth element is adjacent to the third element and is below the third element, and the third value is derived by using difference from the second value, wherein the predetermined initial value is 8.
2. An image encoding apparatus for encoding an image, the apparatus characterized by comprising; a dividing unit configured to divide the image into a plurality of blocks; a quantizing unit configured to perform quantizing processing on an object block using a quantization matrix and a quantization parameter; and an encoding unit configured to encode data for indicating a plurality of values corresponding to at least a part of elements in the quantization matrix, wherein, in a case where a width of the object block is 8 samples and a height of the object block is 16 samples, the data includes: data indicating difference between a predetermined initial value and a first value which is set as a first element corresponding to DC component in the quantization matrix and is not set for a plurality of elements in the quantization matrix used for the quantizing processing; data indicating difference between the first value and a second value which is set as a second element which is adjacent to the first element and is below the first element, wherein the second value is not set for a plurality of elements in the quantization matrix used for the quantizing processing; and data indicating difference between the second value and a third value which is set as a third element and a fourth element, wherein the third element is adjacent to the second element and is below the second element, and the fourth element is adjacent to the third element and is below the third element, wherein the predetermined initial value is 8.
3. An image decoding method for decoding an image from encoded data, the method characterized by comprising: decoding data for indicating a plurality of difference values for a quantization matrix used for inverse quantizing processing which derives transform coefficients from quantized transform coefficients; deriving a plurality of values corresponding to at least a part of elements in the quantization matrix by using the data, and deriving the quantization matrix based on the plurality of values; and performing the inverse quantizing processing on an object block using the quantization matrix and a quantization parameter, wherein, in a case where a width of the object block is 8 samples and a height of the object block is 16 samples, the quantization matrix is derived in the deriving by: setting a first value among the plurality of values as a first element corresponding to DC component in the quantization matrix, wherein the first value is not set for a plurality of elements in the quantization matrix used for the inverse quantizing processing, and the first value is derived by using difference from a predetermined initial value; setting a second value among the plurality of values as a second element which is adjacent to the first element and is below the first element, wherein the second value is not set for a plurality of elements in the quantization matrix used for the inverse quantizing processing, and the second value is derived by using difference from the first value; and setting a third value among the plurality of values as a third element and a fourth element, wherein the third element is adjacent to the second element and is below the second element, the fourth element is adjacent to the third element and is below the third element, and the third value is derived by using difference from the second value, wherein the predetermined initial value is 8.
4. An image encoding method for encoding an image, the method characterized by comprising; dividing the image into a plurality of blocks; performing quantizing processing on an object block using a quantization matrix and a quantization parameter; and encoding data for indicating a plurality of values corresponding to at least a part of elements in the quantization matrix, wherein, in a case where a width of the object block is 8 samples and a height of the object block is 16 samples, the data includes: data indicating difference between a predetermined initial value and a first value which is set as a first element corresponding to DC component in the quantization matrix and is not set for a plurality of elements in the quantization matrix used for the quantizing processing; data indicating difference between the first value and a second value which is set as a second element which is adjacent to the first element and is below the first element, wherein the second value is not set for a plurality of elements in the quantization matrix used for the quantizing processing; and data indicating difference between the second value and a third value which is set as a third element and a fourth element, wherein the third element is adjacent to the second element and is below the second element, and the fourth element is adjacent to the third element and is below the third element, wherein the predetermined initial value is 8.
