Image processing equipment and methods

Abstract

The invention relates to an image processing device comprising a selector configured to select, from transformation units of different sizes, a transformation unit used to perform an inverse orthogonal transformation of the image data to be decoded; a generator configured to generate, from the first quantization matrix corresponding to the transformation unit for the first size, a second quantization matrix corresponding to the transformation unit for the second size; and an inverse quantizer configured to inversely quantize the transformation coefficient data of the image data using the second quantization matrix generated by the generator when the selector selects the transformation unit for the second size. Furthermore, the invention relates to a method for processing images and a computer-readable invariant record containing instructions to perform this method.

Description

[0001] This disclosure relates to an image processing apparatus and an image processing method.

[0002] In H.264 / AVC, one of the standard specifications for video encoding schemes, different quantization steps can be used for each component of the orthogonal transformation coefficients when quantizing image data in profiles of High Profile or higher. The quantization steps for each component of the orthogonal transformation coefficients can be set based on a quantization matrix (also called a scaling list) defined to be the same size as the unit of the orthogonal transformation and a reference step value.

[0003] Figure 19 shows the default values ​​of the four types of quantization matrices predefined in H.264 / AVC. For example, in intra-prediction mode, when the size of the transformation unit is 4x4, matrix SL01 is the default quantization matrix. In inter-prediction mode, when the size of the transformation unit is 4x4, matrix SL02 is the default quantization matrix. In intra-prediction mode, when the size of the transformation unit is 8x8, matrix SL03 is the default quantization matrix. In inter-prediction mode, when the size of the transformation unit is 8x8, matrix SL04 is the default quantization matrix. In addition, the user can specify their own quantization matrices different from the default values ​​shown in Figure 19 in the sequence parameter set or picture parameter set. If no quantization matrix is ​​used, the quantization step used during quantization will be the same value for all components.

[0004] HEVC (High Efficiency Video Coding), which is being standardized as the next-generation video coding method following H.264 / AVC, introduces the concept of a coding unit (CU: Coding Unit), which corresponds to a conventional macroblock (see Non-Patent Literature 1 below). The size range of the coding unit is specified in the sequence parameter set by a pair of powers of 2 values ​​called LCU (Largest Coding Unit) and SCU (Smallest Coding Unit). Then, using split_flag, the specific size of the coding unit within the range specified by LCU and SCU is identified.

[0005] In HEVC, one coding unit can be divided into one or more orthogonal transformation units, i.e., one or more transform units (TUs). As the size of the transform unit, any of 4×4, 8×8, 16×16, and 32×32 can be used. Therefore, the quantization matrix can also be specified for each candidate size of these transform units.

[0006] By contrast, in H.264 / AVC, as specified in the publicly available reference software called JM (Joint Model) (http: / / iphome.hhi.de / suehring / tml / index.htm), it was possible to specify only one quantization matrix for the size of one transform unit within one picture. In contrast, Non-Patent Document 2 below proposes specifying a plurality of quantization matrix candidates for the size of one transform unit within one picture and adaptively selecting a quantization matrix for each block from the viewpoint of rate-distortion (RD) optimization.

[0007] JCTVC-B205, “Test Model under Consideration”, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 2nd Meeting: Geneva, CH, 21-28 July, 2010VCEG-AD06, “Adaptive Quantization Matrix Selection on KTA Software”, ITU - Telecommunications Standardization Sector STUDY GROUP 16 Question 6 Video Coding Experts Group (VCEG) 30th Meeting: Hangzhou, China, 23 - 24 October, 2006

[0008] However, if the number of selectable transformation unit sizes increases, the number of corresponding quantization matrices also increases, and the increase in the code size of the quantization matrices can lead to a decrease in coding efficiency. Furthermore, this decrease in coding efficiency may become more pronounced if the number of quantization matrices that can be specified for each transformation unit size increases from one to multiple.

[0009] Therefore, the present invention aims to provide an image processing apparatus and an image processing method that can suppress the increase in code amount when the number of quantization matrices is large.

[0010] According to one embodiment, an image processing apparatus is provided, comprising: a selection unit that selects a transformation unit to be used for the inverse orthogonal transformation of the image data to be decoded from a plurality of transformation units of different sizes; a generation unit that generates a second quantization matrix corresponding to a transformation unit of a second size from a first quantization matrix corresponding to a transformation unit of a first size; and an inverse quantization unit that, when the second size transformation unit is selected by the selection unit, inversely quantizes the transformation coefficient data of the image data using the second quantization matrix generated by the generation unit.

[0011] The above image processing device can typically be implemented as an image decoding device that decodes an image.

[0012] The generation unit may also generate the second quantization matrix using matrix information that identifies the first quantization matrix, and difference information that represents the difference between the second size prediction matrix predicted from the first quantization matrix and the second quantization matrix.

[0013] The generation unit may also obtain the matrix information and the difference information from the sequence parameter set or the picture parameter set.

[0014] Furthermore, if the generation unit obtains a first flag from the sequence parameter set or picture parameter set indicating that there is no difference between the prediction matrix and the second quantization matrix, the generation unit may use the prediction matrix as the second quantization matrix.

[0015] The first size may also be the smallest size among the sizes of the plurality of conversion units.

[0016] Alternatively, the second size may be larger than the first size, and the generation unit may calculate the prediction matrix by duplicating the first element or the second element as an element between adjacent first and second elements in the first quantization matrix.

[0017] Alternatively, the second size may be larger than the first size, and the generation unit may calculate the prediction matrix by linearly interpolating the elements between the first and second elements that are adjacent to each other in the first quantization matrix.

[0018] The second size may also be twice the first size on one side.

[0019] Alternatively, the second size may be smaller than the first size, and the generation unit may calculate the prediction matrix by decimating the elements of the first quantization matrix.

[0020] The second size may be smaller than the first size, and the generation unit may calculate the prediction matrix by calculating the average of a plurality of adjacent elements in the first quantization matrix.

[0021] The generation unit may also generate the second quantization matrix from the first quantization matrix if a second flag specifying the use of a user-defined matrix for the second quantization matrix is ​​obtained from the sequence parameter set or picture parameter set.

[0022] In another embodiment, an image processing method is provided which includes: selecting a transformation unit to be used for the inverse orthogonal transformation of the image data to be decoded from a plurality of transformation units of different sizes; generating a second quantization matrix corresponding to a second transformation unit of a second size from a first quantization matrix corresponding to a first transformation unit of a first size; and, when the second transformation unit of a second size is selected, inverse quantizing the transformation coefficient data of the image data using the second quantization matrix generated from the first quantization matrix.

[0023] In another embodiment, an image processing device is provided, comprising: a selection unit that selects a conversion unit to be used for orthogonal transformation of the image data to be encoded from a plurality of conversion units of different sizes; a quantization unit that quantizes the conversion coefficient data generated by orthogonally transforming the image data with the conversion unit selected by the selection unit using a quantization matrix corresponding to the selected conversion unit; and an encoding unit that encodes information for generating a second quantization matrix corresponding to a conversion unit of a second size from a first quantization matrix corresponding to a conversion unit of a first size.

[0024] The above image processing device can typically be implemented as an image encoding device that encodes an image.

[0025] In another embodiment, an image processing method is provided which includes: selecting a transformation unit to be used for orthogonal transformation of image data to be encoded from a plurality of transformation units of different sizes; quantizing transformation coefficient data generated by orthogonally transforming the image data with the selected transformation unit using a quantization matrix corresponding to the selected transformation unit; and encoding information for generating a second quantization matrix corresponding to a second transformation unit from a first quantization matrix corresponding to a first transformation unit of a first size.

[0026] As described above, the image processing apparatus and image processing method according to the present disclosure can suppress the increase in the code amount when the number of quantization matrices is large.

[0027] This is a block diagram showing an example of the configuration of an image encoding device according to one embodiment. This is a block diagram showing an example of a detailed configuration of the orthogonal transformation / quantization unit according to one embodiment. This is a block diagram showing an example of a more detailed configuration of the matrix processing unit according to one embodiment. This is an explanatory diagram showing an example of information inserted into the sequence parameter set according to one embodiment. This is an explanatory diagram showing an example of information inserted into the picture parameter set according to one embodiment. This is the first half of a flowchart showing a first example of the processing flow during encoding according to one embodiment. This is the second half of a flowchart showing a first example of the processing flow during encoding according to one embodiment. This is the first half of a flowchart showing a second example of the processing flow during encoding according to one embodiment. This is the second half of a flowchart showing a second example of the processing flow during encoding according to one embodiment. This is a block diagram showing an example of the configuration of an image decoding device according to one embodiment. This is a block diagram showing an example of a detailed configuration of the inverse quantization / inverse orthogonal transformation unit according to one embodiment. This is a block diagram showing an example of a more detailed configuration of the matrix generation unit according to one embodiment. This is the first half of a flowchart showing a first example of the processing flow during decoding according to one embodiment. This is the second half of a flowchart showing a first example of the processing flow during decoding according to one embodiment. This is the first half of a flowchart showing a second example of the processing flow during decoding according to one embodiment. This is the second half of a flowchart showing a second example of the processing flow during decoding according to one embodiment. This is the first half of a flowchart showing an example of the processing flow during encoding according to one modified example. This is the latter half of a flowchart showing an example of the processing flow during encoding in a modified example. This is the first half of a flowchart showing an example of the processing flow during decoding in a modified example. This is the latter half of a flowchart showing an example of the processing flow during decoding in a modified example. This is a block diagram showing an example of a schematic configuration of a television device. This is a block diagram showing an example of a schematic configuration of a mobile phone. This is a block diagram showing an example of a schematic configuration of a recording and playback device. This is a block diagram showing an example of a schematic configuration of an imaging device. This is an explanatory diagram showing the default values ​​of the quantization matrix that are predefined in H.264 / AVC.

[0028] Preferred embodiments of the present disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0029] The "modes for carrying out the invention" will be described in the following order. 1. Example of the configuration of an image encoding device according to one embodiment 1-1. Example of an overall structure 1-2. Example of the configuration of the orthogonal transformation and quantization unit 1-3. Detailed Configuration Example of the Matrix Processing Unit 1-4. Examples of information to be encoded 2. Processing flow during encoding according to one embodiment 3. Example of the configuration of an image decoding device according to one embodiment 3-1. Example of an overall structure 3-2. Example Configuration of Inverse Quantization / Inverse Orthogonal Transformation Unit 3-3. Detailed Configuration Example of the Matrix Generation Unit 4. Processing flow during decoding according to one embodiment 5. Variations 6. Application Examples 7. Summary

[0030] <1. Example of the configuration of an image encoding device according to one embodiment> This section describes an example configuration of an image encoding device according to one embodiment.

[0031] [1-1. Example of overall configuration] Figure 1 is a block diagram showing an example of the configuration of an image encoding device 10 according to one embodiment. Referring to Figure 1, the image encoding device 10 includes an A / D (Analogue to Digital) conversion unit 11, a sorting buffer 12, a subtraction unit 13, an orthogonal transformation / quantization unit 14, a reversible encoding unit 16, a storage buffer 17, a rate control unit 18, an inverse quantization unit 21, an inverse orthogonal transformation unit 22, an addition unit 23, a deblocking filter 24, a frame memory 25, a selector 26, an intra prediction unit 30, a motion search unit 40, and a mode selection unit 50.

[0032] The A / D conversion unit 11 converts the image signal input in analog format into digital image data, rearranges the series of digital image data, and outputs it to the buffer 12.

[0033] The sorting buffer 12 sorts the images included in the series of image data input from the A / D conversion unit 11. After sorting the images according to the GOP (Group of Pictures) structure related to the encoding process, the sorting buffer 12 outputs the sorted image data to the subtraction unit 13, the intra prediction unit 30, and the motion search unit 40.

[0034] The subtraction unit 13 is supplied with image data input from the sorting buffer 12 and predicted image data selected by the mode selection unit 50, which will be described later. The subtraction unit 13 calculates prediction error data, which is the difference between the image data input from the sorting buffer 12 and the predicted image data input from the mode selection unit 50, and outputs the calculated prediction error data to the orthogonal transformation / quantization unit 14.

[0035] The orthogonal transformation and quantization unit 14 performs orthogonal transformation and quantization on the prediction error data input from the subtraction unit 13, and outputs the quantized transformation coefficient data (hereinafter referred to as quantized data) to the reversible coding unit 16 and the inverse quantization unit 21. The bit rate of the quantized data output from the orthogonal transformation and quantization unit 14 is controlled based on the rate control signal from the rate control unit 18. The detailed configuration of the orthogonal transformation and quantization unit 14 will be described in more detail later.

[0036] The reversible coding unit 16 is supplied with quantized data input from the orthogonal transformation / quantization unit 14, information for generating a quantization matrix on the decoding side, and information regarding intra-prediction or inter-prediction selected by the mode selection unit 50. The intra-prediction information may include, for example, prediction mode information indicating the optimal intra-prediction mode for each block. The inter-prediction information may include, for example, prediction mode information for predicting the motion vector for each block, differential motion vector information, and reference image information.

[0037] The reversible coding unit 16 generates a coded stream by performing a reversible coding process on the quantized data. The reversible coding by the reversible coding unit 16 may be, for example, variable-length coding or arithmetic coding. The reversible coding unit 16 also multiplexes information for generating the quantization matrix, which will be explained in detail later, into the header of the coded stream (for example, the sequence parameter set and the picture parameter set). Furthermore, the reversible coding unit 16 multiplexes the intra-prediction information or inter-prediction information described above into the header of the coded stream. The reversible coding unit 16 then outputs the generated coded stream to the storage buffer 17.

[0038] The storage buffer 17 temporarily stores the encoded stream input from the reversible encoding unit 16 using a storage medium such as a semiconductor memory. The storage buffer 17 then outputs the stored encoded stream at a rate corresponding to the bandwidth of the transmission line (or the output line from the image encoding device 10).

[0039] The rate control unit 18 monitors the available capacity of the storage buffer 17. The rate control unit 18 generates a rate control signal according to the available capacity of the storage buffer 17 and outputs the generated rate control signal to the orthogonal transformation / quantization unit 14. For example, when the available capacity of the storage buffer 17 is small, the rate control unit 18 generates a rate control signal to reduce the bit rate of the quantized data. Also, for example, when the available capacity of the storage buffer 17 is sufficiently large, the rate control unit 18 generates a rate control signal to increase the bit rate of the quantized data.

[0040] The inverse quantization unit 21 performs inverse quantization processing on the quantized data input from the orthogonal transformation / quantization unit 14. The inverse quantization unit 21 then outputs the transformation coefficient data obtained by the inverse quantization processing to the inverse orthogonal transformation unit 22.

[0041] The inverse orthogonal transformation unit 22 restores the prediction error data by performing an inverse orthogonal transformation on the transformation coefficient data input from the inverse quantization unit 21. The inverse orthogonal transformation unit 22 then outputs the restored prediction error data to the addition unit 23.

[0042] The adder 23 generates decoded image data by adding the restored prediction error data input from the inverse orthogonal transform 22 and the prediction image data input from the mode selection 50. The adder 23 then outputs the generated decoded image data to the deblocking filter 24 and the frame memory 25.

[0043] The deblocking filter 24 performs filtering to reduce block distortion that occurs during image encoding. The deblocking filter 24 removes block distortion by filtering the decoded image data input from the summing unit 23, and outputs the decoded image data after filtering to the frame memory 25.

[0044] The frame memory 25 stores the decoded image data input from the adder 23 and the decoded image data after filtering input from the deblocking filter 24 using a storage medium.

[0045] The selector 26 reads the decoded image data before filtering, which is used for intra prediction, from the frame memory 25 and supplies the read decoded image data to the intra prediction unit 30 as reference image data. The selector 26 also reads the decoded image data after filtering, which is used for inter prediction, from the frame memory 25 and supplies the read decoded image data to the motion search unit 40 as reference image data.

[0046] The intra prediction unit 30 performs intra prediction processing for each intra prediction mode based on the image data to be encoded input from the sorting buffer 12 and the decoded image data supplied via the selector 26. For example, the intra prediction unit 30 evaluates the prediction results for each intra prediction mode using a predetermined cost function. The intra prediction unit 30 then selects the intra prediction mode with the smallest cost function value, i.e., the intra prediction mode with the highest compression ratio, as the optimal intra prediction mode. Furthermore, the intra prediction unit 30 outputs information related to intra prediction, such as prediction mode information indicating the optimal intra prediction mode, predicted image data, and cost function value, to the mode selection unit 50.

[0047] The motion search unit 40 performs inter-frame prediction processing based on the image data to be encoded input from the sorting buffer 12 and the decoded image data supplied via the selector 26. For example, the motion search unit 40 evaluates the prediction results for each prediction mode using a predetermined cost function. Next, the motion search unit 40 selects the prediction mode with the smallest cost function value, i.e., the prediction mode with the highest compression ratio, as the optimal prediction mode. The motion search unit 40 also generates predicted image data according to the optimal prediction mode. The motion search unit 40 then outputs inter-frame prediction information, including prediction mode information representing the selected optimal prediction mode, predicted image data, and inter-frame prediction information such as the cost function value, to the mode selection unit 50.

[0048] The mode selection unit 50 compares the cost function value for intra prediction input from the intra prediction unit 30 with the cost function value for inter prediction input from the motion search unit 40. The mode selection unit 50 then selects the prediction method with the lower cost function value among intra prediction and inter prediction. If intra prediction is selected, the mode selection unit 50 outputs information related to intra prediction to the reversible coding unit 16 and outputs the predicted image data to the subtraction unit 13 and the addition unit 23. If inter prediction is selected, the mode selection unit 50 outputs the above-mentioned information related to inter prediction to the reversible coding unit 16 and outputs the predicted image data to the subtraction unit 13 and the addition unit 23.

[0049] [1-2. Example of the configuration of the orthogonal transformation and quantization unit] Figure 2 is a block diagram showing an example of a detailed configuration of the orthogonal transformation / quantization unit 14 of the image encoding device 10 shown in Figure 1. Referring to Figure 2, the orthogonal transformation / quantization unit 14 includes a selection unit 110, an orthogonal transformation unit 120, a quantization unit 130, a quantization matrix buffer 140, and a matrix processing unit 15.

[0050] (1) Selection section The selection unit 110 selects a transformation unit (TU) to be used for the orthogonal transformation of the image data to be encoded from a plurality of transformation units of different sizes. Candidate transformation unit sizes that can be selected by the selection unit 110 include, for example, 4×4 and 8×8 for H.264 / AVC, and 4×4, 8×8, 16×16, and 32×32 for HEVC. The selection unit 110 may select any of the transformation units depending on, for example, the size or image quality of the image to be encoded, or the performance of the device. The selection of the transformation unit by the selection unit 110 may be hand-tuned by the user developing the device. The selection unit 110 then outputs information specifying the size of the selected transformation unit to the orthogonal transformation unit 120, the quantization unit 130, the reversible encoding unit 16, and the dequantization unit 21.

[0051] (2) Orthogonal transformation unit The orthogonal transformation unit 120 orthogonally transforms the image data (i.e., prediction error data) supplied from the subtraction unit 13 using the transformation units selected by the selection unit 110. The orthogonal transformation performed by the orthogonal transformation unit 120 may be, for example, a discrete cosine transform (DCT) or a Carunen-Loeb transform. The orthogonal transformation unit 120 then outputs the transformation coefficient data obtained by the orthogonal transformation process to the quantization unit 130.

[0052] (3) Quantization section The quantization unit 130 quantizes the conversion coefficient data generated by the orthogonal transformation unit 120 using the quantization matrix corresponding to the conversion unit selected by the selection unit 110. The quantization unit 130 also changes the bit rate of the output quantized data by switching the quantization step based on the rate control signal from the rate control unit 18.

[0053] The quantization unit 130 also stores a set of quantization matrices corresponding to each of the multiple conversion units that can be selected by the selection unit 110 in the quantization matrix buffer 140. For example, if there are four candidate conversion units of different sizes, such as 4x4, 8x8, 16x16, and 32x32, as in HEVC, then a set of four quantization matrices corresponding to each of these four sizes can be stored in the quantization matrix buffer 140. Note that if a default quantization matrix, as illustrated in Figure 19, is used for a certain size, then only a flag indicating that the default quantization matrix is ​​used (i.e., that a user-defined quantization matrix is ​​not used) may be stored in the quantization matrix buffer 140 in association with that size.

[0054] A set of quantization matrices that may be used by the quantization unit 130 may typically be set for each sequence of the encoded stream. The quantization unit 130 may also update the set of quantization matrices set for each sequence for each picture. Information for controlling the setting and updating of such sets of quantization matrices may be inserted, for example, into a sequence parameter set and a picture parameter set.

[0055] (4) Quantization matrix buffer The quantization matrix buffer 140 temporarily stores a set of quantization matrices corresponding to a plurality of transformation units that can be selected by the selection unit 110, using a storage medium such as semiconductor memory. The set of quantization matrices stored in the quantization matrix buffer 140 is referenced during processing by the matrix processing unit 150, which will be described next.

[0056] (5) Matrix Processing Unit The matrix processing unit 150, for each sequence and picture of the encoded stream, refers to the set of quantization matrices stored in the quantization matrix buffer 140 and generates information for generating quantization matrices corresponding to one or more other sizes of transformation units from a quantization matrix corresponding to one size of transformation unit. The size of the transformation unit that forms the basis for generating the quantization matrices is typically the smallest of the sizes of multiple transformation units. That is, if there are four candidate transformation units of sizes, such as 4x4, 8x8, 16x16, and 32x32, as in HEVC, information for generating quantization matrices of other sizes from a 4x4 quantization matrix can be generated. The information generated by the matrix processing unit 150 may include, for example, the base matrix information and difference matrix information, which will be described later. The information generated by the matrix processing unit 150 is output to the reversible encoding unit 16 and can be inserted into the header of the encoded stream.

[0057] In this specification, we will mainly describe examples in which a larger quantization matrix is ​​generated from the smallest quantization matrix. However, we are not limited to such examples, and a smaller quantization matrix and / or a larger quantization matrix may be generated from a quantization matrix of a size other than the smallest.

[0058] [1-3. Detailed Configuration Example of Matrix Processing Unit] Figure 3 is a block diagram showing a more detailed example of the matrix processing unit 150 of the orthogonal transformation / quantization unit 14 shown in Figure 2. Referring to Figure 3, the matrix processing unit 150 includes a prediction unit 152 and a difference calculation unit 154.

[0059] (1) Prediction unit The prediction unit 152 retrieves a set of quantization matrices stored in the quantization matrix buffer 140 and predicts a larger second quantization matrix from the first quantization matrix included in the retrieved set. For example, a 4x4 quantization matrix SL1 is defined as follows:

[0060]

[0061] The 8x8 prediction matrix PSL2 predicted from the quantization matrix SL1 by the prediction unit 152 can be calculated, for example, according to the following prediction formula (2):

[0062]

[0063] Referring to prediction formula (2), the prediction matrix PSL2 is a matrix generated by replicating either of two adjacent elements in the quantization matrix SL1 as an element between the two elements.

[0064] Instead, the prediction matrix PSL2 may be calculated from the quantization matrix SL1 according to the following prediction formula (3):

[0065]

[0066] Referring to prediction formula (3), the prediction matrix PSL2 is a matrix generated by linearly interpolating an element between two adjacent elements in the quantization matrix SL1 from the two elements. Note that the rightmost element of the prediction matrix PSL2 in prediction formula (3) is replicated from the element one to its left, but those rightmost elements may be calculated by linear extrapolation instead of replication. Similarly, the bottom element of the prediction matrix PSL2 in prediction formula (3) may also be calculated by linear extrapolation instead of being replicated from the element one above it. For example, the 8-by-8 component PSL2 of the prediction matrix PSL2 ８，８ is a in prediction formula (3) ３３ but may also be calculated as follows according to linear extrapolation:

[0067]

[0068] Prediction formula (2) is a formula capable of generating the prediction matrix PSL2 with less computational cost compared to prediction formula (3). On the other hand, by using prediction formula (3), it is possible to obtain a smoother prediction matrix closer to the originally used quantization matrix. Therefore, by using prediction formula (3), each element of the difference matrix to be described later can be made closer to zero, reducing the amount of information to be encoded.

[0069] Note that prediction formulas (2) and (3) are merely examples of available prediction formulas, and any other arbitrary prediction formula may also be used.

[0070] When the prediction unit 152 generates the prediction matrix PSL2 from the quantization matrix SL1, it outputs the generated prediction matrix PSL2 to the difference calculation unit 154. Further, the prediction unit 152 predicts, for example, the 16×16 prediction matrix PSL3 from the 8×8 quantization matrix SL2 included in the set of quantization matrices, and outputs the prediction matrix PSL3 to the difference calculation unit 154. Furthermore, the prediction unit 152 predicts the 32×32 prediction matrix PSL4 from the 16×16 quantization matrix SL3 included in the set of quantization matrices, and outputs the prediction matrix PSL4 to the difference calculation unit 154. The prediction of these prediction matrices PSL3 and PSL4 may also be performed according to a prediction formula equivalent to the above-described prediction formula (2) or (3). Further, the prediction unit 152 outputs the basis matrix information for specifying the 4×4 quantization matrix SL1 on which the generation of the above-described prediction matrices PSL2, PSL3, and PSL4 is based to the reversible encoding unit 16.

[0071] (2) Difference calculation unit The difference calculation unit 154 calculates difference matrices DSL2, DSL3, and DSL4 representing the differences between the prediction matrices PSL2, PSL3, and PSL4 input from the prediction unit 152 and the corresponding quantization matrices SL2, SL3, and SL4 according to formulas (5) to (7), respectively:

[0072]

[0073] Then, the difference calculation unit 154 outputs the difference matrix information representing these difference matrices DSL2, DSL3, and DSL4 to the reversible encoding unit 16.

[0074] Note that when a predetermined quantization matrix is used for a certain size, the matrix processing unit 150 outputs only a flag indicating that the predetermined quantization matrix is used to the reversible encoding unit 16 in association with the corresponding size without performing the prediction and difference calculation of the quantization matrix of that size. Further, when the difference between the prediction matrix and the quantization matrix is zero, the difference calculation unit 154 may output only a flag indicating that there is no difference to the reversible encoding unit 16 instead of outputting the difference matrix information. Further, when the quantization matrix is not updated at the timing of picture switching, the matrix processing unit 150 may output only a flag indicating that the quantization matrix is not updated to the reversible encoding unit 16. <000\0101>

[0075] [1-4. Examples of information to be encoded] (1) Sequence parameter set Figure 4 is an explanatory diagram showing an example of information inserted into the sequence parameter set in this embodiment. Referring to Figure 4, three types of information are shown as information encoded for each size of the quantization matrix (size of the transformation unit (TU)): "matrix type flag," "difference flag," and "(encoded) matrix information."

[0076] The matrix type flag specifies whether a user-defined quantization matrix or a default quantization matrix is ​​used for each size. If the matrix type flag is "1" for a given size, the quantization matrix for that size is user-defined. If the matrix type flag is "0" for a given size, the quantization matrix for that size is the default quantization matrix. When the matrix type flag is "0", the matrix information, the difference matrix information, and the difference flag described below are not encoded.

[0077] The difference flag is a flag that indicates whether or not a difference exists between the prediction matrix and the quantization matrix for each size, when the matrix type flag is "1: user defined". If the difference flag is "1" for a certain size, a difference exists between the prediction matrix and the quantization matrix of that size, and the difference matrix information is encoded. If the difference flag is "0" for a certain size, the difference matrix information of that size is not encoded. Note that for the size that forms the basis of the prediction (for example, 4x4), the difference flag is not encoded regardless of the matrix type flag.

[0078] (2) Picture parameter set Figure 5 is an explanatory diagram showing an example of information inserted into the picture parameter set in this embodiment. Referring to Figure 5, four types of information are shown as information encoded for each size of the quantization matrix (size of the transformation unit (TU)): "update flag," "matrix type flag," "difference flag," and "(encoded) matrix information." Of these, the meanings of the matrix type flag and the difference flag are the same as the flags with the same names in the sequence parameter set explained using Figure 4.

[0079] The update flag indicates whether or not the quantization matrix should be updated at the time of picture switching for each size. If the update flag is "1" for a certain size, the quantization matrix for that size is updated. If the update flag is "0", the quantization matrix for that size is not updated, and the quantization matrix set for the previous picture or the current sequence is used as is. If the update flag is "0", the matrix type flag, difference flag, and difference matrix information (matrix information in the case of 4x4) are not encoded for that size.

[0080] <2. Processing flow during encoding according to one embodiment> Figures 6A and 6B are flowcharts showing a first example of the processing flow during encoding according to this embodiment. The processing shown in the flowchart can be mainly performed by the matrix processing unit 150 and the reversible encoding unit 16 for each sequence of encoded streams.

[0081] Referring to Figure 6A, first the matrix processing unit 150 obtains a set of quantization matrices to be used by the quantization unit 130 in the sequence from the quantization matrix buffer 140 (step S100). Here, as an example, the set of quantization matrices is assumed to include quantization matrices corresponding to sizes 4×4, 8×8, 16×16, and 32×32.

[0082] Next, the matrix processing unit 150 determines whether the 4x4 quantization matrix is ​​a user-defined matrix (step S102). If the 4x4 quantization matrix is ​​a user-defined matrix, the reversible encoding unit 16 encodes a matrix type flag (=1) and basic matrix information representing the 4x4 quantization matrix (step S106). On the other hand, if the 4x4 quantization matrix is ​​a default matrix, the reversible encoding unit 16 encodes only the matrix type flag (=0) (step S108).

[0083] Next, the matrix processing unit 150 determines whether the 8x8 quantization matrix is ​​a user-defined matrix (step S112). If the 8x8 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates an 8x8 prediction matrix from a 4x4 quantization matrix according to, for example, the prediction formula (2) or (3) described above (step S114). The reversible coding unit 16 then encodes a matrix type flag (=1), a difference flag, and difference matrix information (only if there is a difference) representing the difference between the 8x8 quantization matrix and the calculated prediction matrix (step S116). On the other hand, if the 8x8 quantization matrix is ​​a default matrix, the reversible coding unit 16 encodes only the matrix type flag (=0) (step S118).

[0084] Next, referring to Figure 6B, the matrix processing unit 150 determines whether the 16x16 quantization matrix is ​​a user-defined matrix (step S122). If the 16x16 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates a 16x16 prediction matrix from the 8x8 quantization matrix (step S124). The reversible coding unit 16 then encodes the matrix type flag (=1), the difference flag, and difference matrix information (only if there is a difference) representing the difference between the 16x16 quantization matrix and the calculated prediction matrix (step S126). On the other hand, if the 16x16 quantization matrix is ​​a default matrix, the reversible coding unit 16 encodes only the matrix type flag (=0) (step S128).

[0085] Next, the matrix processing unit 150 determines whether the 32x32 quantization matrix is ​​a user-defined matrix (step S132). If the 32x32 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates a 32x32 prediction matrix from the 16x16 quantization matrix (step S134). The reversible coding unit 16 then encodes a matrix type flag (=1), a difference flag, and difference matrix information (only if there is a difference) representing the difference between the 32x32 quantization matrix and the calculated prediction matrix (step S136). On the other hand, if the 32x32 quantization matrix is ​​a default matrix, the reversible coding unit 16 encodes only the matrix type flag (=0) (step S138).

[0086] Figures 7A and 7B are flowcharts showing a second example of the encoding process according to this embodiment. The processing shown in the flowchart can be mainly performed by the matrix processing unit 150 and the reversible encoding unit 16 for each picture of the encoded stream.

[0087] Referring to Figure 7A, first the matrix processing unit 150 obtains a set of quantization matrices to be used by the quantization unit 130 in the picture from the quantization matrix buffer 140 (step S150). Here, as in the examples of Figures 6A and 6B, the set of quantization matrices is assumed to include quantization matrices corresponding to the sizes 4×4, 8×8, 16×16, and 32×32.

[0088] Next, the matrix processing unit 150 determines whether or not the 4x4 quantization matrix is ​​updated in the picture (step S152). If the quantization matrix is ​​not updated, the reversible encoding unit 16 encodes only the update flag (=0) (step S158). On the other hand, if the quantization matrix is ​​updated, the process proceeds to step S154. If the quantization matrix is ​​updated, the matrix processing unit 150 determines whether or not the new 4x4 quantization matrix is ​​a user-defined matrix (step S154). If the 4x4 quantization matrix is ​​a user-defined matrix, the reversible encoding unit 16 encodes the update flag (=1), the matrix type flag (=1), and the basic matrix information representing the 4x4 quantization matrix (step S156). On the other hand, if the 4x4 quantization matrix is ​​a default matrix, the reversible encoding unit 16 encodes the update flag (=1) and the matrix type flag (=0) (step S158).

[0089] Next, the matrix processing unit 150 determines whether or not the 8x8 quantization matrix is ​​updated in the picture (step S160). If the quantization matrix is ​​not updated, the reversible encoding unit 16 encodes only the update flag (=0) (step S168). On the other hand, if the quantization matrix is ​​updated, the process proceeds to step S162. If the quantization matrix is ​​updated, the matrix processing unit 150 determines whether or not the 8x8 quantization matrix is ​​a user-defined matrix (step S162). If the 8x8 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates an 8x8 prediction matrix from the new picture's 4x4 quantization matrix, regardless of whether or not the 4x4 quantization matrix has been updated (step S164). Then, the reversible encoding unit 16 encodes the update flag (=1), the matrix type flag (=1), the difference flag, and difference matrix information (only if there is a difference) that represents the difference between the 8x8 quantization matrix and the calculated prediction matrix (step S166). On the other hand, if the 8x8 quantization matrix is ​​a predetermined matrix, the reversible encoding unit 16 encodes the update flag (=1) and the matrix type flag (=0) (step S168).

[0090] Next, referring to Figure 7B, the matrix processing unit 150 determines whether or not the 16x16 quantization matrix is ​​updated in the picture (step S170). If the quantization matrix is ​​not updated, the reversible encoding unit 16 encodes only the update flag (=0) (step S178). On the other hand, if the quantization matrix is ​​updated, the process proceeds to step S172. If the quantization matrix is ​​updated, the matrix processing unit 150 determines whether or not the 16x16 quantization matrix is ​​a user-defined matrix (step S172). If the 16x16 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates a 16x16 prediction matrix from the new picture's 8x8 quantization matrix, regardless of whether or not the 8x8 quantization matrix has been updated (step S174). Then, the reversible encoding unit 16 encodes the update flag (=1), the matrix type flag (=1), the difference flag, and difference matrix information (only if there is a difference) that represents the difference between the 16x16 quantization matrix and the calculated prediction matrix (step S176). On the other hand, if the 16x16 quantization matrix is ​​a default matrix, the reversible encoding unit 16 encodes the update flag (=1) and the matrix type flag (=0) (step S178).

[0091] Next, the matrix processing unit 150 determines whether or not the 32x32 quantization matrix is ​​updated in the picture (step S180). If the quantization matrix is ​​not updated, the reversible encoding unit 16 encodes only the update flag (=0) (step S188). On the other hand, if the quantization matrix is ​​updated, the process proceeds to step S182. If the quantization matrix is ​​updated, the matrix processing unit 150 determines whether or not the 32x32 quantization matrix is ​​a user-defined matrix (step S182). If the 32x32 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates a 32x32 prediction matrix from the 16x16 quantization matrix of the new picture, regardless of whether or not the 16x16 quantization matrix has been updated (step S184). Then, the reversible encoding unit 16 encodes the update flag (=1), the matrix type flag (=1), the difference flag, and difference matrix information (only if there is a difference) representing the difference between the 32x32 quantization matrix and the calculated prediction matrix (step S186). On the other hand, if the 32x32 quantization matrix is ​​a default matrix, the reversible encoding unit 16 encodes the update flag (=1) and the matrix type flag (=0) (step S188).

[0092] Thus, by using a method that predicts other quantization matrices based on any one of the quantization matrices, the need to transmit multiple quantization matrices corresponding to the sizes of multiple transformation units from the encoding side to the decoding side is eliminated. Therefore, even when the number of quantization matrices is large, the increase in the amount of code is effectively suppressed.

[0093] <3. Example of the configuration of an image decoding device according to one embodiment> This section describes an example configuration of an image decoding device according to one embodiment.

[0094] [3-1. Example of overall configuration] Figure 8 is a block diagram showing an example of the configuration of an image decoding device 60 according to one embodiment. Referring to Figure 8, the image decoding device 60 includes a storage buffer 61, a reversible decoding unit 62, an inverse quantization / inverse orthogonal transformation unit 63, an addition unit 65, a deblocking filter 66, a sorting buffer 67, a D / A (Digital to Analogue) conversion unit 68, a frame memory 69, selectors 70 and 71, an intra prediction unit 80, and a motion compensation unit 90.

[0095] The storage buffer 61 temporarily stores the encoded stream input via the transmission line using a storage medium.

[0096] The reversible decoding unit 62 decodes the encoded stream input from the storage buffer 61 according to the encoding scheme used during encoding. The reversible decoding unit 62 also decodes the information multiplexed in the header area of ​​the encoded stream. The information multiplexed in the header area of ​​the encoded stream may include, for example, the basic matrix information and difference matrix information for generating the quantization matrix described above, as well as information regarding intra-prediction and inter-prediction in the block header. The reversible decoding unit 62 outputs the decoded quantized data and information for generating the quantization matrix to the inverse quantization / inverse orthogonal transformation unit 63. The reversible decoding unit 62 also outputs information regarding intra-prediction to the intra-prediction unit 80. The reversible decoding unit 62 also outputs information regarding inter-prediction to the motion compensation unit 90.

[0097] The inverse quantization / inverse orthogonal transformation unit 63 generates prediction error data by performing inverse quantization and inverse orthogonal transformation on the quantized data input from the reversible decoding unit 62. The inverse quantization / inverse orthogonal transformation unit 63 then outputs the generated prediction error data to the addition unit 65.

[0098] The adder 65 generates decoded image data by adding the prediction error data input from the inverse quantization / inverse orthogonal transformation unit 63 and the prediction image data input from the selector 71. The adder 65 then outputs the generated decoded image data to the deblocking filter 66 and the frame memory 69.

[0099] The deblocking filter 66 removes block distortion by filtering the decoded image data input from the summing unit 65, and outputs the filtered decoded image data to the rearrangement buffer 67 and the frame memory 69.

[0100] The sorting buffer 67 generates a series of time-series image data by sorting the images input from the deblocking filter 66. The sorting buffer 67 then outputs the generated image data to the D / A conversion unit 68.

[0101] The D / A conversion unit 68 converts the digital image data input from the sorting buffer 67 into an analog image signal. The D / A conversion unit 68 then displays the image by outputting the analog image signal to a display (not shown) connected to the image decoding device 60, for example.

[0102] The frame memory 69 stores the decoded image data before filtering input from the adder 65 and the decoded image data after filtering input from the deblocking filter 66 using a storage medium.

[0103] The selector 70 switches the output destination of the image data from the frame memory 69 between the intra prediction unit 80 and the motion compensation unit 90 for each block in the image, according to the mode information acquired by the reversible decoding unit 62. For example, when the intra prediction mode is specified, the selector 70 outputs the decoded image data before filtering supplied from the frame memory 69 to the intra prediction unit 80 as reference image data. Also, when the inter prediction mode is specified, the selector 70 outputs the decoded image data after filtering supplied from the frame memory 69 to the motion compensation unit 90 as reference image data.

[0104] The selector 71 switches the source of the predicted image data to be supplied to the adder 65 between the intra prediction unit 80 and the motion compensation unit 90 for each block in the image, according to the mode information acquired by the reversible decoding unit 62. For example, if the intra prediction mode is specified, the selector 71 supplies the predicted image data output from the intra prediction unit 80 to the adder 65. If the inter prediction mode is specified, the selector 71 supplies the predicted image data output from the motion compensation unit 90 to the adder 65.

[0105] The intra prediction unit 80 predicts the pixel values ​​within the screen based on the intra prediction information input from the reversible decoding unit 62 and the reference image data from the frame memory 69, and generates predicted image data. The intra prediction unit 80 then outputs the generated predicted image data to the selector 71.

[0106] The motion compensation unit 90 performs motion compensation processing based on the inter prediction information input from the reversible decoding unit 62 and the reference image data from the frame memory 69, and generates predicted image data. The motion compensation unit 90 then outputs the generated predicted image data to the selector 71.

[0107] [3-2. Example of the configuration of the inverse quantization / inverse orthogonal transformation unit] Figure 9 is a block diagram showing an example of a detailed configuration of the inverse quantization / inverse orthogonal transformation unit 63 of the image decoding device 60 shown in Figure 8. Referring to Figure 9, the inverse quantization / inverse orthogonal transformation unit 63 includes a matrix generation unit 210, a selection unit 230, an inverse quantization unit 240, and an inverse orthogonal transformation unit 250.

[0108] (1) Matrix generation unit The matrix generation unit 210 generates quantization matrices corresponding to one or more other transformation units of a given size from a quantization matrix corresponding to one transformation unit of a given size for each sequence and picture of the encoded stream. The size of the transformation unit on which the quantization matrices are generated is typically the smallest of the sizes of multiple transformation units. In this embodiment, the matrix generation unit 210 generates 8x8, 16x16, and 32x32 quantization matrices from the smallest size, a 4x4 quantization matrix, using difference matrix information for larger sizes.

[0109] (2) Selection section The selection unit 230 selects a transformation unit (TU) from a plurality of transformation units of different sizes to be used for the inverse orthogonal transformation of the image data to be decoded. Candidate transformation unit sizes that can be selected by the selection unit 230 include, for example, 4×4 and 8×8 for H.264 / AVC, and 4×4, 8×8, 16×16, and 32×32 for HEVC. The selection unit 230 may also select a transformation unit based, for example, on the LCU, SCU, and split_flag included in the header of the encoded stream. The selection unit 230 then outputs information specifying the size of the selected transformation unit to the inverse quantization unit 240 and the inverse orthogonal transformation unit 250.

[0110] (3) Inverse quantization section The inverse quantization unit 240 inversely quantizes the transformation coefficient data that was quantized during image encoding using the quantization matrix corresponding to the transformation unit selected by the selection unit 230. The quantization matrix used for the inverse quantization process includes the matrix generated by the matrix generation unit 210. That is, for example, if the selection unit 230 selects an 8x8, 16x16, or 32x32 transformation unit, the quantization matrix generated by the matrix generation unit 210 from a 4x4 quantization matrix may be used as the quantization matrix corresponding to the selected transformation unit. The inverse quantization unit 240 then outputs the inversely quantized transformation coefficient data to the inverse orthogonal transformation unit 250.

[0111] (4) Inverse orthogonal transformation section The inverse orthogonal transform unit 250 generates prediction error data by performing an inverse orthogonal transform on the conversion coefficient data inversely quantized by the inverse quantization unit 240 according to the orthogonal transform method used during encoding, using the selected transformation units. The inverse orthogonal transform unit 250 then outputs the generated prediction error data to the adder unit 65.

[0112] [3-3. Detailed Configuration Example of the Matrix Generation Unit] Figure 10 is a block diagram showing a more detailed example of the matrix generation unit 210 of the inverse quantization / inverse orthogonal transformation unit 63 shown in Figure 9. Referring to Figure 10, the matrix generation unit 210 includes a base matrix acquisition unit 212, a difference acquisition unit 214, a prediction unit 216, a reconstruction unit 218, and a quantization matrix buffer 220.

[0113] (1) Basic matrix acquisition unit The base matrix acquisition unit 212 acquires base matrix information input from the reversible decoding unit 62. In this embodiment, the base matrix information is information that identifies the 4x4 quantization matrix SL1, which is the smallest size as described above. The base matrix acquisition unit 212 then stores the 4x4 quantization matrix SL1 identified from the acquired base matrix information in the quantization matrix buffer 220. If the matrix type flag acquired for each sequence or each picture is "0", the base matrix acquisition unit 212 stores the default 4x4 quantization matrix in the quantization matrix buffer 220 without acquiring base matrix information. Also, if the update flag acquired for each picture is "0", the base matrix acquisition unit 212 does not update the quantization matrix SL1 stored in the quantization matrix buffer 220 by the previous processing. The base matrix acquisition unit 212 also outputs the 4x4 quantization matrix SL1 to the prediction unit 216.

[0114] (2) Difference acquisition unit The difference acquisition unit 214 acquires difference matrix information input from the reversible decoding unit 62. In this embodiment, as described above, the difference matrix information is information that identifies the difference matrices DSL2, DSL3, and DSL4, which represent the difference between the prediction matrices PSL2, PSL3, and PSL4 predicted from the 4x4 quantization matrix SL1 and the quantization matrices SL2, SL3, and SL4. The difference acquisition unit 214 outputs the difference matrices DSL2, DSL3, and DSL4 identified by the difference matrix information to the reconstruction unit 218. Note that if the matrix type flag acquired for each sequence or each picture is "0" or the difference flag is "0", the difference acquisition unit 214 does not acquire difference matrix information and sets the difference matrix of the corresponding size to a zero matrix. Also, if the update flag acquired for each picture is "0", the difference acquisition unit 214 does not output a difference matrix for the corresponding size.

[0115] (3) Prediction section The prediction unit 216 calculates a larger 8x8 prediction matrix PSL2 from the base matrix input from the base matrix acquisition unit 212, i.e., the 4x4 quantization matrix SL1 in this embodiment, according to the prediction formula used during image encoding (for example, prediction formula (2) or (3) described above). The prediction unit 216 then calculates a 16x16 prediction matrix PSL3 from the quantization matrix SL2 reconstructed by the reconstruction unit 218 using the calculated 8x8 prediction matrix PSL2. Furthermore, the prediction unit 216 calculates a 32x32 prediction matrix PSL4 from the quantization matrix SL3 reconstructed by the reconstruction unit 218 using the calculated 16x16 prediction matrix PSL3. The prediction unit 216 outputs prediction matrices PSL2, PSL3, and PSL4 to the reconstruction unit 218, respectively. Note that the prediction unit 216 does not generate prediction matrices for sizes where the matrix type flag is "0", and uses a default quantization matrix to calculate larger prediction matrices. Furthermore, the base matrix acquisition unit 212 does not generate prediction matrices for sizes where the update flag is "0," and instead uses the quantization matrix generated in a previous process to calculate prediction matrices for larger sizes.

[0116] (4) Reconstruction section The reconstruction unit 218 reconstructs the quantization matrices SL2, SL3, and SL4, respectively, by adding the prediction matrices PSL2, PSL3, and PSL4 input from the prediction unit 216 with the difference matrices DSL2, DSL3, and DSL4 input from the difference acquisition unit 214.

[0117]

[0118] The reconstruction unit 218 then stores the reconstructed 8x8, 16x16, and 32x32 quantization matrices SL2, SL3, and SL4 in the quantization matrix buffer 220. If the matrix type flag acquired for each sequence or each picture is "0", the reconstruction unit 218 stores a default quantization matrix of the corresponding size in the quantization matrix buffer 220. Also, if the update flag acquired for each picture is "0", the base matrix acquisition unit 212 does not update the corresponding size quantization matrices SL2, SL3, or SL4 stored in the quantization matrix buffer 220 by the previous processing.

[0119] (5) Quantization matrix buffer The quantization matrix buffer 220 temporarily stores the quantization matrix SL1 identified by the base matrix acquisition unit 212, and the quantization matrices SL2, SL3, and SL4 reconstructed by the reconstruction unit 218. These quantization matrices SL1, SL2, SL3, and SL4 stored in the quantization matrix buffer 220 are used for inverse quantization processing of the quantized conversion coefficient data by the inverse quantization unit 240.

[0120] The configuration of the inverse quantization / inverse orthogonal transformation unit 63 of the image decoding device 60 described in this section can also be applied to the inverse quantization unit 21 and inverse orthogonal transformation unit 22 of the image encoding device 10 shown in Figure 1.

[0121] <4. Processing flow during decoding according to one embodiment> Figures 11A and 11B are flowcharts showing a first example of the processing flow during decoding according to this embodiment. The processing shown in the flowchart can be mainly performed by the matrix generation unit 210 for each sequence of encoded streams.

[0122] Referring to Figure 11A, first, the matrix generation unit 210 determines whether the 4x4 quantization matrix is ​​a user-defined matrix based on the matrix type flag included in the sequence parameter set of the sequence (step S202). If the 4x4 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 sets the 4x4 quantization matrix using the basic matrix information (i.e., stores it in the quantization matrix buffer 220) (step S204). On the other hand, if the 4x4 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 4x4 quantization matrix (step S206).

[0123] Next, the matrix generation unit 210 determines whether the 8x8 quantization matrix is ​​a user-defined matrix (step S212). If the 8x8 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates an 8x8 prediction matrix from the 4x4 quantization matrix according to, for example, the prediction formula (2) or (3) described above, and adds the calculated prediction matrix to the 8x8 difference matrix. This reconstructs the 8x8 quantization matrix (step S214). If the 8x8 difference flag is "0", the difference matrix is ​​a zero matrix, and the 8x8 prediction matrix can be set as is as the quantization matrix. On the other hand, if the 8x8 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 8x8 quantization matrix (step S216).

[0124] Next, referring to Figure 11B, the matrix generation unit 210 determines whether the 16x16 quantization matrix is ​​a user-defined matrix (step S222). If the 16x16 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates a 16x16 prediction matrix from the 8x8 quantization matrix and adds the calculated prediction matrix to the 16x16 difference matrix. This reconstructs the 16x16 quantization matrix (step S224). If the 16x16 difference flag is "0", the difference matrix is ​​a zero matrix, and the 16x16 prediction matrix can be set as is as the quantization matrix. On the other hand, if the 16x16 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 16x16 quantization matrix (step S226).

[0125] Next, the matrix generation unit 210 determines whether the 32x32 quantization matrix is ​​a user-defined matrix (step S232). If the 32x32 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates a 32x32 prediction matrix from the 16x16 quantization matrix and adds the calculated prediction matrix to the 32x32 difference matrix. This reconstructs the 32x32 quantization matrix (step S234). If the 32x32 difference flag is "0", the difference matrix is ​​a zero matrix, and the 32x32 prediction matrix can be set as the quantization matrix. On the other hand, if the 32x32 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 32x32 quantization matrix (step S236).

[0126] Figures 12A and 12B are flowcharts showing a second example of the processing flow during decoding according to this embodiment. The processing shown in the flowchart can be mainly performed by the matrix generation unit 210 for each picture of the encoded stream.

[0127] Referring to Figure 12A, first, the matrix generation unit 210 determines whether or not the 4x4 quantization matrix is ​​updated in the picture based on the update flag included in the picture parameter set (step S250). If the 4x4 quantization matrix is ​​not updated, steps S252 to S256 are skipped. If the 4x4 quantization matrix is ​​updated, the matrix generation unit 210 determines whether or not the new 4x4 quantization matrix is ​​a user-defined matrix based on the matrix type flag (step S252). If the 4x4 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 sets the 4x4 quantization matrix using the basic matrix information (step S254). On the other hand, if the 4x4 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 4x4 quantization matrix (step S256).

[0128] Next, the matrix generation unit 210 determines whether or not the 8x8 quantization matrix is ​​updated in the picture based on the update flag (step S260). If the 8x8 quantization matrix is ​​not updated, steps S262 to S266 are skipped. If the 8x8 quantization matrix is ​​updated, the matrix generation unit 210 determines whether or not the new 8x8 quantization matrix is ​​a user-defined matrix based on the matrix type flag (step S262). If the 8x8 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates an 8x8 prediction matrix from the 4x4 quantization matrix of the new picture, regardless of whether or not the 4x4 quantization matrix has been updated, and adds the calculated prediction matrix to the 8x8 difference matrix. The 8x8 quantization matrix is ​​then reconstructed (step S264). If the 8x8 difference flag is "0", the difference matrix is ​​a zero matrix, and the 8x8 prediction matrix can be set as the quantization matrix. On the other hand, if the 8x8 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 8x8 quantization matrix (step S266).

[0129] Next, referring to Figure 12B, the matrix generation unit 210 determines whether or not the 16x16 quantization matrix is ​​updated in the picture based on the update flag (step S270). If the 16x16 quantization matrix is ​​not updated, steps S272 to S276 are skipped. If the 16x16 quantization matrix is ​​updated, the matrix generation unit 210 determines whether or not the new 16x16 quantization matrix is ​​a user-defined matrix based on the matrix type flag (step S272). If the 16x16 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates a 16x16 prediction matrix from the 8x8 quantization matrix of the new picture, regardless of whether or not the 8x8 quantization matrix has been updated, and adds the calculated prediction matrix to the 16x16 difference matrix. As a result, the 16x16 quantization matrix is ​​reconstructed (step S274). If the 16x16 difference flag is "0", the difference matrix is ​​a zero matrix, and the 16x16 prediction matrix can be set as the quantization matrix. On the other hand, if the 16x16 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 16x16 quantization matrix (step S276).

[0130] Next, the matrix generation unit 210 determines, based on the update flag, whether or not the 32x32 quantization matrix is ​​updated in the picture (step S280). If the 32x32 quantization matrix is ​​not updated, steps S282 to S286 are skipped. If the 32x32 quantization matrix is ​​updated, the matrix generation unit 210 determines, based on the matrix type flag, whether or not the new 32x32 quantization matrix is ​​a user-defined matrix (step S282). If the 32x32 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates a 32x32 prediction matrix from the 16x16 quantization matrix of the new picture, regardless of whether or not the 16x16 quantization matrix has been updated, and adds the calculated prediction matrix to the 32x32 difference matrix. The 32x32 quantization matrix is ​​then reconstructed (step S284). If the 32x32 difference flag is "0", the difference matrix is ​​a zero matrix, and the 32x32 prediction matrix can be set as the quantization matrix. On the other hand, if the 32x32 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 32x32 quantization matrix (step S286).

[0131] Thus, by using a method that predicts other quantization matrices based on any one of the quantization matrices, the quantization matrix can be appropriately reconstructed on the decoding side even when only the difference information for the quantization matrix to be predicted is transmitted from the encoding side to the decoding side. Therefore, even when the number of quantization matrices is large, the increase in the amount of code is effectively suppressed.

[0132] In this specification, examples have been mainly described in which only one type of quantization matrix is ​​set for the size of one transformation unit. However, the specification is not limited to such examples, and multiple types of quantization matrices may be set for the size of one transformation unit. In that case, the sequence parameter set and the picture parameter set may include additional flags indicating which of the multiple types of quantization matrices should be used as the basis for predicting larger quantization matrices. Alternatively, multiple types of quantization matrices may be set for the size of one transformation unit, and the quantization matrices may be selectively switched for each slice or block within a single picture.

[0133] <5. Variant Example> As mentioned above, the techniques disclosed herein may be implemented by predicting a smaller quantization matrix from a larger quantization matrix. For example, an 8x8 quantization matrix SL2 is defined as follows:

[0134]

[0135] The prediction unit 152 of the orthogonal transformation / quantization unit 14 of the image coding device 10 can calculate a 4x4 prediction matrix PSL1 from such a quantization matrix SL2 according to, for example, the following prediction formula (12):

[0136]

[0137] Referring to prediction formula (12), the prediction matrix PSL1 is a matrix generated by thinning out the elements of the quantization matrix SL2 by every other row and every other column. The positions of the elements to be thinned out are not limited to the example in prediction formula (12), and may be other positions. Furthermore, by increasing the number of elements to be thinned out, it is possible to generate a prediction matrix from the quantization matrix with a side size of one-quarter or less.

[0138] Alternatively, the prediction matrix PSL1 may be calculated from the quantization matrix SL2 according to the following prediction formula (13):

[0139]

[0140] Referring to prediction formula (13), the prediction matrix PSL1 is a matrix generated by calculating the average of four elements adjacent to each other vertically and horizontally in the quantization matrix SL2 as one element of the prediction matrix PSL1. Alternatively, a prediction matrix with a side size of one-quarter or less can be generated from the quantization matrix by calculating the average of more (e.g., 16) elements adjacent to each other vertically and horizontally. Note that instead of the mean as in prediction formula (13), other types of representative values ​​such as the median, minimum, or maximum may be calculated from multiple elements.

[0141] Even when a smaller prediction matrix is ​​calculated from a larger quantization matrix, the difference calculation unit 154 calculates a difference matrix representing the difference between the prediction matrix input from the prediction unit 152 and the corresponding quantization matrix, and outputs the difference matrix information representing the calculated difference matrix to the reversible coding unit 16. Then, the matrix generation unit 210 of the inverse quantization / inverse orthogonal transformation unit 63 of the image decoding device 60 generates a smaller quantization matrix from the quantization matrix specified by the basic matrix information using one of the prediction formulas and difference matrix information described above.

[0142] Figures 13A and 13B are flowcharts showing an example of the processing flow during encoding according to this modified example. The processing shown in the flowchart can be mainly performed by the matrix processing unit 150 and the reversible encoding unit 16 for each sequence of the encoded stream.

[0143] Referring to Figure 13A, first the matrix processing unit 150 obtains a set of quantization matrices to be used by the quantization unit 130 in the sequence from the quantization matrix buffer 140 (step S300). Here, as an example, the set of quantization matrices is assumed to include quantization matrices corresponding to sizes 4×4, 8×8, 16×16, and 32×32.

[0144] Next, the matrix processing unit 150 determines whether the 32x32 quantization matrix is ​​a user-defined matrix (step S302). If the 32x32 quantization matrix is ​​a user-defined matrix, the reversible encoding unit 16 encodes a matrix type flag (=1) and basic matrix information representing the 32x32 quantization matrix (step S306). On the other hand, if the 32x32 quantization matrix is ​​a default matrix, the reversible encoding unit 16 encodes only the matrix type flag (=0) (step S308).

[0145] Next, the matrix processing unit 150 determines whether the 16x16 quantization matrix is ​​a user-defined matrix (step S312). If the 16x16 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates a 16x16 prediction matrix from a 32x32 quantization matrix according to, for example, the prediction formula (12) or (13) described above (step S314). The reversible coding unit 16 then encodes the matrix type flag (=1), the difference flag, and difference matrix information (only if there is a difference) representing the difference between the 16x16 quantization matrix and the calculated prediction matrix (step S316). On the other hand, if the 16x16 quantization matrix is ​​a default matrix, the reversible coding unit 16 encodes only the matrix type flag (=0) (step S318).

[0146] Next, referring to Figure 13B, the matrix processing unit 150 determines whether the 8x8 quantization matrix is ​​a user-defined matrix (step S322). If the 8x8 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates an 8x8 prediction matrix from the 16x16 quantization matrix (step S324). The reversible coding unit 16 then encodes the matrix type flag (=1), the difference flag, and difference matrix information (only if there is a difference) representing the difference between the 8x8 quantization matrix and the calculated prediction matrix (step S326). On the other hand, if the 8x8 quantization matrix is ​​a default matrix, the reversible coding unit 16 encodes only the matrix type flag (=0) (step S328).

[0147] Next, the matrix processing unit 150 determines whether the 4x4 quantization matrix is ​​a user-defined matrix (step S332). If the 4x4 quantization matrix is ​​a user-defined matrix, the matrix processing unit 150 calculates a 4x4 prediction matrix from the 8x8 quantization matrix (step S334). The reversible coding unit 16 then encodes the matrix type flag (=1), the difference flag, and difference matrix information (only if there is a difference) representing the difference between the 4x4 quantization matrix and the calculated prediction matrix (step S336). On the other hand, if the 4x4 quantization matrix is ​​a default matrix, the reversible coding unit 16 encodes only the matrix type flag (=0) (step S338).

[0148] Thus, in this modified example, when defining the quantization matrix using SPS, the prediction matrix can be calculated and encoded in order from the largest quantization matrix. Similarly, when updating the quantization matrix using PPS, the prediction matrix can be calculated and encoded in order from the largest quantization matrix.

[0149] Figures 14A and 14B are flowcharts showing an example of the processing flow during decoding according to this embodiment. The processing shown in the flowchart can be mainly performed by the matrix generation unit 210 for each sequence of encoded streams.

[0150] Referring to Figure 14A, first, the matrix generation unit 210 determines whether the 32x32 quantization matrix is ​​a user-defined matrix based on the matrix type flag included in the sequence parameter set of the sequence (step S402). If the 32x32 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 sets the 32x32 quantization matrix using the basic matrix information (i.e., stores it in the quantization matrix buffer 220) (step S404). On the other hand, if the 32x32 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 32x32 quantization matrix (step S406).

[0151] Next, the matrix generation unit 210 determines whether the 16x16 quantization matrix is ​​a user-defined matrix (step S412). If the 16x16 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates a 16x16 prediction matrix from the 32x32 quantization matrix according to, for example, the prediction formula (12) or (13) described above, and adds the calculated prediction matrix to the 16x16 difference matrix. This reconstructs the 16x16 quantization matrix (step S414). If the 16x16 difference flag is "0", the difference matrix is ​​a zero matrix, and the 16x16 prediction matrix can be set as is as the quantization matrix. On the other hand, if the 16x16 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 16x16 quantization matrix (step S416).

[0152] Next, referring to Figure 14B, the matrix generation unit 210 determines whether the 8x8 quantization matrix is ​​a user-defined matrix (step S422). If the 8x8 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates an 8x8 prediction matrix from the 16x16 quantization matrix and adds the calculated prediction matrix to the 8x8 difference matrix. This reconstructs the 8x8 quantization matrix (step S424). If the 8x8 difference flag is "0", the difference matrix is ​​a zero matrix, and the 8x8 prediction matrix can be set as is as the quantization matrix. On the other hand, if the 8x8 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 8x8 quantization matrix (step S426).

[0153] Next, the matrix generation unit 210 determines whether the 4x4 quantization matrix is ​​a user-defined matrix (step S432). If the 4x4 quantization matrix is ​​a user-defined matrix, the matrix generation unit 210 calculates a 4x4 prediction matrix from the 8x8 quantization matrix and adds the calculated prediction matrix to the 4x4 difference matrix. This reconstructs the 4x4 quantization matrix (step S434). If the 4x4 difference flag is "0", the difference matrix is ​​a zero matrix, and the 4x4 prediction matrix can be set as the quantization matrix. On the other hand, if the 4x4 quantization matrix is ​​a default matrix, the matrix generation unit 210 sets the default 4x4 quantization matrix (step S436).

[0154] Thus, in this modified example, when decoding the quantization matrix from the SPS, the matrices can be reconstructed in order from the largest quantization matrix. Similarly, when updating the quantization matrix with the PPS, the matrices can be reconstructed in order from the largest quantization matrix.

[0155] <6. Application Examples> The image encoding device 10 and image decoding device 60 according to the above embodiment can be applied to various electronic devices such as transmitters or receivers for satellite broadcasting, cable TV and other wired broadcasting, distribution over the Internet, and distribution to terminals via cellular communication; recording devices that record images on media such as optical discs, magnetic discs, and flash memory; or playback devices that reproduce images from these storage media. Four application examples will be described below.

[0156] [6-1. First Application Example] Figure 15 shows an example of a schematic configuration of a television device to which the above-described embodiment is applied. The television device 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, an external interface 909, a control unit 910, a user interface 911, and a bus 912.

[0157] The tuner 902 extracts the signal of the desired channel from the broadcast signal received via the antenna 901 and demodulates the extracted signal. The tuner 902 then outputs the encoded bitstream obtained by demodulation to the demultiplexer 903. In other words, the tuner 902 has the role of a transmission means in the television device 900 that receives the encoded stream in which the image is encoded.

[0158] The demultiplexer 903 separates the video stream and audio stream of the program to be viewed from the encoded bitstream and outputs each separated stream to the decoder 904. The demultiplexer 903 also extracts auxiliary data such as the EPG (Electronic Program Guide) from the encoded bitstream and supplies the extracted data to the control unit 910. The demultiplexer 903 may also descramble the encoded bitstream if it is scrambled.

[0159] The decoder 904 decodes the video stream and audio stream input from the demultiplexer 903. The decoder 904 then outputs the video data generated by the decoding process to the video signal processing unit 905. The decoder 904 also outputs the audio data generated by the decoding process to the audio signal processing unit 907.

[0160] The video signal processing unit 905 plays back the video data input from the decoder 904 and displays the video on the display unit 906. The video signal processing unit 905 may also display an application screen supplied via the network on the display unit 906. The video signal processing unit 905 may also perform additional processing on the video data, such as noise reduction, depending on the settings. Furthermore, the video signal processing unit 905 may generate images of GUI (Graphical User Interface) elements, such as menus, buttons, or cursors, and superimpose the generated images onto the output image.

[0161] The display unit 906 is driven by a drive signal supplied from the video signal processing unit 905 and displays a video or image on the video surface of a display device (for example, a liquid crystal display, plasma display, or OLED).

[0162] The audio signal processing unit 907 performs playback processing such as D / A conversion and amplification on the audio data input from the decoder 904 and outputs sound from the speaker 908. The audio signal processing unit 907 may also perform additional processing on the audio data, such as noise reduction.

[0163] The external interface 909 is an interface for connecting the television device 900 to an external device or network. For example, a video stream or audio stream received via the external interface 909 may be decoded by the decoder 904. That is, the external interface 909 also serves as a transmission means in the television device 900 for receiving an encoded stream in which an image is encoded.

[0164] The control unit 910 has a processor such as a CPU (Central Processing Unit), and memory such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory stores programs executed by the CPU, program data, EPG data, and data acquired via the network. Programs stored in the memory are read and executed by the CPU, for example, when the television device 900 is started up. By executing the programs, the CPU controls the operation of the television device 900, for example, in response to operation signals input from the user interface 911.

[0165] The user interface 911 is connected to the control unit 910. The user interface 911 includes, for example, buttons and switches for the user to operate the television device 900, and a remote control signal receiver. The user interface 911 detects user operations through these components, generates operation signals, and outputs the generated operation signals to the control unit 910.

[0166] The bus 912 interconnects the tuner 902, demultiplexer 903, decoder 904, video signal processing unit 905, audio signal processing unit 907, external interface 909, and control unit 910.

[0167] In the television device 900 configured in this way, the decoder 904 has the function of the image decoding device 60 according to the above embodiment. This makes it possible to suppress the increase in the amount of code when the number of quantization matrices is large when decoding an image in the television device 900.

[0168] [6-2. Second Application Example] Figure 16 shows an example of a schematic configuration of a mobile phone to which the above-described embodiment is applied. The mobile phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a multiplexing / separation unit 928, a recording / playback unit 929, a display unit 930, a control unit 931, an operation unit 932, and a bus 933.

[0169] The antenna 921 is connected to the communication unit 922. The speaker 924 and microphone 925 are connected to the audio codec 923. The operation unit 932 is connected to the control unit 931. The bus 933 interconnects the communication unit 922, the audio codec 923, the camera unit 926, the image processing unit 927, the multiplexing / decompression unit 928, the recording / playback unit 929, the display unit 930, and the control unit 931.

[0170] The mobile phone 920 performs operations such as sending and receiving voice signals, sending and receiving emails or image data, capturing images, and recording data in various operating modes, including voice call mode, data communication mode, shooting mode, and video call mode.

[0171] In voice call mode, the analog voice signal generated by the microphone 925 is supplied to the voice codec 923. The voice codec 923 converts the analog voice signal into voice data, and compresses the converted voice data by A / D conversion. The voice codec 923 then outputs the compressed voice data to the communication unit 922. The communication unit 922 encodes and modulates the voice data to generate a transmission signal. The communication unit 922 then transmits the generated transmission signal to the base station (not shown) via the antenna 921. The communication unit 922 also amplifies and frequency converts the radio signal received via the antenna 921 to acquire the received signal. The communication unit 922 then demodulates and decodes the received signal to generate voice data, and outputs the generated voice data to the voice codec 923. The voice codec 923 decompresses and D / A converts the voice data to generate an analog voice signal. The audio codec 923 then supplies the generated audio signal to the speaker 924 to output sound.

[0172] In data communication mode, for example, the control unit 931 generates character data that constitutes an email in response to an operation by the user via the operation unit 932. The control unit 931 also displays the characters on the display unit 930. The control unit 931 also generates email data in response to a transmission instruction from the user via the operation unit 932 and outputs the generated email data to the communication unit 922. The communication unit 922 encodes and modulates the email data and generates a transmission signal. The communication unit 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. The communication unit 922 also amplifies and frequency converts the radio signal received via the antenna 921 and acquires the received signal. The communication unit 922 then demodulates and decodes the received signal to restore the email data and outputs the restored email data to the control unit 931. The control unit 931 displays the contents of the email on the display unit 930 and stores the email data in the storage medium of the recording and playback unit 929.

[0173] The recording and playback unit 929 has any read / write storage medium. For example, the storage medium may be an internal storage medium such as RAM or flash memory, or an external storage medium such as a hard disk, magnetic disk, magneto-optical disk, optical disk, USB memory, or memory card.

[0174] In shooting mode, for example, the camera unit 926 captures an image of a subject to generate image data and outputs the generated image data to the image processing unit 927. The image processing unit 927 encodes the image data input from the camera unit 926 and stores the encoded stream in the storage medium of the recording and playback unit 929.

[0175] In videophone mode, for example, the multiplexing / decompression unit 928 multiplexes the video stream encoded by the image processing unit 927 and the audio stream input from the audio codec 923, and outputs the multiplexed stream to the communication unit 922. The communication unit 922 encodes and modulates the stream to generate a transmission signal. The communication unit 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. The communication unit 922 also amplifies and frequency-converts the radio signal received via the antenna 921 to obtain a received signal. These transmission and received signals may include encoded bitstreams. The communication unit 922 then demodulates and decodes the received signal to restore the stream and outputs the restored stream to the multiplexing / decompression unit 928. The multiplexing / decompression unit 928 separates the video stream and audio stream from the input stream, and outputs the video stream to the image processing unit 927 and the audio stream to the audio codec 923. The image processing unit 927 decodes the video stream and generates video data. The video data is supplied to the display unit 930, which then displays a series of images. The audio codec 923 expands and performs D / A conversion on the audio stream to generate an analog audio signal. The audio codec 923 then supplies the generated audio signal to the speaker 924 to output sound.

[0176] In the mobile phone 920 configured in this way, the image processing unit 927 has the functions of the image encoding device 10 and image decoding device 60 according to the above-described embodiment. This makes it possible to suppress the increase in the amount of code when the number of quantization matrices is large when encoding and decoding images in the mobile phone 920.

[0177] [6-3. Third Application Example] Figure 17 shows a schematic example of the configuration of a recording and playback device to which the above-described embodiment is applied. The recording and playback device 940, for example, encodes audio data and video data of a received broadcast program and records it on a recording medium. The recording and playback device 940 may also encode audio data and video data acquired from other devices and record it on a recording medium. Furthermore, the recording and playback device 940 plays back the data recorded on the recording medium on a monitor and speaker in response to user instructions. At this time, the recording and playback device 940 decodes the audio data and video data.

[0178] The recording and playback device 940 includes a tuner 941, an external interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) 948, a control unit 949, and a user interface 950.

[0179] The tuner 941 extracts the signal of the desired channel from the broadcast signal received via the antenna (not shown) and demodulates the extracted signal. The tuner 941 then outputs the encoded bitstream obtained by demodulation to the selector 946. In other words, the tuner 941 plays the role of a transmission means in the recording and playback device 940.

[0180] The external interface 942 is an interface for connecting the recording and playback device 940 to an external device or network. The external interface 942 may be, for example, an IEEE 1394 interface, a network interface, a USB interface, or a flash memory interface. For example, video data and audio data received via the external interface 942 are input to the encoder 943. That is, the external interface 942 has the role of a transmission means in the recording and playback device 940.

[0181] The encoder 943 encodes the video data and audio data input from the external interface 942 if they are not encoded. The encoder 943 then outputs the encoded bitstream to the selector 946.

[0182] The HDD944 records compressed encoded bitstreams of content data such as video and audio, various programs, and other data on its internal hard disk. The HDD944 also reads this data from the hard disk when playing back video and audio.

[0183] The disk drive 945 records and reads data from the recording medium that is installed in it. The recording medium installed in the disk drive 945 may be, for example, a DVD disc (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+R, DVD+RW, etc.) or a Blu-ray® disc.

[0184] When recording video and audio, the selector 946 selects the encoded bitstream input from the tuner 941 or encoder 943 and outputs the selected encoded bitstream to the HDD 944 or disk drive 945. When playing back video and audio, the selector 946 outputs the encoded bitstream input from the HDD 944 or disk drive 945 to the decoder 947.

[0185] Decoder 947 decodes the encoded bitstream and generates video data and audio data. Decoder 947 then outputs the generated video data to OSD 948. Decoder 904 also outputs the generated audio data to an external speaker.

[0186] The OSD948 plays back the video data input from the decoder947 and displays the video. The OSD948 may also superimpose GUI images, such as menus, buttons, or cursors, onto the displayed video.

[0187] The control unit 949 has a processor such as a CPU, and memory such as RAM and ROM. The memory stores programs executed by the CPU, and program data. The programs stored in the memory are read and executed by the CPU, for example, when the recording and playback device 940 is started up. By executing the programs, the CPU controls the operation of the recording and playback device 940, for example, in response to operation signals input from the user interface 950.

[0188] The user interface 950 is connected to the control unit 949. The user interface 950 includes, for example, buttons and switches for the user to operate the recording and playback device 940, and a remote control signal receiver. The user interface 950 detects user operations through these components, generates an operation signal, and outputs the generated operation signal to the control unit 949.

[0189] In the recording and playback device 940 configured in this way, the encoder 943 has the function of the image encoding device 10 according to the above-described embodiment. The decoder 947 has the function of the image decoding device 60 according to the above-described embodiment. This makes it possible to suppress the increase in the amount of code when the number of quantization matrices is large when encoding and decoding images in the recording and playback device 940.

[0190] [6-4. Fourth Application Example] Figure 18 shows a schematic example of the configuration of an imaging device to which the above-described embodiment is applied. The imaging device 960 captures an image of a subject to generate an image, encodes the image data, and records it on a recording medium.

[0191] The imaging device 960 comprises an optical block 961, an imaging unit 962, a signal processing unit 963, an image processing unit 964, a display unit 965, an external interface 966, a memory 967, a media drive 968, an OSD 969, a control unit 970, a user interface 971, and a bus 972.

[0192] The optical block 961 is connected to the imaging unit 962. The imaging unit 962 is connected to the signal processing unit 963. The display unit 965 is connected to the image processing unit 964. The user interface 971 is connected to the control unit 970. The bus 972 interconnects the image processing unit 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control unit 970.

[0193] The optical block 961 includes a focus lens and an aperture mechanism. The optical block 961 forms an optical image of the subject on the imaging surface of the imaging unit 962. The imaging unit 962 has an image sensor such as a CCD or CMOS, and converts the optical image formed on the imaging surface into an image signal as an electrical signal by photoelectric conversion. The imaging unit 962 then outputs the image signal to the signal processing unit 963.

[0194] The signal processing unit 963 performs various camera signal processing such as knee correction, gamma correction, and color correction on the image signal input from the imaging unit 962. The signal processing unit 963 outputs the image data after camera signal processing to the image processing unit 964.

[0195] The image processing unit 964 encodes the image data input from the signal processing unit 963 and generates encoded data. The image processing unit 964 then outputs the generated encoded data to the external interface 966 or the media drive 968. The image processing unit 964 also decodes the encoded data input from the external interface 966 or the media drive 968 and generates image data. The image processing unit 964 then outputs the generated image data to the display unit 965. The image processing unit 964 may also output image data input from the signal processing unit 963 to the display unit 965 to display an image. The image processing unit 964 may also superimpose display data acquired from the OSD 969 onto the image output to the display unit 965.

[0196] OSD969 generates GUI images such as menus, buttons, or cursors, and outputs the generated images to the image processing unit 964.

[0197] The external interface 966 is configured, for example, as a USB input / output terminal. The external interface 966 connects the imaging device 960 and a printer, for example, when printing an image. A drive is also connected to the external interface 966 as needed. A removable media such as a magnetic disk or optical disk is mounted on the drive, and a program read from the removable media can be installed on the imaging device 960. Furthermore, the external interface 966 may be configured as a network interface connected to a network such as a LAN or the Internet. That is, the external interface 966 has the role of a transmission means in the imaging device 960.

[0198] The recording medium mounted in the media drive 968 may be any read / write removable media, such as a magnetic disk, magneto-optical disk, optical disk, or semiconductor memory. Alternatively, the recording medium may be fixedly mounted in the media drive 968, and a non-portable storage unit such as an internal hard disk drive or SSD (Solid State Drive) may be configured.

[0199] The control unit 970 has a processor such as a CPU, and memory such as RAM and ROM. The memory stores programs executed by the CPU, and program data. The programs stored in the memory are read and executed by the CPU, for example, when the imaging device 960 is started up. By executing the programs, the CPU controls the operation of the imaging device 960 in response to operation signals input from, for example, the user interface 971.

[0200] The user interface 971 is connected to the control unit 970. The user interface 971 includes, for example, buttons and switches for the user to operate the imaging device 960. The user interface 971 detects user operations through these components, generates operation signals, and outputs the generated operation signals to the control unit 970.

[0201] In the imaging device 960 configured in this way, the image processing unit 964 has the functions of the image encoding device 10 and image decoding device 60 according to the above-described embodiment. This makes it possible to suppress the increase in the amount of code when the number of quantization matrices is large when encoding and decoding images in the imaging device 960.

[0202] <7. Summary> Up to this point, an image encoding device 10 and an image decoding device 60 according to one embodiment have been described with reference to Figures 1 to 18. According to this embodiment, when multiple quantization matrices corresponding to multiple transformation units of different sizes are used, a second quantization matrix corresponding to a second size transformation unit is generated from a first quantization matrix corresponding to a first size transformation unit using a prediction method. Therefore, the need to encode the entire second quantization matrix is ​​eliminated, and even when the number of quantization matrices increases, the increase in the amount of code can be suppressed.

[0203] Furthermore, according to this embodiment, the second quantization matrix is ​​generated using matrix information that identifies the first quantization matrix, and difference information (difference matrix information) that represents the difference between the prediction matrix and the second quantization matrix. Therefore, with respect to the second quantization matrix, an appropriate second quantization matrix can be obtained on the image decoding side by encoding only the difference between the matrix and the prediction matrix.

[0204] Furthermore, according to this embodiment, if a first flag indicating that there is no difference between the prediction matrix and the second quantization matrix is ​​obtained from the sequence parameter set or picture parameter set, the prediction matrix predicted from the second quantization matrix is ​​set as the second quantization matrix. In this case, since even difference information is not encoded for the second quantization matrix, a further reduction in the amount of coding is possible.

[0205] Furthermore, the size of the first quantization matrix may be the smallest size among the sizes of multiple transformation units. With this configuration, since it is not necessary to encode the entire matrix for quantization matrices other than the smallest size quantization matrix, the increase in the amount of code can be suppressed more effectively even when the number of quantization matrices is large.

[0206] In this specification, an example has been described in which information for generating a quantization matrix is ​​multiplexed in the header of the encoded stream and transmitted from the encoding side to the decoding side. However, the method of transmitting this information is not limited to this example. For example, this information may be transmitted or recorded as separate data associated with the encoded bitstream without being multiplexed in the encoded bitstream. Here, the term “associate” means that an image (which may be a part of an image, such as a slice or block) contained in the bitstream can be linked to the information corresponding to that image during decoding. That is, the information may be transmitted on a separate transmission path from the image (or bitstream). The information may also be recorded on a separate recording medium (or a separate recording area of ​​the same recording medium) from the image (or bitstream). Furthermore, the information and the image (or bitstream) may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0207] Preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical idea set forth in the claims, and these will naturally also be understood to fall within the technical scope of the present disclosure.

[0208] 10 Image Processing Device (Image Encoding Device) 16 Encoding section 110 Selection section 120 Orthogonal Transformation Unit 130 Quantization section 60. Image processing device (image decoding device) 210 Matrix generation unit 230 Selection Section 240 Inverse quantization section 250 Inverse orthogonal transformation section

Claims

[1] A selection unit for selecting a transformation unit to be used for the inverse orthogonal transformation of the image data to be decoded from a plurality of transformation units of different sizes, A generation unit that generates a second quantization matrix corresponding to a second size transformation unit from a first quantization matrix corresponding to a first size transformation unit, When the second size conversion unit is selected by the selection unit, the inverse quantization unit inversely quantizes the conversion coefficient data of the image data using the second quantization matrix generated by the generation unit, An image processing device equipped with the following features. [2] The image processing apparatus according to claim 1, wherein the generation unit generates a second quantization matrix using matrix information that identifies the first quantization matrix and difference information that represents the difference between the second quantization matrix and the second quantization matrix predicted from the first quantization matrix. [3] The image processing apparatus according to claim 2, wherein the generation unit acquires the matrix information and the difference information from a sequence parameter set or a picture parameter set. [4] The image processing apparatus according to claim 2, wherein the generation unit sets the prediction matrix to the second quantization matrix when a first flag indicating that there is no difference between the prediction matrix and the second quantization matrix is ​​obtained from the sequence parameter set or the picture parameter set. [5] The image processing apparatus according to claim 1, wherein the first size is the smallest size among the sizes of the plurality of conversion units. [6] The second size is larger than the first size, The generation unit calculates the prediction matrix by duplicating the first element or the second element as an element between adjacent first and second elements in the first quantization matrix. The image processing apparatus according to claim 2. [7] The second size is larger than the first size, The generation unit calculates the prediction matrix by linearly interpolating the elements between adjacent first and second elements in the first quantization matrix. The image processing apparatus according to claim 2. [8] The image processing apparatus according to claim 1, wherein the second size is twice the size of the first size on one side. [9] The second size is smaller than the first size, The generation unit calculates the prediction matrix by decimating the elements of the first quantization matrix. The image processing apparatus according to claim 2. [10] The second size is smaller than the first size, The generation unit calculates the prediction matrix by calculating the average of a plurality of adjacent elements in the first quantization matrix. The image processing apparatus according to claim 2. [11] The image processing apparatus according to claim 1, wherein the generation unit generates the second quantization matrix from the first quantization matrix when a second flag specifying the use of a user-defined matrix for the second quantization matrix is ​​obtained from the sequence parameter set or the picture parameter set. [12] Selecting a transformation unit from several transformation units of different sizes to be used for the inverse orthogonal transformation of the image data to be decoded, The process involves generating a second quantization matrix corresponding to a second size transformation unit from a first quantization matrix corresponding to a first size transformation unit, When the second size conversion unit is selected, the conversion coefficient data of the image data is inversely quantized using the second quantization matrix generated from the first quantization matrix, Image processing methods including [specific details omitted]. [13] A selection unit for selecting a transformation unit to be used for the orthogonal transformation of the image data to be encoded from a plurality of transformation units of different sizes, A quantization unit that quantizes the transformation coefficient data generated by orthogonally transforming the image data using the transformation units selected by the selection unit, using the quantization matrix corresponding to the selected transformation unit, An encoding unit that encodes information for generating a second quantization matrix corresponding to a second size transformation unit from a first quantization matrix corresponding to a first size transformation unit, An image processing device equipped with the following features. [14] Selecting a transformation unit from several transformation units of different sizes to be used for the orthogonal transformation of the image data to be encoded, The transformation coefficient data generated by orthogonally transforming the image data using the selected transformation unit is quantized using the quantization matrix corresponding to the selected transformation unit. Encoding information for generating a second quantization matrix corresponding to a second size transformation unit from a first quantization matrix corresponding to a first size transformation unit, Image processing methods including [specific details omitted].

Images