Image encoding device and image encoding method

Abstract

The present invention provides an image encoding apparatus and an image encoding method. The image encoding apparatus includes an encoding processing unit configured to: perform a primary transform processing to transform prediction error data in the spatial domain, which is the difference between image data and predicted image data, into transform coefficient data in the frequency domain; perform scaling list processing on the transform coefficient data to obtain scaling list coefficient data; perform a secondary transform processing to concentrate the scaling list coefficient data at low frequencies; and encode the scaling list coefficient data that has undergone secondary transform processing and transform information indicating that secondary transform processing has been performed.

Description

[0001] This application is a divisional application of Chinese patent application filed on February 1, 2019, with application number 201780048136.1 and entitled "Image Processing Apparatus, Image Processing Method and Storage Medium". The international filing date of the parent application is July 7, 2017, and the international application number is PCT / JP2017 / 025021. Technical Field

[0002] This disclosure relates to image processing apparatus, image processing methods, and programs. Background Technology

[0003] In image coding, in order to improve the concentration of coefficient energy (concentrating the transform coefficients at low frequencies), it has been disclosed to apply a second transform to the transform coefficients after the primary transform, which is different from the primary transform (see, for example, Non-Patent Literature 1).

[0004] Reference List

[0005] Non-patent literature

[0006] Non-Patent Document 1: Jianle Chen, Elena Alshina, Gary J. Sullivan, Jens-Rainer Ohm, Jill Boyce, “Algorithm Description of Joint Exploration Test Model 3”, JVET-C1001_v3, 3rd Meeting of the Joint Video Exploration Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11: 26 May to 1 June 2016, Geneva, Switzerland Summary of the Invention

[0007] Technical issues

[0008] However, as in Non-Patent Document 1, bandwidth control using a scaling list (quantization matrix) for the frequency domain is affected because the transform coefficients after multiple transformations may be transformed to a domain different from the frequency domain relative to the original image characteristics.

[0009] Therefore, even when multiple transformations are applied, it is desirable to provide a mechanism for bandwidth control using scaling lists.

[0010] Solution to the problem

[0011] According to this disclosure, an image encoding apparatus is provided, including an encoding processing unit configured to: perform a primary transform process to transform prediction error data in the spatial domain, which is the difference between image data and predicted image data, into transform coefficient data in the frequency domain; perform scaling list processing on the transform coefficient data to obtain scaling list coefficient data; perform a secondary transform process to concentrate the scaling list coefficient data at low frequencies; and encode the scaling list coefficient data that has undergone secondary transform processing and transform information indicating that secondary transform processing has been performed.

[0012] According to this disclosure, an image coding method is provided, comprising: performing a primary transform process to transform prediction error data in the spatial domain, which is the difference between image data and predicted image data, into transform coefficient data in the frequency domain; performing a scaling list process on the transform coefficient data to obtain scaling list coefficient data; performing a secondary transform process to concentrate the scaling list coefficient data at low frequencies; and encoding the scaling list coefficient data that has undergone the secondary transform process and transform information indicating that the secondary transform process has been performed.

[0013] According to this disclosure, an image processing apparatus is provided, including: a processing control unit that controls scaling list processing based on transformation information related to a transformation applied to a processing target block.

[0014] Additionally, according to this disclosure, an image processing method is provided, comprising: controlling scaling list processing by a processor based on transformation information relating to the transformation of a processing target block applied to a processing target.

[0015] Additionally, according to this disclosure, a program is provided that enables a computer to perform the following function: control scaling list processing based on transformation information related to the transformation applied to the processing target block.

[0016] Beneficial effects of the present invention

[0017] According to this disclosure as described above, bandwidth control using a scaling list becomes possible even when multiple transformations are applied.

[0018] Note that the effects described above are not necessarily limiting. Any of the effects described herein or other effects that can be understood from this specification can be achieved by using or replacing the effects described above. Attached Figure Description

[0019] Figure 1 This is an illustrative diagram schematically showing the flow of existing processing based on the scaling list.

[0020] Figure 2 It is a graph of Y = Clip3(coeffMin, coeffMax, X).

[0021] Figure 3 This is an explanatory diagram schematically illustrating the process flow according to the first embodiment of this disclosure.

[0022] Figure 4 This is a block diagram illustrating an example configuration of an image encoding device 10 according to an embodiment.

[0023] Figure 5 This is a block diagram illustrating an example of a detailed configuration of the processing unit 14 according to an embodiment.

[0024] Figure 6 This is a flowchart illustrating an example of the processing flow when encoding according to existing technology with a transformation quantity of 1.

[0025] Figure 7 This is a flowchart illustrating an example of the process of encoding according to a new technology according to an embodiment.

[0026] Figure 8 This is a block diagram illustrating an example configuration of an image decoding device 60 according to an embodiment.

[0027] Figure 9 This is a block diagram illustrating an example of the detailed configuration of the reverse processing unit 63 according to an embodiment.

[0028] Figure 10 This is a flowchart illustrating an example of the processing flow when decoding according to existing technology with a transformation quantity of 1.

[0029] Figure 11 This is a flowchart illustrating an example of the process of decoding according to a new technology according to an embodiment.

[0030] Figure 12 This is a block diagram illustrating an example of a detailed configuration of the processing unit 14-2 according to a second embodiment of the present disclosure.

[0031] Figure 13 This is a flowchart illustrating an example of the processing flow during encoding according to an embodiment.

[0032] Figure 14 This is a block diagram illustrating an example of the detailed configuration of the reverse processing unit 63-2 according to an embodiment.

[0033] Figure 15 This is a flowchart illustrating an example of the processing flow during decoding according to an embodiment.

[0034] Figure 16 This is a block diagram illustrating a typical configuration example of a computer.

[0035] Figure 17This is a block diagram illustrating an example of a schematic configuration of a television device.

[0036] Figure 18 This is a block diagram illustrating an example of a schematic configuration of a mobile phone.

[0037] Figure 19 This is a block diagram illustrating an example of a schematic configuration of a recording / reproduction device.

[0038] Figure 20 This is a block diagram illustrating an example of a schematic configuration of an imaging device.

[0039] Figure 21 This is a block diagram illustrating an example of a schematic configuration of a video set.

[0040] Figure 22 This is a block diagram illustrating an example of a schematic configuration of a video processor.

[0041] Figure 23 This is a block diagram illustrating another example of a schematic configuration of a video processor.

[0042] Figure 24 This is a block diagram illustrating an example of a schematic configuration of a network system. Detailed Implementation

[0043] In the following, one or more preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, structural elements having substantially the same function and structure are designated by the same reference numerals, and repeated descriptions of these structural elements are omitted.

[0044] The descriptions below will proceed in the following order.

[0045] 1. First Embodiment

[0046] 2. Second Embodiment

[0047] 3. Hardware Configuration Example

[0048] 4. Application Examples

[0049] 5. Conclusion

[0050] <1. First Embodiment>

[0051] [1-1. Overview]

[0052] For example, in order to improve the concentration of coefficient energy (concentrating the transform coefficients at low frequencies), Non-Patent Document 1 discloses applying a secondary transform to the transform coefficients after a primary transform applied to the frequency domain. However, since the transform coefficients after applying multiple transforms may be transformed to a domain different from the frequency domain relative to the original image characteristics, bandwidth control using a scaling list for the frequency domain (scaling list processing) is affected.

[0053] Figure 1 This is a schematic diagram illustrating the existing processing flow of processing according to the scaling list when applying a second transformation to the transformation coefficients after the primary transformation.

[0054] like Figure 1 As shown, after applying the primary transform (S11), the secondary transform (S12) is applied to the transform coefficients. Next, quantization and scaling list processing (S13) are performed on the transform coefficients after the secondary transform. This is the processing during encoding.

[0055] Next, during decoding, inverse quantization and scaling list processing are performed (S14), followed by inverse quadratic transform (S15), and in addition, inverse primary transform is applied (S16).

[0056] As described above, the method described in Non-Patent Document 1 is used to perform (inverse)quantization and scaling list processing simultaneously or sequentially. For example, in High Efficiency Video Coding (HEVC), inverse quantization and scaling list processing are performed simultaneously. The scaling list processing during decoding in HEVC will be described below.

[0057] In HEVC, the inverse transformation coefficient value d[x][y] at position (x, y) within the processing block is calculated using the scaling factor m[x][y] as shown in Equation (1) below. The scaling factor m[x][y] is the value of the element corresponding to position (x, y) in the scaling list.

[0058] [Formula 1]

[0059] d[x][y]=Clip3(coeffMin, coeffMax, ((TransCoeffLevel[xTbY][yTbY][cIdx][x][y]*m[x][y]*levelScale[qP%6]<<(qP / 6))+(1<<(bdShift-1)))>>bdShift)…(1)

[0060] In formula (1), TransCoeffLevel[xTbY][yTbY][cIdx][x][y] represents the quantization level value (quantized data) at position (x, y) in the processing block. Furthermore, in formula (1), levelScale[qP%6]<<(qP / 6) and (1<<(bdShift-1))>>bdShift) are values ​​related to the quantization processing using the quantization parameter qP. Additionally, Clip3(coeffMin, coeffMax, X) represents the value obtained by rounding the numerical value X to be equal to or greater than the minimum coefficient value coeffMin and equal to or less than the maximum coefficient value coeffMax.

[0061] Figure 2 This is a graph of Y = Clip3(coeffMin, coeffMax, X). Figure 1 As shown, the value of Clip3(coeffMin, coeffMax, X) is equal to or greater than coeffMin, and is also equal to or less than coeffMax.

[0062] As shown in Equation (1), in HEVC, during inverse quantization, scaling list processing is performed by multiplying the scaling list by the scaling factor m[x][y]. Furthermore, similarly, in HEVC, quantization and scaling list processing are performed simultaneously.

[0063] However, as described in, for example, Non-Patent Document 1, the transform coefficients, i.e., the rotation matrix, after applying a quadratic transform are transformed to a domain different from the frequency domain relative to the original image characteristics. Therefore, when performing scaling list processing as part of (inverse)quantization processing using a scaling list for the frequency domain, it is difficult to perform appropriate bandwidth control.

[0064] Therefore, embodiments of this disclosure were created in view of the above circumstances. The image encoding or decoding device according to this embodiment controls bandwidth control through transformation processing or scaling list processing based on transformation information.

[0065] Figure 3 This is an illustrative diagram schematically showing the process flow according to this embodiment. For example... Figure 3 As shown, in this embodiment, scaling list processing (S22) is performed on the transform coefficients after the primary transform (S21). Next, after applying a second transform (S23) to the coefficients after scaling list processing, quantization processing using the quantization parameter qP is performed (S24). The above describes the processing during encoding.

[0066] Next, during decoding, first, inverse quantization is performed (S25), and then inverse quadratic transform is applied (S26). Additionally, after performing scaling list processing (S27), inverse primary transform is applied (S28).

[0067] like Figure 3 As shown in the illustration, in this embodiment, by performing scaling list processing immediately after the primary transformation, scaling list processing is performed on the data existing in the frequency domain relative to the original image characteristics, thereby enabling possible bandwidth control through the scaling list.

[0068] Note that the number of transformations according to this embodiment is not limited to the primary and secondary transformations, and can be a larger number (3 or more). For example, if Figure 1 The processing during encoding and decoding shown is extended to a greater number of transformation processes, each of which can be represented as shown in Equations (2) and (3) below.

[0069] C=Q(SL(Fn*...F2*F1(R)))...(2)

[0070] R=(F'1*F'2*...F'n*SL'(DQ(C)))...(3)

[0071] Note that C represents quantized data, Q represents quantization processing, SL represents scaling list processing during encoding, F1 is the primary transform (such as DCT or DST), F2 is the secondary transform, Fn is the nth transform, * is the convolution operation, and R is the residual image. Additionally, F'a represents the inverse transform of Fa, and SL' represents scaling list processing during decoding.

[0072] Moreover, if based on Figure 3 The processing during encoding and decoding shown in this embodiment is extended to a greater number of transformation processes, each of which can be represented as shown in formulas (4) and (5) below.

[0073] C=Q*Fn*...F2*SL(F1(R))...(4)

[0074] R=(F'1*SL'(F'2*...F'n*DQ(C)))...(5)

[0075] According to this embodiment, even if the number of transformations is 3 or more, bandwidth control of the scaling list becomes possible by performing scaling list processing immediately after the primary transformation, as in formulas (4) and (5) above.

[0076] Note that in this embodiment, the transform coefficient value dn of the inverse quantization is expressed as in the following formula (6). Here, the transform coefficient value dn is the transform coefficient value of the nth inverse transform.

[0077] [Formula 2]

[0078] dn[x][y]=Clip3(coeffMin, coeffMax, ((TransCoeffLevel[xTbY][yTbY][cIdx][x][y]*levelScale[qP%6]<<(qP / 6))+(1<<(bdShift-1)))>>bdShift)…(6)

[0079] As in formula (6) above, in the inverse quantization process of this embodiment, the process using the quantization parameter qP is performed without using the scaling factor m[x][y].

[0080] Furthermore, in this embodiment, the transform coefficient value d0 of the inverse primary transform obtained by scaling list processing during decoding is shown in the following formula (7).

[0081] [Formula 3]

[0082] d0[x][y]=Clip3(coeffMin, coeffMax, ((d1[xTbY][yTbY][cIdx][x][y]*m[x][y]))…(7)

[0083] Here, d1[xTbY][yTbY][cIdx][x][y] represents the coefficient values ​​after the inverse quadratic transformation. Note that when the number of transformations is 1 (only the primary transformation), the transformation coefficient values ​​d0[x][y] obtained by formulas (6) and (7) are essentially the same as the transformation coefficient values ​​d[x][y] obtained by formula (1). However, the two results may differ when the quantization level value (quantized data) TransCoeffLevel[xTbY][yTbY][cIdx][x][y] is not included in the range between coeffMin and coeffMax.

[0084] The overview of this embodiment has been described above. Next, the configuration and processing flow according to this embodiment will be described. Note that for simplicity, the following description is an example in which the number of transformations to be applied is at most 2, but as mentioned above, the number of transformations according to this embodiment is not limited to such an example, and may also be 3 or more.

[0085] [1-2. Configuration of Image Encoding Equipment]

[0086] (1) Overall configuration

[0087] Figure 4This is a block diagram illustrating an example configuration of an image encoding device 10, which is one aspect of an image processing apparatus according to this embodiment. (Reference) Figure 4 The image encoding device 10 is provided with a reordering buffer 11, a control unit 12, a subtraction unit 13, a processing unit 14, a lossless encoding unit 16, an accumulation buffer 17, an inverse processing unit 21, an addition unit 23, a deblocking filter 24, a SAO filter 25, a frame memory 26, a switch 27, a mode setting unit 28, an intra-frame prediction unit 30, and an inter-frame prediction unit 40.

[0088] The reordering buffer 11 reorders the image data of a series of images included in the video to be encoded according to the group of pictures (GOP) structure during the encoding process. The reordering buffer 11 outputs the reordered image data to the control unit 12, the subtraction unit 13, the intra-frame prediction unit 30, and the inter-frame prediction unit 40.

[0089] For example, the control unit 12 determines the coding parameters to be provided to each part based on rate-distortion optimization (RDO). The determined coding parameters are then provided to each block.

[0090] For example, encoding parameters may include transformation information related to the transforms to be applied to the target transform block. For instance, the transformation information may include information indicating whether a quadratic transform should be applied to the target transform block (e.g., see JVET-B1001, 2.5.2 Quadratic Transformation). Furthermore, the transformation information may include information indicating the number of transforms to be applied to the target transform block. Additionally, the transformation information may include information indicating the type of transform to be applied to the target transform block.

[0091] In addition, encoding parameters may include scaling list information (e.g., JCTVC-W1005, 7.3.4 Scaling List Data Syntax), which indicates the scaling list to be used in scaling list processing. Furthermore, encoding parameters may include the quantization parameter (qP) to be used in (inverse)quantization.

[0092] Note that the coding parameters determined by the control unit 12 can be any parameters, and can include various information not limited to the information described above. The coding parameters may include block information indicating how HEVC coding tree units (CTUs), coding units (CUs), transform units (TUs), prediction units (PUs), etc., information related to intra-frame prediction, and information related to inter-frame prediction.

[0093] The subtraction unit 13 calculates the prediction error data, which is the difference between the image data input from the reordering buffer 11 and the predicted image data, and outputs the calculated prediction error data to the processing unit 14.

[0094] The processing unit 14 performs orthogonal transformation processing, scaling list processing, and quantization processing based on the transformation information, scaling list information, quantization parameters, etc., input from the control unit 12. The processing unit 14 outputs the quantized data (hereinafter referred to as quantized data) to the lossless encoding unit 16 and the inverse processing unit 21. Note that a more detailed configuration of the processing unit 14 will be described later.

[0095] The lossless encoding unit 16 generates an encoded stream by encoding the quantized data input from the processing unit 14. Furthermore, the lossless encoding unit 16 encodes the encoding parameters determined by the control unit 12 and inserts the encoded parameters into the header region of the encoded stream. The lossless encoding unit 16 outputs the generated encoded stream to the accumulation buffer 17.

[0096] The accumulation buffer 17 uses a storage medium such as a semiconductor memory to temporarily buffer the encoded stream input from the lossless encoding unit 16. Subsequently, the accumulation buffer 17 outputs the buffered encoded stream to a transmission unit (e.g., a communication interface or connection interface connected to a peripheral device), the rate of which depends on the bandwidth of the transmission channel.

[0097] The inverse processing unit 21 and the addition unit 23 form a local decoder. The local decoder is responsible for reconstructing the original image from the encoded data.

[0098] The inverse processing unit 21 performs the inverse processing of the processing performed by the processing unit 14. For example, based on the transform information, scaling list information, quantization parameters, etc., input from the control unit 12, the inverse processing unit 21 reconstructs the prediction error data by performing inverse quantization processing, scaling list processing, and inverse orthogonal transform processing. Subsequently, the inverse processing unit 21 outputs the reconstructed prediction error data to the addition unit 23. Note that the inverse processing (inverse quantization processing, scaling list processing, and inverse orthogonal transform processing) performed by the inverse processing unit 21 is similar to the inverse processing performed in the image decoding device described later. Therefore, reference will be made later. Figure 9 These inverse processes are described in the description relating to image decoding devices.

[0099] The addition unit 23 adds the recovered prediction error data input from the inverse processing unit 21 to the prediction image data input from the intra-frame prediction unit 30 or the inter-frame prediction unit 40, thereby generating decoded image data (reconstructed image). Then, the addition unit 23 outputs the generated decoded image data to the deblocking filter 24 and the frame memory 26.

[0100] Both the deblocking filter 24 and the SAO filter 25 are loop filters used to improve the image quality of the reconstructed image. The deblocking filter 24 removes block distortion by filtering the decoded image data input from the adder 23, and outputs the filtered decoded image data to the SAO filter 25. The SAO filter 25 removes noise by applying edge offset processing or band offset processing to the decoded image data input from the deblocking filter 24, and outputs the processed decoded image data to the frame memory 26.

[0101] The frame memory 26 stores the unfiltered decoded image data input from the adder 23 and the decoded image data with loop filtering applied input from the SAO filter 25 in the storage medium.

[0102] Switch 27 reads unfiltered decoded image data to be used for intra-frame prediction from frame memory 26 and provides the read decoded image data as reference image data to intra-frame prediction unit 30. Furthermore, switch 27 reads filtered decoded image data output from frame memory 26 for inter-frame prediction and provides the read decoded image data as reference image data to inter-frame prediction unit 40.

[0103] The mode setting unit 28 sets a prediction coding mode for each block based on a comparison between the costs input from the intra-prediction unit 30 and the inter-prediction unit 40. For blocks with the intra-prediction mode set, the mode setting unit 28 outputs the predicted image data generated by the intra-prediction unit 30 to the subtraction unit 13 and outputs information about intra-prediction to the lossless coding unit 16. Furthermore, for blocks with the inter-prediction mode set, the mode setting unit 28 outputs the predicted image data generated by the inter-prediction unit 40 to the subtraction unit 13 and outputs information about inter-prediction to the lossless coding unit 16.

[0104] The intra-prediction unit 30 performs intra-prediction processing on each PU in HEVC based on the raw image data and the decoded image data. For example, the intra-prediction unit 30 evaluates the cost based on the prediction error and the amount of code to be generated for each prediction mode candidate within the search range. Then, the intra-prediction unit 30 selects the prediction mode that minimizes the cost as the optimal prediction mode. In addition, the intra-prediction unit 30 generates predicted image data based on the selected optimal prediction mode. Then, the intra-prediction unit 30 outputs information about the intra-prediction to the mode setting unit 28, which includes prediction mode information indicating the optimal prediction mode, the corresponding cost, and the predicted image data.

[0105] The inter-frame prediction unit 40 performs inter-frame prediction processing (motion compensation) on each PU of HEVC based on the raw image data and the decoded image data. This includes inter-frame prediction processing (motion detection and motion compensation). For example, the inter-frame prediction unit 40 evaluates the cost based on the prediction error and generation bit rate of each prediction mode candidate included in the search range specified by HEVC. Next, the inter-frame prediction unit 40 selects the prediction mode that produces the lowest cost, or in other words, the prediction mode that produces the highest compression ratio, as the optimal prediction mode. Furthermore, the inter-frame prediction unit 40 generates predicted image data based on the selected optimal prediction mode. Subsequently, the inter-frame prediction unit 40 outputs information related to inter-frame prediction, the corresponding cost, and the predicted image data to the mode setting unit 28.

[0106] (2) Processing Department

[0107] Figure 5 It is shown Figure 4 A block diagram illustrating an example of the detailed configuration of the processing unit 14 shown. (See reference...) Figure 5 The processing unit 14 includes a primary transformation unit 141, a scaling list processing unit 142, a processing control unit 143, a secondary transformation unit 144, and a quantization unit 145.

[0108] The primary transformation unit 141 performs a primary transformation process on the prediction error data input from the subtraction unit 13. The primary transformation of the primary transformation unit 141 is preferably a direct transformation process, such as a discrete cosine transform or a discrete sine transform. More specifically, the primary transformation unit 141 transforms the prediction error data input from the subtraction unit 13 from an image signal in the spatial domain to coefficient data in the frequency domain for each TU. Subsequently, the primary transformation unit 141 outputs the transformed coefficient data to the scaling list processing unit 142.

[0109] The scaling list processing unit 142 performs scaling list processing on the transformation coefficient data input from the primary transformation unit 141. For example, the scaling list processing unit 142 can perform scaling list processing by dividing the transformation coefficient data by the scaling list included in the scaling list information determined by the control unit 12. The scaling list processing unit 142 outputs the coefficient data after scaling list processing (hereinafter referred to as scaling list coefficient data) to the processing control unit 143.

[0110] The processing control unit 143 controls the transformation processing of the secondary transformation unit 144 and the quantization processing of the quantization unit 145, which will be described later, based on transformation information related to the transformation of the processing target block input from the control unit 12. For example, according to this embodiment, the processing control unit 143 can determine whether to perform a secondary transformation on the processing target block based on the transformation information.

[0111] Note that when the processing control unit 143 makes the above determination, the information indicating whether to perform a secondary transformation on the processing target block (e.g., a flag) may be included in the transformation information, or may include, for example, information indicating the number and type of transformations on the processing target block.

[0112] When it is determined that a second transformation will be performed on the target block, the processing control unit 143 outputs the scaling list coefficient data input from the scaling list processing unit 142 to the second transformation unit 144. In this case, the transformation coefficient data after the transformation processing of the second transformation unit 144 is input to the quantization unit 145, as described later.

[0113] Furthermore, if it is determined that a secondary transformation will not be performed on the target block, the processing control unit 143 outputs the scaling list coefficient data input from the scaling list processing unit 142 to the quantization unit 145. In this case, since the input of data to the secondary transformation unit 144 is skipped and the transformation processing of the secondary transformation unit 144 is not performed, the scaling list coefficient data is input to the quantization unit 145.

[0114] In other words, the processing control unit 143 can control the inputs to the transformation processing of the secondary transformation unit 144 and the quantization processing of the quantization unit 145 based on the transformation information.

[0115] When scaling list coefficient data is input from the processing control unit 143, the secondary transformation unit 144 performs a transformation process (secondary transformation) based on a different transformation (secondary transformation) than the primary transformation. The secondary transformation process according to this embodiment is not particularly limited, but it can be a transformation process that transforms to a domain other than the frequency domain, for example, the secondary transformation process described in Non-Patent Document 1. Furthermore, the secondary transformation process to be performed by the secondary transformation unit 144 can be specified from a plurality of pre-prepared transformation processes based on the transformation information determined by the control unit 12. The secondary transformation unit 144 outputs the transformation coefficient data after the secondary transformation process to the quantization unit 145.

[0116] The quantization unit 145 performs quantization processing based on the quantization parameter qP determined by the control unit 12. Under the control of the processing control unit 143, the quantization unit 145 performs quantization processing by taking the transform coefficient data or scaling list coefficient data after the transform processing of the quadratic transform unit 144 as input data. The quantization unit 145 outputs the quantized data after quantization processing to the lossless encoding unit 16 and the inverse processing unit 21.

[0117] According to the configuration of the processing unit 14 described above, the scaling list processing can be performed immediately after the transformation to the frequency domain by the primary transform unit 141. Therefore, bandwidth control through the scaling list processing becomes possible, regardless of the number and type of transforms to be applied to the target block.

[0118] [Processing flow during decoding 1-3]

[0119] (1) Existing technology

[0120] Figure 6 This is a flowchart illustrating an example of the process when encoding according to existing technology with a transformation quantity of 1.

[0121] refer to Figure 6 First, a transformation process (S32) is performed, followed by quantization and scaling of the list (S34). Next, an encoding process is performed (S36).

[0122] (2) New technologies

[0123] Figure 7 This is a flowchart illustrating an example of the process of encoding according to the new technology in this embodiment described above.

[0124] refer to Figure 7 The primary transformation unit 141 performs primary transformation processing on the prediction error data input from the subtraction unit 13 (S102). Next, the scaling list processing unit 142 performs scaling list processing on the transformation coefficient data input from the primary transformation unit 141 (S104). Next, the processing control unit 143 determines whether to perform a secondary transformation based on the transformation information (S106).

[0125] If it is determined that a second transformation will be performed (as in S106), the scaling list coefficient data is output to the second transformation unit 144, and the second transformation unit 144 performs the second transformation (S108). Next, the quantization unit 145 performs quantization processing on the transformed coefficient data after the transformation processing of the second transformation unit 144 (S110).

[0126] On the other hand, if it is determined that no secondary transformation is performed (No in S106), the scaling list coefficient data is output to the quantization unit 145, and the quantization unit 145 performs quantization processing on the scaling list coefficient data (S112).

[0127] Finally, the lossless encoding unit 16 encodes the quantized data obtained through the processing in step S110 or step S112. Furthermore, at this time, the lossless encoding unit 16 encodes various encoding parameters, including transformation information.

[0128] Note that the processing unit for each of the above processes can be any unit and does not have to be the same as each other. Therefore, the processing in each step can also be performed in parallel with the processing in another step, or the order in which the processes are performed can be rearranged.

[0129] By performing each process as described above, the image encoding device 10 can perform scaling list processing immediately after the frequency domain transformation, regardless of the number and type of transforms to be applied to the target block. Therefore, bandwidth control via scaling list processing becomes possible, regardless of the number and type of transforms to be applied to the target block.

[0130] [1-4. Configuration of Image Decoding Equipment]

[0131] (1) Overall configuration

[0132] Next, the decoding of the encoded data as described above will be described. Figure 8 This is a block diagram illustrating an example configuration of an image decoding device 60, which is one aspect of an image processing apparatus according to this embodiment. (Reference) Figure 8 It provides an accumulation buffer 61, a lossless decoding unit 62, an inverse processing unit 63, an addition unit 65, a deblocking filter 66, a SAO filter 67, a reordering buffer 68, a digital-to-analog (D / A) conversion unit 69, a frame memory 70, selectors 71a and 71b, an intra-frame prediction unit 80, and an inter-frame prediction unit 90.

[0133] The accumulation buffer 61 uses a storage medium to temporarily buffer the encoded stream received from the image encoding device 10 via a transmission unit (e.g., a communication interface or connection interface connected to a peripheral device) not shown.

[0134] The lossless decoding unit 62 decodes the quantized data in the encoded stream input from the accumulation buffer 61 according to the encoding scheme used during encoding. Furthermore, the lossless decoding unit 62 decodes the encoding parameters inserted into the header region of the encoded stream. The encoding parameters decoded by the lossless decoding unit 62 may include, for example, the transform information, scaling list information, quantization parameters, information related to intra-frame prediction, and information related to inter-frame prediction, as described above.

[0135] The lossless decoding unit 62 outputs quantized data, transform information, scaling list information, and quantization parameters to the inverse processing unit 63. Furthermore, the lossless decoding unit 62 outputs information related to intra-frame prediction to the intra-frame prediction unit 80. Additionally, the lossless decoding unit 62 outputs information related to inter-frame prediction to the inter-frame prediction unit 90.

[0136] The inverse processing unit 63 performs the operations performed by the processing unit 14 during the encoding of the quantized data input from the lossless decoding unit 62. Figure 4 , Figure 5 The inverse processing unit 63 performs the reverse processing of the previous processing and generates prediction error data. The inverse processing unit 63 outputs the generated prediction error data to the addition unit 65. Note that a more detailed configuration of the inverse processing unit 63 will be described later.

[0137] The adder 65 generates decoded image data by adding the prediction error data input from the inverse processing unit 63 to the prediction image data input from the selector 71b. Then, the adder 65 outputs the generated decoded image data to the deblocking filter 66 and the frame memory 70.

[0138] The deblocking filter 66 removes block distortion by filtering the decoded image data input from the adder 65, and outputs the filtered decoded image data to the SAO filter 67.

[0139] SAO filter 67 removes noise by applying edge offset processing or band offset processing to the decoded image data input from deblocking filter 66, and outputs the processed decoded image data to reorder buffer 68 and frame memory 70.

[0140] The reordering buffer 68 reorders the images input from the SAO filter 67 to generate a time-series image data sequence. Then, the reordering buffer 68 outputs the generated image data to the D / A converter 69.

[0141] The D / A converter 69 converts the digital image data input from the reordering buffer 68 into an analog image signal. Then, for example, the D / A converter 69 outputs the analog image signal to a display (not shown) connected to the image decoding device 60, thereby displaying the decoded video.

[0142] The frame memory 70 stores the unfiltered decoded image data input from the adder 65 and the filtered decoded image data input from the SAO filter 67 in the storage medium.

[0143] Based on the mode information acquired by the lossless decoding unit 62, selector 71a switches the output destination of image data from frame memory 70 between intra-prediction unit 80 and inter-prediction unit 90 for each block in the image. If an intra-prediction mode has been specified, for example, selector 71a outputs unfiltered decoded image data provided from frame memory 70 as reference image data to intra-prediction unit 80. Conversely, if an inter-prediction mode has been specified, selector 71a outputs filtered decoded image data as reference image data to inter-prediction unit 90.

[0144] Selector 71b switches the output source of the predicted image data to be provided to adder 65 between intra-frame prediction unit 80 and inter-frame prediction unit 90 based on the mode information acquired by lossless decoding unit 62. For example, if an intra-frame prediction mode has been specified, selector 71b provides the predicted image data output from intra-frame prediction unit 80 to adder 65. Furthermore, if an inter-frame prediction mode has been specified, selector 71b provides the predicted image data output from inter-frame prediction unit 90 to adder 65.

[0145] The intra-prediction unit 80 performs intra-prediction processing based on information about intra-prediction and reference image data from the frame memory 70, thereby generating predicted image data. Then, the intra-prediction unit 80 outputs the generated predicted image data to the selector 71b.

[0146] The inter-frame prediction unit 90 performs inter-frame prediction processing based on information about inter-frame prediction input from the lossless decoding unit 62 and reference image data from the frame memory 70, thereby generating predicted image data. Then, the inter-frame prediction unit 90 outputs the generated predicted image data to the selector 71b.

[0147] (2) Reverse Processing Unit

[0148] Figure 9 It is shown Figure 8 A block diagram illustrating an example of the detailed configuration of the reverse processing unit 63 shown. (See reference...) Figure 9 The inverse processing unit 63 includes an inverse quantization unit 631, a processing control unit 632, an inverse quadratic transformation unit 633, a scaling list processing unit 634, and an inverse primary transformation unit 635.

[0149] The inverse quantization unit 631 inverse-quantizes the quantized data input from the lossless decoding unit 62 with the same quantization parameter qP used during encoding, and reconstructs the coefficient data. For example, the inverse quantization unit 631 can perform the inverse quantization process as described in reference formula (6). The inverse quantization unit 631 outputs the reconstructed coefficient data to the processing control unit 632. Note that the coefficient data reconstructed by the inverse quantization unit 631 can be transform coefficient data or scaled list coefficient data processed according to the quadratic transform, depending on whether the quadratic transform was performed during encoding.

[0150] Based on transformation information related to the transformation of the target block, the processing control unit 632 controls the inverse transformation processing of the inverse quadratic transformation unit 633 (described later) and the scaling list processing of the scaling list processing unit 634. For example, according to this embodiment, the processing control unit 632 can determine whether to perform an inverse quadratic transformation on the target block based on the transformation information.

[0151] Note that when the processing control unit 632 makes the above determination, the information indicating whether a secondary transformation is performed on the target block during encoding (e.g., a flag) may be included in the transformation information, or it may include information indicating the number and type of transformations on the target block. For example, if a secondary transformation has been performed during encoding, the processing control unit 632 may determine that an inverse secondary transformation is performed, and if no secondary transformation is performed during encoding, the processing control unit 632 may determine that an inverse secondary transformation is not performed.

[0152] When it is determined that an inverse quadratic transform will be performed on the target block, the processing control unit 632 outputs the coefficient data input from the inverse quantization unit 631 to the inverse quadratic transform unit 633. In this case, the coefficient data after the inverse quadratic transform processing by the inverse quadratic transform unit 633 is input into the scaling list processing unit 634, as described later.

[0153] If it is determined that no inverse quadratic transform will be performed on the target block, the processing control unit 632 will output the coefficient data input from the inverse quantization unit 631 to the scaling list processing unit 634. In this case, since the input of data to the inverse quadratic transform unit 633 is skipped and the inverse transform processing of the inverse quadratic transform unit 633 is not performed, the inverse-quantized coefficient data is input to the scaling list processing unit 634.

[0154] In other words, the processing control unit 632 can control the inputs to the inverse transformation processing of the inverse quadratic transformation unit 633 and the scaling list processing of the scaling list processing unit 634 based on the above determination based on the transformation information.

[0155] When coefficient data is input from the processing control unit 632, the inverse quadratic transform unit 633 performs the inverse transform processing (inverse quadratic transform processing) of the quadratic transform performed during encoding. The inverse quadratic transform unit 633 outputs the coefficient data after the quadratic transform processing to the scaling list processing unit 634.

[0156] The scaling list processing unit 634 performs scaling list processing based on the scaling list information input from the lossless decoding unit 62. Under the control of the processing control unit 632, the scaling list processing unit 634 performs scaling list processing by taking the coefficient data after transformation processing by the inverse quadratic transform unit 633 or the coefficient data after inverse quantization processing by the inverse quantization unit 631 as input data.

[0157] For example, the scaling list processing unit 634 can perform scaling list processing as described in reference formula (7). The scaling list processing unit 634 outputs the transformed coefficient data after scaling list processing to the inverse primary transform unit 635.

[0158] The inverse primary transform unit 635 generates prediction error data by performing an inverse primary transform on the transform coefficient data input from the scaling list processing unit 634, based on the transform scheme of the primary transform processing used during encoding. The inverse primary transform unit 635 outputs the generated prediction error data to the adder unit 65.

[0159] According to the configuration of the reverse processing unit 63 described above, decoding based on the bandwidth control of the scaling list processing becomes possible, regardless of the number and type of transformations applied to the target block being processed.

[0160] [1-5. Processing flow during decoding]

[0161] (1) Existing technology

[0162] Figure 10 This is a flowchart illustrating an example of the processing flow during decoding according to existing techniques when the number of transformations is 1. Note that from the processing flow during decoding, Figure 10 The process flow, from decoding to generating prediction error data, is shown.

[0163] refer to Figure 10 First, decoding is performed (S52). Next, inverse quantization and scaling list processing are performed (S54). After that, inverse transform processing is performed to generate prediction error data (S56).

[0164] (2) New technologies

[0165] Figure 11 This is a flowchart illustrating an example of the processing flow during decoding according to the new technology in this embodiment described above. Note that from the decoding processing flow, Figure 11 The process flow, from decoding to generating prediction error data, is shown.

[0166] refer to Figure 11 First, the lossless decoding unit 62 performs decoding processing to obtain (decode) quantized data and encoding parameters (S202). At this time, transform information, scaling list information, quantization parameters qP, etc., can be included in the obtained encoding parameters.

[0167] Next, the inverse quantization unit 631 performs inverse quantization on the quantized data input from the lossless decoding unit 62 using the same quantization parameter qP as used during encoding (S204). Next, the processing control unit 632 determines whether to perform an inverse quadratic transform based on the transform information (S206).

[0168] If it is determined that an inverse quadratic transform will be performed (as in S206), the coefficient data after inverse quantization is output to the inverse quadratic transform unit 633, and the inverse quadratic transform unit 633 performs the inverse quadratic transform (S208). Next, the scaling list processing unit 634 performs scaling list processing on the coefficient data after the transform processing by the inverse quadratic transform unit 633 (S210).

[0169] On the other hand, if it is determined that the inverse quadratic transform will not be performed (No in S206), the coefficient data after inverse quantization is output to the scaling list processing unit 634, and the scaling list processing unit 634 performs scaling list processing on the coefficient data after inverse quantization (S212).

[0170] Subsequently, the inverse primary transformation unit 635 performs an inverse primary transformation on the transformation coefficient data obtained through the processing in step S210 or step S212, and generates prediction error data (S214).

[0171] Note that the processing unit for each of the above processes can be any unit and does not have to be the same as each other. Therefore, the processing in each step can also be performed in parallel with the processing in another step, or the order in which the processes are performed can be rearranged.

[0172] By performing each of the processes described above, the image decoding device 60 is able to perform decoding based on the bandwidth control processed by the scaling list, regardless of the number and type of transforms applied to the target block being processed.

[0173] <2. Second Embodiment>

[0174] [2-1. Overview]

[0175] The first embodiment of this disclosure has been described above. Next, as a second embodiment of this disclosure, the case where multiple transformations are represented by a single transformation via convolution operations will be described below.

[0176] For example, assuming Fx is a convolution of n transformations Fn*...F2*F1, the encoding and decoding processes according to this embodiment can be expressed as each of the following formulas (8) and (9).

[0177] C=Q(SLx(Fx(R))......(8)

[0178] R=(F'x*SLx'(DQ(C)))...(9)

[0179] Note that here, SL x The scaling list SL determined by the control unit 12 described in the first embodiment is used. conv It is represented by the following formula (10).

[0180] SLx = Fx(SL) conv )......(10)

[0181] For example, when the number of transformations is 2, the scaling list SL con The second transformation F2 is represented by the following formulas (11) and (12), respectively, SL x By scaling list SL conv It is obtained by the inner product operation of the second transformation F2, for example, as shown in formula (13) below. Note, for example, that the specific coefficients of the second transformation F2 are determined based on the transformation information in multiple transformations.

[0182] [Formula 4]

[0183]

[0184] The overview of this embodiment has been described above. Next, the configuration and processing flow according to this embodiment will be described. Note that for simplicity, the following description is an example in which the number of transformations to be applied is at most 2, but as mentioned above, the number of transformations according to this embodiment is not limited to such an example, and may also be 3 or more.

[0185] [2-2. Configuration of Image Encoding Equipment]

[0186] Compared to the image encoding device 10 according to the first embodiment, the image encoding device 10 according to this embodiment differs only in the functional configuration of the processing unit and the inverse processing unit. Therefore, the configuration of the processing unit 14-2 provided in the image encoding device 10 according to this embodiment will be described below. Note that since the functional configuration of the inverse processing unit provided in the image encoding device 10 according to this embodiment is similar to that of the inverse processing unit provided in the image decoding device 60 according to this embodiment, which will be described later, the description is omitted here.

[0187] Figure 12 This is a block diagram illustrating an example of the detailed configuration of the processing unit 14-2 provided in the image encoding apparatus 10 according to this embodiment. (See reference...) Figure 12 The processing unit 14-2 includes a processing control unit 146, a transformation unit 147, and a quantization / scaling list processing unit 148.

[0188] The processing control unit 146 controls the transformation processing of the transformation unit 147 and the quantization processing and scaling list processing of the quantization / scaling list processing unit 148, which will be described later, based on the transformation information related to the transformation of the processing target block input from the control unit 12.

[0189] For example, similar to reference Figure 5The processing control unit 143 described herein, and the processing control unit 146 according to this embodiment, can determine whether to perform a secondary transformation on the processing target block based on the transformation information.

[0190] Furthermore, according to this embodiment, the processing control unit 146 specifies the transformation F to be applied by the transformation unit 147 based on the transformation information. For example, when performing a secondary transformation, the transformation F may be the transformation Fx = F2*F1 obtained by the convolution operation of the primary transformation F1 and the secondary transformation F2. On the other hand, when not performing a secondary transformation, the transformation F may be the primary transformation F1. The processing control unit 146 provides information about the specified transformation F to the transformation unit 147.

[0191] Furthermore, according to this embodiment, the processing control unit 146 specifies the scaling list SL to be used in the scaling list processing performed by the quantization / scaling list processing unit 148 based on the transformation information. For example, in the case of performing a quadratic transformation, the specified scaling list SL may be the scaling list SL determined by the control unit 12. conv The scaling list SL obtained by the second transformation F2 in formula (13) x On the other hand, without performing a secondary transformation, the specified scaling list SL can be the scaling list SL specified by the control unit 12. conv The processing control unit 146 provides information about the specified scaling list SL to the quantization / scaling list processing unit 148.

[0192] The transformation unit 147 performs transformation processing on the prediction error data input from the subtraction unit 13 based on the information about transformation F provided by the processing control unit 146. The transformation unit 147 outputs the transformed coefficient data after transformation processing to the quantization / scaling list processing unit 148.

[0193] The quantization / scaling list processing unit 148 performs quantization and scaling list processing based on information about the scaling list SL provided by the processing control unit 146. Note that the quantization / scaling list processing unit 148 can perform scaling list processing and quantization processing simultaneously or continuously using the quantization parameter qP and the scaling list SL. The quantization / scaling list processing unit 148 outputs the quantized data after quantization processing to the lossless encoding unit 16 and the inverse processing unit 21.

[0194] According to the configuration of the processing unit 14-2 described above, bandwidth control through scaling list processing becomes possible, regardless of the number and type of transforms applied to the processing target block. Furthermore, since multiple transforms are represented as a single transform through convolution operations, the processing load can be reduced.

[0195] [2-3. Processing flow during encoding]

[0196] Figure 13This is a flowchart illustrating an example of the processing flow during encoding according to the above embodiment.

[0197] refer to Figure 13 First, the processing control unit 146 determines whether to perform a secondary transformation on the processing target block (S302). If it is determined that a secondary transformation should be performed (Yes in S302), the transformation unit 147 performs transformation processing by convolution operation of the primary transformation and the secondary transformation (S304). Next, the scaling list SL is calculated by the processing control unit 146. x (S306), and the quantization / scaling list processing unit 148 uses the calculated scaling list SL x To perform quantization and list scaling processing (S308).

[0198] On the other hand, if it is determined that a secondary transformation will not be performed (No in S302), the transformation unit 147 performs a primary transformation (S310), and the quantization / scaling list processing unit 148 uses the scaling list SL determined by the control unit 12. conv To perform quantization and list scaling processing (S312).

[0199] Finally, the lossless encoding unit 16 encodes the quantized data obtained through the processing in step S308 or step S312. Furthermore, at this time, the lossless encoding unit 16 encodes various encoding parameters, including transformation information.

[0200] Note that the processing unit for each of the above processes can be any unit and does not have to be the same as each other. Therefore, the processing in each step can also be performed in parallel with the processing in another step, or the order in which the processes are performed can be rearranged.

[0201] By performing each of the above processes, the image encoding device 10 becomes capable of performing bandwidth control through scaling list processing, regardless of the number and type of transforms applied to the target block being processed. Furthermore, since multiple transforms are represented as a single transform through convolution operations, the amount of processing can be reduced.

[0202] [2-4. Configuration of Image Decoding Equipment]

[0203] Next, the decoding of the encoded data as described above will be described. Compared with the image decoding device 60 according to the first embodiment, the image decoding device 60 according to this embodiment differs only in the functional configuration of the inverse processing unit. Therefore, the configuration of the inverse processing unit 63-2 provided in the image decoding device 60 according to this embodiment will be described below.

[0204] Figure 14 This is a block diagram illustrating an example of the detailed configuration of the reverse processing unit 63-2 provided in the image decoding device 60 according to this embodiment. (See reference...) Figure 14 The inverse processing unit 63-2 includes a processing control unit 636, an inverse quantization / scaling list processing unit 637, and an inverse transformation unit 638.

[0205] Based on the transformation information related to the transformation of the target block, the processing control unit 636 controls the quantization processing and scaling list processing through the inverse quantization / scaling list processing unit 637, and controls the inverse transformation processing through the inverse transformation unit 638.

[0206] For example, similar to reference Figure 9 The processing control unit 632 described herein, and the processing control unit 636 according to this embodiment, can determine whether to perform an inverse quadratic transformation on the processing target block based on transformation information.

[0207] Furthermore, according to this embodiment, the processing control unit 636 specifies the scaling list SL to be used in the scaling list processing performed by the inverse quantization / scaling list processing unit 637 based on the transformation information. For example, in the case of performing an inverse quadratic transformation, the specified scaling list SL may be the scaling list SL included in the scaling list information decoded by the lossless decoding unit 62. conv The scaled list SL obtained by the inner product operation of the second transformation F2. x As shown in formula (13). On the other hand, without performing the inverse quadratic transformation, the specified scaling list SL can be the scaling list SL included in the scaling list information decoded by the lossless decoding unit 62. conv The processing control unit 636 provides information about the specified scaling list SL to the inverse quantization / scaling list processing unit 637.

[0208] Furthermore, according to this embodiment, the processing control unit 636 specifies the inverse transformation F' to be applied by the inverse transformation unit 638 based on the transformation information. For example, when performing an inverse quadratic transformation, the transformation F' may be the inverse transformation F'x obtained by the convolution operation of the primary transformation F1 and the quadratic transformation F2, which is Fx = F2 * F1. On the other hand, when not performing an inverse quadratic transformation, the inverse transformation F may be the inverse transformation of the primary transformation F1, i.e., the inverse primary transformation F'1. The processing control unit 146 provides information about the specified inverse transformation F' to the inverse transformation unit 638.

[0209] The inverse quantization / scaling list processing unit 637 processes the scaling list based on the information provided by the processing control unit 636.

[0210] The information from SL is used to perform inverse quantization and list scaling. Note that the inverse quantization / list scaling part...

[0211] 637 can use the quantization parameter qP and the scaling list SL to perform quantization and scaling list processing simultaneously or continuously. The inverse quantization / scaling list processing unit 637 outputs the transform coefficient data after inverse quantization to the inverse transform unit 638.

[0212] The inverse transform unit 638 generates prediction error data by performing inverse transform processing on the transform coefficient data input from the inverse quantization / scaling list processing unit 637 based on information about the inverse transform F' provided by the processing control unit 636. The inverse transform unit 638 outputs the generated prediction error data to the addition unit 65.

[0213] According to the configuration of the inverse processing unit 63-2 described above, decoding becomes possible based on bandwidth control processed by scaling lists, regardless of the number and type of transforms applied to the target block being processed. Furthermore, since multiple transforms are represented as a single transform through convolution operations, the amount of processing can be reduced.

[0214] [2-5. Processing flow during decoding]

[0215] Figure 15 This is a flowchart illustrating an example of the processing flow during decoding according to the above embodiment. Note that from the processing flow during decoding, Figure 15 The process flow, focusing on the process from decoding to generating prediction error data, is shown.

[0216] refer to Figure 15 First, the lossless decoding unit 62 performs decoding processing to obtain (decode) quantized data and encoding parameters (S402). At this time, transform information, scaling list information, quantization parameters qP, etc., can be included in the obtained encoding parameters.

[0217] Next, the processing control unit 636 determines whether to perform an inverse quadratic transformation on the processing target block based on the transformation information (S404).

[0218] If it is determined that an inverse quadratic transformation will be performed (as in S404), the scaling list SL is calculated by the processing control unit 636. x (S406). Next, the inverse quantization / scaling list processing unit 637 uses the calculated scaling list SL x The inverse quantization and scaling list processing are performed (S408). Next, the inverse transform unit 638 performs the inverse transform processing by convolution operation of the primary transform and the secondary transform to generate prediction error data (S412).

[0219] On the other hand, if it is determined that the inverse quadratic transform will not be performed (No in S404), the inverse quantization / scaling list processing unit 637 uses the scaling list SL included in the scaling list information decoded in step S402.conv To perform inverse quantization and list scaling (S414).

[0220] Next, the inverse transform unit 638 performs inverse primary transform processing to generate prediction error data (S416).

[0221] Note that the processing unit for each of the above processes can be any unit and does not have to be the same as each other. Therefore, the processing in each step can also be performed in parallel with the processing in another step, or the order in which the processes are performed can be rearranged.

[0222] By performing each of the processes described above, the image decoding device 60 is able to perform decoding based on the bandwidth control processed by the scaling list, regardless of the number and type of transforms applied to the target block. Furthermore, since multiple transforms are represented as a single transform through convolution operations, the amount of processing can be reduced.

[0223] <3. Hardware Configuration Example>

[0224] The above series of processes can be performed by hardware or by software. In the case of performing the series of processes by software, a program forming the software is installed on the computer. Here, the term "computer" includes computers built into dedicated hardware, computers capable of performing various functions by installing various programs on them, such as general-purpose personal computers.

[0225] Figure 16 This is a block diagram illustrating an exemplary hardware configuration of a computer that performs the series of processes described above according to a program.

[0226] exist Figure 16 In the computer 800 shown, the central processing unit (CPU) 801, read-only memory (ROM) 802 and random access memory (RAM) 803 are interconnected via bus 804.

[0227] Additionally, the input / output interface 810 is also connected to the bus 804. The input unit 811, output unit 812, storage unit 813, communication unit 814, and driver 815 are connected to the input / output interface 810.

[0228] Input unit 811 includes, for example, a keyboard, mouse, microphone, touchpad, input terminals, etc. Output unit 812 includes, for example, a display, speaker, output terminals, etc. Storage unit 813 includes, for example, a hard disk, RAM disk, non-volatile memory, etc. Communication unit 814 includes, for example, a network interface. Driver 815 drives removable medium 821, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0229] In a computer configured as described above, for example, the series of processes described above are performed by having the CPU 801 load the program stored in the storage unit 813 into the RAM 803 via the input / output interface 810 and the bus 804, and the program is executed. Additionally, the data required by the CPU 801 to perform various processes is also appropriately stored in the RAM 803.

[0230] For example, an application can be applied by recording a program executed by a computer (CPU 801) on a removable medium 821 as a packaging medium, etc. In this case, the program can be installed in the storage unit 813 via the input / output interface 810 by inserting the removable medium 821 into the drive 815.

[0231] Alternatively, the program can be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit 814 and installed in the storage unit 813.

[0232] Alternatively, the program can be pre-installed in ROM 802 or storage unit 813.

[0233] <4. Application Examples>

[0234] The image encoding device 10 and image decoding device 60 according to the above embodiments can be applied to various electronic devices, such as transmitters or receivers for satellite broadcasting, cable broadcasting such as cable television, distribution over the Internet via cellular communication, and distribution to terminals; recording devices for recording images on media such as optical discs, magnetic disks, and flash memory; or reproduction devices for reproducing images from the above storage media.

[0235] (1) First application example: TV receiver

[0236] Figure 17 An example of a schematic configuration of a television device applying the above embodiments is shown. The television device 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, an external interface (I / F) 909, a control unit 910, a user interface (I / F) 911, and a bus 912.

[0237] Tuner 902 extracts the desired channel signal from the broadcast signal received via antenna 901 and demodulates the extracted signal. Then, tuner 902 outputs the demodulated coded bitstream to demultiplexer 903. That is, tuner 902 functions as the transmission unit of television device 900 that receives the coded stream of coded images.

[0238] Demultiplexer 903 demultiplexes the video and audio streams of the program to be viewed from the encoded bitstream and outputs the demultiplexed stream to decoder 904. Additionally, demultiplexer 903 extracts auxiliary data such as Electronic Program Guide (EPG) from the encoded bitstream and provides the extracted data to control unit 910. Note that demultiplexer 903 can perform descrambling even when the encoded bitstream has been scrambled.

[0239] Decoder 904 decodes the video and audio streams input from demultiplexer 903. Then, decoder 904 outputs the video data generated from the decoding process to video signal processing unit 905. Additionally, decoder 904 outputs the audio data generated from the decoding process to audio signal processing unit 907.

[0240] The video signal processing unit 905 reproduces the video data input from the decoder 904 so that the display unit 906 can display the video. Additionally, the video signal processing unit 905 can cause the display unit 906 to display an application screen provided via a network. Furthermore, the video signal processing unit 905 can perform additional processing on the video data according to settings, such as noise reduction. Moreover, the video signal processing unit 905 can generate images of a graphical user interface (GUI), such as menus, buttons, or cursors, and overlay the generated images onto the output image.

[0241] The display unit 906 is driven by the drive signal provided from the video signal processing unit 905, and video or images are displayed on the video plane of the display device (e.g., liquid crystal display, plasma display, organic light-emitting display (OLED)).

[0242] The audio signal processing unit 907 performs reproduction processing, including D / A conversion and amplification, on the audio data input from the decoder 904, and outputs sound from the speaker 908. Additionally, the audio signal processing unit 907 can perform further processing on the audio data, such as noise removal.

[0243] External interface 909 is an interface used to connect television device 900 to external devices or networks. For example, video or audio streams received via external interface 909 can be decoded by decoder 904. In other words, external interface 909 also functions as a transmission unit for television device 900 that receives encoded streams of encoded images.

[0244] The control unit 910 has a processor such as a CPU and a memory such as RAM and ROM. The memory stores programs executed by the CPU, program data, EPG data, and data acquired via a network. The programs stored in the memory are read and executed by the CPU when, for example, the television device 900 is started. The CPU controls the operation of the television device 900 by executing programs in response to operation signals input from, for example, the user interface unit 911.

[0245] The user interface unit 911 is connected to the control unit 910. The user interface unit 911 includes, for example, buttons and switches for user operation of the television device 900, a receiving unit for remote control signals, etc. The user interface unit 911 generates an operation signal by detecting the user's operation via any of the aforementioned components, and outputs the generated operation signal to the control unit 910.

[0246] Bus 912 connects tuner 902, demultiplexer 903, decoder 904, video signal processing unit 905, audio signal processing unit 907, external interface 909 and control unit 910 to each other.

[0247] In the television device 900 configured in this way, the decoder 904 may also include the functions of the image decoding device 60 described above. In other words, the decoder 904 can be configured to decode encoded data according to the methods described in each of the above embodiments. With this arrangement, the television device 900 becomes capable of performing decoding according to bandwidth control processed by the scaling list.

[0248] Furthermore, in the television device 900 configured in this way, the video signal processing unit 905 is capable of encoding image data provided from the decoder 904 and outputting the obtained encoded data from the outside to the television device 900 via the external interface 909. Additionally, the video signal processing unit 905 may also include the functions of the image encoding device 10 described above. In other words, the video signal processing unit 905 can be configured to encode image data provided from the decoder 904 according to the methods described in each of the above embodiments. With this arrangement, the television device 900 becomes capable of performing bandwidth control through scaling list processing.

[0249] (2) Second application example: mobile phone

[0250] Figure 18 An example of a schematic configuration of a mobile phone applying the above embodiments is shown. The mobile phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a multiplexing / demultiplexing unit 928, a recording / playback unit 929, a display unit 930, a control unit 931, an operation unit 932, and a bus 933.

[0251] Antenna 921 is connected to communication unit 922. Speaker 924 and microphone 925 are connected to audio codec 923. Operation unit 932 is connected to control unit 931. Bus 933 interconnects communication unit 922, audio codec 923, camera unit 926, image processing unit 927, multiplexing / demultiplexing unit 928, recording / reproducing unit 929, display unit 930, and control unit 931.

[0252] Mobile phone 920 performs actions such as sending / receiving audio signals, sending / receiving emails or image data, capturing images, and recording data in various operating modes including audio call mode, data communication mode, photography mode, and video call mode.

[0253] In audio call mode, the analog audio signal generated by microphone 925 is provided to audio codec 923. Audio codec 923 then converts the analog audio signal into audio data, performs A / D conversion on the converted audio data, and compresses the data. Audio codec 923 then outputs the compressed audio data to communication unit 922. Communication unit 922 encodes and modulates the audio data to generate a transmission signal. Then, communication unit 922 transmits the generated transmission signal to base station (not shown) via antenna 921. Furthermore, communication unit 922 amplifies the radio signal received via antenna 921, performs frequency conversion, and acquires the received signal. Then, communication unit 922 demodulates and decodes the received signal to generate audio data and outputs the generated audio data to audio codec 923. Audio codec 923 expands the audio data, performs D / A conversion on the data, and generates an analog audio signal. Audio codec 923 then provides the generated audio signal to speaker 924 for audio output.

[0254] In data communication mode, for example, control unit 931 generates character data for configuring emails based on user operations detected by operation unit 932. Control unit 931 also displays the characters on display unit 930. Furthermore, control unit 931 generates email data based on instructions obtained from the user via operation unit 932 to send email data, and outputs the generated email data to communication unit 922. Communication unit 922 encodes and modulates the email data to generate a transmission signal. Then, communication unit 922 transmits the generated transmission signal to a base station (not shown) via antenna 921. Communication unit 922 also amplifies the radio signals received via antenna 921, performs frequency conversion, and acquires the received signal. Subsequently, communication unit 922 demodulates and decodes the received signal, recovers the email data, and outputs the recovered email data to control unit 931. Control unit 931 displays the content of the email on display unit 930 and provides the email data to the storage medium of recording / reproducing unit 929 so that the data is recorded on the medium.

[0255] Recording / reproducing unit 929 includes any readable and writable storage medium. For example, the storage medium can be an internal storage medium, such as RAM or flash memory, or it can be an externally installed storage medium, such as a hard disk, magnetic disk, magneto-optical disk, optical disk, USB storage device, or memory card.

[0256] In photographic mode, for example, camera unit 926 images the object to generate image data and outputs the generated image data to image processing unit 927. Image processing unit 927 encodes the image data input from camera unit 926 and provides the encoded stream to the storage medium of recording / reproducing unit 929 so that the encoded stream is recorded in the medium.

[0257] Furthermore, in image display mode, the recording / reproducing unit 929 reads the encoded stream recorded on the storage medium and outputs it to the image processing unit 927. The image processing unit 927 decodes the encoded stream input from the recording / reproducing unit 929, provides image data to the display unit 930, and displays the image.

[0258] In videophone mode, for example, multiplexing / demultiplexing unit 928 multiplexes the video stream encoded by image processing unit 927 and the audio stream input from audio codec 923, and outputs the multiplexed stream to communication unit 922. Communication unit 922 encodes and modulates the stream to generate a transmission signal. Then, communication unit 922 transmits the generated transmission signal to base station (not shown) via antenna 921. Furthermore, communication unit 922 amplifies the radio signal received via antenna 921, performs frequency conversion, and acquires the received signal. The transmission and received signals may include coded bitstreams. Communication unit 922 therefore demodulates and decodes the received signal to recover the stream, and outputs the recovered stream to multiplexing / demultiplexing unit 928. Multiplexing / demultiplexing unit 928 demultiplexes the video and audio streams from the input stream and outputs the video and audio streams to image processing unit 927 and audio codec 923, respectively. Image processing unit 927 decodes the video stream to generate video data. The video data is then provided to the display unit 930, which displays a series of images. The audio codec 923 extends and performs D / A conversion on the audio stream to generate an analog audio signal. The audio codec 923 then provides the generated audio signal to the speaker 924 to output audio.

[0259] In the mobile phone 920 configured in this way, the image processing unit 927 may include, for example, the functions of the image encoding device 10 described above. In other words, the image processing unit 927 may be configured to encode image data according to the methods described in each of the above embodiments. With this arrangement, the mobile phone 920 becomes capable of performing bandwidth control through scaling list processing.

[0260] Furthermore, in the mobile phone 920 configured in this way, the image processing unit 927 may include, for example, the functions of the image decoding device 60 described above. In other words, the image processing unit 927 may be configured to decode encoded data according to the methods described in each of the above embodiments. With this arrangement, the mobile phone 920 becomes capable of performing decoding according to bandwidth control processed by the scaling list.

[0261] (3) Third application example: Recording / reproduction devices

[0262] Figure 19An example of a schematic configuration of a recording / reproduction apparatus applying the above embodiments is shown. For example, the recording / reproduction apparatus 940 encodes audio and video data of received broadcast programs and records the data onto a recording medium. For example, the recording / reproduction apparatus 940 can also encode audio and video data acquired from another device and record the data onto a recording medium. The recording / reproduction apparatus 940, for example, in response to a user instruction, reproduces the data recorded on the recording medium on a monitor and speakers. In this case, the recording / reproduction apparatus 940 decodes the audio and video data.

[0263] The recording / reproduction device 940 includes a tuner 941, an external interface 942, an encoder 943, a hard disk drive (HDD) 944, a disk drive 945, a selector 946, a decoder 947, an on-screen display (OSD) 948, a control unit 949, and a user interface 950.

[0264] Tuner 941 extracts the signal of the desired channel from the broadcast signal received through an antenna (not shown) and demodulates the extracted signal. Then, tuner 941 outputs the coded bit stream obtained by demodulation to selector 946. That is, tuner 941 functions as a transmission device in recording / reproducing apparatus 940.

[0265] External interface 942 is an interface for connecting recording / reproduction device 940 to external devices or networks. External interface 942 can be, for example, an IEEE 1394 interface, a network interface, a USB interface, or a flash memory interface. For example, video and audio data received through external interface 942 are input to encoder 943. In other words, external interface 942 functions as a transmission device within recording / reproduction device 940.

[0266] Encoder 943 encodes the video and audio data input from external interface 942 without encoding them. Encoder 943 then outputs the encoded bitstream to selector 946.

[0267] The HDD 944 records an encoded bitstream to its internal hard drive, compressing content data such as video and audio, various programs, and other data within that bitstream. When the video and audio are reproduced, the HDD 944 reads this data from the hard drive.

[0268] The disk drive 945 records and reads data from a recording medium attached to the disk drive. The recording medium attached to the disk drive 945 may be, for example, a DVD disk (e.g., DVD video, DVD-RAM, DVD-R, DVD-RW, DVD+R, or DVD+RW) or a Blu-ray disc (registered trademark).

[0269] Selector 946 selects the encoded bitstream input from tuner 941 or encoder 943 when recording video and audio, and outputs the selected encoded bitstream to HDD 944 or disk drive 945. When reproducing video and audio, on the other hand, selector 946 outputs the encoded bitstream input from HDD 944 or disk drive 945 to decoder 947.

[0270] Decoder 947 decodes the encoded bitstream to generate video and audio data. Then, decoder 904 outputs the generated video data to OSD 948 and the generated audio data to an external speaker.

[0271] The OSD 948 reproduces the video data input from the decoder 947 and displays the video. The OSD 948 can also overlay images of GUI elements such as menus, buttons, or cursors onto the displayed video.

[0272] The control unit 949 includes a processor such as a CPU and memories such as RAM and ROM. The memories store programs executed by the CPU, as well as program data. The programs stored in the memories are read and executed by the CPU when the recording / reproduction device 940 is started. By executing the program, the CPU controls the operation of the recording / reproduction device 940 according to operating signals input from, for example, the user interface 950.

[0273] User interface 950 is connected to control unit 949. User interface 950 includes buttons and switches for user operation recording / replay device 940, and a receiver for receiving, for example, remote control signals. User interface 950 detects user operations through these components to generate operation signals, and outputs the generated operation signals to control unit 949.

[0274] In the recording / reproducing apparatus 940 configured in this manner, the encoder 943 includes the functions of the image encoding apparatus 10 according to the above embodiment. Additionally, the decoder 947 includes the functions of the image decoding apparatus 60 according to the above embodiment. With this arrangement, bandwidth control via scaling list processing becomes possible when the recording / reproducing apparatus 940 applies multiple (inverse)transformations for encoding or decoding.

[0275] (4) Fourth application example: imaging equipment

[0276] Figure 20 An example of a schematic configuration of an imaging apparatus applying the above embodiments is shown. The imaging apparatus 960 images an object to generate an image, encodes the image data, and records the data into a recording medium.

[0277] The imaging device 960 includes an optical block 961, an imaging unit 962, a signal processing unit 963, an image processing unit 964, a display unit 965, an external interface 966, a memory 967, a media driver 968, an OSD 969, a control unit 970, a user interface 971, and a bus 972.

[0278] Optical block 961 is connected to imaging unit 962. Imaging unit 962 is connected to signal processing unit 963. Display unit 965 is connected to image processing unit 964. User interface 971 is connected to control unit 970. Bus 972 interconnects image processing unit 964, external interface 966, memory 967, media driver 968, OSD 969 and control unit 970.

[0279] Optical block 961 includes a focusing lens and an aperture mechanism. Optical block 961 forms an optical image of the object on the imaging plane of imaging unit 962. Imaging unit 962 includes an image sensor such as a CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor) and performs photoelectric conversion to convert the optical image formed on the imaging plane into an image signal as an electrical signal. Imaging unit 962 then outputs the image signal to signal processing unit 963.

[0280] The signal processing unit 963 performs various camera signal processing operations on the image signal input from the imaging unit 962, such as inflection point correction, gamma correction, and color correction. The signal processing unit 963 then outputs the image data that has undergone camera signal processing to the image processing unit 964.

[0281] Image processing unit 964 encodes image data input from signal processing unit 963 and generates encoded data. Then, image processing unit 964 outputs the generated encoded data to external interface 966 or media driver 968. Image processing unit 964 also decodes encoded data input from external interface 966 or media driver 968 to generate image data. Then, image processing unit 964 outputs the generated image data to display unit 965. Furthermore, image processing unit 964 can output image data input from signal processing unit 963 to display unit 965 so that display unit 965 displays an image. Additionally, image processing unit 964 can overlay display data acquired from OSD 969 onto the image output on display unit 965.

[0282] OSD 969 generates images of GUI elements such as menus, buttons, or cursors, and outputs the generated images to image processing unit 964.

[0283] External interface 966 is configured, for example, as a USB input / output terminal. For instance, external interface 966 connects imaging device 960 to a printer when printing images. Furthermore, a drive can be connected to external interface 966 as needed. For example, a removable medium such as a disk or optical disc can be attached to the drive, allowing programs read from the removable medium to be installed onto imaging device 960. External interface 966 can also be configured as a network interface for connecting to a network such as a LAN or the Internet. In other words, external interface 966 functions as a transmission device within imaging device 960.

[0284] The recording medium attached to the media drive 968 can be any removable medium that is both readable and writable, such as a magnetic disk, magneto-optical disk, optical disk, or semiconductor memory. Alternatively, the recording medium can be fixedly attached to the media drive 968, allowing for the configuration, for example, of a non-removable storage unit such as a built-in hard disk drive or a solid-state drive (SSD).

[0285] The control unit 970 includes a processor such as a CPU and memories such as RAM and ROM. The memories store programs executed by the CPU, as well as program data. The programs stored in the memories are read and executed by the CPU when the imaging device 960 is started. By executing the programs, the CPU controls the operation of the imaging device 960 according to operating signals input from, for example, the user interface 971.

[0286] User interface 971 is connected to control unit 970. User interface 971 includes buttons and switches for, for example, user operation of imaging device 960. User interface 971 detects user operations through these components to generate operation signals and outputs the generated operation signals to control unit 970.

[0287] In the imaging device 960 configured in this manner, the image processing unit 964 includes the functions of the image encoding device 10 and the image decoding device 60 according to the above embodiments. With this arrangement, bandwidth control via scaling list processing becomes possible when the imaging device 960 applies multiple (inverse)transforms for encoding or decoding.

[0288] (5) Fifth application example: video collection

[0289] In addition, this technology can also be implemented as any type of configuration of a device included in any device or system, such as a processor provided as a large-scale integrated (LSI) chip, a module using multiple processors, a unit using multiple modules, a collection of additional functions added to the unit (i.e., a configuration of part of a device), etc. Figure 21 An example of a schematic configuration of a video set applying this technology is shown.

[0290] Recently, electronic devices are becoming more multifunctional, and in the development and manufacture of such electronic devices, in the case of implementing a partial configuration for sale, supply, etc., it has become common not only to implement the method as a configuration including a single function, but also to combine multiple configurations including related functions and implement the method as a single set including multiple functions.

[0291] Figure 21 The video set 1300 shown is a multifunctional configuration and is a combination of means including functions related to image encoding and decoding (one or both) and means including other functions related to these functions.

[0292] like Figure 21 As shown, the video set 1300 includes a group of modules such as a video module 1311, an external memory 1312, a power management module 1313, and a front-end module 1314, and Figure 21 The device also includes related functions such as connection 1321, camera 1322 and sensor 1323.

[0293] A module is a component that integrates several interrelated functional parts into a unified function. The specific physical configuration can be any configuration, but for example, it is conceivable to set up and integrate multiple processors with their own functions, electronic circuit elements such as resistors and capacitors, other devices, etc., onto a circuit board. It is also conceivable to combine a module with another module, processor, etc., to create a new module.

[0294] exist Figure 21 In the example case, video module 1311 is a combination of configurations including image processing-related functions, and includes an application processor, a video processor, a broadband modem 1333, and an RF module 1334.

[0295] A processor is a configuration with predetermined functions integrated into a semiconductor chip as a System-on-Chip (SoC), and can also be specified as, for example, a Large-Scale Integration (LSI) chip. The configuration with predetermined functions can be logic circuitry (hardware configuration), but it can also be a CPU, ROM, RAM, etc., and programs executed using these (software configuration), or a combination of both. For example, a processor can include logic circuitry and a CPU, ROM, RAM, etc., and can be configured to implement a subset of functions using the logic circuitry (hardware configuration) while implementing other functions using programs executed on the CPU (software configuration).

[0296] Figure 21The application processor 1331 is a processor that executes applications related to image processing. In order to achieve the intended functions, the applications executing in the application processor 1331 are not only able to perform computational processing, but also able to control the internal and external configurations of the video module 1311, such as the video processor 1332, when necessary.

[0297] The video processor 1332 is a processor that includes functions related to image encoding / decoding (one or both).

[0298] The broadband modem 1333 performs digital modulation, etc., to convert data (digital signals) transmitted through wired or wireless (or both) broadband communications performed via a broadband connection such as the Internet or public telephone networks into analog signals, and also performs demodulation to convert analog signals received through such broadband communications into data (digital signals). The broadband modem 1333 processes any type of information, such as image data processed by the video processor 1332, streams of encoded image data, application and settings data.

[0299] RF module 1334 is a module that performs frequency conversion, modulation / demodulation, amplification, filtering, and other processing on radio frequency (RF) signals transmitted and received via an antenna. For example, RF module 1334 generates RF signals by performing frequency conversion and other processes on baseband signals generated by broadband modem 1333. Furthermore, for example, RF module 1334 generates baseband signals by performing frequency conversion and other processes on RF signals received via front-end module 1314.

[0300] Note that, as Figure 21 As shown by the dashed line 1341, the application processor 1331 and the video processor 1332 can also be unified and configured as a single processor.

[0301] External memory 1312 is a module provided outside of video module 1311, which includes storage devices used by video module 1311. The storage devices of external memory 1312 can be implemented in any kind of physical configuration, but since storage devices are typically used to store large amounts of data such as image data in units of frames, it is desirable to implement storage devices with relatively inexpensive and high-capacity semiconductor memory (e.g., dynamic random access memory (DRAM)).

[0302] The power management module 1313 manages and controls the power supply to the video module 1311 (each configuration within the video module 1311).

[0303] Front-end module 1314 is a module that provides front-end functions (circuit on the antenna-side transmit / receive port) to RF module 1334. For example... Figure 21As shown, the front-end module 1314 includes, for example, an antenna unit 1351, a filter 1352, and an amplifier unit 1353.

[0304] Antenna unit 1351 includes an antenna for transmitting and receiving wireless signals, and its peripheral configuration. Antenna unit 1351 transmits the signal provided from amplification unit 1353 as a wireless signal, and provides the received wireless signal as an electrical signal (RF signal) to filter 1352. Filter 1352 performs filtering and other processing on the RF signal received by antenna unit 1351, and provides the processed RF signal to RF module 1334. Amplification unit 1353 amplifies the RF signal provided from RF module 1334 and provides it to antenna unit 1351.

[0305] Connection 1321 is a module that includes functions related to external connectivity. The physical configuration of connection 1321 can be any configuration. For example, connection 1321 includes a configuration with communication functions other than those supported by communication standards such as broadband modem 1333 and external input / output terminals.

[0306] For example, connection 1321 may include a module with communication capabilities conforming to wireless communication standards (e.g., Bluetooth, IEEE 802.11 (e.g., Wi-Fi), Near Field Communication (NFC), or Infrared Data Association (IrDA)), and an antenna for transmitting and receiving signals conforming to those standards. Additionally, connection 1321 may include a module with communication capabilities conforming to wired communication standards (e.g., Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI)), and a port conforming to those standards. Furthermore, connection 1321 may include the ability to transmit another type of data (signal), such as an analog input / output terminal.

[0307] Note that connection 1321 may include a destination device for transmitting data (signals). For example, connection 1321 may include a drive (not only drives for removable media, but also hard disks, solid-state drives (SSDs), network attached storage (NAS), etc.) that reads and writes data about recording media such as disks, optical disks, magneto-optical disks, or semiconductor memory. Furthermore, connection 1321 may include devices for outputting images and sound (such as monitors and speakers).

[0308] Camera 1322 is a module that has the function of imaging an object and obtaining image data of the object. For example, the image data obtained by imaging through camera 1322 is provided to video processor 1332 and encoded.

[0309] Sensor 1323 is a module with any type of sensor function, such as a sound sensor, ultrasonic sensor, light sensor, illuminance sensor, infrared sensor, image sensor, rotation sensor, angle sensor, angular velocity sensor, speed sensor, acceleration sensor, tilt sensor, magnetic field sensor, vibration sensor, or temperature sensor. For example, the data detected by sensor 1323 is provided to application processor 1331 and utilized by applications, etc.

[0310] The configuration described as a module above can also be implemented as a processor, and conversely, the configuration described as a processor can also be implemented as a module.

[0311] In a video set 1300 with the above configuration, this technology can be applied to a video processor 1332, which will be described later. Therefore, the video set 1300 can be implemented as a collection of applications of this technology.

[0312] (Example configuration of a video processor)

[0313] Figure 22 The video processor 1332 using this technology is shown. Figure 21 An example of a schematic configuration.

[0314] exist Figure 22 In the example cases, such as Figure 22 As shown, the video processor 1332 includes the functions of receiving input video signals and audio signals and encoding these signals according to a predetermined method, as well as the functions of decoding the encoded video data and audio data and reproducing and outputting the video signals and audio signals.

[0315] like Figure 22 As shown, the video processor 1332 includes a video input processing unit 1401, a first image magnification / reduction unit 1402, a second image magnification / reduction unit 1403, a video output processing unit 1404, a frame memory 1405, and a memory control unit 1406. Furthermore, the video processor 1332 includes an encoding / decoding engine 1407, video elementary stream (ES) buffers 1408A and 1408B, and audio ES buffers 1409A and 1409B. Additionally, the video processor 1332 includes an audio encoder 1410, an audio decoder 1411, a multiplexer (MUX) 1412, a demultiplexer (DMUX) 1413, and a stream buffer 1414.

[0316] The video input processing unit 1401, for example, acquires data from connection 1321 ( Figure 21The first image magnification / reduction unit 1402 performs format conversion and image magnification / reduction processing on the image data. The second image magnification / reduction unit 1403 performs image data magnification or reduction processing according to the format of the destination output through the video output processing unit 1404, and performs format conversion and image magnification / reduction processing similar to the first image magnification / reduction unit 1402. The video output processing unit 1404 performs format conversion and conversion to analog signal on the image data, and outputs the result to the connection 1321, for example, as a reproduced video signal.

[0317] The frame memory 1405 is a memory for image data shared by the video input processing unit 1401, the first image magnification / reduction unit 1402, the second image magnification / reduction unit 1403, the video output processing unit 1404, and the encoding / decoding engine 1407. The frame memory 1405 is implemented, for example, as a semiconductor memory such as DRAM.

[0318] The memory control unit 1406 receives a synchronization signal from the encoding / decoding engine 1407 and controls access to, writing to, and reading from the frame memory 1405 according to the access schedule of the frame memory 1405 written in the access management table 1406A. The memory control unit 1406 updates the access management table 1406A according to the processing performed by the encoding / decoding engine 1407, the first image magnification / reduction unit 1402, the second image magnification / reduction unit 1403, etc.

[0319] Encoding / decoding engine 1407 performs processing of encoded image data and decoding of video streams, whereby the video stream is data encoded from the image data. For example, encoding / decoding engine 1407 encodes image data read from frame memory 1405 and continuously writes the encoded data as a video stream to video ES buffer 1408A. Additionally, for example, encoding / decoding engine 1407 continuously reads and decodes the video stream from video ES buffer 1408B and writes the decoded data as image data to frame memory 1405. During this encoding and decoding, encoding / decoding engine 1407 uses frame memory 1405 as its working area. Furthermore, for example, encoding / decoding engine 1407 outputs a synchronization signal to memory control unit 1406 at the start of processing each macroblock.

[0320] The video ES buffer 1408A buffers and provides the video stream generated by the encoding / decoding engine 1407 to the multiplexer (MUX) 1412. The video ES buffer 1408B buffers the video stream provided from the demultiplexer (DMUX) 1413 and provides that video stream to the encoding / decoding engine 1407.

[0321] Audio ES buffer 1409A buffers and provides the audio stream generated by audio encoder 1410 to multiplexer (MUX) 1412. Audio ES buffer 1409B buffers the audio stream provided from demultiplexer (DMUX) 1413 and provides the audio stream to audio decoder 1411.

[0322] The audio encoder 1410 performs digital conversion on audio signals input from the connection 1321, and encodes the audio signals according to a predetermined method, such as the MPEG audio method or the Audio Code 3 (AC3) method. The audio encoder 1410 continuously writes the audio stream (in which the audio signal is encoded data) into the audio ES buffer 1409A. The audio decoder 1411 decodes the audio stream provided from the audio ES buffer 1409B, for example, performing a conversion to an analog signal, and provides the result to the connection 1321, for example, as a reproduced audio signal.

[0323] The multiplexer (MUX) 1412 multiplexes video and audio streams. The multiplexing method (i.e., the format of the bitstream generated by multiplexing) can be any method. Furthermore, during this multiplexing, the multiplexer (MUX) 1412 can also add predetermined header information, etc., to the bitstream. In other words, the multiplexer (MUX) 1412 can convert the format of the stream through multiplexing. For example, by multiplexing video and audio streams, the multiplexer (MUX) 1412 converts the streams into a transport stream, which is a bitstream in a format for transmission. Additionally, for example, by multiplexing video and audio streams, the multiplexer (MUX) 1412 converts the streams into data (file data) in a file format for recording.

[0324] Demultiplexer (DMUX) 1413 demultiplexes the bitstream that has multiplexed the video and audio streams according to a method corresponding to the multiplexing method of multiplexer (MUX) 1412. In other words, demultiplexer (DMUX) 1413 extracts the video and audio streams from the bitstream read from stream buffer 1414 (separating the video and audio streams). In other words, demultiplexer (DMUX) 1413 can convert the format of the stream by demultiplexing (the inverse conversion of multiplexer (MUX) 1412). For example, demultiplexer (DMUX) 1413 can obtain the transport stream provided from connection 1321, broadband modem 1333, etc., for example via stream buffer 1414, and can convert the transport stream into video and audio streams by demultiplexing. Additionally, for example, the demultiplexer (DMUX) 1413 can acquire file data read from any type of recording medium of various types of recording media via the connection 1321, for example, via the stream buffer 1414, and can convert the file data into video and audio streams through demultiplexing.

[0325] Stream buffer 1414 buffers the bit stream. For example, stream buffer 1414 buffers the transport stream provided from multiplexer (MUX) 1412 and provides the transport stream to connection 1321, broadband modem 1333, etc., at a predetermined time or based on external requests.

[0326] Additionally, for example, the stream buffer 1414 buffers file data provided from the multiplexer (MUX) 1412, and provides the file data to the connection 1321 at a predetermined time or based on an external request, for example, so that the file data is recorded in any type of recording medium of various types of recording media.

[0327] In addition, the stream buffer 1414 buffers, for example, the transport streams acquired via the connection 1321, the broadband modem 1333, etc., and provides the transport streams to the demultiplexer (DMUX) 1413 at a predetermined time or based on an external request.

[0328] Additionally, the stream buffer 1414 buffers file data read from any type of recording medium from various types of recording media, such as in the connection 1321, and provides the file data to the demultiplexer (DMUX) 1413 at a predetermined time or based on an external request.

[0329] Next, an example of the operation of the video processor 1332 with this configuration will be described. For example, a video signal input to the video processor 1332 from a connection such as 1321 is converted into digital image data in a predetermined format (e.g., 4:2:2Y / Cb / Cr format) in the video input processing unit 1401 and continuously written to the frame memory 1405. The digital image data is read out to the first image magnification / reduction unit 1402 or the second image magnification / reduction unit 1403, undergoes format conversion to a predetermined format (e.g., 4:2:0Y / Cb / Cr, etc.) and magnification / reduction processing, and is written to the frame memory 1405 again. The image data is encoded by the encoding / decoding engine 1407 and written as a video stream to the video ES buffer 1408A.

[0330] In addition, the audio signals input from the connection 1321 to the video processor 1332 are encoded by the audio encoder 1410 and written as an audio stream to the audio ES buffer 1409A.

[0331] The video stream in the video ES buffer 1408A and the audio stream in the audio ES buffer 1409A are read and multiplexed by the multiplexer (MUX) 1412, and converted into transport streams, file data, etc. The transport stream generated by the multiplexer (MUX) 1412 is buffered in the stream buffer 1414, and then output to an external network via, for example, a connection 1321, a broadband modem 1333, etc. Furthermore, the file data generated by the multiplexer (MUX) 1412 is buffered in the stream buffer 1414, and then output to, for example, a connection 1321, and recorded on any type of recording medium of various types of recording media.

[0332] Furthermore, transport streams input to video processor 1332 from external networks, such as via connection 1321 or broadband modem 1333, are buffered in stream buffer 1414 and then demultiplexed by demultiplexer (DMUX) 1413. Additionally, file data read from any type of recording medium from various types of recording media, such as via connection 1321, is input to video processor 1332 and buffered in buffer 1414, then demultiplexed by demultiplexer (DMUX) 1413. In other words, the demultiplexer (DMUX) 1413 separates the transport stream or file data input to video processor 1332 into video and audio streams.

[0333] The audio stream is provided to the audio decoder 1411 via the audio ES buffer 1409B and decoded, and the audio signal is reproduced. Meanwhile, the video stream, after being written to the video ES buffer 1408B, is continuously read and decoded by the encoding / decoding engine 1407 and written to the frame memory 1405. The decoded image data undergoes magnification / reduction processing by the second image magnification / reduction unit 1403 and is written to the frame memory 1405. Then, the decoded image data is read out to the video output processing unit 1404, the format is converted to a predetermined format (e.g., 4:2:2Y / Cb / Cr format), further converted to an analog signal, and the video signal is reproduced and output.

[0334] When applying this technology to the video processor 1332 configured in this manner, it is sufficient to apply the technology according to each of the above embodiments to the encoding / decoding engine 1407. In other words, for example, the encoding / decoding engine 1407 may include the functions of the image encoding device 10 or the image decoding device 60 described above, or both. With this arrangement, the video processor 1332 is able to achieve the same performance as described above. Figures 1 to 15 Each of the described embodiments has similar effects.

[0335] Note that in the encoding / decoding engine 1407, this technology (i.e., the function of the image encoding device 10, the function of the image decoding device 60, or both) can be implemented by hardware such as logic circuits, or by software such as embedded programs, or by both of the above.

[0336] (Another exemplary configuration of the video processor)

[0337] Figure 23 Another example of a schematic configuration of a video processor 1332 applying this technology is shown. Figure 23 In the example case, the video processor 1332 includes the function of encoding / decoding video data according to a predetermined method.

[0338] More specifically, such as Figure 23 As shown, the video processor 1332 includes a control unit 1511, a display interface 1512, a display engine 1513, an image processing engine 1514, and internal memory 1515. Furthermore, the video processor 1332 includes a codec engine 1516, a memory interface 1517, a multiplexer / demultiplexer (MUX / DMUX) 1518, a network interface 1519, and a video interface 1520.

[0339] The control unit 1511 controls the operation of each processing unit in the video processor 1332, such as the display interface 1512, the display engine 1513, the image processing engine 1514, and the codec engine 1516.

[0340] like Figure 23 As shown, the control unit 1511 includes, for example, a main CPU 1531, a secondary CPU 1532, and a system controller 1533. The main CPU 1531 executes programs, etc., for controlling the operation of each processing unit in the video processor 1332. The main CPU 1531 generates control signals according to the programs, etc., and provides the control signals to each processing unit (in other words, controls the operation of each processing unit). The secondary CPU 1532 performs a supplementary role to the main CPU 1531. For example, the secondary CPU 1532 executes subprocesses, subroutines, etc., of the programs, etc., executed by the main CPU 1531. The system controller 1533 controls the operation of the main CPU 1531 and the secondary CPU 1532, for example, specifying the programs to be executed by the main CPU 1531 and the secondary CPU 1532.

[0341] For example, under the control of the control unit 1511, the display interface 1512 outputs image data to the connection 1321, etc. For example, the display interface 1512 converts digital image data into analog signals and outputs analog signals, or directly outputs digital image data as a reproduced video signal to the monitor device connected to 1321, etc.

[0342] Under the control of the control unit 1511, the display engine 1513 performs various conversion processes on the image data, such as format conversion, size conversion and color gamut conversion, to match the hardware specifications of the monitor device or other device to which the image is to be displayed.

[0343] Under the control of the control unit 1511, the image processing engine 1514 performs predetermined image processing on the image data, such as filtering processing to improve image quality.

[0344] Internal memory 1515 is memory provided within video processor 1332 and shared by display engine 1513, image processing engine 1514, and codec engine 1516. For example, internal memory 1515 is used to exchange data between display engine 1513, image processing engine 1514, and codec engine 1516. For example, internal memory 1515 stores data provided from display engine 1513, image processing engine 1514, or codec engine 1516, and provides data to display engine 1513, image processing engine 1514, or codec engine 1516 as needed (e.g., in response to a request). Internal memory 1515 can be implemented by any type of storage device; however, since storage devices are typically used to store small amounts of data such as image data in blocks, parameters, etc., it is desirable to implement a storage device with a relatively small capacity (e.g., compared to external memory 1312) but a fast response time, such as semiconductor memory (e.g., static random access memory (SRAM)).

[0345] Codec engine 1516 performs processing related to the encoding and decoding of image data. The encoding / decoding methods supported by codec engine 1516 can be any method, and one or more such methods may exist. For example, codec engine 1516 may be provided with codec functionality for multiple encoding / decoding methods and can be configured to encode or decode image data by selecting from multiple methods.

[0346] exist Figure 23 In the example shown, for instance, codec engine 1516 includes MPEG-2 video 1541, AVC / H.264 1542, HEVC / H.265 1543, HEVC / H.265 (Scalable) 1544, HEVC / H.265 (Multi-View) 1545, and MPEG-DASH 1551 as functional blocks for codec-related processing.

[0347] MPEG-2 Video 1541 is a function block for encoding and decoding image data according to the MPEG-2 method. AVC / H.264 1542 is a function block for encoding and decoding image data according to the AVC method. HEVC / H.265 1543 is a function block for encoding and decoding image data according to the HEVC method. HEVC / H.265 (Scalable) 1544 is a function block for scaling encoding and decoding image data according to the HEVC method. HEVC / H.265 (Multi-View) 1545 is a function block for multi-view encoding and decoding image data according to the HEVC method.

[0348] MPEG-DASH 1551 is a function block for sending and receiving image data according to the MPEG Dynamic Adaptive HTTP Streaming (MPEG-DASH) method. MPEG-DASH is a technology for streaming video using the Hypertext Transfer Protocol (HTTP), characterized by selecting and transmitting appropriate encoded data from pre-prepared segments of encoded data with different resolutions, etc. MPEG-DASH 1551 performs standard-compliant stream generation, transmission control, etc., while for encoding / decoding image data, it uses MPEG-2 Video 1541 to HEVC / H.265 (Multi-View) 1545.

[0349] Memory interface 1517 is the interface to external memory 1312. Data provided from image processing engine 1514 and codec engine 1516 is provided to external memory 1312 through memory interface 1517. In addition, data read from external memory 1312 is provided to video processor 1332 (image processing engine 1514 or codec engine 1516) through memory interface 1517.

[0350] The multiplexer / demultiplexer (MUX DMUX) 1518 multiplexes and demultiplexes various image-related data, such as bitstreams of encoded data, image data, video signals, etc. The multiplexing / demultiplexing method can be any method. For example, when multiplexing, the multiplexer / demultiplexer (MUX DMUX) 1518 can not only collect multiple data into a single data, but also add predetermined header information, etc., to the data. Furthermore, when demultiplexing, the multiplexer / demultiplexer (MUX DMUX) 1518 can not only divide a single data into multiple data, but also add predetermined header information, etc., to each segment of data. In other words, the multiplexer / demultiplexer (MUX DMUX) 1518 can convert data formats through multiplexing / demultiplexing. For example, by multiplexing a bitstream, the multiplexer / demultiplexer (MUX DMUX) 1518 can convert the bitstream into a transport stream (which is a bitstream in a format for transmission), or into file-format data (file data) for recording. Clearly, inverse transformation is also possible through demultiplexing.

[0351] Network interface 1519 is, for example, an interface for a broadband modem 1333, a connection 1321, etc. Video interface 1520 is, for example, an interface for connecting 1321, a camera 1322, etc.

[0352] Next, an example of the operation of this video processor 1332 will be described. For example, when a transport stream is received from an external network via connection 1321, broadband modem 1333, etc., the transport stream is provided to multiplexer / demultiplexer (MUX / DMUX) 1518 via network interface 1519 and demultiplexed, and then decoded by codec engine 1516. For example, the image data obtained by decoding by codec engine 1516 undergoes predetermined image processing by image processing engine 1514, undergoes predetermined conversion by display engine 1513, and is provided to connection 1321 via display interface 1512, for example, and the image is displayed on a monitor. Alternatively, for example, the image data obtained by decoding by codec engine 1516 is re-encoded by codec engine 1516, multiplexed by multiplexer / demultiplexer (MUX / DMUX) 1518 and converted into file data, for example, output to connection 1321 via video interface 1520, etc., and recorded on any type of recording medium of various types of recording media.

[0353] Furthermore, for example, the file data of the encoded image data read from a recording medium (not shown) via connection 1321 is provided to a multiplexer / demultiplexer (MUX / DMUX) 1518 via video interface 1520 and demultiplexed, and then decoded by codec engine 1516. The image data obtained by decoding by codec engine 1516 undergoes predetermined image processing by image processing engine 1514, undergoes predetermined conversion by display engine 1513, and is provided to connection 1321 via display interface 1512, etc., and the image is displayed on a monitor. Additionally, for example, the image data obtained by decoding by codec engine 1516 is re-encoded by codec engine 1516, multiplexed by multiplexer / demultiplexer (MUX / DMUX) 1518 and converted into a transport stream, provided to connection 1321, broadband modem 1333, etc., via network interface 1519, and sent to another device (not shown).

[0354] Note that, for example, the exchange of image data and other data between each processing unit within the video processor 1332 is performed by utilizing internal memory 1515 and external memory 1312. Additionally, the power management module 1313 controls, for example, the power supply to the control unit 1511.

[0355] When applying this technology to the video processor 1332 configured in this manner, it is sufficient to apply the technology according to each of the above embodiments to the codec engine 1516. In other words, for example, it is sufficient for the codec engine 1516 to include the functions of the image encoding device 10 or the image decoding device 60, or both. With this arrangement, the video processor 1332 is able to achieve the same level of performance as described above. Figures 1 to 15 Each of the described embodiments has similar effects.

[0356] Note that in the codec engine 1516, this technology (i.e., the function of the image encoding device 10) can be implemented by hardware such as logic circuits, by software such as embedded programs, or by both.

[0357] The above examples show two configurations of the video processor 1332, but the video processor 1332 can be configured in any way, and may be any configuration other than the two examples described above. Furthermore, the video processor 1332 can be configured as a single semiconductor chip, but it can also be configured as multiple semiconductor chips. For example, a three-dimensional stacked LSI chip with multiple semiconductors stacked is possible. Moreover, a configuration implemented with multiple LSI chips is possible.

[0358] (Application example of the device)

[0359] Video set 1300 can be embedded in any type of device across various types of devices that process image data. For example, video set 1300 can be embedded in a television device 900 (…). Figure 17 ), mobile phone 920 ( Figure 18 ), Recording / Reproduction Equipment 940 ( Figure 19 ), Imaging equipment 960 ( Figure 20 ), etc. By embedding video set 1300, the device is able to obtain the same information as the reference above. Figures 1 to 15 Each of the described embodiments has similar effects.

[0360] Note that as long as video processor 1332 is included, even a portion of each configuration of the aforementioned video set 1300 can be executed as a configuration applying this technology. For example, only video processor 1332 can be executed as a video processor applying this technology. Furthermore, for example, the processor, video module 1311, etc., shown by the dashed line 1341 as described above, can be executed as a processor, module, etc., applying this technology. Furthermore, for example, video module 1311, external memory 1312, power management module 1313, and front-end module 1314 can also be combined and executed as a video unit 1361 applying this technology. Using any of these configurations, a configuration similar to the one referenced above can be obtained. Figures 1 to 15 Each of the described embodiments has similar effects.

[0361] In other words, as long as video processor 1332 is included, any type of configuration can be embedded in various types of devices that process image data, similar to the case of video set 1300. For example, video processor 1332, the processor shown by dashed line 1341, video module 1311, or video unit 1361 can be embedded in television device 900 (…). Figure 17 ), mobile phone 920 ( Figure 18 ), Recording / Reproduction Equipment 940 ( Figure 19 ), Imaging equipment 960 ( Figure 20 ), etc. Furthermore, by embedding any configuration that applies this technology, the device can obtain the same information as the above reference. Figures 1 to 15 Each of the described embodiments has a similar effect, similar to video set 1300.

[0362] <Sixth Application Example: Network Systems>

[0363] In addition, this technology is also applicable to network systems that include multiple devices. Figure 24 An example of a schematic configuration of a network system applying this technology is shown.

[0364] Figure 24The network system 1600 shown is a system in which devices exchange information related to images (moving images) with each other via a network. The cloud service 1601 of the network system 1600 is a system that provides image (moving image) related services to terminals such as computers 1611, audiovisual (AV) devices 1612, mobile information processing terminals 1613, and Internet of Things (IoT) devices 1614 communicatively connected to the cloud service 1601. For example, the cloud service 1601 provides a service of providing image (moving image) content to terminals, such as so-called video streaming (on-demand or live). As another example, the cloud service 1601 provides a backup service for receiving and storing image (moving image) content from terminals. As yet another example, the cloud service 1601 provides a service for mediating the exchange of image (moving image) content between terminals.

[0365] The physical configuration of Cloud Service 1601 can be any configuration. For example, Cloud Service 1601 may include various servers (such as servers for storing and managing motion images, servers for delivering motion images to terminals, servers for acquiring motion images from terminals, and servers for managing users (terminals) and payments), and any type of network, such as the Internet or LAN.

[0366] Computer 1611 includes information processing devices such as personal computers, servers, or workstations. AV devices 1612 include image processing devices such as television receivers, hard disk recorders, game consoles, or cameras. Mobile information processing terminals 1613 include mobile information processing devices such as laptop computers, tablets, mobile phones, or smartphones. IoT devices 1614 include any object performing image-related processing, such as machines, appliances, furniture, other items, IC tags, or card-shaped devices. These terminals all include communication capabilities and are able to connect to (establish a session with) cloud service 1601 and exchange information with (i.e., communicate with) cloud service 1601. Furthermore, each terminal is also able to communicate with another terminal. Communication between terminals can be performed via cloud service 1601 or without cloud service 1601.

[0367] When this technology is applied to network system 1600 as described above, and image (moving image) data is exchanged between terminals or between a terminal and cloud service 1601, the image data can be encoded / decoded as described in each embodiment. In other words, the terminals (from computer 1611 to IoT device 1614) and cloud service 1601 can each include the functions of the image encoding device 10 and image decoding device 60 described above. With this arrangement, bandwidth control via scaling list processing becomes possible when multiple (inverse)transformations are applied for encoding or decoding.

[0368] <5. Conclusion>

[0369] According to embodiments of the present disclosure as described above, bandwidth control can be performed using a scaling list even when multiple transformations are applied.

[0370] Preferred embodiments of this disclosure have been described above with reference to the accompanying drawings, but this disclosure is not limited to the examples described above. Various changes and modifications will be found by those skilled in the art within the scope of the appended claims, and it should be understood that they will naturally fall within the technical scope of this disclosure.

[0371] Control information relating to the technology described in each of the above embodiments can be sent from the encoding side to the decoding side. For example, control information can be sent to control whether to allow (or deny) the application of the technology described above. In addition, for example, control information can be sent specifying an upper limit, a lower limit, or both of the block size for allowing (or denying) the application of the technology described above.

[0372] This technique can be applied to any type of image encoding / decoding that performs primary transforms, secondary transforms, and encoding (decoding, inverse secondary transform, and inverse primary transform). In other words, the specifications for transform (inverse transform), quantization (inverse quantization), encoding (decoding), prediction, etc., can be any specifications and are not limited to the examples above. For example, in the transform (inverse transform), inverse transforms can be performed other than the inverse primary and inverse secondary transforms (in other words, three or more inverse transforms). Furthermore, encoding (decoding) can be a lossless or lossy method. Additionally, quantization (inverse quantization), prediction, etc., can be omitted. Furthermore, processing not described above, such as filtering, can also be performed.

[0373] Furthermore, the effects described in this specification are merely illustrative or exemplary and are not intended to be limiting. That is, other effects that are apparent to those skilled in the art can be achieved from the description of this specification by utilizing or replacing the aforementioned effects, based on the technology disclosed herein.

[0374] In addition, this technology can also be configured as follows. (1)

[0376] An image processing device, comprising:

[0377] The processing control unit controls the scaling list processing based on transformation information related to the transformations applied to the processing target block. (2)

[0379] According to the image processing apparatus described in (1), wherein,

[0380] The processing control unit also controls the inverse transformation processing related to the inverse transformation applied to the processing target block based on the transformation information. (3)

[0382] According to the image processing device described in (2), wherein,

[0383] The processing control unit controls the inputs to the inverse transformation process based on the transformation information. (4)

[0385] The image processing apparatus according to any one of (1) to (3), wherein,

[0386] The processing control unit determines whether to perform inverse transformation processing related to another inverse transformation based on the transformation information. This other transformation is different from the transformation with respect to prediction error data, which is the difference between the image data and the predicted image data. (5)

[0388] According to the image processing device described in (4), wherein,

[0389] The processing control unit determines the input to the scaling list processing based on the input. (6)

[0391] According to the image processing apparatus described in (5), wherein,

[0392] When it is determined that the inverse transformation process will be performed, the processing control unit controls the input in the scaling list processing so that the coefficient data after the inverse transformation process becomes the input data in the scaling list processing. (7)

[0393] According to the image processing apparatus described in (5) or (6), wherein,

[0394] If it is determined that the inverse transformation process will not be performed, the processing control unit controls the input in the scaling list processing, so that the inverse quantized coefficient data becomes the input data in the scaling list processing. (8)

[0396] According to the image processing device described in (4), wherein,

[0397] The processing control unit determines the scaling list to be used in the scaling list processing. (9)

[0399] According to the image processing apparatus described in (8), wherein,

[0400] When it is determined that inverse transformation processing will be performed, the processing control unit specifies the scaling list to be used in the scaling list processing by performing an inner product operation between a predetermined scaling list and another transformation. (10)

[0402] According to the image processing device described in (8) or (9),

[0403] The inverse transform process is performed based on the convolution operation. (11)

[0405] According to the image processing apparatus described in (1), wherein,

[0406] Transformation information includes information indicating whether a predetermined transformation has been applied to the target block being processed. (12)

[0408] According to the image processing apparatus described in (1), wherein,

[0409] Transformation information includes information indicating the number of transformations applied to the target block being processed. (13)

[0411] An image processing method, comprising:

[0412] The processor controls the scaling list processing based on transformation information related to the transformations applied to the processing target block. (14)

[0414] A program that causes a computer to perform the following functions:

[0415] The scaling list processing is controlled based on transformation information related to the transformations applied to the target block being processed.

[0416] Reference tag list

[0417] 10 Image Encoding Devices

[0418] 12 Control Department

[0419] 13 Subtraction Section

[0420] 14 Processing Department

[0421] 16 Lossless Encoding Department

[0422] 17 Accumulator Buffer

[0423] 21 Reverse Processing Department

[0424] 23 Addition Department

[0425] 30-frame prediction unit

[0426] 40-frame prediction unit

[0427] 60 Image Decoding Devices

[0428] 61 Accumulator Buffer

[0429] 62 lossless decoding unit

[0430] 63 Reverse Processing Department

[0431] 65 Addition Department

[0432] 70-frame memory

[0433] 80-frame prediction unit

[0434] 90-frame prediction unit

[0435] 141 Primary Transformer

[0436] 142 Zoom List Processing Department

[0437] 143 Processing Control Department

[0438] 144 Secondary Transformation Unit

[0439] Quantitative Department 145

[0440] 146 Processing Control Department

[0441] 147 Transformer

[0442] 148 Quantization / Scaling List Processing Department

[0443] 631 Anti-Quantization Department

[0444] 632 Processing and Control Department

[0445] 633 Inverse Second Transform Unit

[0446] 634 Zoom List Processing Department

[0447] 635 Inverse Primary Transformer

[0448] 636 Processing and Control Department

[0449] 637 Inverse Quantization / Scaling List Processing Unit

[0450] 638 Inverse Transformer.

Claims

1. An image encoding device (10), comprising an encoding processing unit (14), configured to: Determine whether to perform a second transformation. If the second transformation is to be performed, transformation processing is performed by convolution of the first transformation and the second transformation. A second scaling list is calculated based on the inner product of the first scaling list and the second transformation. Quantization processing and scaling list processing are then performed using the calculated second scaling list. If it is determined that the secondary transformation will not be performed, the primary transformation is performed, and the first scaling list is used to perform quantization and scaling list processing. The obtained quantized data and the transformation information indicating that the second transformation has been performed are encoded.

2. The image encoding device according to claim 1, wherein, The transformation information includes information indicating the number of transformation processes, including the primary transformation and the secondary transformation.

3. An image encoding method, comprising: Determine whether to perform a second transformation; When it is determined that the second transformation is to be performed, the transformation processing is performed by the convolution operation of the primary transformation and the second transformation, the second scaling list is calculated based on the inner product operation of the first scaling list and the second transformation, and the calculated second scaling list is used to perform quantization processing and scaling list processing. If it is determined that the secondary transformation will not be performed, the primary transformation is performed, and the first scaling list is used to perform quantization and scaling list processing. as well as The obtained quantized data and the transformation information indicating that the second transformation has been performed are encoded.

4. The image encoding method according to claim 3, wherein, The transformation information includes information indicating the number of transformation processes, including the primary transformation and the secondary transformation.

Images