Image processing apparatus and image processing method

Abstract

To enable increasing prediction accuracy.SOLUTION: When intra-prediction using a matrix operation is performed, a prediction image of a current prediction block is generated by performing interpolation processing using original pixels of a reference image as above adjacent pixels that are adjacent on the prediction image of the current prediction block of a target of encoding / decoding. And, encoding / decoding of the current prediction block is performed using the prediction image. The present technology can be applied to a case where encoding and decoding of an image is performed, for example.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present technology relates to an image processing apparatus and an image processing method, and more particularly, to an image processing apparatus and an image processing method that can, for example, improve prediction accuracy.

Background Art

[0002] In JVET (Joint Video Experts Team), a joint standardization body of ITU-T and ISO / IEC, standardization work on VVC (Versatile Video Coding), a next-generation image coding method, is being advanced with the aim of further improving coding efficiency compared to H.265 / HEVC.

[0003] In the standardization work of VVC, Non-Patent Document 1 proposes generating a predicted image by averaging the pixels (pixel values) of a reference image, performing matrix multiplication using the averaged pixels obtained by averaging, and performing interpolation processing using the result of the matrix multiplication and using the averaged pixels as upper adjacent pixels adjacent to the predicted image of the prediction block.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the generation of the predicted image described in Non-Patent Document 1, interpolation processing is always performed using the averaged pixel as the upper adjacent pixel. Therefore, the prediction accuracy of the predicted image may not be improved.

[0006] The present technology has been made in view of such a situation, and aims to improve the prediction accuracy of the predicted image.

Means for Solving the Problem

[0007] The first image processing apparatus of the present technology, when performing intra prediction using matrix operation, performs interpolation processing using the original pixel of the reference image as the upper adjacent pixel adjacent to the upper side of the predicted image of the current prediction block to be encoded, thereby generating an intra prediction unit for generating a predicted image of the current prediction block, and an encoding unit for encoding the current prediction block using the predicted image generated by the intra prediction unit. Another first image processing apparatus of the present technology includes a setting unit that sets identification data for identifying whether to perform interpolation processing using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current prediction block for intra prediction using matrix operations, and an encoding unit that encodes the current prediction block and generates a bitstream including the identification data set by the setting unit.

[0008] The first image processing method of the present technology, when performing intra prediction using matrix operation, performs interpolation processing using the original pixel of the reference image as the upper adjacent pixel adjacent to the upper side of the predicted image of the current prediction block to be encoded, thereby generating an intra prediction step for generating a predicted image of the current prediction block, and an encoding step for encoding the current prediction block using the predicted image generated in the intra prediction step. Another first image processing method of the present technology includes setting identification data for identifying whether to perform interpolation processing using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current prediction block for intra prediction using matrix operations, and encoding the current prediction block to generate a bitstream including the identification data.

[0009] In the first image processing apparatus and the image processing method of the present technology, when performing intra prediction using matrix operation, interpolation processing is performed using the original pixel of the reference image as the upper adjacent pixel adjacent to the upper side of the predicted image of the current prediction block to be encoded, thereby generating a predicted image of the current prediction block. Then, the current prediction block is encoded using the predicted image. In another first image processing apparatus and image processing method of the present technology, identification data for identifying whether to perform interpolation processing using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current prediction block for intra prediction using matrix operations is set. Then, the current prediction block is encoded, and a bitstream including the identification data is generated.

[0010] When performing intra prediction using matrix operations, the second image processing apparatus of the present technology performs interpolation processing using the original pixels of the reference image as the upper adjacent pixels adjacent to the predicted image of the current prediction block to be decoded, thereby generating the predicted image of the current prediction block. An intra prediction unit, and a decoding unit that decodes the current prediction block using the predicted image generated by the intra prediction unit. Another second image processing apparatus of the present technology includes a parsing unit that parses the identification data from a bitstream including the identification data for identifying whether to perform interpolation processing using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current prediction block for intra prediction using matrix operations, an intra prediction unit that generates a predicted image of the current prediction block using the identification data parsed by the parsing unit, and a decoding unit that decodes the current prediction block using the predicted image generated by the intra prediction unit.

[0011] The second image processing method of the present technology includes an intra prediction step of generating a predicted image of the current prediction block by performing interpolation processing using the original pixels of the reference image as the upper adjacent pixels adjacent to the predicted image of the current prediction block to be decoded when performing intra prediction using matrix operations, and a decoding step of decoding the current prediction block using the predicted image generated in the intra prediction step. Another second image processing method of the present technology includes parsing the identification data from a bitstream including the identification data for identifying whether to perform interpolation processing using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current prediction block for intra prediction using matrix operations, generating a predicted image of the current prediction block using the identification data, and decoding the current prediction block using the predicted image.

[0012] In the second image processing apparatus and image processing method of the present technology, when performing intra prediction using matrix operations, the predicted image of the current prediction block is generated by performing interpolation processing using the original pixels of the reference image as the upper adjacent pixels adjacent to the predicted image of the current prediction block to be decoded. Then, the current prediction block is decoded using the predicted image. In another second image processing apparatus and image processing method of the present technology, identification data for identifying whether to perform interpolation processing using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current prediction block for intra prediction using matrix operations is parsed from a bitstream including the identification data. Then, using the identification data, a predicted image of the current prediction block is generated, and using the predicted image, the current prediction block is decoded.

[0013] Note that the image processing apparatus may be an independent apparatus or an internal block constituting one apparatus.

[0014] Also, the image processing apparatus can be realized by causing a computer to execute a program. The program can be provided by recording it on a recording medium or transmitting it via a transmission medium.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0016] <References>

[0017] The scope disclosed in this specification is not limited to the content of the embodiments. The content of the following reference documents REF1 - REF6, which were known at the time of filing, is also incorporated herein by reference. That is, the content described in the following reference documents REF1 - REF6 is also used as a basis for judging support requirements. For example, even if Quad - Tree Block Structure, QTBT (Quad Tree Plus Binary Tree) Block Structure, and MTT (Multi - type Tree) Block Structure are not directly defined in the detailed description of the invention, they are within the scope of this disclosure and are considered to meet the support requirements of the claims. Also, for example, with respect to technical terms such as Parsing, Syntax, Semantics, etc., even if they are not directly defined in the detailed description of the invention, they are within the scope of this disclosure and are considered to meet the support requirements of the claims.

[0018] REF1: Recommendation ITU - T H.264 (04 / 2017) “Advanced video coding for generic audiovisual services”, April 2017 REF2: Recommendation ITU - T H.265 (02 / 2018) “High efficiency video coding”, February 2018 REF3: Benjamin Bross, Jianle Chen, Shan Liu, Versatile Video Coding (Draft 5), JVET - N1001 - v7 (version 7 - date 2019 - 05 - 29) REF4: Jianle Chen, Yan Ye, Seung Hwan Kim, Algorithm description for Versatile Video Coding and Test Model 5 (VTM 5), JVET - N1002 - v1 REF5: JVET-N0217-v3: CE3: Affine linear weighted intra prediction (CE3-4.1, CE3-4.2) (version 7 - date 2019-01-17) REF6: JVET-M0043-v2: CE3: Affine linear weighted intra prediction (test 1.2.1, test 1.2.2) (version 2 - date 2019-01-09)

[0019] <Definition>

[0020] Adjacent means not only the case where it is adjacent to the current pixel of interest by one pixel (one line), but also the case where it is adjacent by a plurality of pixels (a plurality of lines). Therefore, adjacent pixels include, in addition to the pixel at the position of one pixel directly adjacent to the current pixel, the pixels at the positions of a plurality of pixels continuously adjacent to the current pixel.

[0021] Downsampling means reducing the number of pixels. Therefore, downsampling includes reducing the number of pixels using operations such as averaging and median, and reducing the number of pixels without operations.

[0022] A prediction block means a block (PU (Prediction Unit)) that is a processing unit when performing intra prediction, and includes sub-blocks within the prediction block. When the prediction block, the orthogonal transform block (TU (Transform Unit)) that is a processing unit when performing orthogonal transform, and the coding block (CU (Coding Unit)) that is a processing unit when performing coding are unified into the same block, the prediction block, the orthogonal transform block, and the coding block mean the same block.

[0023] The prediction mode of intra prediction means information comprehensively including various matters related to intra prediction, such as the mode number when performing intra prediction, the block size of the prediction block, the mode number of intra prediction using matrix operations (MIP (Matrix-based Intra Prediction)), the type of matrix used when performing matrix operations, and the type of matrix size used when performing matrix operations.

[0024] In this technology, identification data for identifying a plurality of patterns can be set as the syntax of a bitstream obtained by encoding an image. The bitstream can include identification data for identifying various patterns.

[0025] As the identification data, for example, in the generation of a predicted image of intra prediction, data for identifying whether to use the original pixel (pixel value) of the reference image or the downsampled pixel (pixel value) as the adjacent pixel adjacent to the prediction block can be adopted. Further, as the identification data, for example, data for identifying whether to use the original pixel of the reference image or the downsampled pixel as the upper adjacent pixel adjacent above the predicted image of the prediction block and the left adjacent pixel adjacent to the left, respectively, can be adopted.

[0026] When the identification data is included in the bitstream, a decoder that decodes the bitstream can perform processing more efficiently by parsing and referring to the identification data.

[0027] <Prediction image generation method proposed in JVET-N0217>

[0028] FIG. 1 is a diagram for explaining a method for generating a predicted image of MIP (Matrix-based Intra Prediction) proposed in JVET-N0217 (Reference Document REF5).

[0029] In JVET-N0217, in MIP, it is proposed to generate a predicted image by averaging the pixels (pixel values) of the reference image (the decoded image that has become the reference), performing matrix-vector multiplication using the averaged pixels obtained by averaging, and performing interpolation using the result of the matrix multiplication and the averaged pixels.

[0030] Here, the original pixels of the reference image adjacent above the current prediction block, which is the prediction block to be encoded / decoded, are also referred to as upper original pixels. Also, the original pixels of the reference image adjacent to the left of the current prediction block are also referred to as left original pixels.

[0031] Furthermore, let the horizontal size (width size) of the block be represented by W, and the vertical size (height size) be represented by H.

[0032] In FIG. 1, a block of W×H = 8×8 pixels is adopted as the current prediction block.

[0033] In averaging, for the current prediction block, the upper original pixels (pixel values) bdry of the reference image top are averaged, and a plurality of averaged pixels (pixel values) bdry as downsampled pixels red are generated.

[0034] Furthermore, in averaging, for the current prediction block, the left original pixels bdry of the reference image left are averaged, and a plurality of averaged pixels bdry as downsampled pixels red are generated.

[0035] When the current prediction block is a block of W×H = 8×8 pixels, the averaging of the upper original pixels bdry top and the left original pixels bdry leftIn the averaging, by taking the average of two adjacent original pixels of the reference image, averaging pixels bdry are generated four at a time. red are generated.

[0036] In the matrix operation, according to the prediction mode k of the intra prediction, the matrix A used for the matrix operation k and the offset b k are set. Then, in the matrix operation, the matrix A k and the vector bdry red having the averaging pixels bdry obtained by averaging as elements red are multiplied. Further, in the matrix operation, the offset b k is added to the result of the multiplication. Thereby, some pixels pred red of the predicted image of the current prediction block are generated.

[0037] In the interpolation process, interpolation is performed using the upper adjacent pixel adjacent above the predicted image of the current prediction block, the left adjacent pixel adjacent to the left of the predicted image of the current prediction block, and some pixels pred red of the predicted image of the current prediction block generated by the matrix operation, and the remaining pixels of the predicted image are generated.

[0038] As the upper adjacent pixel of the predicted image, among the averaging pixels bdry red of the reference image, the averaging pixel bdry top generated using the upper original pixel bdry top red is used.

[0039] As the left adjacent pixel of the predicted image, the left original pixel bdry left of the reference image is used.

[0040] Here, the position of the pixel at the x-th from the left and the y-th from the top is represented as (x - 1, y - 1), and the pixel at the position (x - 1, y - 1) is also described as pixel (x - 1, y - 1).

[0041] When the current prediction block is a block of W×H = 8×8 pixels, some pixels pred of the predicted image generated by matrix operation red are the pixels (x - 1, y - 1) (shown hatched in the figure) at the positions where x - 1 and y - 1 are odd among the pixels of the predicted image of the current prediction block.

[0042] In the interpolation process, as four upper adjacent pixels at the positions where x - 1 is odd adjacent to the predicted image of the current prediction block, four averaged pixels bdry top red are arranged. Further, as eight left adjacent pixels adjacent to the left of the predicted image of the current prediction block, eight left original pixels bdry left are arranged.

[0043] And, by vertical (perpendicular) interpolation using the averaged pixel bdry top red as the upper adjacent pixel and the pixel pred generated by matrix operation red which is the pixel (x - 1, y - 1) at the position where x - 1 and y - 1 are odd, pixels at the positions where x - 1 is odd and y - 1 is even in the predicted image are generated.

[0044] Furthermore, by horizontal (horizontal direction) interpolation using the left original pixel bdry left as the left adjacent pixel, the pixel pred generated by matrix operation red and the pixels generated by vertical interpolation, the remaining pixels of the predicted image are generated.

[0045] And, by combining the pixels generated by the interpolation process and the pixels generated by the matrix operation, the predicted image (pred) of the current prediction block is generated.

[0046] In JVET - N0217, in the interpolation process, as the upper adjacent pixels adjacent to the predicted image of the current prediction block, the averaged pixels bdry top generated using the upper original pixels bdry top red of the reference image are used.

[0047] That is, the interpolation process is performed using pixels that are not the actual pixels (original pixels) of the reference image itself.

[0048] Therefore, it is necessary to hold the averaged pixel bdry used when performing the interpolation process after the matrix operation, and a storage area (memory) for that holding is required. top red It is necessary to hold the averaged pixel bdry used when performing the interpolation process after the matrix operation, and a storage area (memory) for that holding is required.

[0049] Furthermore, since the interpolation process is always performed using the averaged pixel bdry top red as the upper adjacent pixel, the correlation between the pixels of the prediction block and the pixels of the prediction image generated using the averaged pixel bdry top red decreases, and the prediction accuracy of the intra prediction, that is, the prediction accuracy of the prediction image cannot be improved, or there is a risk that the prediction accuracy will decrease.

[0050] Therefore, in this technology, in the interpolation process, the upper original pixel bdry of the reference image is used as the upper adjacent pixel top to improve the prediction accuracy of the prediction image.

[0051] <Image processing system to which this technology is applied>

[0052] FIG. 2 is a block diagram showing a configuration example of an embodiment of an image processing system to which this technology is applied.

[0053] The image processing system 10 includes an image processing apparatus as an encoder 11 and an image processing apparatus as a decoder 51.

[0054] The encoder 11 encodes the original image to be encoded supplied thereto and outputs an encoded bit stream obtained by the encoding. The encoded bit stream is supplied to the decoder 51 via a recording medium or a transmission medium (not shown).

[0055] The decoder 51 decodes the encoded bit stream supplied thereto and outputs the decoded image obtained by the decoding.

[0056] <Configuration example of the encoder 11>

[0057] Figure 3 is a block diagram showing a configuration example of the encoder 11 in Figure 2.

[0058] Regarding the block diagrams described below, in order to avoid complication of the figures, the description of the lines supplying the information (data) required for the processing of each block is appropriately omitted.

[0059] In Figure 3, the encoder 11 includes an A / D conversion unit 21, a rearrangement buffer 22, an arithmetic unit 23, an orthogonal conversion unit 24, a quantization unit 25, a reversible encoding unit 26, and an accumulation buffer 27. Further, the encoder 11 includes an inverse quantization unit 28, an inverse orthogonal conversion unit 29, an arithmetic unit 30, a frame memory 32, a selection unit 33, an intra prediction unit 34, a motion prediction compensation unit 35, a predicted image selection unit 36, and a rate control unit 37. Also, the encoder 11 includes a deblocking filter 31a, an adaptive offset filter 41, and an ALF (adaptive loop filter) 42.

[0060] The A / D conversion unit 21 performs A / D conversion on the original image (encoding target) of the analog signal into the original image of the digital signal and supplies it to the rearrangement buffer 22 for storage. When the original image of the digital signal is supplied to the encoder 11, the encoder 11 can be configured without providing the A / D conversion unit 21.

[0061] The rearrangement buffer 22 rearranges the frames of the original image from the display order to the encoding (decoding) order according to the GOP (Group Of Picture) and supplies them to the arithmetic unit 23, the intra prediction unit 34, and the motion prediction compensation unit 35.

[0062] The arithmetic unit 23 subtracts the predicted image supplied from the intra prediction unit 34 or the motion prediction compensation unit 35 via the predicted image selection unit 36 from the original image from the rearrangement buffer 22, and supplies the residual (prediction residual) obtained by the subtraction to the orthogonal transformation unit 24.

[0063] The orthogonal transformation unit 24 performs an orthogonal transformation such as a discrete cosine transformation or a Karhunen - Loeve transformation on the residual supplied from the arithmetic unit 23, and supplies the orthogonal transformation coefficients obtained by the orthogonal transformation to the quantization unit 25.

[0064] The quantization unit 25 quantizes the orthogonal transformation coefficients supplied from the orthogonal transformation unit 24. The quantization unit 25 sets quantization parameters based on the target value of the code amount (code amount target value) supplied from the rate control unit 37, and performs quantization of the orthogonal transformation coefficients. The quantization unit 25 supplies the encoded data, which is the quantized orthogonal transformation coefficients, to the reversible encoding unit 26.

[0065] The reversible encoding unit 26 encodes the quantized orthogonal transformation coefficients as the encoded data from the quantization unit 25 using a predetermined reversible encoding method.

[0066] Also, the reversible encoding unit 26 acquires, from each block, the encoding information necessary for decoding in the decoder 170 among the encoding information regarding predictive encoding in the encoder 11.

[0067] Here, examples of the encoding information include prediction modes such as intra prediction and inter prediction, motion information such as motion vectors, code amount target values, quantization parameters, picture types (I, P, B), filter parameters of the deblocking filter 31a and the adaptive offset filter 41, and the like.

[0068] The prediction mode can be acquired from the intra prediction unit 34 or the motion prediction compensation unit 35. The motion information can be acquired from the motion prediction compensation unit 35. The filter parameters of the deblocking filter 31a and the adaptive offset filter 41 can be acquired from the deblocking filter 31a and the adaptive offset filter 41, respectively.

[0069] The reversible encoding unit 26 encodes the encoded information using variable length encoding such as CAVLC (Context-Adaptive Variable Length Coding) or CABAC (Context-Adaptive Binary Arithmetic Coding), or other reversible encoding methods, and generates an encoded bit stream (multiplexed) including the encoded information after encoding and the encoded data from the quantization unit 25, and supplies it to the accumulation buffer 27.

[0070] Here, the above operation units 23 to the reversible encoding unit 26 function as an encoding unit for encoding an image.

[0071] The accumulation buffer 27 temporarily accumulates the encoded bit stream supplied from the reversible encoding unit 26. The encoded bit stream accumulated in the accumulation buffer 27 is read out and transmitted at a predetermined timing.

[0072] The encoded data, which is the orthogonal transform coefficient quantized in the quantization unit 25, is supplied not only to the reversible encoding unit 26 but also to the inverse quantization unit 28. The inverse quantization unit 28 inverse-quantizes the quantized orthogonal transform coefficient by a method corresponding to the quantization by the quantization unit 25, and supplies the orthogonal transform coefficient obtained by the inverse quantization to the inverse orthogonal transform unit 29.

[0073] The inverse orthogonal transform unit 29 inverse-orthogonally transforms the orthogonal transform coefficient supplied from the inverse quantization unit 28 by a method corresponding to the orthogonal transform process by the orthogonal transform unit 24, and supplies the residual obtained as a result of the inverse orthogonal transform to the operation unit 30.

[0074] The operation unit 30 adds the prediction image supplied from the intra prediction unit 34 or the motion prediction compensation unit 35 via the prediction image selection unit 36 to the residual supplied from the inverse orthogonal transform unit 29, thereby obtaining (a part of) the decoded image obtained by decoding the original image and outputting it.

[0075] The decoded image output by the arithmetic unit 30 is supplied to the deblocking filter 31a or the frame memory 32.

[0076] The frame memory 32 temporarily stores the decoded image supplied from the arithmetic unit 30 and the decoded image (filtered image) to which the deblocking filter 31a, the adaptive offset filter 41, and the ALF 42 are applied and which is supplied from the ALF 42. The decoded image stored in the frame memory 32 is supplied to the selection unit 33 as a reference image used for generating a predicted image at a necessary timing.

[0077] The selection unit 33 selects the supply destination of the reference image supplied from the frame memory 32. When intra prediction is performed in the intra prediction unit 34, the selection unit 33 supplies the reference image supplied from the frame memory 32 to the intra prediction unit 34. When inter prediction is performed in the motion prediction compensation unit 35, the selection unit 33 supplies the reference image supplied from the frame memory 32 to the motion prediction compensation unit 35.

[0078] The intra prediction unit 34 performs intra prediction (intra-frame prediction) using the original image supplied from the rearrangement buffer 22 and the reference image supplied from the frame memory 32 via the selection unit 33. The intra prediction unit 34 selects an optimal intra prediction mode based on a predetermined cost function, and supplies the predicted image generated from the reference image in the optimal intra prediction mode to the predicted image selection unit 36. Further, the intra prediction unit 34 appropriately supplies the intra prediction mode selected based on the cost function to the reversible encoding unit 26 and the like.

[0079] The motion prediction compensation unit 35 performs motion prediction using the original image supplied from the rearrangement buffer 22 and the reference image supplied from the frame memory 32 via the selection unit 33. Further, the motion prediction compensation unit 35 performs motion compensation according to the motion vector detected by the motion prediction, and generates a predicted image. The motion prediction compensation unit 35 performs inter prediction in a plurality of prepared inter prediction modes, and generates a predicted image from the reference image.

[0080] The motion prediction compensation unit 35 selects the optimal prediction mode of the inter prediction from among the prediction modes of a plurality of inter predictions based on a predetermined cost function. Further, the motion prediction compensation unit 35 supplies the prediction image generated in the optimal prediction mode of the inter prediction to the prediction image selection unit 36.

[0081] Also, the motion prediction compensation unit 35 supplies motion information such as the optimal prediction mode of the inter prediction selected based on the cost function and the motion vector necessary for decoding the encoded data encoded in the prediction mode of the inter prediction to the reversible encoding unit 26.

[0082] The prediction image selection unit 36 selects the source of the prediction image to be supplied to the arithmetic unit 23 and the arithmetic unit 30 from among the intra prediction unit 34 and the motion prediction compensation unit 35, and supplies the prediction image supplied from the selected source to the arithmetic unit 23 and the arithmetic unit 30.

[0083] The rate control unit 37 controls the rate of the quantization operation of the quantization unit 25 so that overflow or underflow does not occur based on the amount of codes of the encoded bit stream accumulated in the accumulation buffer 27. That is, the rate control unit 37 sets the target amount of codes of the encoded bit stream so that overflow and underflow of the accumulation buffer 27 do not occur, and supplies it to the quantization unit 25.

[0084] The deblocking filter 31a applies a deblocking filter to the decoded image from the arithmetic unit 30 as necessary, and supplies the decoded image (filtered image) to which the deblocking filter has been applied, or the decoded image to which the deblocking filter has not been applied, to the adaptive offset filter 41.

[0085] The adaptive offset filter 41 applies an adaptive offset filter to the decoded image from the deblocking filter 31a as necessary, and supplies the decoded image (filtered image) to which the adaptive offset filter has been applied, or the decoded image to which the adaptive offset filter has not been applied, to the ALF 42.

[0086] ALF42 applies the ALF to the decoded image from the adaptive offset filter 41 as necessary, and supplies the decoded image to which the ALF is applied, or the decoded image to which the ALF is not applied, to the frame memory 32.

[0087] <Encoding process>

[0088] Figure 4 is a flowchart for explaining an example of the encoding process of the encoder 11 in Figure 3.

[0089] Note that the order of each step of the encoding process shown in Figure 4 is for convenience of explanation, and each step of the actual encoding process is performed in parallel and in the necessary order as appropriate. The same applies to the processes described later.

[0090] In the encoder 11, in step S11, the A / D conversion unit 21 A / D-converts the original image and supplies it to the rearrangement buffer 22, and the process proceeds to step S12.

[0091] In step S12, the rearrangement buffer 22 stores the original image from the A / D conversion unit 21, rearranges it in the encoding order, and outputs it, and the process proceeds to step S13.

[0092] In step S13, the intra prediction unit 34 performs intra prediction, and the process proceeds to step S14. In step S14, the motion prediction compensation unit 35 performs inter prediction for motion prediction and motion compensation, and the process proceeds to step S15.

[0093] In the intra prediction by the intra prediction unit 34 and the inter prediction by the motion prediction compensation unit 35, cost functions of various prediction modes are calculated and a predicted image is generated.

[0094] In step S15, the prediction image selection unit 36 determines an optimal prediction mode based on each cost function obtained by the intra prediction unit 34 and the motion prediction compensation unit 35. Then, the prediction image selection unit 36 selects and outputs the prediction image of the optimal prediction mode from among the prediction image generated by the intra prediction unit 34 and the prediction image generated by the motion prediction compensation unit 35, and the process proceeds from step S15 to step S16.

[0095] In step S16, the arithmetic unit 23 calculates the residual between the target image to be encoded, which is the source image output by the rearrangement buffer 22, and the prediction image output by the prediction image selection unit 36, and supplies it to the orthogonal transformation unit 24, and the process proceeds to step S17.

[0096] In step S17, the orthogonal transformation unit 24 orthogonally transforms the residual from the arithmetic unit 23, and supplies the resulting orthogonal transformation coefficients to the quantization unit 25, and the process proceeds to step S18.

[0097] In step S18, the quantization unit 25 quantizes the orthogonal transformation coefficients from the orthogonal transformation unit 24, and supplies the quantization coefficients obtained by the quantization to the reversible encoding unit 26 and the inverse quantization unit 28, and the process proceeds to step S19.

[0098] In step S19, the inverse quantization unit 28 inverse quantizes the quantization coefficients from the quantization unit 25, and supplies the resulting orthogonal transformation coefficients to the inverse orthogonal transformation unit 29, and the process proceeds to step S20. In step S20, the inverse orthogonal transformation unit 29 inverse orthogonally transforms the orthogonal transformation coefficients from the inverse quantization unit 28, and supplies the resulting residual to the arithmetic unit 30, and the process proceeds to step S21.

[0099] In step S21, the arithmetic unit 30 adds the residual from the inverse orthogonal transformation unit 29 and the prediction image output by the prediction image selection unit 36, and generates a decoded image corresponding to the source image that was the target of the residual calculation in the arithmetic unit 23. The arithmetic unit 30 supplies the decoded image to the deblocking filter 31a, and the process proceeds from step S21 to step S22.

[0100] In step S22, the deblocking filter 31a applies a deblocking filter to the decoded image from the arithmetic unit 30, supplies the resulting filtered image to the adaptive offset filter 41, and the process proceeds to step S23.

[0101] In step S23, the adaptive offset filter 41 applies an adaptive offset filter to the filtered image from the deblocking filter 31a, supplies the resulting filtered image to the ALF 42, and the process proceeds to step S24.

[0102] In step S24, the ALF 42 applies an ALF to the filtered image from the adaptive offset filter 41, supplies the resulting filtered image to the frame memory 32, and the process proceeds to step S25.

[0103] In step S25, the frame memory 32 stores the filtered image supplied from the ALF 42, and the process proceeds to step S26. The filtered image stored in the frame memory 32 is used as a reference image for generating a prediction image in steps S13 and S14.

[0104] In step S26, the reversible encoding unit 26 encodes the encoded data, which is the quantized coefficient from the quantization unit 25, and generates an encoded bitstream including the encoded data. Further, the reversible encoding unit 26 encodes encoding information such as the quantization parameter used for quantization in the quantization unit 25, the prediction mode obtained by intra prediction in the intra prediction unit 34, the prediction mode and motion information obtained by inter prediction in the motion prediction compensation unit 35, and the filter parameters of the deblocking filter 31a and the adaptive offset filter 41 as necessary, and includes them in the encoded bitstream.

[0105] Then, the reversible encoding unit 26 supplies the encoded bitstream to the accumulation buffer 27, and the process proceeds from step S26 to step S27.

[0106] In step S27, the accumulation buffer 27 accumulates the encoded bit stream from the reversible encoding unit 26, and the process proceeds to step S28. The encoded bit stream accumulated in the accumulation buffer 27 is read out as appropriate and transmitted.

[0107] In step S28, the rate control unit 37 controls the rate of the quantization operation of the quantization unit 25 based on the amount of codes (generated code amount) of the encoded bit stream accumulated in the accumulation buffer 27 so that overflow or underflow does not occur, and the encoding process ends.

[0108] <Configuration example of decoder 51>

[0109] FIG. 5 is a block diagram showing a detailed configuration example of the decoder 51 in FIG. 2.

[0110] In FIG. 5, the decoder 51 includes an accumulation buffer 61, a reversible decoding unit 62, an inverse quantization unit 63, an inverse orthogonal transformation unit 64, an arithmetic unit 65, a rearrangement buffer 67, and a D / A conversion unit 68. Further, the decoder 51 includes a frame memory 69, a selection unit 70, an intra prediction unit 71, a motion prediction compensation unit 72, and a selection unit 73. Also, the decoder 51 includes a deblocking filter 31b, an adaptive offset filter 81, and an ALF 82.

[0111] The accumulation buffer 61 temporarily accumulates the encoded bit stream transmitted from the encoder 11 and supplies the encoded bit stream to the reversible decoding unit 62 at a predetermined timing.

[0112] The reversible decoding unit 62 receives the encoded bit stream from the accumulation buffer 61 and decodes it in a manner corresponding to the encoding method of the reversible encoding unit 26 in FIG. 3.

[0113] Then, the reversible decoding unit 62 supplies the quantization coefficients as encoded data included in the decoding result of the encoded bit stream to the inverse quantization unit 63.

[0114] In addition, the reversible decoding unit 62 has a function of performing parsing. The reversible decoding unit 62 parses the necessary encoded information included in the decoding result of the encoded bit stream, and supplies the encoded information to the intra prediction unit 71, the motion prediction compensation unit 72, the deblocking filter 31b, the adaptive offset filter 81, and other necessary blocks.

[0115] The inverse quantization unit 63 inverse-quantizes the quantization coefficients as the encoded data from the reversible decoding unit 62 in a method corresponding to the quantization method of the quantization unit 25 in FIG. 3, and supplies the orthogonal transform coefficients obtained by the inverse quantization to the inverse orthogonal transform unit 64.

[0116] The inverse orthogonal transform unit 64 inverse-orthogonally transforms the orthogonal transform coefficients supplied from the inverse quantization unit 63 in a method corresponding to the orthogonal transform method of the orthogonal transform unit 24 in FIG. 3, and supplies the resulting residual to the arithmetic unit 65.

[0117] In addition to the residual being supplied from the inverse orthogonal transform unit 64 to the arithmetic unit 65, a predicted image is supplied to the arithmetic unit 65 from the intra prediction unit 71 or the motion prediction compensation unit 72 via the selection unit 73.

[0118] The arithmetic unit 65 adds the residual from the inverse orthogonal transform unit 64 and the predicted image from the selection unit 73 to generate a decoded image, and supplies the decoded image to the deblocking filter 31b.

[0119] Here, the above-described reversible decoding unit 62 to arithmetic unit 65 constitute a decoding unit that decodes an image.

[0120] The rearrangement buffer 67 temporarily stores the decoded image supplied from the ALF82, rearranges the order of the frames (pictures) of the decoded image from the encoding (decoding) order to the display order, and supplies the rearranged image to the D / A conversion unit 68.

[0121] The D / A conversion unit 68 performs D / A conversion on the decoded image supplied from the rearrangement buffer 67, and outputs the converted image to a display (not shown) for display. When the device connected to the decoder 51 receives a digital signal image, the decoder 51 can be configured without providing the D / A conversion unit 68.

[0122] The frame memory 69 temporarily stores the decoded image supplied from the ALF 82. Further, the frame memory 69 supplies the decoded image to the selection unit 70 as a reference image used for generating a predicted image at a predetermined timing or based on an external request such as the intra prediction unit 71 or the motion prediction compensation unit 72.

[0123] The selection unit 70 selects the destination of the reference image supplied from the frame memory 69. When decoding an image encoded by intra prediction, the selection unit 70 supplies the reference image supplied from the frame memory 69 to the intra prediction unit 71. Also, when decoding an image encoded by inter prediction, the selection unit 70 supplies the reference image supplied from the frame memory 69 to the motion prediction compensation unit 72.

[0124] The intra prediction unit 71 performs intra prediction using the reference image supplied from the frame memory 69 via the selection unit 70 in the same manner as the intra prediction unit 34 in FIG. 3 according to the prediction mode included in the encoded information supplied from the reversible decoding unit 62. Then, the intra prediction unit 71 supplies the predicted image obtained by the intra prediction to the selection unit 73.

[0125] The motion prediction compensation unit 72 performs inter prediction using the reference image supplied from the frame memory 69 via the selection unit 70 in the same manner as the motion prediction compensation unit 35 in FIG. 3 according to the prediction mode included in the encoded information supplied from the reversible decoding unit 62. The inter prediction is performed using the motion information etc. included in the encoded information supplied from the reversible decoding unit 62 as necessary.

[0126] The motion prediction compensation unit 72 supplies the predicted image obtained by the inter prediction to the selection unit 73.

[0127] The selection unit 73 selects the predicted image supplied from the intra prediction unit 71 or the predicted image supplied from the motion prediction compensation unit 72 and supplies it to the arithmetic unit 65.

[0128] The deblocking filter 31b applies a deblocking filter to the decoded image from the arithmetic unit 65 according to the filter parameters included in the encoded information supplied from the reversible decoding unit 62. The deblocking filter 31b supplies the decoded image (filtered image) to which the deblocking filter has been applied, or the decoded image to which the deblocking filter has not been applied, to the adaptive offset filter 81.

[0129] The adaptive offset filter 81 applies an adaptive offset filter to the decoded image from the deblocking filter 31b as necessary according to the filter parameters included in the encoded information supplied from the reversible decoding unit 62. The adaptive offset filter 81 supplies the decoded image (filtered image) to which the adaptive offset filter has been applied, or the decoded image to which the adaptive offset filter has not been applied, to the ALF 82.

[0130] The ALF 82 applies the ALF to the decoded image from the adaptive offset filter 81 as necessary, and supplies the decoded image to which the ALF has been applied, or the decoded image to which the ALF has not been applied, to the rearrangement buffer 67 and the frame memory 69.

[0131] <Decoding process>

[0132] FIG. 6 is a flowchart for explaining an example of the decoding process of the decoder 51 in FIG. 5.

[0133] In the decoding process, in step S51, the accumulation buffer 61 temporarily accumulates the encoded bit stream transmitted from the encoder 11, and supplies it to the reversible decoding unit 62 as appropriate, and the process proceeds to step S52.

[0134] In step S52, the reversible decoding unit 62 receives and decodes the encoded bit stream supplied from the accumulation buffer 61, and supplies the quantization coefficient as encoded data included in the decoding result of the encoded bit stream to the inverse quantization unit 63.

[0135] Further, the reversible decoding unit 62 parses the encoded information included in the decoding result of the encoded bit stream. Then, the reversible decoding unit 62 supplies the necessary encoded information to the intra prediction unit 71, the motion prediction compensation unit 72, the deblocking filter 31b, the adaptive offset filter 81, and other necessary blocks.

[0136] Then, the process proceeds from step S52 to step S53, and the intra prediction unit 71 or the motion prediction compensation unit 72 performs intra prediction or inter prediction to generate a predicted image according to the reference image supplied from the frame memory 69 via the selection unit 70 and the encoded information supplied from the reversible decoding unit 62. Then, the intra prediction unit 71 or the motion prediction compensation unit 72 supplies the predicted image obtained by the intra prediction or the inter prediction to the selection unit 73, and the process proceeds from step S53 to step S54.

[0137] In step S54, the selection unit 73 selects the predicted image supplied from the intra prediction unit 71 or the motion prediction compensation unit 72 and supplies it to the arithmetic unit 65, and the process proceeds to step S55.

[0138] In step S55, the inverse quantization unit 63 inverse quantizes the quantization coefficients from the reversible decoding unit 62 and supplies the resulting orthogonal transformation coefficients to the inverse orthogonal transformation unit 64, and the process proceeds to step S56.

[0139] In step S56, the inverse orthogonal transformation unit 64 inverse orthogonally transforms the orthogonal transformation coefficients from the inverse quantization unit 63 and supplies the resulting residual to the arithmetic unit 65, and the process proceeds to step S57.

[0140] In step S57, the arithmetic unit 65 generates a decoded image by adding the residual from the inverse orthogonal transformation unit 64 and the predicted image from the selection unit 73. Then, the arithmetic unit 65 supplies the decoded image to the deblocking filter 31b, and the process proceeds from step S57 to step S58.

[0141] In step S58, the deblocking filter 31b applies a deblocking filter to the decoded image from the arithmetic unit 65 in accordance with the filter parameters included in the encoded information supplied from the inverse decoder 62. The deblocking filter 31b supplies the filter image obtained as a result of the application of the deblocking filter to the adaptive offset filter 81, and the process proceeds from step S58 to step S59.

[0142] In step S59, the adaptive offset filter 81 applies an adaptive offset filter to the filter image from the deblocking filter 31b in accordance with the filter parameters included in the encoded information supplied from the inverse decoder 62. The adaptive offset filter 81 supplies the filter image obtained as a result of the application of the adaptive offset filter to the ALF82, and the process proceeds from step S59 to step S60.

[0143] The ALF82 applies the ALF to the filter image from the adaptive offset filter 81, and supplies the resulting filter image to the rearrangement buffer 67 and the frame memory 69, and the process proceeds to step S61.

[0144] In step S61, the frame memory 69 temporarily stores the filter image supplied from the ALF82, and the process proceeds to step S62. The filter image (decoded image) stored in the frame memory 69 is used as a reference image for generating a predicted image in the intra prediction or inter prediction in step S53.

[0145] In step S62, the rearrangement buffer 67 rearranges the filter images supplied from the ALF82 in the display order, and supplies them to the D / A conversion unit 68, and the process proceeds to step S63.

[0146] In step S63, the D / A conversion unit 68 performs D / A conversion on the filter image from the rearrangement buffer 67, and the decoding process ends. The D / A converted filter image (decoded image) is output to a display (not shown) and displayed.

[0147] <Configuration Example of Intra Prediction Unit 34>

[0148] FIG. 7 is a block diagram showing a configuration example of the intra prediction unit 34.

[0149] Note that in FIG. 7, only the prediction image generation unit 110, which is a part of the intra prediction unit 34 that generates the prediction image of MIP, is shown. The intra prediction unit 71 also has a prediction image generation unit similar to the prediction image generation unit 110.

[0150] The prediction image generation unit 110 includes an averaging unit 111, a matrix-vector multiplication unit 112, and an interpolation unit 113.

[0151] The averaging unit 111 is supplied with a reference image (decoded image as a reference image) from the selection unit 33 for the current prediction block.

[0152] The averaging unit 111 averages the upper original pixels (pixel values) of the reference image for the current prediction block and generates a plurality of averaged pixels (pixel values) as downsampled pixels.

[0153] Also, the averaging unit 111 averages the left original pixels of the reference image for the current prediction block and generates a plurality of averaged pixels as downsampled pixels.

[0154] The averaging unit 111 supplies the averaged pixels to the matrix-vector multiplication unit 112.

[0155] The matrix-vector multiplication unit 112 is supplied with the prediction mode k of the intra prediction.

[0156] The matrix-vector multiplication unit 112 sets a matrix A k and an offset b which is a vector k for use in matrix operations according to the prediction mode k. Then, the matrix-vector multiplication unit 112 performs, as a matrix operation, the matrix A kIt performs multiplication with a vector having the averaged pixels of the averaging unit 111 as elements. Further, the matrix-vector multiplication unit 112, as a matrix operation, adds the offset b to the result of the multiplication, thereby generating some pixels of the predicted image of the current prediction block and supplying them to the interpolation unit 113. k Thereby, some pixels of the predicted image of the current prediction block are generated and supplied to the interpolation unit 113.

[0157] The interpolation unit 113 is supplied with a reference image.

[0158] The interpolation unit 113 performs interpolation processing using the upper adjacent pixels adjacent above the predicted image of the current prediction block, the left adjacent pixels adjacent to the left of the predicted image of the current prediction block, and some pixels of the predicted image of the current prediction block from the matrix-vector multiplication unit 112.

[0159] The interpolation unit 113 generates the remaining pixels of the predicted image of the current prediction block by the interpolation process, and together with some pixels of the predicted image from the matrix-vector multiplication unit 112, generates (completes) the predicted image of the current prediction block.

[0160] The interpolation unit 113 uses the upper original pixel of the reference image as the upper adjacent pixel of the predicted image, and uses the left original pixel of the reference image as the left adjacent pixel of the predicted image.

[0161] FIG. 8 is a flowchart for explaining an example of the process of generating the predicted image of MIP performed by the predicted image generation unit 110.

[0162] In step S111, the averaging unit 111 averages the upper original pixel of the reference image and the left original pixel of the reference image to generate averaged pixels. The averaging unit 111 supplies the averaged pixels to the matrix-vector multiplication unit 112, and the process proceeds from step S111 to step S112.

[0163] In step S112, the matrix-vector multiplication unit 112, according to the prediction mode k, uses the matrix A used for the matrix operation k and the offset b kSet it. Further, the matrix-vector multiplication unit 112 uses the matrix A k and the offset b k to perform a matrix operation on the vector having the averaged pixels from the averaging unit 111 as elements.

[0164] That is, the matrix-vector multiplication unit 112 performs, as a matrix operation, the multiplication of the matrix A k and the vector having the averaged pixels of the averaging unit 111 as elements. Further, the matrix-vector multiplication unit 112 performs, as a matrix operation, the addition of the offset b k to the result of the multiplication. The matrix-vector multiplication unit 112 generates some pixels of the predicted image of the current prediction block by the above matrix operation, supplies them to the interpolation unit 113, and the process proceeds from step S112 to step S113.

[0165] In step S113, the interpolation unit 113 uses the upper original pixel of the reference image as the upper adjacent pixel and the left original pixel of the reference image as the left adjacent pixel, and performs interpolation processing using these upper and left adjacent pixels and some pixels of the predicted image of the current prediction block from the matrix-vector multiplication unit 112.

[0166] The interpolation unit 113 generates the remaining pixels of the predicted image of the current prediction block by the interpolation process, and generates the predicted image of the current prediction block in combination with some pixels of the predicted image from the matrix-vector multiplication unit 112.

[0167] FIG. 9 is a diagram for explaining a method of generating the predicted image of the MIP of the predicted image generation unit 110.

[0168] In FIG. 9, similar to FIG. 1, a block of W×H = 8×8 pixels is adopted as the current prediction block. However, the prediction block is not limited to a block of W×H = 8×8 pixels.

[0169] The averaging unit 111, similar to JVET-N0217, for the current prediction block, the upper original pixel bdry of the reference image topAverage them to generate a plurality of averaged pixels bdry as downsampled pixels red is generated.

[0170] Furthermore, similar to JVET-N0217, the averaging unit 111 averages the left original pixels bdry of the reference image for the current prediction block left Average them to generate a plurality of averaged pixels bdry as downsampled pixels red is generated.

[0171] When the current prediction block is a block of W×H = 8×8 pixels, the averaging of the upper original pixels bdry top is performed by taking the average of two upper original pixels bdry adjacent in the horizontal direction of the reference image top Thereby, four averaged pixels bdry red are generated. Similarly, the averaging of the left original pixels bdry left is performed by taking the average of two left original pixels bdry adjacent in the vertical direction of the reference image left Thereby, four averaged pixels bdry red are generated.

[0172] Here, in the present embodiment, as the downsampled pixels, averaged pixels obtained by taking the average of the original pixels are adopted, but the downsampled pixels are not limited to the averaged pixels. That is, the downsampling can be performed by taking the average of a plurality of pixels, performing an operation other than the average such as median, or simply thinning out the pixels.

[0173] The matrix-vector multiplication unit 112, similar to JVET-N0217, uses a matrix A k and an offset b k to perform an expression pred red as a matrix operation on the vector bdry red having the averaged pixels bdry red =A k ·bdry red +bk Perform the operation.

[0174] That is, the matrix-vector multiplication unit 112 performs, as a matrix operation, the multiplication of the matrix A k and the vector bdry having the averaged pixel bdry red as elements. red That is, the matrix-vector multiplication unit 112 performs the multiplication A k ·bdry red Furthermore, the matrix-vector multiplication unit 112 adds the offset b k ·bdry red to the result of the multiplication A k ·bdry. Thereby, the matrix-vector multiplication unit 112 generates some pixels pred red of the predicted image of the current prediction block.

[0175] Similar to JVET-N0217, the interpolation unit 113 performs interpolation processing using the upper adjacent pixel adjacent to the predicted image of the current prediction block, the left adjacent pixel adjacent to the left of the predicted image of the current prediction block, and some pixels pred red of the predicted image of the current prediction block generated by matrix operation.

[0176] The interpolation unit 113 generates the remaining pixels (pixels shown as white in the figure) of the predicted image by the interpolation process, and combines them with the pixels pred red generated by the matrix operation to generate the predicted image (pred) of the current prediction block.

[0177] However, in the interpolation process of JVET-N0217, as described with reference to FIG. 1, as the upper adjacent pixel of the predicted image, the averaged pixel bdry red among the averaged pixels bdry top generated using the upper original pixel bdry top red is used.

[0178] In contrast, in the interpolation process of the interpolation unit 113, as the upper adjacent pixel of the predicted image, instead of the averaged pixel bdry top red the upper original pixel bdry topIt itself is used.

[0179] As described above, in the predicted image generation unit 110, as the upper adjacent pixel, the upper original pixel bdry of the reference image is used. top Therefore, since the upper original pixel bdry of the reference image is used as the upper adjacent pixel, it is not necessary to hold the averaged pixel (pixel value) bdry that is required when using the averaged pixel as the upper adjacent pixel. top red Furthermore, when using the upper original pixel bdry of the reference image as the upper adjacent pixel, it can be expected that the prediction accuracy of the intra prediction will be improved as compared with the case of using the averaged pixel bdry. top When using top red the averaged pixel bdry

[0180] <Another configuration example of the intra prediction unit 34>

[0181] FIG. 10 is a block diagram showing another configuration example of the intra prediction unit 34.

[0182] Note that in FIG. 10, similar to FIG. 7, only the prediction image generation unit 120, which is the part that generates the prediction image of the MIP in the intra prediction unit 34, is shown. The intra prediction unit 71 also has a prediction image generation unit similar to the prediction image generation unit 120.

[0183] Also, in the figure, the parts corresponding to the prediction image generation unit 110 in FIG. 7 are given the same reference numerals, and the description thereof will be omitted as appropriate below.

[0184] The prediction image generation unit 120 includes an averaging unit 111, a matrix-vector multiplication unit 112, and an interpolation unit 123.

[0185] Therefore, the prediction image generation unit 120 has the averaging unit 111 and the matrix-vector multiplication unit 112 in common with the prediction image generation unit 110 in FIG. 7. However, the prediction image generation unit 120 is different from the prediction image generation unit 110 in that it has an interpolation unit 123 instead of the interpolation unit 113.

[0186] In the prediction image generation unit 120, the interpolation unit 123 is supplied with a reference image and a part of the pixels of the prediction image of the current prediction block generated by the matrix-vector multiplication unit 112, and is also supplied with a prediction mode k and the averaged pixels generated by the averaging unit 111.

[0187] Similar to the interpolation unit 113, the interpolation unit 123 performs interpolation processing using the upper adjacent pixel and the left adjacent pixel of the prediction image, and a part of the pixels of the prediction image of the current prediction block from the matrix-vector multiplication unit 112, and combines the remaining pixels of the prediction image generated by the interpolation processing with a part of the pixels of the prediction image from the matrix-vector multiplication unit 112 to generate a prediction image.

[0188] However, the interpolation unit 123 can select whether to use the upper original pixel of the reference image or the averaged pixel as the downsampled pixel as the upper adjacent pixel according to the prediction mode k. Also, the interpolation unit 123 can select whether to use the left original pixel of the reference image or the averaged pixel as the downsampled pixel as the left adjacent pixel according to the prediction mode k.

[0189] FIG. 11 is a diagram for explaining a method of generating a prediction image of the MIP of the prediction image generation unit 120.

[0190] In FIG. 11, similar to FIG. 9, a block of W×H = 8×8 pixels is adopted as the current prediction block. However, the prediction block is not limited to a block of W×H = 8×8 pixels.

[0191] In the prediction image generation unit 120, in the averaging unit 111, similar to the prediction image generation unit 110, for the current prediction block, the averaged pixel bdry obtained by averaging the upper original pixel bdry of the reference image top and the averaged pixel bdry obtained by averaging the left original pixel bdry red are generated. left and the averaged pixel bdry obtained by averaging the left original pixel bdry red are generated.

[0192] Furthermore, in the predicted image generation unit 120, in the matrix-vector multiplication unit 112, similar to the predicted image generation unit 110, the averaged pixel bdry red vector bdry red serves as the target for the matrix operation formula pred red =A k ·bdry red +b k is calculated, and a part of the pixels pred red of the predicted image of the current prediction block is generated.

[0193] Then, in the predicted image generation unit 120, in the interpolation unit 123, similar to the predicted image generation unit 110, the upper adjacent pixel adjacent above the predicted image of the current prediction block, the left adjacent pixel adjacent to the left of the predicted image of the current prediction block, and the part of the pixels pred red of the predicted image of the current prediction block generated by the matrix operation are used for interpolation processing, thereby generating the predicted image (pred) of the current prediction block.

[0194] However, in the interpolation unit 123, depending on the prediction mode k, either the upper original pixel bdry top of the reference image is used as the upper adjacent pixel, or the averaged pixel bdry top red as the downsampled pixel can be selected. Also, in the interpolation unit 123, depending on the prediction mode k, either the left original pixel bdry left of the reference image is used as the left adjacent pixel, or the averaged pixel bdry left red as the downsampled pixel can be selected.

[0195] The averaged pixel bdry top red is the averaged pixel generated by using the upper original pixel bdry red among the averaged pixels bdry top by taking the average of the upper original pixels bdry top . The averaged pixel bdry left redis the averaged pixel generated using the left original pixel bdry left is.

[0196] The interpolation unit 123 has selectors 131 and 132.

[0197] The selector 131 is supplied with the upper original pixel bdry top and the averaged pixel bdry top red . The selector 131 selects and outputs either the upper original pixel bdry top or the averaged pixel bdry top red according to the prediction mode k. In the interpolation unit 123, among the upper original pixel bdry top and the averaged pixel bdry top red , the pixel output by the selector 131 is used as the upper adjacent pixel for interpolation processing.

[0198] The selector 132 is supplied with the left original pixel bdry left and the averaged pixel bdry left red . The selector 132 selects and outputs either the left original pixel bdry left or the averaged pixel bdry left red according to the prediction mode k. In the interpolation unit 123, among the left original pixel bdry left and the averaged pixel bdry left red , the pixel output by the selector 132 is used as the left adjacent pixel for interpolation processing.

[0199] As described above, in the prediction image generation unit 120, when it is possible to select whether to use the upper original pixel of the reference image or the averaged pixel as the upper adjacent pixel, and / or whether to use the left original pixel of the reference image or the averaged pixel as the left adjacent pixel, it can be expected that the prediction accuracy of the intra prediction will be further improved.

[0200] FIG. 12 is a diagram showing examples of pixels selected as the upper adjacent pixel and the left adjacent pixel according to prediction mode k.

[0201] Here, the upper adjacent pixel, the left adjacent pixel, or both are also referred to as adjacent pixels. Also, the upper original pixel, the left original pixel, or both are also referred to as original pixels.

[0202] In FIG. 12, when prediction mode k includes information representing the MIP mode (mode number) and the size identifier MipSizeId, the pixels selected as adjacent pixels (upper adjacent pixel, left adjacent pixel) according to the MIP mode and the size identifier MipSizeId are shown.

[0203] In FIG. 12, "Original" represents the original pixel (upper original pixel, left original pixel), and "Averaged" represents the averaged pixel. According to FIG. 12, for example, when the MIP mode is 1 and the size identifier MipSizeId is 0, the left original pixel (Original) is selected as the left adjacent pixel, and the averaged pixel (Averaged) is selected as the upper adjacent pixel.

[0204] The size identifier MipSizeId is an identifier representing the size of matrix A k used in MIP, and is set according to the block size of the current prediction block. Therefore, when prediction mode k includes information representing the size identifier MipSizeId, it can be said that prediction mode k includes information representing the block size of the current prediction block.

[0205] Prediction mode k can include information representing the prediction direction of intra prediction, directional prediction, non-directional prediction (e.g., player prediction, DC prediction). The prediction direction of intra prediction includes the reference direction of directional prediction.

[0206] When the prediction mode k includes information representing directional prediction, as adjacent pixels, an original pixel or an averaged pixel can be selected. For example, when the prediction mode k includes information representing directional prediction, according to the reference direction of the directional prediction, as adjacent pixels, an original pixel or an averaged pixel can be selected.

[0207] Specifically, for example, when there is a pattern such as a vertical edge in the current prediction block and the reference direction of the directional prediction is close to the vertical direction, the upper original pixel can be selected as the upper adjacent pixel. Also, for example, when there is a pattern such as a horizontal edge in the current prediction block and the reference direction of the directional prediction is close to the horizontal direction, the left original pixel can be selected as the left adjacent pixel. In this case, the prediction accuracy can be improved.

[0208] When the prediction mode k includes information representing non-directional prediction, for example, when DC prediction is performed as intra prediction, the averaged pixel can be selected as the upper adjacent pixel and the left adjacent pixel.

[0209] In addition, the selection of the pixel to be the adjacent pixel adjacent to the predicted image is performed according to the prediction mode k, and in the encoder 11, according to whether or not an in-loop filter is applied to the decoded image (local decoded image) serving as the reference image, that is, whether or not a part or all of the deblocking filter 31a, the adaptive offset filter 41, and the ALF 42 are applied. For example, through simulation, according to whether or not the in-loop filter is applied, the selection of the pixel to be the adjacent pixel can be performed so that the cost is reduced.

[0210] The intra prediction unit 34 can function as a setting unit that sets identification data for identifying whether to use the original pixel of the reference image or the averaged pixel as the downsampled pixel as the adjacent pixel adjacent to the predicted image in the generation of the predicted image of the intra prediction. The identification data set by the intra prediction unit 34 can be included in the encoded bitstream, for example, as part of the encoded information.

[0211] The identification data can include data for identifying whether to use the upper original pixel of the reference image or the averaged pixel as the downsampled pixel as the upper adjacent pixel. Further, the identification data can include data for identifying whether to use the left original pixel of the reference image or the averaged pixel as the downsampled pixel as the left adjacent pixel.

[0212] Note that whether to use the original pixel of the reference image or the averaged pixel as the downsampled pixel as the adjacent pixel is not selected according to the prediction mode or the like, but is treated as a separate prediction mode, and the one with a smaller cost can be selected.

[0213] <Description of a computer to which the present technology is applied>

[0214] Next, the series of processes of the above-described encoder 11 and decoder 51 can be performed by hardware or by software. When the series of processes are performed by software, the program constituting the software is installed in a general-purpose computer or the like.

[0215] FIG. 13 is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.

[0216] The program can be pre-recorded in a hard disk 905 or a ROM 903 as a recording medium built in the computer.

[0217] Alternatively, the program can be stored (recorded) on a removable recording medium 911 driven by a drive 909. Such a removable recording medium 911 can be provided as so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, a semiconductor memory, and the like.

[0218] In addition to being installed from the removable recording medium 911 as described above into the computer, the program can be downloaded to the computer via a communication network or a broadcast network and installed on the built-in hard disk 905. That is, the program can be wirelessly transferred to the computer from, for example, a download site via an artificial satellite for digital satellite broadcasting, or can be wiredly transferred to the computer via a network such as a LAN (Local Area Network) or the Internet.

[0219] The computer incorporates a CPU (Central Processing Unit) 902, and an input / output interface 910 is connected to the CPU 902 via a bus 901.

[0220] When a command is input by operating the input unit 907 by the user via the input / output interface 910, the CPU 902 executes the program stored in the ROM (Read Only Memory) 903 accordingly. Alternatively, the CPU 902 loads and executes the program stored in the hard disk 905 into the RAM (Random Access Memory) 904.

[0221] As a result, the CPU 902 performs the processes according to the flowchart described above or the processes performed according to the configuration of the block diagram described above. Then, the CPU 902 outputs the processing result from the output unit 906 via the input / output interface 910 as necessary, or transmits it from the communication unit 908, or further records it on the hard disk 905, etc.

[0222] Note that the input unit 907 is composed of a keyboard, a mouse, a microphone, etc. Also, the output unit 906 is composed of an LCD (Liquid Crystal Display), a speaker, etc.

[0223] Here, in this specification, the processes performed by the computer according to the program do not necessarily have to be performed in time series in the order described as a flowchart. That is, the processes performed by the computer according to the program also include processes that are executed in parallel or individually (for example, parallel processing or object-based processing).

[0224] Also, the program may be processed by one computer (processor), or may be processed in a distributed manner by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

[0225] Furthermore, in this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device in which a plurality of modules are housed in one housing are both systems.

[0226] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0227] For example, the present technology can adopt a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

[0228] In addition, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices.

[0229] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in that one step can be executed by one device or can be shared and executed by a plurality of devices.

[0230] Also, the effects described in this specification are merely illustrative and not limiting, and there may be other effects.

Description of Reference Numerals

[0231] 10 Image processing system, 11 Encoder, 21 A / D conversion unit, 22 Rearrangement buffer 22, 23 Arithmetic unit, 24 Orthogonal transformation unit, 25 Quantization unit, 26 Reversible coding unit, 27 Accumulation buffer, 28 Inverse quantization unit, 29 Inverse orthogonal transformation unit, 30 Arithmetic unit, 31a, 31b Deblocking filter, 32 Frame memory, 33 Selection unit, 34 Intra prediction unit, 35 Motion prediction compensation unit, 36 Predicted image selection unit, 37 Rate control unit, 41 Adaptive offset filter, 42 ALF, 51 Decoder, 61 Accumulation buffer, 62 Reversible decoding unit, 63 Inverse quantization unit, 64 Inverse orthogonal transformation unit, 65 Arithmetic unit, 67 Rearrangement buffer, 68 D / A conversion unit, 69 Frame memory, 70 Selection unit, 71 Intra prediction unit, 72 Motion prediction compensation unit, 73 Selection unit, 81 Adaptive offset filter, 82 ALF, 110 Predicted image generation unit, 111 Averaging unit, 112 Matrix-vector multiplication unit, 113 Interpolation unit, 120 Predicted image generation unit, 123 Interpolation unit, 901 Bus, 902 CPU, 903 ROM, 904 RAM, 905 Hard disk, 906 Output unit, 907 Input unit, 908 Communication unit, 909 Drive, 910 Input / output interface, 911 Removable recording medium

Claims

1. A setting unit that sets identification data for identifying whether to perform an interpolation process using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current predicted block for intra prediction using a matrix operation; an encoding unit that encodes the current prediction block to generate a bit stream including the identification data set by the setting unit; An image processing device comprising:

2. The identification data is data for identifying whether an original pixel of the reference image is used as an upper adjacent pixel adjacent to an upper side of the predicted image of the current prediction block. The image processing device according to claim 1 .

3. The identification data is data for identifying whether to use an original pixel of the reference image as a left adjacent pixel adjacent to the left of the predicted image of the current prediction block. The image processing device according to claim 1 .

4. The identification data is data for identifying whether to use original pixels of the reference image depending on a prediction mode of intra prediction. The image processing device according to claim 1 .

5. The identification data is data for identifying whether to use original pixels of the reference image depending on whether an in-loop filter has been applied to the locally decoded image serving as the reference image. The image processing device according to claim 1 .

6. Setting identification data for identifying whether to perform an interpolation process using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current predicted block for intra prediction using a matrix operation; encoding the current prediction block to generate a bitstream including the identification data; An image processing method comprising:

7. A parsing unit for parsing identification data from a bit stream including the identification data for identifying whether to perform an interpolation process using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current predicted block for intra prediction using a matrix operation; an intra prediction unit that generates a predicted image of the current prediction block using the identification data parsed by the parsing unit; a decoding unit that decodes the current prediction block by using the prediction image generated by the intra prediction unit; An image processing device comprising:

8. Parsing identification data from a bitstream including the identification data for identifying whether to perform an interpolation process using original pixels of a reference image as adjacent pixels adjacent to a predicted image of a current predicted block for intra prediction using a matrix operation; generating a predicted image of the current prediction block using the identification data; decoding the current prediction block using the predicted image; An image processing method comprising:

Images