Image processing device and image processing method

Abstract

This technology relates to an image processing device and method capable of improving prediction accuracy. When performing intra-frame prediction using matrix operations, a predicted image for the current prediction block is generated by interpolating original pixels of a reference image as the upper adjacent pixels of the predicted image of the current prediction block being encoded and decoded. The predicted image is then used to encode and decode the current prediction block. This technology can be applied, for example, to image encoding and decoding.

Description

Technical Field

[0001] The present technology relates to an image processing device and an image processing method, and more particularly, to an image processing device and an image processing method capable of improving prediction accuracy, for example. Background Art

[0002] The Joint Video Experts Team (JVET), a joint standardization organization of ITU-T and ISO / IEC, aims to further improve coding efficiency compared to H.265 / HEVC and is studying the standardization of Versatile Video Coding (VVC) as a next-generation image coding method.

[0003] In the standardization work of VVC, non-patent document 1 discloses generating a predicted image by averaging the pixels (pixel values) of a reference image, performing a matrix operation (matrix product) using the average pixel obtained by averaging, using the result of the matrix operation, and performing interpolation processing using the average pixel as the upper adjacent pixel of the predicted image of the prediction block.

[0004] Reference List

[0005] Non-patent literature

[0006] Non-Patent Document 1: JVET-N0217-v3: CE3: Affine linear weighted intra prediction (CE3-4.1, CE3-4.2) (Version 7 - Date 2019-01-17) Summary of the Invention

[0007] Problems to be solved by the present invention

[0008] In the generation of the predicted image described in Non-Patent Document 1, interpolation processing is always performed using the average pixel as the upper adjacent pixel. Therefore, the prediction accuracy of the predicted image cannot be improved.

[0009] The present technology has been made in view of such circumstances, and enables improvement in the prediction accuracy of a predicted image.

[0010] Solution to the problem

[0011] According to the first aspect of the present technology, the image processing device includes: an intra-frame prediction unit, which is configured to: when performing intra-frame prediction using matrix operation, generate a predicted image of a current prediction block by performing interpolation processing using original pixels of a reference image as upper adjacent pixels above the predicted image of the current prediction block to be encoded; and an encoding unit, which is configured to encode the current prediction block using the predicted image generated by the intra-frame prediction unit.

[0012] According to the image processing method of the first aspect of the present technology, the image processing method includes: intra-frame prediction processing, when intra-frame prediction is performed using matrix operation, the intra-frame prediction processing generates a predicted image of a current prediction block by performing interpolation processing using original pixels of a reference image as upper adjacent pixels of the predicted image of the current prediction block to be encoded; and encoding processing of encoding the current prediction block using the predicted image generated in the intra-frame prediction processing.

[0013] According to the image processing device and image processing method of the first aspect of the present technology, when performing intra-frame prediction using matrix operations, a predicted image for the current prediction block to be encoded is generated by performing interpolation processing using original pixels of a reference image as upper adjacent pixels of the predicted image of the current prediction block to be encoded. The current prediction block is then encoded using the predicted image.

[0014] According to the second aspect of the present technology, the image processing device includes: an intra-frame prediction unit, which is configured to: when performing intra-frame prediction using matrix operation, generate a predicted image of a current prediction block by performing interpolation processing using original pixels of a reference image as upper adjacent pixels of the predicted image of the current prediction block to be decoded; and a decoding unit, which is configured to decode the current prediction block using the predicted image generated by the intra-frame prediction unit.

[0015] According to the image processing method of the second aspect of the present technology, the image processing method includes: intra-frame prediction processing, when intra-frame prediction is performed using matrix operation, the intra-frame prediction processing generates a predicted image of a current prediction block by performing interpolation processing using original pixels of a reference image as upper adjacent pixels of the predicted image of the current prediction block to be decoded; and decoding processing of decoding the current prediction block using the predicted image generated in the intra-frame prediction processing.

[0016] According to the second aspect of the present technology, an image processing device and an image processing method generate a predicted image for a current prediction block by performing interpolation processing using original pixels of a reference image as upper adjacent pixels of the predicted image for the current prediction block to be decoded when performing intra prediction using matrix operations. The predicted image is then used to decode the current prediction block.

[0017] Note that the image processing apparatus may be an independent device or an internal block constituting one apparatus.

[0018] In addition, the image processing apparatus can be realized by causing a computer to execute the program. The program can be provided by recording it on a recording medium or transmitting it via a transmission medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram for describing a method of generating a predicted image using matrix-based intra prediction (MIP) proposed by JVET-N0217.

[0020] Figure 2 is a block diagram illustrating a configuration example of an embodiment of an image processing system to which the present technology is applied.

[0021] Figure 3 is a block diagram showing a configuration example of the encoder 11 .

[0022] Figure 4 is a flowchart illustrating an example of encoding processing by the encoder 11 .

[0023] Figure 5 : is a block diagram showing a configuration example of the decoder 51 .

[0024] Figure 6 is a flowchart illustrating an example of decoding processing by the decoder 51 .

[0025] Figure 7 : is a block diagram showing a configuration example of the intra prediction unit 34 .

[0026] Figure 8 : is a flowchart illustrating an example of a process of generating a predicted image of an MIP, which is performed by the predicted image generating unit 110 .

[0027] Figure 9 3 is a diagram illustrating a method of generating a predicted image of an MIP by the predicted image generating unit 110 .

[0028] Figure 10 : is a block diagram showing another configuration example of the intra prediction unit 34 .

[0029] Figure 11 3 is a diagram illustrating a method of generating a predicted image of an MIP by the predicted image generating unit 120 .

[0030] Figure 12 is a diagram showing an example of pixels selected as upper neighboring pixels and left neighboring pixels according to prediction mode k.

[0031] Figure 13 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION

[0032] References

[0033] The scope of the disclosure herein is not limited to the contents of the embodiments, and the contents of the following references REF1 to REF6 known at the time of submission are also incorporated herein by reference. In other words, the contents described in the following references REF1 to REF6 are also the basis for judging the support conditions. For example, even if the quadtree block structure, quadtree plus binary tree (QTBT) block structure, and multi-type tree (MTT) block structure are not directly defined in the detailed description of the present invention, they are within the scope of the present disclosure and should meet the support conditions of the claims. In addition, even when technical terms such as parsing, grammar, and semantics are not directly defined in the detailed description of the present invention, similarly, they are within the scope of the present disclosure and should meet the support conditions of the claims.

[0034] REF1: Recommendation ITU-T H.264 (04 / 2017) “Advanced video coding for generic audiovisual services”, April 2017

[0035] REF2: Recommendation ITU-T H.265 (02 / 2018) “High efficiency video coding”, February 2018

[0036] REF3: Benjamin Bross, Jianle Chen, Shan Liu, Versatile Video Coding (Draft 5), JVET-N1001-v7 (Version 7 - Date 2019-05-29)

[0037] REF4: Jianle Chen, Yan Ye, Seung Hwan Kim, Algorithm description for Versatile Video Coding and Test Model 5 (VTM5), JVET-N1002-v1

[0038] REF5: JVET-N0217-v3: CE3: Affine linear weighted intra prediction (CE3-4.1, CE3-4.2) (Version 7 - Date 2019-01-17)

[0039] REF6: JVET-M0043-v2: CE3: Affine linear weighted intra prediction (test 1.2.1, test 1.2.2) (version 2 - date 2019-01-09)

[0040] <Definition>

[0041] Adjacent includes not only the case where one pixel (one row) is adjacent to the current pixel of interest, but also the case where multiple pixels (multiple rows) are adjacent. Therefore, adjacent pixels include not only the pixel at the position of one pixel directly adjacent to the current pixel, but also the pixels at the positions of multiple pixels continuously adjacent to the current pixel.

[0042] Downsampling means reducing the number of pixels. Thus, downsampling includes reducing the number of pixels by using calculations such as averaging or median, or reducing the number of pixels without performing calculations.

[0043] A prediction block refers to a block (prediction unit (PU)) that is a processing unit when performing intra prediction, and also includes subblocks in the prediction block. When a prediction block, an orthogonal transform block (transform unit (TU)) that is a processing unit for performing orthogonal transform, and a coding block (coding unit (CU)) that is a processing unit for encoding are unified into the same block, the prediction block, the orthogonal transform block, and the coding block mean the same block.

[0044] The prediction mode of intra-frame prediction means information that comprehensively includes various matters related to intra-frame prediction, such as the mode number when performing intra-frame prediction, the block size of the prediction block, the mode number of matrix-based intra-frame prediction (MIP) using matrix operations, the type of matrix used when performing matrix operations, and the type of size of the matrix used when performing matrix operations.

[0045] In the present technology, identification data identifying a plurality of modes may be set as a bitstream syntax obtained by encoding an image.The bitstream may include identification data identifying various modes.

[0046] As identification data, for example, in generating a predicted image for intra-frame prediction, data for identifying whether to use original pixels (pixel values) of a reference image or whether to use (pixel values ​​of) downsampled pixels as neighboring pixels adjacent to the prediction block can be used. Furthermore, as identification data, for example, data for separately identifying whether to use original pixels of a reference image or downsampled pixels as upper neighboring pixels adjacent to the upper side of the predicted image of the prediction block and as left neighboring pixels adjacent to the left side of the predicted image of the prediction block can be used.

[0047] When identification data is included in a bitstream, a decoder that decodes the bitstream can perform processing more efficiently by parsing and referring to the identification data.

[0048] <Method for generating predicted images proposed in JVET-N0217>

[0049] Figure 1This is a diagram for describing a method of generating a predicted image using matrix-based intra prediction (MIP) proposed by JVET-N0217 (reference REF5).

[0050] In JVET-N0217, in MIP, it has been proposed to generate a predicted image by averaging pixels (pixel values) of a reference image (decoded image), performing matrix operations (matrix-vector-multiplication) using the average pixels obtained by averaging, and performing interpolation processing (interpolation) using the result of the matrix operation and the average pixels.

[0051] Here, the original pixels of the reference image adjacent to the upper side of the current prediction block as the prediction block to be encoded / decoded are also referred to as upper original pixels. In addition, the original pixels of the reference image adjacent to the left side of the current prediction block are also referred to as left original pixels.

[0052] Furthermore, the horizontal size of the block (horizontal size) is represented by W, and the vertical size of the block (vertical size) is represented by H.

[0053] exist Figure 1 In , a block of W×H=8×8 pixels is used as the current prediction block.

[0054] In averaging, for the current prediction block, the original pixel (pixel value) above the reference image is averaged. top Average and generate multiple average pixels (pixel values) as downsampled pixels bdry red .

[0055] In addition, when averaging, for the current prediction block, the original pixel bdry on the left side of the reference image left Average and generate multiple average pixels bdry as downsampled pixels red .

[0056] When the current prediction block is a block of W×H=8×8 pixels, the original pixel bdry above top The average and the original pixel on the left bdryl eft In the averaging, four average pixels bdry are generated by averaging two adjacent original pixels of the reference image. red .

[0057] In the matrix operation, the matrix A used for the matrix operation is set according to the prediction mode k of the intra prediction k and offset b k Then, in matrix operations, the matrix A k Multiply by the average pixel bdry obtained by averaging red Vector bdry redAs an element. In addition, in matrix operations, the offset b k is added to the result of the multiplication. As a result, some pixels pred of the predicted image of the current prediction block are generated. red .

[0058] In the interpolation process, the upper adjacent pixels adjacent to the upper side of the predicted image of the current prediction block, the left adjacent pixels adjacent to the left side of the predicted image of the current prediction block, and some pixels pred of the predicted image of the current prediction block generated by matrix operation are used. red to perform interpolation and generate the remaining pixels of the predicted image.

[0059] As the upper neighboring pixels of the predicted image, the average pixel bdry in the reference image red Among them, use the original pixel bdry above top The average pixel bdry generated top red .

[0060] As the left neighboring pixel of the predicted image, use the left original pixel bdry of the reference image left .

[0061] Here, the position of the x-th pixel from the left and the y-th pixel from the top is expressed as (x-1, y-1), and the pixel at the position (x-1, y-1) is described as pixel (x-1, y-1).

[0062] When the current prediction block is a block of W×H=8×8 pixels, some pixels pred of the prediction image generated by matrix operation are red is x-1 among the pixels of the predicted image of the current prediction block that is pixel (x-1, y-1) (shown by a diagonal line in the figure), and the pixel (x-1, y-1) is at an odd position among the pixels of the predicted image of the current prediction block where x-1 and y-1 are.

[0063] In the interpolation process, the four average pixels bdry top red Arranged as four upper adjacent pixels, at the position adjacent to x-1, adjacent to the upper side of the predicted image of the current prediction block. In addition, eight left original pixels bdry left Eight left-neighboring pixels arranged adjacent to the left side of the predicted image of the current prediction block.

[0064] Then, use pixel bdry as the average of the adjacent pixels above top red And the pixel pred generated by matrix operation redThe pixel (x-1, y-1) at the position where x-1 and y-1 are odd numbers is generated by longitudinal (vertical) interpolation to generate the pixel at the position where x-1 is odd and y-1 is even number of the predicted image.

[0065] In addition, using the left original pixel bdry as the left adjacent pixel left , pixel pred generated by matrix operation red And the pixels generated by the vertical interpolation are interpolated horizontally to generate the remaining pixels of the predicted image.

[0066] Then, a predicted image (pred) of the current prediction block is generated by combining pixels generated by the interpolation process and pixels generated by the matrix operation.

[0067] In JVET-N0217, in the interpolation process, the original pixel bdry above the reference image is used top The average pixel bdry generated top red Used as the upper neighboring pixel of the predicted image of the current prediction block.

[0068] That is, the interpolation process is performed using pixels that are not actual pixels (original pixels) of the reference image itself.

[0069] Therefore, after the matrix operation, the average pixel bdry used when performing the interpolation process must be saved top red , and is needed to save the average pixel bdry top red storage area (memory).

[0070] In addition, since the average pixel bdry is always used top red The interpolation process is performed as the upper adjacent pixels, so there is a risk that the pixels of the prediction block are different from those obtained by using the average pixel bdry. top red The correlation between pixels of the generated predicted image decreases, and the prediction accuracy of the intra prediction (that is, the prediction accuracy of the predicted image) cannot be improved, or the prediction accuracy may decrease.

[0071] Therefore, in the present technology, in the interpolation process, by using the upper original pixel bdry of the reference image top as the upper adjacent pixels to improve the prediction accuracy of the predicted image.

[0072] <Image Processing System Applying This Technology>

[0073] Figure 2is a block diagram illustrating a configuration example of an embodiment of an image processing system to which the present technology is applied.

[0074] The image processing system 10 includes an image processing device as an encoder 11 and an image processing device as a decoder 51 .

[0075] The encoder 11 encodes the original image to be encoded supplied thereto and outputs an encoded bit stream obtained by the encoding. The encoded bit stream is supplied to the decoder 51 via a recording medium or a transmission medium (not shown).

[0076] The decoder 51 decodes the encoded bit stream supplied thereto, and outputs a decoded image obtained by the decoding.

[0077] <Configuration Example of Encoder 11>

[0078] Figure 3 It shows Figure 2 A block diagram of a configuration example of the encoder 11 is shown.

[0079] Note that, in the block diagrams described below, description of lines that provide information (data) necessary for processing of each block is omitted as appropriate in order to avoid complicating the block diagrams.

[0080] exist Figure 3 , the encoder 11 includes an A / D conversion unit 21, a reordering buffer 22, a calculation unit 23, an orthogonal transform unit 24, a quantization unit 25, a reversible encoding unit 26, and an accumulation buffer 27. In addition, the encoder 11 includes an inverse quantization unit 28, an inverse orthogonal transform unit 29, a calculation unit 30, a frame memory 32, a selection unit 33, an intra-frame prediction unit 34, a motion prediction / compensation unit 35, a predicted image selection unit 36, and a rate control unit 37. In addition, the encoder 11 has a deblocking filter 31a, an adaptive offset filter 41, and an adaptive loop filter (ALF) 42.

[0081] The A / D conversion unit 21 A / D converts the original image (encoding target) of the analog signal into the original image of the digital signal, and supplies and stores the A / D-converted original image into the reordering buffer 22. Note that when the original image of the digital signal is supplied to the encoder 11, the encoder 11 may be configured without providing the A / D conversion unit 21.

[0082] The reordering buffer 22 rearranges the frames of the original image from display order to encoding (decoding) order according to the group of pictures (GOP), and supplies the frames to the calculation unit 23 , the intra prediction unit 34 , and the motion prediction / compensation unit 35 .

[0083] The calculation unit 23 subtracts the predicted image supplied from the intra prediction unit 34 or the motion prediction / compensation unit 35 from the original image from the rearrangement buffer 22 via the predicted image selection unit 36 ​​, and supplies the residual (prediction residual) obtained by the subtraction to the orthogonal transformation unit 24 .

[0084] The orthogonal transform unit 24 performs an orthogonal transform such as a discrete cosine transform or a Karhunen-Loève transform on the residual supplied from the calculation unit 23 , and supplies an orthogonal transform coefficient obtained by the orthogonal transform to the quantization unit 25 .

[0085] The quantization unit 25 quantizes the orthogonal transform coefficients supplied from the orthogonal transform unit 24. The quantization unit 25 sets a quantization parameter based on the target value of the code amount (code amount target value) supplied from the rate control unit 37, and performs quantization of the orthogonal transform coefficients. The quantization unit 25 supplies the encoded data, which is the quantized orthogonal transform coefficient, to the reversible encoding unit 26.

[0086] The reversible encoding unit 26 encodes the quantized orthogonal transform coefficient into the encoded data from the quantization unit 25 by a predetermined reversible encoding method.

[0087] Furthermore, the reversible encoding unit 26 acquires encoding information necessary for decoding by the decoding device 170 from each block among the encoding information related to the predictive encoding in the encoder 11 .

[0088] Here, as encoding information, for example, there is a prediction mode of intra-frame prediction or inter-frame prediction, motion information such as a motion vector, a target value of the amount of code, a quantization parameter, a picture type (I, P, B), filter parameters such as the deblocking filter 31a and the adaptive offset filter 41, and the like.

[0089] The prediction mode may be acquired from the intra prediction unit 34 or the motion prediction / compensation unit 35. The motion information may be acquired from the motion prediction / compensation unit 35. The filter parameters of the deblocking filter 31a and the adaptive offset filter 41 may be acquired from the deblocking filter 31a and the adaptive offset filter 41, respectively.

[0090] The reversible coding unit 26 encodes the coding information by, for example, variable length coding or arithmetic coding (such as context adaptive variable length coding (CAVLC) or context adaptive binary arithmetic coding (CABAC) or other reversible coding methods), and after encoding, generates a coded bit stream including (multiplexed) the coded information and the coded data from the quantization unit 25, and provides the generated coded bit stream to the accumulation buffer 27.

[0091] Here, the above calculation unit 23 or the reversible encoding unit 26 functions as an encoding unit for encoding an image.

[0092] The accumulation buffer 27 temporarily accumulates the encoded bit stream supplied from the reversible encoding unit 26. The encoded bit stream stored in the accumulation buffer 27 is read and transmitted at a predetermined timing.

[0093] The encoded data, which is the orthogonal transform coefficient quantized in the quantization unit 25, is supplied not only to the reversible encoding unit 26 but also to the inverse quantization unit 28. The inverse quantization unit 28 inversely quantizes the quantized orthogonal transform coefficient by a method corresponding to the quantization by the quantization unit 25, and transmits the orthogonal transform coefficient obtained by the inverse quantization to the inverse orthogonal transform unit 29.

[0094] The inverse orthogonal transform unit 29 performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization unit 28 by a method corresponding to the orthogonal transform process of the orthogonal transform unit 24 , and supplies a residual obtained by the inverse orthogonal transform to the calculation unit 30 .

[0095] The calculation unit 30 adds the predicted image supplied from the intra prediction unit 34 or the motion prediction / compensation unit 35 via the predicted image selection unit 36 ​​and the residual supplied from the inverse orthogonal transform unit 29 , thereby obtaining and outputting (a portion of) a decoded image that decodes the original image.

[0096] The decoded image output by the calculation unit 30 is supplied to the deblocking filter 31 a or the frame memory 32 .

[0097] The frame memory 32 temporarily stores the decoded image supplied from the calculation unit 30 and the decoded image (filtered image) supplied from the ALF 42, and the decoded image is applied with the deblocking filter 31a, the adaptive offset filter 41, and the ALF 42. The decoded image stored in the frame memory 32 is supplied to the selection unit 33 as a reference image for generating a predicted image at a desired timing.

[0098] The selection unit 33 selects a supply destination of the reference image supplied from the frame memory 32. When intra prediction is performed in the intra prediction unit 34, the selection unit 33 supplies the reference image supplied from the frame memory 32 to the intra prediction unit 34. When inter prediction is performed in the motion prediction / compensation unit 35, the selection unit 33 supplies the reference image supplied from the frame memory 32 to the motion prediction / compensation unit 35.

[0099] The intra prediction unit 34 performs intra prediction (intra-screen prediction) using the original image supplied from the reordering buffer 22 and the reference image supplied from the frame memory 32 via the selection unit 33. The intra prediction unit 34 selects an optimal prediction mode for intra prediction based on a predetermined cost function, and supplies a predicted image generated from the reference image in the prediction mode for the optimal intra prediction to the predicted image selection unit 36. Furthermore, the intra prediction unit 34 appropriately supplies the prediction mode for intra prediction selected based on the cost function to the reversible encoding unit 26 and the like.

[0100] The motion prediction / compensation unit 35 performs motion prediction using the original image supplied from the reordering buffer 22 and the reference image supplied from the frame memory 32 via the selection unit 33. Furthermore, the motion prediction / compensation unit 35 performs motion compensation based on the motion vector detected by the motion prediction, and generates a predicted image. The motion prediction / compensation unit 35 performs inter-frame prediction in a prediction mode among a plurality of inter-frame predictions prepared in advance, and generates a predicted image from the reference image.

[0101] The motion prediction / compensation unit 35 selects an optimal inter prediction mode from among the prediction modes of the inter prediction based on a predetermined cost function, and supplies a predicted image generated in the optimal inter prediction mode to the predicted image selection unit 36 ​​.

[0102] In addition, the motion prediction / compensation unit 35 provides the reversible encoding unit 26 with the optimal inter prediction prediction mode selected based on the cost function or motion information (such as a motion vector required to decode the encoded data encoded in the inter prediction prediction mode).

[0103] The predicted image selection unit 36 ​​selects a supply source of a predicted image to be supplied from the intra prediction unit 34 and the motion prediction / compensation unit 35 to the computing unit 23 and the computing unit 30 , and selects a predicted image to be supplied from the selected supply source to the computing unit 23 and the computing unit 30 .

[0104] The rate control unit 37 controls the rate of the quantization operation of the quantization unit 25 so that overflow or underflow does not occur based on the code amount of the encoded bit stream accumulated in the accumulation buffer 27. That is, the rate control unit 37 sets the target code amount of the encoded bit stream and supplies the set target code amount to the quantization unit 25 so that overflow and underflow of the accumulation buffer 27 do not occur.

[0105] The deblocking filter 31 a applies a deblocking filter to the decoded image from the calculation unit 30 as necessary, and supplies the decoded image to which the deblocking filter is applied (filtered image) or the decoded image to which the deblocking filter is not applied to the adaptive offset filter 41 .

[0106] The adaptive offset filter 41 applies the adaptive offset filter to the decoded image from the deblocking filter 31 a as needed, and supplies the decoded image (filtered image) to which the adaptive offset filter is applied or the decoded image to which the adaptive offset filter is not applied to the ALF 42 .

[0107] The ALF 42 applies ALF to the decoded image from the adaptive offset filter 41 as necessary, and supplies the decoded image to which ALF is applied or the decoded image to which ALF is not applied to the frame memory 32 .

[0108] <Encoding Processing>

[0109] Figure 4 It shows Figure 3 Flowchart of an example of encoding processing of the encoder 11.

[0110] Figure 4 The order of each step of the encoding process shown in FIG is for convenience of explanation, and each step of the actual encoding process is appropriately executed in parallel in the necessary order. The same applies to the process described later.

[0111] In step S11 , in the encoder 11 , the A / D conversion unit 21 A / D-converts the original image and supplies the A / D-converted original image to the rearrangement buffer 22 , and the process proceeds to step S12 .

[0112] In step S12 , the rearrangement buffer 22 stores the original images from the A / D conversion unit 21 , rearranges the original images in encoding order, and outputs the rearranged original images, and the process proceeds to step S13 .

[0113] In step S13, the intra prediction unit 34 performs intra prediction, and the process proceeds to step S 14. In step S14, the motion prediction / compensation unit 35 performs inter prediction for motion prediction or motion compensation, and the process proceeds to step S15.

[0114] In the intra prediction by the intra prediction unit 34 and the inter prediction by the motion prediction / compensation unit 35 , cost functions of various prediction modes are calculated and a predicted image is generated.

[0115] In step S15, the predicted image selection unit 36 ​​determines the optimal prediction mode based on each cost function obtained by the intra prediction unit 34 and the motion prediction / compensation unit 35. Then, the predicted image selection unit 36 ​​selects and outputs the predicted image of the optimal prediction mode from the predicted image generated by the intra prediction unit 34 and the predicted image generated by the motion prediction / compensation unit 35, and the process proceeds from step S15 to step S16.

[0116] In step S16, the calculation unit 23 calculates the residual between the target image to be encoded which is the original image output by the rearrangement buffer 22 and the predicted image output by the predicted image selection unit 36, and provides the calculated residual to the orthogonal transformation unit 24, and the processing proceeds to step S17.

[0117] In step S17 , the orthogonal transform unit 24 performs an orthogonal transform on the residual from the calculation unit 23 , and supplies the resultant orthogonal transform coefficient to the quantization unit 25 , and the process proceeds to step S18 .

[0118] In step S18 , the quantization unit 25 quantizes the orthogonal transform coefficient from the orthogonal transform unit 24 , and supplies the quantized coefficient obtained by the quantization to the reversible encoding unit 26 and the inverse quantization unit 28 , and the process proceeds to step S19 .

[0119] In step S19, the inverse quantization unit 28 inversely quantizes the quantized coefficients from the quantization unit 25 and supplies the obtained orthogonal transform coefficients to the inverse orthogonal transform unit 29, and the process proceeds to step S20. In step S20, the inverse orthogonal transform unit 29 inversely orthogonally transforms the orthogonal transform coefficients from the inverse quantization unit 28 and supplies the obtained residual to the calculation unit 30, and the process proceeds to step S21.

[0120] In step S21, the calculation unit 30 adds the residual from the inverse orthogonal transform unit 29 to the predicted image output by the predicted image selection unit 36, and generates a decoded image corresponding to the original image that is the target of calculation of the residual by the calculation unit 23. The calculation unit 30 supplies the decoded image to the deblocking filter 31a, and the process proceeds from step S21 to step S22.

[0121] In step S22 , the deblocking filter 31 a applies a deblocking filter to the decoded image from the calculation unit 30 , and supplies the resulting filtered image to the adaptive offset filter 41 , and the process proceeds to step S23 .

[0122] In step S23 , the adaptive offset filter 41 applies the adaptive offset filter to the filtered image from the deblocking filter 31 a and supplies the resultant filtered image to the ALF 42 , and the process proceeds to step S24 .

[0123] In step S24 , the ALF 42 applies ALF to the filtered image from the adaptive offset filter 41 and supplies the resultant filtered image to the frame memory 32 , and the process proceeds to step S25 .

[0124] In step S25, the frame memory 32 stores the filtered image supplied from the ALF 42, and the process proceeds to step S26. The filtered image stored in the frame memory 32 is used as a reference image from which a predicted image is generated in steps S13 and S14.

[0125] In step S26, the reversible encoding unit 26 encodes the encoded data, which is the quantization coefficient from the quantization unit 25, and generates an encoded bit stream including the encoded data. In addition, the reversible encoding unit 26 encodes encoding information such as the quantization parameter used for quantization in the quantization unit 25, the prediction mode obtained by intra-frame prediction in the intra-frame prediction unit 34, the prediction mode or motion information obtained by inter-frame prediction in the motion prediction / compensation unit 35, or the filter parameters of the deblocking filter 31a and the adaptive offset filter 41, as needed, and includes the encoded information in the encoded bit stream.

[0126] Then, the reversible encoding unit 26 supplies the encoded bit stream to the accumulation buffer 27, and the process proceeds from step S26 to step S27.

[0127] In step S27, the accumulation buffer 27 accumulates the encoded bit stream from the reversible encoding unit 26, and the process proceeds to step S28. The encoded bit stream accumulated in the accumulation buffer 27 is read and transmitted as appropriate.

[0128] In step S28 , the rate control unit 37 controls the quantization operation of the quantization unit 25 based on the code amount (generated code amount) of the encoded bit stream accumulated in the accumulation buffer 27 so that overflow or underflow does not occur, and the encoding process ends.

[0129] <Configuration Example of Decoder 51>

[0130] Figure 5 It shows Figure 2 A block diagram of a configuration example of the decoder 51 is shown.

[0131] exist Figure 5 In the embodiment, the decoder 51 has an accumulation buffer 61, a reversible decoding unit 62, an inverse quantization unit 63, an inverse orthogonal transform unit 64, a calculation unit 65, a rearrangement buffer 67, and a D / A conversion unit 68. In addition, the decoder 51 has a frame memory 69, a selection unit 70, an intra-frame prediction unit 71, a motion prediction / compensation unit 72, and a selection unit 73. In addition, the decoder 51 has a deblocking filter 31b, an adaptive offset filter 81, and an ALF 82.

[0132] The accumulation buffer 61 temporarily accumulates the encoded bit stream transmitted from the encoder 11 , and supplies the encoded bit stream to the reversible decoding unit 62 at a predetermined timing.

[0133] The reversible decoding unit 62 receives the encoded bit stream from the accumulation buffer 61 and Figure 3 The received encoded bit stream is decoded using a method corresponding to the encoding method of the reversible encoding unit 26.

[0134] Then, the reversible decoding unit 62 supplies the quantized coefficient as encoded data included in the decoding result of the encoded bit stream to the inverse quantization unit 63 .

[0135] In addition, the reversible decoding unit 62 has a function of performing parsing. The reversible decoding unit 62 parses necessary encoding information included in the decoding result of the encoded bit stream and provides the encoding information to the intra-frame prediction unit 71, the motion prediction / compensation unit 72, the deblocking filter 31b, the adaptive offset filter 81, and other necessary blocks.

[0136] The inverse quantization unit 63 is connected with Figure 3 The quantized coefficients as the encoded data from the reversible decoding unit 62 are inversely quantized by a method corresponding to the quantization method of the quantization unit 25 in , and the orthogonal transformation coefficients obtained by the inverse quantization are supplied to the inverse orthogonal transformation unit 64 .

[0137] The inverse orthogonal transform unit 64 is connected to Figure 3 The orthogonal transform unit 24 performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization unit 63 by a method corresponding to the orthogonal transform method in , and supplies the obtained residual to the calculation unit 65 .

[0138] In addition to the residual supplied from the inverse orthogonal transform unit 64 to the calculation unit 65 , a predicted image is supplied from the intra prediction unit 71 or the motion prediction / compensation unit 72 via the selection unit 73 .

[0139] The calculation unit 65 adds the residual from the inverse orthogonal transform unit 64 and the predicted image from the selection unit 73 , generates a decoded image, and supplies the generated decoded image to the deblocking filter 31 b .

[0140] Here, the above reversible decoding unit 62 or the calculation unit 65 constitutes a decoding unit for decoding an image.

[0141] The rearrangement buffer 67 temporarily stores the decoded image supplied from the ALF 82 , rearranges the arrangement of frames (pictures) of the decoded image from encoding (decoding) order to display order, and supplies the rearranged frames to the D / A conversion unit 68 .

[0142] The D / A conversion unit 68 performs D / A conversion on the decoded image supplied from the rearrangement buffer 67 and outputs the D / A-converted decoded image to a display (not shown) for display. Note that when the device connected to the decoder 51 accepts an image of a digital signal, the decoder 51 may be configured without providing the D / A conversion unit 68.

[0143] The frame memory 69 temporarily stores the decoded image supplied from the ALF 82. In addition, the frame memory 69 supplies the decoded image as a reference image to the selection unit 70 for generating a predicted image at a predetermined timing or based on an external request such as the intra prediction unit 71 or the motion prediction / compensation unit 72.

[0144] The selection unit 70 selects a supply destination of the reference image supplied from the frame memory 69. When decoding an image encoded by intra prediction, the selection unit 70 supplies the reference image supplied from the frame memory 69 to the intra prediction unit 71. In addition, when decoding an image encoded by inter prediction, the selection unit 70 supplies the reference image supplied from the frame memory 69 to the motion prediction / compensation unit 72.

[0145] Similar to Figure 3 The intra prediction unit 34 performs intra prediction using the reference image supplied from the frame memory 69 via the selection unit 70 according to the prediction mode included in the encoding information supplied from the reversible decoding unit 62. The intra prediction unit 71 then supplies the predicted image obtained by the intra prediction to the selection unit 73.

[0146] Similar to Figure 3 The motion prediction / compensation unit 35 of the reversible decoding unit 62 performs inter-frame prediction using the reference image supplied from the frame memory 69 via the selection unit 70 according to the prediction mode included in the encoding information supplied from the reversible decoding unit 62. The inter-frame prediction is performed by using the motion information and the like included in the encoding information supplied from the reversible decoding unit 62 as needed.

[0147] The motion prediction / compensation unit 72 supplies the predicted image obtained by the inter prediction to the selection unit 73 .

[0148] The selection unit 73 selects the predicted image supplied from the intra prediction unit 71 or the predicted image supplied from the motion prediction / compensation unit 72 , and supplies the predicted image to the calculation unit 65 .

[0149] The deblocking filter 31b applies a deblocking filter to the decoded image from the calculation unit 65 according to the filter parameters included in the encoding information supplied from the reversible decoding unit 62. The deblocking filter 31b supplies the decoded image (filtered image) to which the deblocking filter is applied or the decoded image to which the deblocking filter is not applied to the adaptive offset filter 81.

[0150] The adaptive offset filter 81 applies an adaptive offset filter to the decoded image from the deblocking filter 31b as needed, based on the filter parameters included in the encoded information supplied from the reversible decoding unit 62. The adaptive offset filter 81 supplies the decoded image (filtered image) to which the adaptive offset filter has been applied or the decoded image to which the adaptive offset filter has not been applied to the ALF 82.

[0151] The ALF 82 applies ALF to the decoded image from the adaptive offset filter 81 as needed, and supplies the decoded image to which ALF is applied or the decoded image to which ALF is not applied to the reordering buffer 67 and the frame memory 69 .

[0152] <Decoding Process>

[0153] Figure 6 It shows Figure 5 Flowchart of an example of decoding processing of the decoder 51.

[0154] In step S51 , in the decoding process, the accumulation buffer 61 temporarily stores the encoded bit stream transmitted from the encoder 11 , and supplies the stored encoded bit stream to the reversible decoding unit 62 as appropriate, and the process proceeds to step S52 .

[0155] In step S52 , the reversible decoding unit 62 receives the encoded bit stream supplied from the accumulation buffer 61 and decodes it, and supplies the quantization coefficient as encoded data included in the decoded result of the encoded bit stream to the inverse quantization unit 63 .

[0156] In addition, the reversible decoding unit 62 parses the encoding information included in the decoding result of the encoded bit stream. Then, the reversible decoding unit 62 provides the necessary encoding information to the intra prediction unit 71, the motion prediction / compensation unit 72, the deblocking filter 31b, the adaptive offset filter 81 and other necessary blocks.

[0157] Then, the process proceeds from step S52 to step S53, and the intra prediction unit 71 or the motion prediction / compensation unit 72 performs intra prediction or inter prediction for generating a predicted image based on the reference image supplied from the frame memory 69 via the selection unit 70 and the encoding information supplied from the reversible decoding unit 62. Then, the intra prediction unit 71 or the motion prediction / compensation unit 72 supplies the predicted image obtained by the intra prediction or inter prediction to the selection unit 73, and the process proceeds from step S53 to step S54.

[0158] In step S54 , the selection unit 73 selects the predicted image supplied from the intra prediction unit 71 or the motion prediction / compensation unit 72 , and supplies the selected predicted image to the calculation unit 65 , and the process proceeds to step S55 .

[0159] In step S55 , the inverse quantization unit 63 inversely quantizes the quantized coefficient from the reversible decoding unit 62 , and supplies the resultant orthogonal transform coefficient to the inverse orthogonal transform unit 64 , and the process proceeds to step S56 .

[0160] In step S56 , the inverse orthogonal transform unit 64 performs inverse orthogonal transform on the orthogonal transform coefficient from the inverse quantization unit 63 , and supplies the resultant residual to the calculation unit 65 , and the process proceeds to step S57 .

[0161] In step S57, the calculation unit 65 generates a decoded image by adding the residual from the inverse orthogonal transform unit 64 and the predicted image from the selection unit 73. Then, the calculation unit 65 supplies the decoded image to the deblocking filter 31b, and the process proceeds from step S57 to step S58.

[0162] In step S58, the deblocking filter 31b applies a deblocking filter to the decoded image from the calculation unit 65 based on the filter parameters included in the encoded information supplied from the reversible decoding unit 62. The deblocking filter 31b supplies the filtered image obtained by applying the deblocking filter to the adaptive offset filter 81, and the process proceeds from step S58 to step S59.

[0163] In step S59, the adaptive offset filter 81 applies an adaptive offset filter to the filtered image from the deblocking filter 31b based on the filter parameters included in the encoded information supplied from the reversible decoding unit 62. The adaptive offset filter 81 supplies the filtered image obtained by applying the adaptive offset filter to the ALF 82, and the process proceeds from step S59 to step S60.

[0164] The ALF 82 applies the ALF to the filtered image from the adaptive offset filter 81 and supplies the resulting filtered image to the reordering buffer 67 and the frame memory 69, and the process proceeds to step S61.

[0165] In step S61, the frame memory 69 temporarily stores the filtered image supplied from the ALF 82, and the process proceeds to step S62. The filtered image (decoded image) stored in the frame memory 69 is used as a reference image from which a predicted image is generated by intra prediction or inter prediction in step S53.

[0166] In step S62 , the rearrangement buffer 67 rearranges the filtered images supplied from the ALF 82 in display order, and supplies the rearranged filtered images to the D / A conversion unit 68 , and the process proceeds to step S63 .

[0167] In step S63, the D / A conversion unit 68 D / A converts the filtered image from the rearrangement buffer 67, and this process ends the decoding process. The D / A-converted filtered image (decoded image) is output and displayed on a display (not shown).

[0168] <Configuration Example of Intra Prediction Unit 34>

[0169] Figure 7 : is a block diagram showing a configuration example of the intra prediction unit 34 .

[0170] Notice, Figure 7 Only the predicted image generation unit 110 of the intra prediction unit 34 , which is a portion that generates a predicted image for the MIP, is shown. The intra prediction unit 71 also has a predicted image generation unit similar to the predicted image generation unit 110 .

[0171] The predicted image generation unit 110 includes an averaging unit 111 , a matrix-vector multiplication unit 112 , and an interpolation unit 113 .

[0172] The reference image (as a decoded image) of the current prediction block is supplied from the selection unit 33 to the averaging unit 111 .

[0173] The averaging unit 111 averages (the pixel values ​​of) the upper original pixels of the reference image of the current prediction block, and generates a plurality of average pixels (the pixel values ​​of) as down-sampled pixels.

[0174] In addition, the averaging unit 111 averages the left original pixels of the reference image of the current prediction block and generates a plurality of average pixels as down-sampled pixels.

[0175] The averaging unit 111 provides the average pixel to the matrix-vector multiplication unit 112 .

[0176] The prediction mode k of the intra prediction is supplied to the matrix-vector multiplication unit 112 .

[0177] The matrix vector multiplication unit 112 sets the matrix A used for matrix operation according to the prediction mode k. k and vector offset b k Then, the matrix vector multiplication unit 112 performs the matrix A k The multiplication between the average pixel of the averaging unit 111 and the vector as an element is performed as a matrix operation. In addition, the matrix vector multiplication unit 112 shifts the pixel by the average pixel. k The result of the multiplication as the matrix operation is added, thereby generating some pixels of the predicted image of the current prediction block, and the pixels are supplied to the interpolation unit 113 .

[0178] The reference image is supplied to the interpolation unit 113 .

[0179] The interpolation unit 113 performs interpolation processing by using the upper adjacent pixels adjacent to the upper side of the predicted image of the current prediction block, the left adjacent pixels adjacent to the left side of the predicted image of the current prediction block, and some pixels of the predicted image of the current prediction block from the matrix-vector multiplication unit 112.

[0180] The interpolation unit 113 generates the remaining pixels of the predicted image of the current prediction block through an interpolation process, and generates (completes) the predicted image of the current prediction block by combining with some pixels of the predicted image from the matrix-vector multiplication unit 112 .

[0181] The interpolation unit 113 uses the upper original pixels of the reference image as the upper adjacent pixels of the predicted image, and uses the left original pixels of the reference image as the left adjacent pixels of the predicted image.

[0182] Figure 8 is a flowchart for describing an example of a process of generating a predicted image of an MIP, which is performed by the predicted image generating unit 110 .

[0183] In step S111, the averaging unit 111 averages the upper original pixels of the reference image and averages the left original pixels of the reference image to generate an average pixel. The averaging unit 111 provides the average pixel to the matrix vector multiplication unit 112, and the process proceeds from step S111 to step S112.

[0184] In step S112, the matrix vector multiplication unit 112 sets the matrix A for matrix operation according to the prediction mode k. k and vector offset b kIn addition, the matrix vector multiplication unit 112 uses the matrix A k and offset b k Matrix operations are performed on a vector whose elements are the average pixels from the averaging unit 111 .

[0185] That is, the matrix vector multiplication unit 112 performs the matrix A k The multiplication between the average pixel of the averaging unit 111 and the vector as an element is performed as a matrix operation. In addition, the matrix vector multiplication unit 112 shifts the pixel by the average pixel. k The matrix vector multiplication unit 112 generates some pixels of the predicted image of the current prediction block through the above matrix operation and supplies the generated pixels to the interpolation unit 113, and the process proceeds from step S112 to step S113.

[0186] In step S113, the interpolation unit 113 uses the upper original pixels of the reference image as the upper adjacent pixels and uses the left original pixels of the reference image as the left adjacent pixels, and uses the upper adjacent pixels, the left adjacent pixels and some pixels of the predicted image of the current prediction block from the matrix-vector multiplication unit 112 to perform interpolation processing.

[0187] The interpolation unit 113 generates the remaining pixels of the predicted image of the current prediction block through an interpolation process, and generates the predicted image of the current prediction block by combining with some pixels of the predicted image from the matrix-vector multiplication unit 112 .

[0188] Figure 9 3 is a diagram illustrating a method of generating a predicted image of an MIP by the predicted image generating unit 110 .

[0189] and Figure 1 Similar, in Figure 9 In FIG, a block of W×H=8×8 pixels is used as a current prediction block. However, the prediction block is not limited to a block of W×H=8×8 pixels.

[0190] Similar to JVET-N0217, the averaging unit 111 averages the original pixels bdry above the reference image of the current prediction block. top Average and generate multiple average pixels bdry red as downsampled pixels.

[0191] In addition, similar to JVET-N0217, the averaging unit 111 averages the left original pixel bdry of the reference image of the current prediction block. left Average and generate multiple average pixels bdry red as downsampled pixels.

[0192] When the current prediction block is a block of W×H=8×8 pixels, by performing the operations on two upper original pixels bdry adjacent to each other in the horizontal direction of the reference image, top Averaging to perform bdry on the original pixel above top The average of the four average pixels bdry red Similarly, by performing the vertical division of the two left original pixels bdry adjacent to the reference image left Averaging to perform bdry on the left original pixel left The average of the four average pixels bdry red .

[0193] Here, in this embodiment, the average pixel obtained by averaging the original pixels is used as the downsampled pixel, but the downsampled pixel is not limited to the average pixel. That is, downsampling can be performed by averaging multiple pixels, performing calculations other than median average, or simply thinning the pixels.

[0194] Similar to JVET-N0217, the matrix vector multiplication unit 112 uses the matrix A set according to the prediction mode k of the intra prediction k and offset b k , and calculate the expression pred red =A k ·bdry red +b k As vector bdry red Matrix operation, where the average pixel bdry red As an element.

[0195] That is, the matrix vector multiplication unit 112 performs the matrix A k With vector bdry red Multiplication A between k ·bdry red , where the average pixel bdry red As an element of matrix operation. In addition, the matrix vector multiplication unit 112 will offset b k and A as the result of multiplication k ·bdry red Therefore, the matrix vector multiplication unit 112 generates some pixels pred of the predicted image of the current prediction block red .

[0196] Similar to JVET-N0217, the interpolation unit 113 calculates the predicted image of the current prediction block by using the upper adjacent pixels adjacent to the upper side of the predicted image of the current prediction block, the left adjacent pixels adjacent to the left side of the predicted image of the current prediction block, and some pixels pred of the predicted image of the current prediction block generated by matrix operation.red to perform interpolation processing.

[0197] The interpolation unit 113 generates the remaining pixels (white pixels in the figure) of the predicted image by interpolation processing, and generates the pixels pred generated by matrix operation by interpolation processing. red Combined to generate the predicted image (pred) of the current prediction block.

[0198] However, in the interpolation process of JVET-N0217, as Figure 1 The above will be done by using the average pixel bdry of the reference image red The original pixel above in bdry top The average pixel bdry generated top red Used as the upper neighboring pixels of the predicted image.

[0199] On the other hand, in the interpolation process of the interpolation unit 113, the upper original pixel bdry of the reference image is used. top itself instead of the average pixel bdry top red As the upper neighboring pixels of the predicted image.

[0200] As described above, since the prediction image generation unit 110 uses the upper original pixel bdry of the reference image top As the upper adjacent pixel, so when using the average pixel as the upper adjacent pixel, there is no need to save the required average pixel bdry top red (pixel value). In addition, when the original pixel bdry above the reference image top When used as the upper neighboring pixel, it can be expected that the average pixel bdry top red Compared with the case of , the prediction accuracy of intra prediction is improved.

[0201] <Another Configuration Example of the Intra Prediction Unit 34>

[0202] Figure 10 : is a block diagram showing another configuration example of the intra prediction unit 34 .

[0203] Note that with Figure 7 similar, Figure 10 Only the predicted image generating unit 120 that generates the predicted image of the MIP in the intra prediction unit 34 is shown. The intra prediction unit 71 also has a predicted image generating unit similar to the predicted image generating unit 120.

[0204] In addition, in the accompanying drawings, Figure 7The corresponding parts of the predicted image generation unit 110 in FIG. 1 are denoted by the same reference numerals, and description thereof is appropriately omitted below.

[0205] The predicted image generation unit 120 includes an averaging unit 111 , a matrix-vector multiplication unit 112 , and an interpolation unit 123 .

[0206] Therefore, the predicted image generation unit 120 and Figure 7 The predicted image generating unit 110 is common to the predicted image generating unit 120 in that the predicted image generating unit 120 has an averaging unit 111 and a matrix-vector multiplication unit 112. However, the predicted image generating unit 120 is different from the predicted image generating unit 110 in that the predicted image generating unit 120 has an interpolation unit 123 instead of the interpolation unit 113.

[0207] The predicted image generation unit 120 provides the reference image and some pixels of the predicted image of the current prediction block generated by the matrix-vector multiplication unit 112 to the interpolation unit 123 , and provides the prediction mode k and the average pixel generated by the averaging unit 111 to the interpolation unit 123 .

[0208] Similar to the interpolation unit 113, the interpolation unit 123 performs interpolation processing by using the upper adjacent pixels and the left adjacent pixels of the predicted image and some pixels of the predicted image of the current prediction block from the matrix-vector multiplication unit 112, and generates a predicted image by combining the remaining pixels of the predicted image generated by the interpolation processing with some pixels of the predicted image from the matrix-vector multiplication unit 112.

[0209] However, the interpolation unit 123 may select whether to use the upper original pixel of the reference image pixel as the upper adjacent pixel or whether to use the average pixel as the downsampled pixel according to the prediction mode k. In addition, the interpolation unit 123 may select whether to use the left original pixel of the reference image pixel as the left adjacent pixel or whether to use the average pixel as the downsampled pixel according to the prediction mode k.

[0210] Figure 11 3 is a diagram illustrating a method of generating a predicted image of an MIP by the predicted image generating unit 120 .

[0211] and Figure 9 Similar, in Figure 11 In FIG, a block of W×H=8×8 pixels is used as a current prediction block. However, the prediction block is not limited to a block of W×H=8×8 pixels.

[0212] Similar to the prediction image generation unit 110, the prediction image generation unit 120 generates the image by averaging the upper original pixel bdry of the reference image in the averaging unit 111. top Averaging to generate the average pixel bdryred , and by averaging the original pixels bdry on the left side of the current prediction block in the averaging unit 111 left Averaging to generate the average pixel bdry red .

[0213] In addition, similar to the predicted image generation unit 110, the predicted image generation unit 120 calculates the expression pred red =A k ·bdry red +b k As a matrix operation to generate some pixels pred of the predicted image of the current prediction block red , this matrix operation will average the pixels bdry red As elements in the matrix-vector multiplication unit 112.

[0214] Then, similar to the predicted image generation unit 110, the predicted image generation unit 120 uses the upper adjacent pixels adjacent to the upper side of the predicted image of the current predicted block, the left adjacent pixels adjacent to the left side of the predicted image of the current predicted block, and some pixels pred of the predicted image of the current predicted block generated by matrix operation. red An interpolation process is performed to generate a predicted image (pred) of the current prediction block in the interpolation unit 123.

[0215] However, the interpolation unit 123 may select whether to use the original pixel bdry above the reference image pixel according to the prediction mode k. top Use as the upper neighbor pixel, or whether to average the pixels bdry top red In addition, the interpolation unit 123 can select whether to use the left original pixel bdry of the reference image pixel according to the prediction mode k. left Used as the left neighboring pixel, or whether to average the pixels bdry left red Used as downsampled pixels.

[0216] Average pixel bdry top red Is to use the original pixel bdry above top (By comparing the original pixel bdry above top Average pixel bdry generated by averaging red Average pixel. Average pixel bdry left red Is to use the original pixel bdry on the left left The resulting average pixel.

[0217] The interpolation unit 123 has selectors 131 and 132 .

[0218] The upper original pixel bdry is provided to the selector 131 top and average pixel bdry top red The selector 131 selects and outputs the upper original pixel bdry according to the prediction mode k top or average pixel bdry top red The interpolation unit 123 uses the upper original pixel bdry output by the selector 131 top and bdry as the average pixel of the adjacent pixels above top red The interpolation process is performed on the pixels in .

[0219] The left original pixel bdry is provided to the selector 132 left and average pixel bdry left red The selector 132 selects and outputs the left original pixel bdry according to the prediction mode k left or average pixel bdry left red The interpolation unit 123 uses the left original pixel bdry output by the selector 132 left and bdry as the average pixel of the left neighboring pixels left red The interpolation process is performed on the pixels in .

[0220] As described above, when the predicted image generation unit 120 can choose whether to use the upper original pixel of the reference image as the upper adjacent pixel or whether to use the average pixel and / or choose whether to use the left original pixel of the reference image as the left adjacent pixel or whether to use the average pixel, it can be expected that the prediction accuracy of the intra-frame prediction can be further improved.

[0221] Figure 12 is a diagram showing an example of pixels selected as upper neighboring pixels and left neighboring pixels according to prediction mode k.

[0222] Here, the upper adjacent pixel, the left adjacent pixel, or both the upper adjacent pixel and the left adjacent pixel are also referred to as adjacent pixels. In addition, the upper original pixel, the left original pixel, or both the upper original pixel and the left original pixel are also referred to as original pixels.

[0223] exist Figure 12 , when the prediction mode k includes information indicating (the mode number of) the MIP mode and the size identifier MipSizeId, pixels selected as neighboring pixels (above neighboring pixels and left neighboring pixels) according to the MIP mode and the size identifier MipSizeId are shown.

[0224] exist Figure 12 In the table, “Original” refers to the original pixels (the original pixels on the top and the original pixels on the left), and “Average” refers to the average pixels. Figure 12 As shown, for example, when the MIP mode is 1 and the size identifier MipSizeId is 0, the left original pixel (Original) is selected as the left adjacent pixel, and the average pixel (Average) is selected as the upper adjacent pixel.

[0225] The size identifier MipSizeId is the matrix A used in the MIP k The size identifier of the prediction mode k is set according to the block size of the current prediction block. Therefore, when the prediction mode k includes information indicating the size identifier MipSizeId, it can be said that the prediction mode k includes information indicating the block size of the current prediction block.

[0226] The prediction mode k may include information indicating a prediction direction of intra prediction, directional prediction, and non-directional prediction (eg, player prediction or DC prediction). The prediction direction of intra prediction includes a reference direction of directional prediction.

[0227] When the prediction mode k includes information indicating directional prediction, the original pixel or the average pixel may be selected as the adjacent pixel. For example, when the prediction mode k includes information indicating directional prediction, the original pixel or the average pixel may be selected as the adjacent pixel according to the reference direction of the directional prediction.

[0228] Specifically, for example, when a pattern such as a vertical edge exists in the current prediction block and the reference direction of directional prediction is close to the vertical direction, the upper original pixel can be selected as the upper neighboring pixel. Furthermore, for example, when a pattern such as a horizontal edge exists in the current prediction block and the reference direction of directional prediction is close to the horizontal direction, the left original pixel can be selected as the left neighboring pixel. In this case, prediction accuracy can be improved.

[0229] When the prediction mode k contains information indicating non-directional prediction, for example, when DC prediction is performed as intra prediction, average pixels may be selected as the upper neighboring pixel and the left neighboring pixel.

[0230] In addition, the selection of pixels as neighboring pixels adjacent to the predicted image may be performed not only according to the prediction mode k, but also depending on whether an in-loop filter is applied to the decoded image (local decoded image) as the reference image in the encoder 11 (that is, whether part or all of the deblocking filter 31a, the adaptive offset filter 41, and the ALF 42 have been applied). For example, simulation may be performed and pixels may be selected as neighboring pixels so that the cost is reduced depending on whether the in-loop filter is applied.

[0231] The intra-frame prediction unit 34 can function as a setting unit for setting identification data for identifying whether to use original pixels of a reference image as adjacent pixels adjacent to the predicted image or whether to use average pixels as downsampled pixels in generating a predicted image for intra-frame prediction. For example, the identification data set by the intra-frame prediction unit 34 can be included in the encoded bitstream as part of the encoding information.

[0232] The identification data may include data for identifying whether to use the upper original pixel of the reference image as the upper adjacent pixel or whether to use the average pixel as the downsampled pixel. In addition, the identification data may include data for identifying whether to use the left original pixel of the reference image as the left adjacent pixel or whether to use the average pixel as the downsampled pixel.

[0233] Note that whether to use the original pixel or the average pixel of the reference image as the downsampled pixel as the adjacent pixel is not selected based on the prediction mode, etc., the prediction mode, etc. are regarded as separate prediction modes, and for the original pixel and the adjacent pixel, one with a smaller cost can be selected.

[0234] <Description of Computer to Which the Present Technology is Applied>

[0235] Next, a series of processes of the encoder 11 and the decoder 51 described above can be executed by hardware or software. When a series of processes is executed by software, a program constituting the software is installed on a general-purpose computer or the like.

[0236] Figure 13 : is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0237] The program can be recorded in advance on the hard disk 905 or the ROM 903 as a recording medium built in the computer.

[0238] Alternatively, the program may be stored (recorded) in a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 may be provided as so-called packaged software. Here, examples of the removable recording medium 911 include a floppy disk, a compact disc read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a magnetic disk, a semiconductor memory, and the like.

[0239] Note that the program can be installed on the computer not only from the removable recording medium 911 as described above, but also downloaded to the computer via a communication network or a broadcast network and installed on the built-in hard disk 905. That is, for example, the program can be wirelessly transmitted from a download site to the computer via an artificial satellite for digital satellite broadcasting, or can be transmitted to the computer by wire via a network such as a local area network (LAN) or the Internet.

[0240] The computer has a built-in central processing unit (CPU) 902 , and an input / output interface 910 is connected to the CPU 902 via a bus 901 .

[0241] When a user inputs a command via the input / output interface 910 by operating the input unit 907 or the like, the CPU 902 accordingly executes a program stored in a read-only memory (ROM) 903. Alternatively, the CPU 902 loads a program stored in the hard disk 905 into a random access memory (RAM) 904 and executes the loaded program.

[0242] Thus, the CPU 902 executes the processing according to the above flowchart or the processing executed according to the configuration of the above block diagram. Then, the CPU 902 outputs the processing result from the output unit 906 from the communication unit 908, or transmits the processing result from the communication unit 908 via, for example, the input / output interface 910 as needed, and also records the processing result on the hard disk 905.

[0243] Note that the input unit 907 is composed of a keyboard, a mouse, a microphone, etc. In addition, the output unit 906 is composed of a liquid crystal display (LCD), a speaker, and the like.

[0244] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. That is, the processing performed by a computer according to a program also includes processing performed in parallel or individually (for example, parallel processing or processing performed by an object).

[0245] In addition, the program may be processed by one computer (processor), or may be distributed and processed by a plurality of computers. In addition, the program may be sent to a remote computer and executed.

[0246] In addition, in this specification, a system means a group of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, any of a plurality of devices housed in separate housings and connected via a network and a device in which a plurality of modules are housed in a single housing is a system.

[0247] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made without departing from the gist of the present technology.

[0248] For example, the present technology may be configured as cloud computing in which one function is shared by a plurality of devices via a network and is jointly processed.

[0249] Furthermore, each step described in the above flowchart may be performed by one device, and may also be shared and performed by a plurality of devices.

[0250] Furthermore, when one step includes a plurality of processes, the plurality of processes included in one step may be executed by one device or may be shared and executed by a plurality of devices.

[0251] In addition, the effects described in this specification are merely exemplary and non-limiting, and other effects may be obtained.

[0252] Reference Signs List

[0253] 10 Image Processing System

[0254] 11 Encoder

[0255] 21 A / D conversion unit

[0256] 22 Reorder Buffer 22

[0257] 23 computing units

[0258] 24 Orthogonal Transformation Unit

[0259] 25 Quantization Units

[0260] 26 reversible coding units

[0261] 27 Accumulation Buffer

[0262] 28 Inverse Quantization Unit

[0263] 29 Inverse Orthogonal Transformation Unit

[0264] 30 computing units

[0265] 31a, 31b Deblocking filter

[0266] 32 frame memories

[0267] 33 Select Units

[0268] 34 intra prediction units

[0269] 35 Motion prediction / compensation unit

[0270] 36 Prediction image selection unit

[0271] 37 Rate Control Unit

[0272] 41 Adaptive Offset Filter

[0273] 42 ALF

[0274] 51 Decoder

[0275] 61 Accumulation Buffer

[0276] 62 reversible decoding units

[0277] 63 Inverse quantization unit

[0278] 64 Inverse Orthogonal Transformation Units

[0279] 65 computing units

[0280] 67 Reorder Buffer

[0281] 68 D / A conversion units

[0282] 69 frame memory

[0283] 70 Select Units

[0284] 71 intra prediction unit

[0285] 72 Motion prediction / compensation unit

[0286] 73 Select Units

[0287] 81 Adaptive Offset Filter

[0288] 82 ALF

[0289] 110 Prediction image generation unit

[0290] 111 Average Unit

[0291] 112 matrix-vector multiplication units

[0292] 113 Interpolation Unit

[0293] 120 Prediction image generation unit

[0294] 123 interpolation unit

[0295] 901 Bus

[0296] 902 CPU

[0297] 903 ROM

[0298] 904 RAM

[0299] 905 Hard Drive

[0300] 906 Output Unit

[0301] 907 Input Unit

[0302] 908 Communication Unit

[0303] 909 Driver

[0304] 910 Input / Output Interface

[0305] 911 Removable Recording Media

Claims

1. An image processing device, comprising: an intra-frame prediction unit configured to, when performing intra-frame prediction using a matrix operation, generate a predicted image of the current prediction block by performing an interpolation process using original pixels of a reference image or downsampled pixels obtained by reducing the number of pixels of the reference image as upper adjacent pixels of the predicted image of the current prediction block to be encoded; as well as an encoding unit configured to encode the current prediction block using the prediction image generated by the intra prediction unit, The intra-frame prediction unit selects whether to use the original pixels of the reference image or the downsampled pixels as the upper adjacent pixels according to the prediction mode of the intra-frame prediction during the interpolation process.

2. The image processing apparatus according to claim 1, wherein: The intra prediction unit generates the predicted image by performing the interpolation process using original pixels of the reference image as left adjacent pixels adjacent to the left side of the predicted image of the current prediction block.

3. The image processing apparatus according to claim 1, wherein: The intra prediction unit selects whether to use original pixels of the reference image or the downsampled pixels as left neighboring pixels adjacent to the left side of the predicted image of the current prediction block according to a prediction mode of the intra prediction in the interpolation process.

4. The image processing apparatus according to claim 3, wherein: The prediction mode of the intra prediction includes information indicating a prediction direction of the intra prediction.

5. The image processing apparatus according to claim 4, wherein: The prediction mode of the intra prediction includes information indicating a block size of the current prediction block. The image processing apparatus according to claim 3 , wherein: The downsampled pixels are pixels obtained by averaging the original pixels.

7. The image processing apparatus according to claim 3, wherein: When the prediction mode of the intra prediction includes information indicating directional prediction, the intra prediction unit selects whether to use original pixels of the reference image or the downsampled pixels as neighboring pixels adjacent to the predicted image of the current prediction block.

8. The image processing apparatus according to claim 7, wherein: When the reference direction of the directional prediction is close to a longitudinal direction, the intra prediction unit uses the original pixel as the upper neighboring pixel.

9. The image processing apparatus according to claim 7, wherein: When a reference direction of the directional prediction is close to a horizontal direction, the intra prediction unit uses the original pixel as the left neighboring pixel.

10. The image processing apparatus according to claim 3, wherein: When the prediction mode of the intra prediction includes information indicating non-directional prediction, the intra prediction unit uses downsampled pixels of the reference image as the upper neighboring pixels. The image processing apparatus according to claim 10 , wherein: When the prediction mode of the intra prediction includes information indicating non-directional prediction, the intra prediction unit uses down-sampled pixels of the reference image as the left neighboring pixels.

12. The image processing apparatus according to claim 1, wherein: The intra prediction unit selects whether to use original pixels of the reference image or the downsampled pixels as the upper neighboring pixels according to whether an in-loop filter is applied to the local decoded image serving as the reference image.

13. The image processing apparatus according to claim 12, wherein: The intra prediction unit selects whether to use original pixels of the reference image or the downsampled pixels as left neighboring pixels adjacent to the left side of the predicted image of the current prediction block according to whether the in-loop filter is applied to the local decoded image.

14. The image processing apparatus according to claim 3, further comprising: A setting unit is configured to set identification data for identifying whether to use original pixels of the reference image or the downsampled pixels as adjacent pixels adjacent to the predicted image of the current prediction block.

15. The image processing apparatus according to claim 14, wherein: The identification data is data for identifying whether to use the original pixel of the reference image or the downsampled pixel as the upper adjacent pixel.

16. The image processing apparatus according to claim 14, wherein: The identification data is data for identifying whether to use the original pixel of the reference image or the downsampled pixel as the left adjacent pixel.

17. An image processing method, comprising: an intra-frame prediction process of generating a predicted image of a current prediction block to be encoded by performing an interpolation process using original pixels of a reference image or downsampled pixels obtained by reducing the number of pixels of the reference image as upper adjacent pixels of the predicted image of the current prediction block to be encoded when performing intra-frame prediction using a matrix operation; and an encoding process for encoding the current prediction block using the prediction image generated in the intra prediction process, The intra-frame prediction process further includes: selecting whether to use the original pixels of the reference image or the downsampled pixels as the upper adjacent pixels according to the prediction mode of the intra-frame prediction in the interpolation process.

18. An image processing apparatus, comprising: an intra-frame prediction unit configured to: generate a predicted image of the current prediction block to be decoded by performing an interpolation process using original pixels of a reference image or downsampled pixels obtained by reducing the number of pixels of the reference image as upper adjacent pixels of the predicted image of the current prediction block to be decoded when performing intra-frame prediction using a matrix operation; as well as a decoding unit configured to decode the current prediction block using the prediction image generated by the intra prediction unit, The intra-frame prediction unit selects whether to use the original pixels of the reference image or the downsampled pixels as the upper adjacent pixels according to the prediction mode of the intra-frame prediction during the interpolation process.

19. An image processing method, comprising: an intra-frame prediction process for generating a predicted image of a current prediction block to be decoded by performing an interpolation process using original pixels of a reference image or downsampled pixels obtained by reducing the number of pixels of the reference image when performing intra-frame prediction using a matrix operation; as well as a decoding process for decoding the current prediction block using the prediction image generated in the intra prediction process, The intra-frame prediction process further includes: selecting whether to use the original pixels of the reference image or the downsampled pixels as the upper adjacent pixels according to the prediction mode of the intra-frame prediction in the interpolation process.

Images