Image processing equipment and methods

Abstract

The invention relates to an image processing device and method that enables denoising of images and bit rates. The low-pass filter setter (93) sets, from the filter coefficients stored in the attached filter coefficient memory (94), the filter coefficient corresponding to the prediction mode information in the image and the quantization parameter. The neighbor image setter (81) uses the filter coefficient set by the low-pass filter setter (93) to process the filter of the neighbor pixel values ​​of the current block from the frame memory (72). The prediction image generator (82) performs prediction in the image using the filtered neighbor pixel values, from the neighbor image setter (81), and generates the prediction image. For example, the invention is applicable to an image encoding device to perform encoding using the H.264 / AVC format.

Description

[0001] The present invention relates to an image processing apparatus and method, and more particularly to an image processing apparatus and method that can achieve noise reduction according to the image and bitrate and improve prediction efficiency.

[0002] In recent years, devices that handle image information digitally and compress images using encoding methods that utilize the redundancy inherent in image information and compress it through orthogonal transformations such as discrete cosine transforms and motion compensation, with the aim of efficient information transmission and storage, are becoming widespread. Examples of such encoding methods include MPEG (Moving Picture Experts Group).

[0003] In particular, MPEG2 (ISO / IEC 13818-2) is defined as a general-purpose image encoding scheme and is a standard that covers both interleaved and sequential scanned images, as well as standard-resolution and high-resolution images. For example, MPEG2 is currently widely used in a wide range of professional and consumer applications. By using the MPEG2 compression scheme, for example, a standard-resolution interleaved scanned image with 720 x 480 pixels can be assigned a code amount (bitrate) of 4 to 8 Mbps. Also, by using the MPEG2 compression scheme, for example, a high-resolution interleaved scanned image with 1920 x 1088 pixels can be assigned a code amount (bitrate) of 18 to 22 Mbps. This makes it possible to achieve high compression ratio and good image quality.

[0004] MPEG2 was primarily intended for high-quality encoding suitable for broadcasting, but it did not support encoding methods with lower bitrates, i.e., higher compression ratios, than MPEG1. With the spread of mobile devices, the need for such encoding methods is expected to increase in the future, and in response to this, the MPEG4 encoding method was standardized. Regarding the image encoding method, its standard was approved as an international standard in December 1998 as ISO / IEC 14496-2.

[0005] Furthermore, in recent years, standardization has been underway for a standard called H.26L (ITU-T Q6 / 16 VCEG), initially intended for video encoding for video conferencing. Compared to conventional encoding methods such as MPEG2 and MPEG4, H.26L requires more computation for encoding and decoding, but it is known to achieve higher encoding efficiency. Currently, as part of the MPEG4 activities, standardization is being carried out as the Joint Model of Enhanced-Compression Video Coding, which is based on H.26L but incorporates features not supported by H.26L to achieve even higher encoding efficiency. In terms of the standardization schedule, H.264 and MPEG-4 Part 10 (Advanced Video Coding, hereinafter referred to as H.264 / AVC) became international standards in March 2003.

[0006] Furthermore, as an extension of this, the standardization of FRExt (Fidelity Range Extension), which includes encoding tools necessary for professional use such as RGB, 4:2:2, and 4:4:4, as well as the 8x8 DCT and quantization matrix specified in MPEG-2, was completed in February 2005. As a result, H.264 / AVC has become an encoding method that can accurately represent film noise contained in movies, and has come to be used in a wide range of applications such as Blu-ray Discs (trademark).

[0007] However, in recent years, there has been a growing need for even higher compression ratios, such as the ability to compress images with four times the resolution of high-definition images, to about 4000 x 2000 pixels. Alternatively, there is a growing need for even higher compression ratios, such as the ability to deliver high-definition images in environments with limited transmission capacity, like the internet. For this reason, the aforementioned VCEG (=Video Coding Expert Group) under the ITU-T is continuing to study ways to improve coding efficiency.

[0008] One of the factors that enables the H.264 / AVC method to achieve higher encoding efficiency compared to conventional methods such as MPEG2 is the adoption of an intra prediction method.

[0009] In the intra prediction method, for the luminance signal, nine types of intra prediction modes for 4x4 pixel and 8x8 pixel block units, and four types of intra prediction modes for 16x16 pixel macroblock units are defined. For the color difference signal, four types of intra prediction modes for 8x8 pixel block units are defined. The intra prediction mode for the color difference signal can be set independently of the intra prediction mode for the luminance signal.

[0010] There is a specific pattern in how the residuals after intra prediction are generated for each intra prediction mode.

[0011] As a method to eliminate such redundancy and further improve encoding efficiency, Non-Patent Document 1 proposes the following method.

[0012] In other words, in advance offline processing, the training signal is used to encode the intra image using the normal H.264 / AVC method, and for each intra prediction mode for each block, an orthogonal transformation such as the Carunen-Lobe transform is performed to calculate the optimal transformation coefficient.

[0013] Then, in the actual encoding process, instead of the orthogonal transform defined in the H.264 / AVC scheme, processing is performed using orthogonal transform coefficients optimized for each mode by the Carunen-Loewe transform described above.

[0014] Non-patent document 2 also proposes a method of combining the above-mentioned intra prediction with inter prediction.

[0015] In other words, in Non-Patent Literature 2, difference information is generated not only for the target block but also for the adjacent pixel values ​​surrounding the target block, based on the motion vector information obtained in the interpretation. Second-order difference information is generated by performing an intraprediction between the difference information about the target block and the difference information about adjacent pixels that is thus generated. The generated second-order difference information is then orthogonally transformed and quantized, and output to the next stage along with the compressed image.

[0016] This further improves the encoding efficiency.

[0017] Furthermore, as mentioned above, in the H.264 / AVC format, the macroblock size is 16 x 16 pixels. However, setting the macroblock size to 16 x 16 pixels is not optimal for large image sizes such as UHD (Ultra High Definition; 4000 x 2000 pixels), which are the target of next-generation encoding formats.

[0018] In this regard, Non-Patent Document 3 and others have proposed expanding the macroblock size to a size such as 32 x 32 pixels.

[0019] ''Improved Intra Coding'',VCEG-AF15,ITU-Telecommunications Standardization Sector STUDY GROUP Question 6 Video coding Experts Group(VCEG), 20-21 April 2007''Second Order Prediction (SOP) in P Slice'', Sijia Chen, JinpengWang, Shangwen Li and, Lu Yu ,VCEG-AD09,ITU-Telecommunications Standardization Sector STUDY GROUP Question 6 Video coding Experts Group(VCEG), 16-18 July 2008''Video Coding Using Extended Block Sizes'',VCEG-AD09,ITU-Telecommunications Standardization Sector STUDY GROUP Question 16 - Contribution 123, Jan 2009

[0020] In the H.264 / AVC scheme, prior to performing the 8x8 pixel block-based intra prediction described above, the pixel values ​​of adjacent pixels are subjected to low-pass filtering. This removes noise contained in adjacent pixels and improves correlation, thereby enabling higher coding efficiency.

[0021] However, although the degree of noise varies depending on the input image, quantization parameter values, and intra-prediction mode, the H.264 / AVC format uses a fixed low-pass filter for noise removal. In other words, this low-pass filter was not optimized according to the input image, quantization parameter values, intra-prediction mode, etc.

[0022] Furthermore, in the H.264 / AVC system, the noise reduction of adjacent pixel values ​​described above was only performed in the intra-prediction mode for 8x8 pixel blocks and was not applied to other modes.

[0023] The present invention has been made in view of these circumstances, and aims to achieve noise reduction according to the image and bitrate, thereby improving prediction efficiency.

[0024] An image processing apparatus according to a first aspect of the present invention comprises: a decoding means for decoding an image of a target block to be decoded; a filter setting means for setting coefficients used for filtering applied to adjacent pixels of the target block, which are used for intra-prediction of the target block, according to encoding parameters, corresponding to the target block; and an intra-prediction means for performing the filtering on the adjacent pixels using the coefficients set by the filter setting means, and performing intra-prediction of the target block.

[0025] The coding parameter includes the intra-prediction mode of the target block or the quantization parameter of the target block, the decoding means decodes the intra-prediction mode of the target block or the quantization parameter of the target block, and the filter setting means can set the coefficient according to the intra-prediction mode or the quantization parameter decoded by the decoding means.

[0026] The system further comprises a filter coefficient storage means for storing the coefficients, wherein the coefficients are calculated in such a way that the residual between the learning block to be encoded and the predicted image obtained by performing intra-prediction on the learning block in accordance with the intra-prediction mode of the learning block or the quantization parameter of the learning block is minimized, and the coefficients are stored in the filter coefficient storage means, and the filter setting means can set a coefficient from among the coefficients stored in the filter coefficient storage means that corresponds to the intra-prediction mode of the target block or the quantization parameter of the target block as the coefficient.

[0027] The filter coefficient storage means can store the coefficients as n-bit values ​​(where n is an integer) depending on the register length of the processor.

[0028] The decoding means decodes the coefficients calculated on the encoding side such that the residual between the predicted image obtained by performing intra-prediction on the target block using the adjacent pixels before the filtering process is applied is minimized, and the coefficients are set in correspondence with the target block, the intra-prediction mode of the target block, or the quantization parameter of the target block. The filter setting means can set the coefficients corresponding to the intra-prediction mode of the target block or the quantization parameter of the target block from among the coefficients decoded by the decoding means.

[0029] The coefficient can consist of a filter coefficient and an offset value.

[0030] The filter coefficient can be configured with three taps.

[0031] The filter coefficients can have symmetry around the coefficient for the zero phase.

[0032] The decoding means decodes the intra-prediction mode of the target block, and the filter setting means sets the coefficient corresponding to the first class if the intra-prediction mode of the target block belongs to the first class, with the vertical mode or horizontal mode being a first class and the other modes being a second class, and sets the coefficient corresponding to the second class if the decoded intra-prediction mode of the target block belongs to the second class.

[0033] The filter setting means further includes a receiving means for receiving flag information indicating whether to perform the filter processing on the adjacent pixels, and the filter setting means can set whether to perform the filter processing on the adjacent pixels based on the flag information received by the receiving means.

[0034] The receiving means receives the flag information on a macroblock basis, and the filter setting means can set whether to perform the filter processing on the adjacent pixels on a macroblock basis based on the flag information received by the receiving means.

[0035] The receiving means receives the flag information in block units, and the filter setting means can set whether to perform the filter processing on adjacent pixels in block units based on the flag information received by the receiving means.

[0036] A first aspect of the present invention is an image processing method which includes the steps of an image processing device decoding an image of a target block to be decoded, setting coefficients used for filtering applied to adjacent pixels of the target block, which are used for intra-prediction of the target block, according to the encoding parameters, applying the filtering to the adjacent pixels using the set coefficients, and performing intra-prediction of the target block.

[0037] An image processing apparatus according to a second aspect of the present invention includes a filter setting means for setting coefficients used for filtering applied to adjacent pixels of a target block, which are used for intra-prediction of the target block, according to the encoding parameters, for a target block that is the target of the encoding process; an intra-prediction means for performing the filtering process on the adjacent pixels using the coefficients set by the filter setting means to perform intra-prediction of the target block; and an encoding means for encoding an image of the target block.

[0038] The encoding parameter includes the intra-prediction mode of the target block or the quantization parameter of the target block, the filter setting means sets the coefficient according to the intra-prediction mode of the target block or the quantization parameter of the target block, and the encoding means can encode the corresponding intra-prediction mode of the target block or the quantization parameter of the target block.

[0039] The system further comprises a filter coefficient storage means for storing the coefficients, wherein the coefficients are calculated in a learning image such that the residual between the learning block to be encoded and the predicted image obtained by performing intra-prediction on the learning block in accordance with the intra-prediction mode of the learning block or the quantization parameter of the learning block is minimized, and the coefficients are stored in the filter coefficient storage means, and the filter setting means can set as the coefficients from among the coefficients stored in the filter coefficient storage means that correspond to the intra-prediction mode of the target block or the quantization parameter of the target block.

[0040] The filter coefficient calculation means further comprises a filter coefficient calculation means that calculates the coefficients such that the residual between the target block and the predicted image obtained by performing intra-prediction on the target block is minimized, using the adjacent pixels before the filtering process is applied in correspondence with the target block and the intra-prediction mode of the target block or the quantization parameter of the target block, the filter setting means sets the coefficients that correspond to the intra-prediction mode of the target block or the quantization parameter of the target block from among the coefficients calculated by the filter coefficient calculation means as the coefficients, and the encoding means can further encode the coefficients.

[0041] The coefficient can consist of a filter coefficient and an offset value.

[0042] The filter setting means can set whether to perform the filter processing on the adjacent pixels, and the encoding means can encode flag information indicating whether to perform the filter processing set by the filter setting means.

[0043] A second aspect of the present invention is an image processing method which includes the steps of an image processing device setting coefficients used for filtering applied to adjacent pixels of a target block, which are used for intra-prediction of the target block, according to the encoding parameters, applying the filtering to the adjacent pixels using the set coefficients, performing intra-prediction of the target block, and encoding an image of the target block.

[0044] In a first aspect of the present invention, an image of a target block to be decoded is decoded, and coefficients used for filtering applied to adjacent pixels of the target block, which are used for intra-prediction of the target block, are set according to the encoding parameters. Then, the filtering is applied to the adjacent pixels using the set coefficients, and intra-prediction of the target block is performed.

[0045] In a second aspect of the present invention, coefficients used for filtering applied to adjacent pixels of a target block, which is used for intra-prediction of the target block, are set according to the encoding parameters. Then, the filtering is applied to the adjacent pixels using the set coefficients, intra-prediction of the target block is performed, and the image of the target block is encoded.

[0046] Each of the above-mentioned image processing devices may be an independent device, or it may be an internal block that constitutes a single image encoding device or image decoding device.

[0047] According to the first aspect of the present invention, an image can be decoded. Furthermore, according to the second aspect of the present invention, noise reduction can be performed according to the image and bitrate.

[0048] According to the second aspect of the present invention, an image can be encoded. Furthermore, according to the first aspect of the present invention, noise reduction can be performed according to the image and bitrate.

[0049] This is a block diagram showing the configuration of one embodiment of an image encoding device to which the present invention is applied. This diagram illustrates the processing order in the case of a 16x16 pixel intra-prediction mode. This diagram shows the types of intra-prediction modes for 4x4 pixels of luminance signals. This diagram shows the types of intra-prediction modes for 4x4 pixels of luminance signals. This diagram illustrates the direction of intra-prediction for 4x4 pixels. This diagram illustrates intra-prediction for 4x4 pixels. This diagram illustrates the coding of the intra-prediction mode for 4x4 pixels of luminance signals. This diagram shows the types of intra-prediction modes for 8x8 pixels of luminance signals. This diagram shows the types of intra-prediction modes for 8x8 pixels of luminance signals. This diagram shows the types of intra-prediction modes for 16x16 pixels of luminance signals. This diagram illustrates intra-prediction for 16x16 pixels. This diagram shows the types of intra-prediction modes for chrominance signals. This is a block diagram showing an example configuration of the intra-prediction unit and the adjacent pixel interpolation filter switching unit of Figure 1. This diagram illustrates the calculation of filter coefficients. This is a flowchart illustrating the encoding process of the image encoding device of Figure 1. This is a flowchart illustrating the prediction process in step S21 of Figure 16. This is a flowchart illustrating the intra-prediction process in step S31 of Figure 17. This is a flowchart illustrating the inter-motion prediction process in step S32 of Figure 17. This is a block diagram showing another configuration example of the intra-prediction unit and adjacent pixel interpolation filter switching unit in Figure 1. This is a flowchart illustrating another example of the intra-prediction process in step S31 of Figure 17. This is a block diagram showing the configuration of one embodiment of an image decoding device to which the present invention is applied. This is a block diagram showing an example of the configuration of the intra-prediction unit and adjacent pixel interpolation filter switching unit in Figure 22. This is a flowchart illustrating the decoding process of the image decoding device in Figure 22. This is a flowchart illustrating the prediction process in step S138 of Figure 24. This is a block diagram showing another configuration example of the intra-prediction unit and adjacent pixel interpolation filter switching unit in Figure 22. This is a flowchart illustrating another example of the prediction process in step S138 of Figure 24. This is a block diagram showing the configuration of one embodiment of a learning device to which the present invention is applied. This is a block diagram showing an example of the configuration of the intra-prediction unit and adjacent pixel interpolation filter calculation unit in Figure 28.It is a flowchart for explaining the intra prediction process by the learning device of FIG. 28. It is a block diagram showing the configuration of another embodiment of the image encoding device to which the present invention is applied. It is a diagram for explaining the secondary prediction process. It is a block diagram showing the configuration of another embodiment of the image decoding device to which the present invention is applied. It is a block diagram showing the configuration of still another embodiment of the image encoding device to which the present invention is applied. It is a block diagram showing a configuration example of the intra prediction unit of FIG. 34. It is a flowchart for explaining another example of the intra prediction process in step S31 of FIG. 17. It is a flowchart for explaining still another example of the intra prediction process in step S31 of FIG. 17. It is a flowchart for explaining another example of the intra prediction process in step S31 of FIG. 17. It is a block diagram showing the configuration of still another embodiment of the image decoding device to which the present invention is applied. It is a block diagram showing a configuration example of the intra prediction unit and the adjacent pixel interpolation filter control unit of FIG. 39. It is a flowchart for explaining still another example of the prediction process in step S138 of FIG. 24. It is a block diagram showing the configuration of another embodiment of the image encoding device to which the present invention is applied. It is a block diagram showing the configuration of another embodiment of the image decoding device to which the present invention is applied. It is a diagram showing an example of an extended block size. It is a block diagram showing a configuration example of the hardware of a computer. It is a block diagram showing a main configuration example of a television receiver to which the present invention is applied. It is a block diagram showing a main configuration example of a mobile phone to which the present invention is applied. It is a block diagram showing a main configuration example of a hard disk recorder to which the present invention is applied. It is a block diagram showing a main configuration example of a camera to which the present invention is applied.

[0050] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The description will be made in the following order. 1. First Embodiment (Adjacent Pixel Interpolation Filter Switching: Example of Intra Prediction) 2. Second Embodiment (Adjacent Pixel Interpolation Filter Switching: Example of Secondary Prediction) 3. Third Embodiment (Adjacent Pixel Interpolation Filter On / Off Control: Example of Intra Prediction) 4. Fourth Embodiment (Adjacent Pixel Interpolation Filter On / Off Control: Example of Secondary Prediction)

[0051] <1. First Embodiment> [Configuration Example of Image Encoding Device] (0000096) Fig. 1 shows the configuration of an embodiment of an image encoding device as an image processing device to which the present invention is applied. (0052) This image encoding device 51 compresses and encodes an image, for example, in the H.264 and MPEG-4 Part10 (Advanced Video Coding) (hereinafter referred to as H.264 / AVC) format. (0053) In the example of Fig. 1, the image encoding device 51 includes an A / D conversion unit 61, a screen rearrangement buffer 62, an arithmetic unit 63, an orthogonal conversion unit 64, a quantization unit 65, a reversible encoding unit 66, an accumulation buffer 67, an inverse quantization unit 68, an inverse orthogonal conversion unit 69, an arithmetic unit 70, a deblocking filter 71, a frame memory 72, a switch 73, an intra prediction unit 74, an adjacent pixel interpolation filter switching unit 75, a motion prediction / compensation unit 76, a predicted image selection unit 77, and a rate control unit 78. (0054) The A / D conversion unit 61 performs A / D conversion on the input image and outputs and stores it in the screen rearrangement buffer 62. The screen rearrangement buffer 62 rearranges the stored images of the frames in the display order according to the GOP (Group of Picture) into the order of the frames for encoding. (0055) The arithmetic unit 63 subtracts the predicted image from the intra prediction unit 74 or the predicted image from the motion prediction / compensation unit 76 selected by the predicted image selection unit 77 from the image read from the screen rearrangement buffer 62, and outputs the difference information to the orthogonal conversion unit 64. The orthogonal conversion unit 64 performs an orthogonal conversion such as a discrete cosine transform or a Karunen - Loeve transform on the difference information from the arithmetic unit 63 and outputs the conversion coefficient. The quantization unit 65 quantizes the conversion coefficient output by the orthogonal conversion unit 64. (0056) The quantized conversion coefficient, which is the output of the quantization unit 65, is input to the reversible encoding unit 66, where reversible encoding such as variable - length encoding or arithmetic encoding is performed and compressed.

[0057] The reversible coding unit 66 obtains information indicating intra prediction from the intra prediction unit 74 and information indicating inter prediction mode from the motion prediction / compensation unit 76. The information indicating intra prediction will also be referred to as intra prediction mode information below. The information indicating inter prediction mode will also be referred to as inter prediction mode information below.

[0058] The reversible encoding unit 66 encodes the quantized conversion coefficients, as well as information indicating intra-prediction, information indicating inter-prediction mode, and quantization parameters, and incorporates them as part of the header information in the compressed image. The reversible encoding unit 66 supplies the encoded data to the storage buffer 67 for storage.

[0059] For example, the reversible coding unit 66 performs reversible coding processing such as variable-length coding or arithmetic coding. Examples of variable-length coding include CAVLC (Context-Adaptive Variable Length Coding) as defined in the H.264 / AVC scheme. Examples of arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).

[0060] The storage buffer 67 outputs the data supplied from the reversible encoding unit 66 as a compressed image encoded in the H.264 / AVC format to, for example, a subsequent recording device or transmission line (not shown).

[0061] The quantized conversion coefficients output from the quantization unit 65 are also input to the inverse quantization unit 68, where they are inversely quantized and then further inversely orthogonal transformed in the inverse orthogonal transformation unit 69. The output of the inversely orthogonal transformation is added by the calculation unit 70 to the predicted image supplied from the predicted image selection unit 77 to obtain a locally decoded image. The deblocking filter 71 removes block distortion from the decoded image and then supplies it to the frame memory 72 for storage. The frame memory 72 also receives and stores the image before it is deblocked by the deblocking filter 71.

[0062] The switch 73 outputs the reference image stored in the frame memory 72 to the motion prediction and compensation unit 76 or the intra prediction unit 74.

[0063] In this image encoding device 51, for example, the I-picture, B-picture, and P-picture from the screen sorting buffer 62 are supplied to the intra-prediction unit 74 as images to be intra-predicted (also referred to as intra-processing). Also, the B-picture and P-picture read from the screen sorting buffer 62 are supplied to the motion prediction and compensation unit 76 as images to be inter-predicted (also referred to as inter-processing).

[0064] The intra prediction unit 74 performs intra prediction processing for all candidate intra prediction modes based on the intra prediction image read from the screen sorting buffer 62 and the reference image supplied from the frame memory 72, and generates a predicted image.

[0065] Prior to this intra prediction processing, the intra prediction unit 74 applies a filter to adjacent pixels that are used for intra prediction of each target block and are adjacent to the target block in a predetermined positional relationship. For this filter processing, filter coefficients set by the adjacent pixel interpolation filter switching unit 75 are used, corresponding to the intra prediction mode supplied by the intra prediction unit 74. That is, in the intra prediction unit 74, for the intra prediction processing of all candidate intra prediction modes, adjacent pixels that have been filtered with filter coefficients set by the adjacent pixel interpolation filter switching unit 75 are used.

[0066] The intra prediction unit 74 calculates a cost function value for the intra prediction mode that generated the predicted image, and selects the intra prediction mode that gives the minimum calculated cost function value as the optimal intra prediction mode. The intra prediction unit 74 supplies the predicted image generated in the optimal intra prediction mode and the cost function value calculated for the corresponding optimal intra prediction mode to the predicted image selection unit 77.

[0067] When the prediction image selection unit 77 selects a prediction image generated in the optimal intra prediction mode, the intra prediction unit 74 supplies information indicating the optimal intra prediction mode to the reversible encoding unit 66. When the reversible encoding unit 66 receives information from the intra prediction unit 74, it encodes this information and uses it as part of the header information in the compressed image.

[0068] The adjacent pixel interpolation filter switching unit 75 stores the filter coefficients corresponding to the quantization parameters and intra-prediction mode, which are obtained by learning using the training image in the learning device 251 shown in Figure 28, which will be described later.

[0069] The adjacent pixel interpolation filter switching unit 75 is supplied with quantization parameters from the rate control unit 78 and intra-prediction mode information from the intra-prediction unit 74. The adjacent pixel interpolation filter switching unit 75 sets filter coefficients corresponding to the quantization parameters from the rate control unit 78 and the intra-prediction mode from the intra-prediction unit 74. The adjacent pixel interpolation filter switching unit 75 supplies the set filter coefficients to the intra-prediction unit 74.

[0070] In addition, in the adjacent pixel interpolation filter switching unit 75, it is also possible to learn and store the filter coefficients so that they correspond to either the quantization parameters or the intra prediction mode, rather than both.

[0071] In addition, the adjacent pixel interpolation filter switching unit 75 stores filter coefficients that have been learned offline in advance, but it is also possible to calculate the filter coefficients online instead. In this case, in order to send them to the decoding side, the filter coefficients set by the adjacent pixel interpolation filter switching unit 75 are output to the reversible coding unit 66, as shown by the dotted arrow.

[0072] The motion prediction and compensation unit 76 performs motion prediction and compensation processing for all candidate interpretation modes. That is, the motion prediction and compensation unit 76 is supplied with the image to be interpreted read from the screen sorting buffer 62 and a reference image from the frame memory 72 via the switch 73. Based on the image to be interpreted and the reference image, the motion prediction and compensation unit 76 detects motion vectors for all candidate interpretation modes, performs compensation processing on the reference image based on the motion vectors, and generates a predicted image.

[0073] The motion prediction and compensation unit 76 also calculates cost function values ​​for all candidate inter prediction modes. The motion prediction and compensation unit 76 determines the prediction mode that gives the minimum value among the calculated cost function values ​​as the optimal inter prediction mode.

[0074] The motion prediction and compensation unit 76 supplies the predicted image and its cost function value, generated in the optimal interpretation mode, to the predicted image selection unit 77. If the predicted image selection unit 77 selects a predicted image generated in the optimal interpretation mode, the motion prediction and compensation unit 76 outputs information indicating the optimal interpretation mode (interpretation mode information) to the reversible coding unit 66.

[0075] If necessary, motion vector information, flag information, and reference frame information are also output to the reversible coding unit 66. The reversible coding unit 66 also performs reversible coding processing on the information from the motion prediction and compensation unit 76, such as variable-length coding and arithmetic coding, and inserts it into the header portion of the compressed image.

[0076] The prediction image selection unit 77 determines the optimal prediction mode from the optimal intra prediction mode and the optimal inter prediction mode based on the cost function values ​​output from the intra prediction unit 74 or the motion prediction / compensation unit 76. The prediction image selection unit 77 then selects the prediction image of the determined optimal prediction mode and supplies it to the calculation units 63 and 70. At this time, the prediction image selection unit 77 supplies the prediction image selection information to the intra prediction unit 74 or the motion prediction / compensation unit 76.

[0077] The rate control unit 78 controls the rate of the quantization operation of the quantization unit 65 by means of quantization parameters, based on the compressed image stored in the storage buffer 67, so as to prevent overflow or underflow.

[0078] The quantization parameters used to control the rate of the quantization unit 65 are supplied to the reversible coding unit 66, where they are reversibly coded and inserted into the header portion of the compressed image. These quantization parameters are also supplied to the adjacent pixel interpolation filter switching unit 75 and used to set the filter coefficients used for filtering applied to adjacent pixels.

[0079] [H. Explanation of intra-prediction processing in the 264 / AVC method] First, we will explain the various intra-prediction modes defined in the H.264 / AVC scheme.

[0080] First, the intra-prediction modes for the luminance signal will be explained. Three types of intra-prediction modes for the luminance signal are defined: intra-4x4 prediction mode, intra-8x8 prediction mode, and intra-16x16 prediction mode. These are modes that define block units and are set for each macroblock. In addition, for the color difference signal, it is possible to set an intra-prediction mode for each macroblock that is independent of the luminance signal.

[0081] Furthermore, in the intra 4x4 prediction mode, one prediction mode can be set from nine types of prediction modes for each 4x4 pixel target block. In the intra 8x8 prediction mode, one prediction mode can be set from nine types of prediction modes for each 8x8 pixel target block. Also, in the intra 16x16 prediction mode, one prediction mode can be set from four types of prediction modes for each 16x16 pixel target macroblock.

[0082] Hereinafter, the intra 4x4 prediction mode, intra 8x8 prediction mode, and intra 16x16 prediction mode will also be appropriately referred to as the 4x4 pixel intra prediction mode, the 8x8 pixel intra prediction mode, and the 16x16 pixel intra prediction mode, respectively.

[0083] In the example in Figure 2, the numbers -1 to 25 assigned to each block represent the bitstream order (processing order on the decoding side) of that block. For the luminance signal, the macroblock is divided into 4x4 pixels, and 4x4 pixel DCT is performed. Then, only in the intra 16x16 prediction mode, as shown in block -1, the DC components of each block are collected to generate a 4x4 matrix, and an orthogonal transform is further applied to this matrix.

[0084] On the other hand, with respect to the color difference signal, the macroblock is divided into 4x4 pixels, and after 4x4 pixel DCT is performed, the DC components of each block are collected as shown in blocks 16 and 17 to generate a 2x2 matrix, which is then subjected to an orthogonal transformation.

[0085] This applies to the intra 8x8 prediction mode only when an 8x8 orthogonal transformation is applied to the target macroblock with a high profile or higher profile.

[0086] Figures 3 and 4 show the intra-prediction modes (Intra_4x4_pred_mode) for 4x4 pixels of nine types of luminance signals. The eight modes other than mode 2, which shows mean value (DC) prediction, correspond to the directions indicated by numbers 0, 1, 3 to 8 in Figure 5.

[0087] The nine types of Intra_4x4_pred_mode will be explained with reference to Figure 6. In the example in Figure 6, pixels a to p represent pixels of the target block to be processed intra, and pixel values ​​A to M represent the pixel values ​​of pixels belonging to adjacent blocks. That is, pixels a to p are the image to be processed read from the screen sorting buffer 62, and pixel values ​​A to M are the pixel values ​​of the decoded image read from the frame memory 72 and referenced.

[0088] In each intra prediction mode shown in Figures 3 and 4, the predicted pixel values ​​of pixels a to p are generated using the pixel values ​​A to M of pixels belonging to adjacent blocks, as follows. Note that a pixel value being "available" means that it is available for use, without any reason such as being at the edge of the frame or not yet encoded. Conversely, a pixel value being "unavailable" means that it is not available for use, for reasons such as being at the edge of the frame or not yet encoded.

[0089] Mode 0 is Vertical Prediction mode and is applied only when pixel values ​​A to D are ''available''. In this case, the predicted pixel values ​​of pixels a to p are generated as shown in the following equation (1). Predicted pixel values ​​for pixels a, e, i, and m = A Predicted pixel values ​​for pixels b, f, j, and n = B Predicted pixel values ​​for pixels c, g, k, and o = C Predicted pixel values ​​for pixels d, h, l, and p = D ... (1)

[0090] Mode 1 is Horizontal Prediction mode and is applied only when pixel values ​​I to L are ''available''. In this case, the predicted pixel values ​​of pixels a to p are generated as shown in the following equation (2). Predicted pixel values ​​for pixels a, b, c, and d = I Predicted pixel values ​​for pixels e, f, g, and h = J Predicted pixel values ​​for pixels i, j, k, l = K Predicted pixel values ​​for pixels m, n, o, and p = L ... (2)

[0091] Mode 2 is DC Prediction mode, and when pixel values ​​A, B, C, D, I, J, K, L are all ''available'', the predicted pixel values ​​are generated as shown in equation (3). (A+B+C+D+I+J+K+L+4) >> 3...(3)

[0092] Also, when pixel values ​​A, B, C, and D are all ''unavailable'', the predicted pixel values ​​are generated as shown in equation (4). (I+J+K+L+2) >> 2...(4)

[0093] Also, when pixel values ​​I, J, K, and L are all ''unavailable'', the predicted pixel values ​​are generated as shown in equation (5). (A+B+C+D+2) >> 2...(5)

[0094] When all pixel values ​​A, B, C, D, I, J, K, and L are "unavailable", 128 is used as the predicted pixel value.

[0095] Mode 3 is Diagonal_Down_Left Prediction mode and is applied only when pixel values ​​A, B, C, D, I, J, K, L, M are ''available''. In this case, the predicted pixel values ​​of pixels a to p are generated as shown in the following equation (6). Predicted pixel value of pixel a = (A + 2B + C + 2) >> 2 Predicted pixel values ​​for pixels b and e = (B + 2C + D + 2) >> 2 Predicted pixel values ​​for pixels c, f, and i = (C + 2D + E + 2) >> 2 Predicted pixel values ​​for pixels d, g, j, and m = (D + 2E + F + 2) >> 2 Predicted pixel values ​​for pixels h, k, and n = (E + 2F + G + 2) >> 2 Predicted pixel values ​​for pixels l and o = (F + 2G + H + 2) >> 2 Predicted pixel value of pixel p = (G + 3H + 2) >> 2 ... (6)

[0096] Mode 4 is Diagonal_Down_Right Prediction mode and is applied only when pixel values ​​A, B, C, D, I, J, K, L, M are ''available''. In this case, the predicted pixel values ​​for pixels a to p are generated as shown in the following equation (7). Predicted pixel value of pixel m = (J + 2K + L + 2) >> 2 Predicted pixel values ​​for pixels i and n = (I + 2J + K + 2) >> 2 Predicted pixel values ​​for pixels e, j, and o = (M + 2I + J + 2) >> 2 Predicted pixel values ​​for pixels a, f, k, and p = (A + 2M + I + 2) >> 2 Predicted pixel values ​​for pixels b, g, and l = (M + 2A + B + 2) >> 2 Predicted pixel values ​​for pixels c and h = (A + 2B + C + 2) >> 2 Predicted pixel value of pixel d = (B + 2C + D + 2) >> 2 ... (7)

[0097] Mode 5 is Diagonal_Vertical_Right Prediction mode and is applied only when pixel values ​​A, B, C, D, I, J, K, L, M are ''available''. In this case, the predicted pixel values ​​of pixels a to p are generated as shown in the following equation (8). Predicted pixel values ​​for pixels a and j = (M + A + 1) >> 1 Predicted pixel values ​​for pixels b and k = (A + B + 1) >> 1 Predicted pixel values ​​for pixels c and l = (B + C + 1) >> 1 Predicted pixel value of pixel d = (C + D + 1) >> 1 Predicted pixel values ​​for pixels e and n = (I + 2M + A + 2) >> 2 Predicted pixel values ​​for pixels f and o = (M + 2A + B + 2) >> 2 Predicted pixel values ​​for pixels g and p = (A + 2B + C + 2) >> 2 Predicted pixel value of pixel h = (B + 2C + D + 2) >> 2 Predicted pixel value of pixel i = (M + 2I + J + 2) >> 2 Predicted pixel value of pixel m = (I + 2J + K + 2) >> 2 ... (8)

[0098] Mode 6 is Horizontal_Down Prediction mode and is applied only when pixel values ​​A, B, C, D, I, J, K, L, M are ''available''. In this case, the predicted pixel values ​​of pixels a to p are generated as shown in the following equation (9). Predicted pixel values ​​for pixels a and g = (M + I + 1) >> 1 Predicted pixel values ​​for pixels b and h = (I + 2M + A + 2) >> 2 Predicted pixel value of pixel c = (M + 2A + B + 2) >> 2 Predicted pixel value of pixel d = (A + 2B + C + 2) >> 2 Predicted pixel values ​​for pixels e and k = (I + J + 1) >> 1 Predicted pixel values ​​for pixels f and l = (M + 2I + J + 2) >> 2 Predicted pixel values ​​for pixels i and o = (J + K + 1) >> 1 Predicted pixel values ​​for pixels j and p = (I + 2J + K + 2) >> 2 Predicted pixel value of pixel m = (K + L + 1) >> 1 Predicted pixel value of pixel n = (J + 2K + L + 2) >> 2 ... (9)

[0099] Mode 7 is Vertical_Left Prediction mode and is applied only when pixel values ​​A, B, C, D, I, J, K, L, M are ''available''. In this case, the predicted pixel values ​​of pixels a to p are generated as shown in the following equation (10). Predicted pixel value of pixel a = (A + B + 1) >> 1 Predicted pixel values ​​for pixels b and i = (B + C + 1) >> 1 Predicted pixel values ​​for pixels c and j = (C + D + 1) >> 1 Predicted pixel values ​​for pixels d and k = (D + E + 1) >> 1 Predicted pixel value of pixel l = (E + F + 1) >> 1 Predicted pixel value of pixel e = (A + 2B + C + 2) >> 2 Predicted pixel values ​​for pixels f and m = (B + 2C + D + 2) >> 2 Predicted pixel values ​​for pixels g and n = (C + 2D + E + 2) >> 2 Predicted pixel values ​​for pixels h and o = (D + 2E + F + 2) >> 2 Predicted pixel value of pixel p = (E + 2F + G + 2) >> 2 ... (10)

[0100] Mode 8 is Horizontal_Up Prediction mode and is applied only when pixel values ​​A, B, C, D, I, J, K, L, M are ''available''. In this case, the predicted pixel values ​​for pixels a through p are generated as shown in equation (11) below. Predicted pixel value of pixel a = (I + J + 1) >> 1 Predicted pixel value of pixel b = (I + 2J + K + 2) >> 2 Predicted pixel values ​​for pixels c and e = (J + K + 1) >> 1 Predicted pixel values ​​for pixels d and f = (J + 2K + L + 2) >> 2 Predicted pixel values ​​for pixels g and i = (K + L + 1) >> 1 Predicted pixel values ​​for pixels h and j = (K + 3L + 2) >> 2 Predicted pixel values ​​for pixels k, l, m, n, o, p = L ... (11)

[0101] Next, with reference to Figure 7, the coding method for the 4x4 pixel intra-prediction mode (Intra_4x4_pred_mode) of the luminance signal will be described. In the example in Figure 7, a target block C consisting of 4x4 pixels to be coded is shown, and blocks A and B, consisting of 4x4 pixels adjacent to target block C, are also shown.

[0102] In this case, it is considered that there is a high correlation between the Intra_4x4_pred_mode in target block C and the Intra_4x4_pred_mode in blocks A and B. By using this correlation and performing the encoding process as follows, higher encoding efficiency can be achieved.

[0103] In other words, in the example of Figure 7, the Intra_4x4_pred_mode in block A and block B are defined as Intra_4x4_pred_modeA and Intra_4x4_pred_modeB, respectively, and MostProbableMode is defined as the following equation (12). MostProbableMode=Min(Intra_4x4_pred_modeA, Intra_4x4_pred_modeB) ... (12)

[0104] In other words, the one of Block A and Block B that is assigned the smaller mode_number is set as MostProbableMode.

[0105] Two values, prev_intra4x4_pred_mode_flag[luma4x4BlkIdx] and rem_intra4x4_pred_mode[luma4x4BlkIdx], are defined in the bitstream as parameters for the target block C. Decryption is performed by processing based on the pseudocode shown in the following equation (13), and the values ​​of Intra_4x4_pred_mode and Intra4x4PredMode[luma4x4BlkIdx] for the target block C can be obtained.

[0106] if(prev_intra4x4_pred_mode_flag[luma4x4BlkIdx]) Intra4x4PredMode[luma4x4BlkIdx] = MostProbableMode else if(rem_intra4x4_pred_mode[luma4x4BlkIdx] < MostProbableMode) Intra4x4PredMode[luma4x4BlkIdx]=rem_intra4x4_pred_mode[luma4x4BlkIdx] else Intra4x4PredMode[luma4x4BlkIdx]=rem_intra4x4_pred_mode[luma4x4BlkIdx] + 1 ... (13)

[0107] Next, we will explain the 8x8 pixel intra prediction mode. Figures 8 and 9 show the 8x8 pixel intra prediction modes (Intra_8x8_pred_mode) for nine types of luminance signals.

[0108] Let p[x,y] (0≦x≦7; 0≦y≦7) be the pixel value in the target 8x8 block, and let p[-1,-1],…,p[-1,15],p[-1,0],…,[p-1,7] be the pixel values ​​of adjacent blocks.

[0109] For the 8x8 pixel intra prediction mode, low-pass filtering is applied to adjacent pixels before generating the predicted values. Here, the pixel values ​​before low-pass filtering are denoted as p[-1,-1],…,p[-1,15],p[-1,0],…p[-1,7], and the pixel values ​​after processing are denoted as p'[-1,-1],…,p'[-1,15],p'[-1,0],…p'[-1,7].

[0110] First, p'[0,-1] is calculated as shown in equation (14) below if p[-1,-1] is ''available'', and as shown in equation (15) below if it is ''not available''. p'[0,-1] = (p[-1,-1] + 2*p[0,-1] + p[1,-1] + 2) >> 2 ... (14) p'[0,-1] = (3*p[0,-1] + p[1,-1] + 2) >> 2 ... (15)

[0111] p'[x,-1] (x=0,…,7) is calculated as shown in equation (16) below. p'[x,-1] = (p[x-1,-1] + 2*p[x,-1] + p[x+1,-1] + 2) >>2 ... (16)

[0112] p'[x,-1] (x=8,…,15) is calculated as shown in equation (17) below if p[x,-1] (x=8,…,15) is ''available''. p'[x,-1] = (p[x-1,-1] + 2*p[x,-1] + p[x+1,-1] + 2) >>2 p'[15,-1] = (p[14,-1] + 3*p[15,-1] + 2) >>2 ... (17)

[0113] p'[-1,-1] is calculated as follows when p[-1,-1] is available. That is, when both p[0,-1] and p[-1,0] are available, p'[-1,-1] is calculated as shown in equation (18), and when p[-1,0] is not available, it is calculated as shown in equation (19). Also, when p[0,-1] is not available, p'[-1,-1] is calculated as shown in equation (20). p'[-1,-1] = (p[0,-1] + 2*p[-1,-1] + p[-1,0] + 2) >>2 ... (18) p'[-1,-1] = (3*p[-1,-1] + p[0,-1] + 2) >>2 ... (19) p'[-1,-1] = (3*p[-1,-1] + p[-1,0] + 2) >>2 ... (20)

[0114] p'[-1,y] (y=0, … ,7) is calculated as follows when p[-1,y] (y=0, … ,7) is ''available''. That is, first, p'[-1,0] is calculated as shown in equation (21) below when p[-1,-1] is ''available'', and as shown in equation (22) when it is ''unavailable''. p'[-1,0] = (p[-1,-1] + 2*p[-1,0] + p[-1,1] + 2) >>2 ... (21) p'[-1,0] = (3*p[-1,0] + p[-1,1] + 2) >>2 ... (22)

[0115] Also, p'[-1,y] (y=1,...,6) is calculated as shown in equation (23), and p'[-1,7] is calculated as shown in equation (24). p[-1,y] = (p[-1,y-1] + 2*p[-1,y] + p[-1,y+1] + 2) >>2 ... (23) p'[-1,7] = (p[-1,6] + 3*p[-1,7] + 2) >>2 ... (24)

[0116] Using the p' calculated in this way, the predicted values ​​for each intra prediction mode shown in Figures 8 and 9 are generated as follows.

[0117] Mode 0 is Vertical Prediction mode and is applied only when p[x,-1] (x=0, … ,7) is ''available''. Predicted value pred8x8 L [x,y] is generated as shown in equation (25) below. pred8x8 L[x,y] = p'[x,-1] x,y=0,...,7 ... (25)

[0118] Mode 1 is the Horizontal Prediction mode and is applied only when p[-1,y] (y=0, … ,7) is ''available''. Predicted value pred8x8 L [x,y] is generated as shown in equation (26) below. pred8x8 L [x,y] = p'[-1,y] x,y=0,...,7 ... (26)

[0119] Mode 2 is DC Prediction mode, and the predicted value is pred8x8 L [x,y] is generated as follows: That is, if both p[x,-1] (x=0, … ,7) and p[-1,y] (y=0, … ,7) are ''available'', the predicted value pred8x8 L [x,y] is generated as shown in equation (27) below.

[0120] If p[x,-1] (x=0, … ,7) is ''available'' but p[-1,y] (y=0, … ,7) is ''unavailable'', then the predicted value pred8x8 L [x,y] is generated as shown in equation (28) below.

[0121] If p[x,-1] (x=0, … ,7) is ''unavailable'' but p[-1,y] (y=0, … ,7) is ''available'', then the predicted value pred8x8 L [x,y] is generated as shown in equation (29) below. When both p[x, -1] (x = 0, …, 7) and p[-1, y] (y = 0, …, 7) are ''unavailable'', the predicted value pred8x8 L [x, y] is generated as in the following equation (30). pred8x8 L [x, y] = 128 ・・・(30) However, equation (30) represents the case of 8-bit input.

[0123] Mode 3 is the Diagonal_Down_Left_prediction mode, and the predicted value pred8x8 L [x, y] is generated as follows. That is, the Diagonal_Down_Left_prediction mode is applied only when p[x, -1], x = 0, …, 15 are ''available''. The predicted pixel value where x = 7 and y = 7 is generated as in the following equation (31), and the other predicted pixel values are generated as in the following equation (32). pred8x8 L [x, y] = (p'[14, -1] + 3 * p[15, -1] + 2) >> 2 ・・・(31) red8x8 L [x, y] = (p'[x + y, -1] + 2 * p'[x + y + 1, -1] + p'[x + y + 2, -1] + 2) >> 2 ・・・(32)

[0124] Mode 4 is the Diagonal_Down_Right_prediction mode, and the predicted value pred8x8 L[x,y] is generated as follows. That is, the Diagonal_Down_Right_prediction mode is applied only when p[x,-1],x=0,...,7 and p[-1,y],y=0,...,7 are ''available'', and predicted pixel values ​​where x > y are generated as shown in equation (33), and predicted pixel values ​​where x < y are generated as shown in equation (34). Also, predicted pixel values ​​where x = y are generated as shown in equation (35). pred8x8 L [x,y] = (p'[xy-2,-1] + 2*p'[xy-1,-1] + p'[xy,-1] + 2) >> 2 ... (33) pred8x8 L [x,y] = (p'[-1,yx-2] + 2*p'[-1,yx-1] + p'[-1,yx] + 2) >> 2 ... (34) pred8x8 L [x,y] = (p'[0,-1] + 2*p'[-1,-1] + p'[-1,0] + 2) >> 2 ... (35)

[0125] Mode 5 is Vertical_Right_prediction mode, and the predicted value is pred8x8 L [x,y] is generated as follows. That is, Vertical_Right_prediction mode is applied only when p[x,-1],x=0,...,7 and p[-1,y],y=-1,...,7 are ''available''. Now, we define zVR as shown in equation (36) below. zVR = 2*x - y ... (36)

[0126] In this case, when zVR is 0, 2, 4, 6, 8, 10, 12, 14, the pixel prediction value is generated as shown in equation (37), and when zVR is 1, 3, 5, 7, 9, 11, 13, the pixel prediction value is generated as shown in equation (38). pred8x8 L [x,y] = (p'[x-(y>>1)-1,-1] + p'[x-(y>>1),-1] + 1) >> 1 ... (37) pred8x8 L [x,y] = (p'[x-(y>>1)-2,-1] + 2*p'[x-(y>>1)-1,-1] + p'[x-(y>>1),-1] + 2) >> 2 ... (38)

[0127] Furthermore, when zVR is -1, the pixel prediction value is generated as shown in equation (39), and in all other cases, i.e., when zVR is -2, -3, -4, -5, -6, -7, the pixel prediction value is generated as shown in equation (40). pred8x8 L [x,y] = (p'[-1,0] + 2*p'[-1,-1] + p'[0,-1] + 2) >> 2 ... (39) pred8x8 L [x,y] = (p'[-1,y-2*x-1] + 2*p'[-1,y-2*x-2] + p'[-1,y-2*x-3] + 2) >> 2 ... (40)

[0128] Mode 6 is Horizontal_Down_prediction mode, and the predicted value is pred8x8 L[x,y] is generated as follows. That is, the Horizontal_Down_prediction mode is applied only when p[x,-1],x=0,...,7 and p[-1,y],y=-1,...,7 are ''available''. Now, let's define zVR as shown in equation (41) below. zHD = 2*y - x ... (41)

[0129] In this case, when zHD is 0, 2, 4, 6, 8, 10, 12, or 14, the predicted pixel value is generated as shown in equation (42), and when zHD is 1, 3, 5, 7, 9, 11, or 13, the predicted pixel value is generated as shown in equation (43). pred8x8 L [x,y] = (p'[-1,y-(x>>1)-1] + p'[-1,y-(x>>1) + 1] >> 1 ... (42) pred8x8 L [x,y] = (p'[-1,y-(x>>1)-2] + 2*p'[-1,y-(x>>1)-1] + p'[-1,y-(x>>1)] + 2) >> 2 ... (43)

[0130] Furthermore, when zHD is -1, the predicted pixel value is generated as shown in equation (44), and when zHD is any other value, i.e., -2, -3, -4, -5, -6, -7, the predicted pixel value is generated as shown in equation (45). pred8x8 L [x,y] = (p'[-1,0] + 2*p[-1,-1] + p'[0,-1] + 2) >> 2 ... (44) pred8x8 L[x,y] = (p'[x-2*y-1,-1] + 2*p'[x-2*y-2,-1] + p'[x-2*y-3,-1] + 2) >> 2 ... (45)

[0131] Mode 7 is Vertical_Left_prediction mode, and the predicted value is pred8x8 L [x,y] is generated as follows. That is, Vertical_Left_prediction mode is applied only when p[x,-1], x=0,...,15 is ''available'', and when y=0,2,4,6 the predicted pixel values ​​are generated as shown in equation (46), and in all other cases, i.e., when y=1,3,5,7 the predicted pixel values ​​are generated as shown in equation (47). pred8x8 L [x,y] = (p'[x+(y>>1),-1] + p'[x+(y>>1)+1,-1] + 1) >> 1 ... (46) pred8x8 L [x,y] = (p'[x+(y>>1),-1] + 2*p'[x+(y>>1)+1,-1] + p'[x+(y>>1)+2,-1] + 2) >> 2 ... (47)

[0132] Mode 8 is Horizontal_Up_prediction mode, and the predicted value is pred8x8 L [x,y] is generated as follows. That is, the Horizontal_Up_prediction mode is applied only when p[-1,y], y=0,...,7 is ''available''. Below, zHU is defined as shown in equation (48). zHU = x + 2*y ... (48)

[0133] When the zHU value is 0, 2, 4, 6, 8, 10, or 12, the predicted pixel value is generated as shown in equation (49) below, and when the zHU value is 1, 3, 5, 7, 9, or 11, the predicted pixel value is generated as shown in equation (50) below. pred8x8 L [x,y] = (p'[-1,y+(x>>1)] + p'[-1,y+(x>>1)+1] + 1) >> 1 ... (49) pred8x8 L [x,y] = (p'[-1,y+(x>>1)] ... (50)

[0134] Furthermore, when the value of zHU is 13, the predicted pixel value is generated as shown in equation (51), and otherwise, i.e., when the value of zHU is greater than 13, the predicted pixel value is generated as shown in equation (52). pred8x8 L [x,y] = (p'[-1,6] + 3*p'[-1,7] + 2) >> 2 ... (51) pred8x8 L [x,y] = p'[-1,7] ... (52)

[0135] Next, we will explain the 16x16 pixel intra prediction mode. Figures 10 and 11 show the 16x16 pixel intra prediction mode (Intra_16x16_pred_mode) for four types of luminance signals.

[0136] The four types of intra prediction modes will be explained with reference to Figure 12. In the example in Figure 12, the target macroblock A to be processed intra is shown, and P(x,y);x,y=-1,0,...,15 represents the pixel values ​​of pixels adjacent to the target macroblock A.

[0137] Mode 0 is Vertical Prediction mode and is applied only when P(x,-1); x,y=-1,0,…,15 is ''available''. In this case, the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in equation (53) below. Pred(x,y) = P(x,-1);x,y=0,…,15 ... (53)

[0138] Mode 1 is Horizontal Prediction mode and is applied only when P(-1,y); x,y=-1,0,…,15 is ''available''. In this case, the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in the following equation (54). Pred(x,y) = P(-1,y);x,y=0,…,15 ... (54)

[0139] Mode 2 is DC Prediction mode, and if P(x,-1) and P(-1,y); x,y=-1,0,...,15 are all ''available'', the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in equation (55) below.

[0140] Also, if P(x,-1); x,y=-1,0,…,15 is ''unavailable'', the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in the following equation (56).

[0141] P(-1,y); If x,y=-1,0,...,15 is ''unavailable'', the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in the following equation (57).

[0142] P(x,-1) and P(-1,y); If x,y=-1,0,...,15 are all ''unavailable'', then 128 is used as the predicted pixel value.

[0143] Mode 3 is Plane Prediction mode and is applied only when P(x,-1) and P(-1,y); x,y=-1,0,...,15 are all ''available''. In this case, the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in the following equation (58).

[0144] Next, the intra-prediction modes for color difference signals will be explained. Figure 13 shows the intra-prediction modes (Intra_chroma_pred_mode) for four types of color difference signals. The intra-prediction mode for color difference signals can be set independently of the intra-prediction mode for luminance signals. The intra-prediction mode for color difference signals is the same as the 16x16 pixel intra-prediction mode for luminance signals described above.

[0145] However, while the 16x16 pixel intra-prediction mode for the luminance signal targets 16x16 pixel blocks, the intra-prediction mode for the chrominance signal targets 8x8 pixel blocks. Furthermore, as shown in Figures 10 and 13 above, the mode numbers do not correspond between the two.

[0146] Here, referring to Figure 12, we follow the definitions of the pixel values ​​of the target macroblock A and adjacent pixel values ​​for the 16x16 pixel intra prediction mode of the luminance signal described above. For example, let P(x,y); x,y=-1,0,…,7 be the pixel values ​​of the pixels adjacent to the target macroblock A (8x8 pixels in the case of a color difference signal) that is processed intra-.

[0147] Mode 0 is DC Prediction mode, and if P(x,-1) and P(-1,y); x,y=-1,0,...,7 are all ''available'', the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in equation (59) below.

[0148] Also, if P(-1,y); x,y=-1,0,…,7 is ''unavailable'', the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in the following equation (60).

[0149] Also, if P(x,-1); x,y=-1,0,…,7 is ''unavailable'', the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in equation (61) below.

[0150] Mode 1 is Horizontal Prediction mode and is applied only when P(-1,y); x,y=-1,0,…,7 is ''available''. In this case, the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in equation (62) below. Pred(x,y) = P(-1,y);x,y=0,…,7 ... (62)

[0151] Mode 2 is Vertical Prediction mode and is applied only when P(x,-1); x,y=-1,0,…,7 is ''available''. In this case, the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in equation (63) below. Pred(x,y) = P(x,-1);x,y=0,…,7 ... (63)

[0152] Mode 3 is Plane Prediction mode and is applied only when P(x,-1) and P(-1,y) ; x,y=-1,0,…,7 are ''available''. In this case, the predicted pixel value Pred(x,y) for each pixel of the target macroblock A is generated as shown in equation (64) below.

[0153] As described above, the intra-prediction modes for the luminance signal include nine types of 4x4 pixel and 8x8 pixel block unit prediction modes, as well as four types of 16x16 pixel macroblock unit prediction modes. These block unit modes are set for each macroblock unit. The intra-prediction modes for the color difference signal include four types of 8x8 pixel block unit prediction modes. These intra-prediction modes for the color difference signal can be set independently of the intra-prediction modes for the luminance signal.

[0154] In addition, for the 4x4 pixel intra prediction mode (intra 4x4 prediction mode) and the 8x8 pixel intra prediction mode (intra 8x8 prediction mode) of the luminance signal, one intra prediction mode is set for each block of 4x4 pixels and 8x8 pixels of the luminance signal. For the 16x16 pixel intra prediction mode (intra 16x16 prediction mode) of the luminance signal and the intra prediction mode of the chrominance signal, one prediction mode is set for each macroblock.

[0155] The types of prediction modes correspond to the directions indicated by numbers 0, 1, 3 to 8 in Figure 5 above. Prediction mode 2 is average value prediction.

[0156] As described above, in intra prediction in the H.264 / AVC system, as shown in equations (14) to (24), filtering of the pixel values ​​of adjacent pixels is performed with predetermined filter coefficients only before intra prediction is performed in 8x8 pixel block units. In contrast, in the image encoding device 51, filtering of the pixel values ​​of adjacent pixels is performed with filter coefficients set according to the block to be predicted before intra prediction is performed in all intra prediction modes.

[0157] [Example of configuration of the intra prediction unit and adjacent pixel interpolation filter switching unit] Figure 14 is a block diagram showing a detailed configuration example of the intra prediction unit 74 and the adjacent pixel interpolation filter switching unit 75 of Figure 1.

[0158] In the example shown in Figure 14, the intra prediction unit 74 is composed of an adjacent image setting unit 81, a prediction image generation unit 82, and an optimal prediction mode determination unit 83.

[0159] The adjacent pixel interpolation filter switching unit 75 is composed of a prediction mode buffer 91, a quantization parameter buffer 92, and a low-pass filter setting unit 93. The low-pass filter setting unit 93 has a built-in filter coefficient memory 94.

[0160] The adjacent image setting unit 81 is supplied with adjacent pixel values ​​of the target block for intra prediction from the frame memory 72. In the case of Figure 14, the switch 73 is not shown, but in reality, the adjacent pixel values ​​are supplied from the frame memory 72 to the adjacent image setting unit 81 via the switch 73. In the case of intra prediction, the adjacent pixel values ​​used are the pixel values ​​that have not been deblocked by the deblocking filter 71.

[0161] The adjacent pixel setting unit 81 uses the filter coefficients set by the low-pass filter setting unit 93 to apply a filter to the adjacent pixel values ​​of the target block from the frame memory 72, and supplies the filtered adjacent pixel values ​​to the prediction image generation unit 82.

[0162] The prediction image generation unit 82 supplies information to the prediction mode buffer 91 indicating which intra prediction mode is currently being processed. The prediction image generation unit 82 uses the filtered adjacent pixel values ​​from the adjacent pixel setting unit 81 to perform intra prediction of the target block in the intra prediction mode supplied to the prediction mode buffer 91 and generates a prediction image. The generated prediction image, along with the intra prediction mode information, is supplied to the optimal prediction mode determination unit 83.

[0163] The optimal prediction mode determination unit 83 is supplied with the intra-prediction image read from the screen sorting buffer 62, the prediction image generated by the prediction image generation unit 82, and its intra-prediction mode information.

[0164] The optimal prediction mode determination unit 83 uses the supplied information to calculate a cost function value for the intra prediction mode that generated the prediction image, and determines the intra prediction mode that gives the minimum calculated cost function value as the optimal intra prediction mode. The optimal prediction mode determination unit 83 outputs the prediction image of the optimal intra prediction mode and the corresponding cost function value to the prediction image selection unit 77.

[0165] In addition, if the prediction image selected by the prediction image selection unit 77 is selected by the optimal prediction mode determination unit 83, the unit 83 supplies information indicating the optimal intra prediction mode to the reversible coding unit 66.

[0166] The prediction mode buffer 91 stores intra-prediction mode information from the prediction image generation unit 82. The quantization parameter buffer 92 stores quantization parameters from the rate control unit 78.

[0167] The low-pass filter setting unit 93 reads intra-prediction mode information of the target block from the prediction mode buffer 91 and reads quantization parameters for the target block from the quantization parameter buffer 92. The low-pass filter setting unit 93 sets filter coefficients corresponding to this information from among the filter coefficients stored in the built-in filter coefficient memory 94 and supplies the set filter coefficients to the adjacent pixel setting unit 81.

[0168] The filter coefficient memory 94 stores the filter coefficients corresponding to the quantization parameters and intra-prediction mode, which are obtained by learning using the training images in the learning device 251 shown in Figure 28, which will be described later. These filter coefficients are calculated and stored for each slice, for example, as described below.

[0169] [Explanation of Calculation of Optimal Filter Coefficients] Next, referring to Figure 15, we will explain how to calculate the optimal filter coefficients used for filtering adjacent pixels. Note that while Figure 15 shows an example of vertical prediction for a 4x4 pixel block, the following explanation is applicable to any intra-prediction mode.

[0170] In the 8x8 pixel block-based intra prediction described above in equations (14) to (24), a three-tap filter coefficient {1,2,1} is defined as a low-pass filter for adjacent pixels. However, we will consider {c0,c1,c2} as a general form of the three taps. Furthermore, in this invention, a fourth parameter c3 is also introduced as an offset value.

[0171] In the following explanation, it is assumed that this 3-tap filter can be set for each slice, but it is not limited to this, and for example, it may be set for the entire sequence or for each GOP.

[0172] In the example in Figure 15, a km (0≦k,m≦3) is the pixel value of a pixel included in the target block, and bm(-1≦m≦4) is the pixel value of an adjacent pixel used for vertical prediction.

[0173] First, the adjacent pixel value b m By applying a three-tap filter, the following equation (65) is generated: b'm(0≦m≦3). In other words, when filtering is performed and filter coefficients are used, the corresponding offset value is also used, as shown in equation (65), unless otherwise specified. In other words, the filter coefficient and the offset value are the coefficients used in filtering. Similarly, when the filter coefficient is encoded and sent to the decoding side, the corresponding offset value is also encoded and sent.

[0174] Now, when the intra prediction mode is n, the predicted pixel value is p ij (b' m Assuming that 0≦i,j≦3, the intra-predicted pixel values ​​are, as described above with reference to Figures 2 to 14, since the predicted pixels are generated by a linear equation, the following equation (66) holds.

[0175] At this time, a ij Using the original image pixel values, the squared prediction error for the target block Ω is expressed by the following equation (67).

[0176] Here, if we denote the set of intrablocks encoded as intraprediction mode n within the target slice as Φ, the sum of the squared prediction errors for the blocks belonging to Φ is expressed by the following equation (68).

[0177] In equation (68) above, Err(Ω∈Φ) is considered a function of c0, c1, c2, c3, i.e., Err(Ω∈Φ; c0, c1, c2, c3), and the values ​​of c0, c1, c2, c3 that minimize Err(Ω∈Φ; c0, c1, c2, c3) are the optimal filter coefficient values ​​for the target slice. In other words, we just need to find c0, c1, c2, c3 such that the following equation (69) holds.

[0178] In other words, equation (69) yields the system of equations shown in equation (70).

[0179] Using matrices, equation (70) can be rewritten as equation (71).

[0180] By solving equation (71), the optimal filter coefficients and offset values ​​{c0, c1, c2, c3} for the target slice can be determined.

[0181] The optimal filter coefficients and offset values ​​{c0, c1, c2, c3} are obtained as floating-point values ​​by solving the system of equations in equation (70). However, when applied to the image encoding device 51 in Figure 1 and the corresponding image decoding device 151 in Figure 22, they are rounded to, for example, 8-bit coefficients.

[0182] In other words, even if the filter coefficients are floating-point numbers, they are stored in the filter coefficient memory 94, for example, as n-bit values ​​(where n is an integer) depending on the register length of the processor.

[0183] By applying the same method as described above to other intra prediction modes, the optimal filter coefficients can be determined for other intra prediction modes as well. Furthermore, the same method can be used to determine the optimal filter coefficients not only for the intra 4x4 prediction mode, but also for the intra 8x8 prediction mode, the intra 16x16 prediction mode, and the intra prediction mode for color difference signals.

[0184] In the above explanation, one filter coefficient is calculated for each intra prediction mode, but this is not limited to this, and one filter coefficient may be calculated for all intra prediction modes. In particular, the intra prediction modes described above with reference to Figures 2 to 14 have different characteristics because the Vertical and Horizontal modes use the predicted pixel values ​​as they are, while the other modes undergo some kind of averaging or weighted averaging process to generate the predicted pixels. For this reason, by classifying the modes into two classes, Vertical and Horizontal modes and other modes, and calculating a filter coefficient for each class, further improvements in coding efficiency can be achieved. Also, for example, for the luminance signal, one filter coefficient may be calculated for the intra 4x4 prediction mode, one for the intra 8x8 prediction mode, and one for the intra 16x16 prediction mode. For the chrominance signal, for example, separate filter coefficients may be calculated for Cb and Cr.

[0185] In the above explanation, we used three taps {c0, c1, c2} as the filter coefficients for low-pass filtering, but you can use any number of taps in the filter, not just three. That is, the filter coefficients plus the offset value for the number of taps can be calculated. However, the degree of the system of equations to be solved increases with the number of taps.

[0186] Furthermore, different filter coefficients can be prepared depending on the frame size, such as CIF (Common Intermediate Format) / QCIF (Quarter CIF), SD (Standard Definition), HD (High Definition), etc., and these can be applied.

[0187] In the method described above, the filter coefficients are calculated by minimizing the intra-prediction residual (prediction squared error). However, the method for calculating the filter coefficients is not limited to this, and for example, if it is necessary to send the filter coefficients to the decoding side, optimization may be performed to include bits for sending those filter coefficients.

[0188] Furthermore, in the filter coefficients described above, we assume symmetry of the coefficients as shown in the following equation (72). C0 = C2 ... (72)

[0189] In other words, the filter coefficients are calculated such that they have symmetry around the coefficient for the zero phase, as in {c0, c1, c0}. This makes it possible to reduce the three simultaneous equations shown in equation (70) above to two. As a result, the amount of computation can be reduced.

[0190] By using the above method, it becomes possible to set appropriate filter coefficients for the input image and adaptively perform low-pass filtering on adjacent pixels, thereby enabling encoding using a predicted image suitable for the image, quantization parameters, and prediction mode. This improves encoding efficiency.

[0191] Two methods are possible for calculating the optimal filter coefficients as described above. The first method is an offline process in which, prior to performing the encoding process, filter coefficients that optimize the entire image signal are calculated in advance using the image signal for training. This offline learning process will be described later with reference to Figure 28, and the filter coefficients and offset values ​​calculated by this learning process are stored in the filter coefficient memory 94 in Figure 14.

[0192] The second method is an online process that sequentially calculates the optimal filter coefficients for each slice. In this case, the calculated filter coefficients and offset values ​​are sent to the decoding side. An example of the second method, the online process, will be described later with reference to Figure 20.

[0193] [Explanation of the encoding process of the image encoding device] Next, the encoding process of the image encoding device 51 in Figure 1 will be explained with reference to the flowchart in Figure 16.

[0194] In step S11, the A / D conversion unit 61 performs A / D conversion on the input image. In step S12, the screen sorting buffer 62 stores the images supplied by the A / D conversion unit 61 and sorts the pictures from the display order to the encoding order.

[0195] In step S13, the calculation unit 63 calculates the difference between the images rearranged in step S12 and the predicted images. The predicted images are supplied to the calculation unit 63 via the predicted image selection unit 77 from the motion prediction and compensation unit 76 in the case of interpretation, and from the intraprediction unit 74 in the case of intraprediction.

[0196] The difference data is smaller in size compared to the original image data. Therefore, the amount of data can be compressed compared to encoding the image as is.

[0197] In step S14, the orthogonal transformation unit 64 performs an orthogonal transformation on the difference information supplied from the calculation unit 63. Specifically, an orthogonal transformation such as a discrete cosine transform or a Carunen-Loebe transform is performed, and the transformation coefficients are output. In step S15, the quantization unit 65 quantizes the transformation coefficients. During this quantization, the rate is controlled as will be explained in the processing of step S25 described later.

[0198] The quantized difference information is then locally decoded as follows. That is, in step S16, the inverse quantization unit 68 inversely quantizes the conversion coefficients quantized by the quantization unit 65 using characteristics corresponding to the characteristics of the quantization unit 65. In step S17, the inverse orthogonal transformation unit 69 inversely orthogonal transforms the conversion coefficients inversely quantized by the inverse quantization unit 68 using characteristics corresponding to the characteristics of the orthogonal transformation unit 64.

[0199] In step S18, the calculation unit 70 adds the predicted image input via the predicted image selection unit 77 to the locally decoded difference information to generate a locally decoded image (an image corresponding to the input to the calculation unit 63). In step S19, the deblocking filter 71 filters the image output from the calculation unit 70. This removes block distortion. In step S20, the frame memory 72 stores the filtered image. Images that have not been filtered by the deblocking filter 71 are also supplied from the calculation unit 70 and stored in the frame memory 72.

[0200] In step S21, the intra prediction unit 74 and the motion prediction / compensation unit 76 each perform image prediction processing. That is, in step S21, the intra prediction unit 74 performs intra prediction processing in intra prediction mode. The motion prediction / compensation unit 76 performs motion prediction / compensation processing in inter prediction mode.

[0201] Details of the prediction process in step S21 will be described later with reference to Figure 17, but in this process, prediction processing is performed for all candidate prediction modes and cost function values ​​are calculated for all candidate prediction modes. Based on the calculated cost function values, the optimal intra prediction mode is selected, and the predicted image generated by the intra prediction of the optimal intra prediction mode and its cost function value are supplied to the predicted image selection unit 77.

[0202] At this time, the intra prediction unit 74 performs filtering on adjacent pixels used for intra prediction of each target block using filter coefficients set by the adjacent pixel interpolation filter switching unit 75, prior to the intra prediction process. Then, the intra prediction unit 74 performs intra prediction using the filtered adjacent pixels to generate a predicted image.

[0203] In step S22, the prediction image selection unit 77 determines one of the optimal intra prediction mode and the optimal inter prediction mode as the optimal prediction mode based on the cost function values ​​output from the intra prediction unit 74 and the motion prediction / compensation unit 76. The prediction image selection unit 77 then selects the prediction image of the determined optimal prediction mode and supplies it to the calculation units 63 and 70. This prediction image is used in the calculations in steps S13 and S18 as described above.

[0204] This selection information for the predicted image is supplied to the intra prediction unit 74 or the motion prediction / compensation unit 76. If the predicted image for the optimal intra prediction mode is selected, the intra prediction unit 74 supplies information indicating the optimal intra prediction mode (i.e., intra prediction mode information) to the reversible coding unit 66.

[0205] When the prediction image of the optimal interpretation mode is selected, the motion prediction and compensation unit 76 outputs information indicating the optimal interpretation mode and, if necessary, information corresponding to the optimal interpretation mode to the reversible coding unit 66. The information corresponding to the optimal interpretation mode includes motion vector information, flag information, and reference frame information. That is, when the prediction image by the interpretation mode is selected as the optimal interpretation mode, the motion prediction and compensation unit 76 outputs interpretation mode information, motion vector information, and reference frame information to the reversible coding unit 66.

[0206] In step S23, the reversible coding unit 66 encodes the quantized conversion coefficients output from the quantization unit 65. That is, the difference image is reversibly coded and compressed using variable-length coding, arithmetic coding, etc. At this time, the information from the intra prediction unit 74, or the information from the motion prediction / compensation unit 76, and the quantization parameters from the rate control unit 78, which were input to the reversible coding unit 66 in step S22 described above, are also coded and added to the header information.

[0207] In step S24, the storage buffer 67 stores the difference image as a compressed image. The compressed image stored in the storage buffer 67 is read out as appropriate and transmitted to the decoding side via the transmission line.

[0208] In step S25, the rate control unit 78 controls the rate of the quantization operation of the quantization unit 65 by means of quantization parameters, based on the compressed image stored in the storage buffer 67, so as to prevent overflow or underflow.

[0209] The quantization parameters used to control the rate of the quantization unit 65 are supplied to the reversible coding unit 66, and in step S23 described above, they are reversibly coded and inserted into the header portion of the compressed image. These quantization parameters are also supplied to the adjacent pixel interpolation filter switching unit 75, and in step S21 described above, they are used to set the filter coefficients used in the filtering process applied to adjacent pixels, which is performed prior to intra prediction.

[0210] [Explanation of prediction processing] Next, the prediction process in step S21 of Figure 16 will be explained with reference to the flowchart in Figure 17.

[0211] If the image to be processed supplied from the screen sorting buffer 62 is an image of a block to be processed intra-processed, the decoded image to be referenced is read from the frame memory 72 and supplied to the intra-prediction unit 74 via the switch 73.

[0212] In step S31, the intra prediction unit 74 uses the supplied image to intra-predict the pixels of the block to be processed using all candidate intra prediction modes. The decoded pixels used for reference are those that have not been deblock filtered by the deblock filter 71.

[0213] Details of the intra prediction process in step S31 will be described later with reference to Figure 18, but in this process, the optimal filter coefficient is set, and the set filter coefficient is used to perform filtering on adjacent pixels. Then, intra prediction is performed using the filtered adjacent pixels to generate a predicted image.

[0214] The above processing is performed for all candidate intra-prediction modes, cost function values ​​are calculated for all candidate intra-prediction modes, and the optimal intra-prediction mode is determined based on the calculated cost function values. The generated prediction image and the cost function value of the optimal intra-prediction mode are supplied to the prediction image selection unit 77.

[0215] If the image to be processed supplied from the screen sorting buffer 62 is an image to be interprocessed, the referenced image is read from the frame memory 72 and supplied to the motion prediction and compensation unit 76 via the switch 73. Based on these images, in step S32, the motion prediction and compensation unit 76 performs intermotion prediction processing. That is, the motion prediction and compensation unit 76 refers to the image supplied from the frame memory 72 and performs motion prediction processing for all candidate intermotion prediction modes.

[0216] Details of the inter motion prediction process in step S32 will be described later with reference to Figure 19, but in this process, motion prediction processing is performed for all candidate inter prediction modes and cost function values ​​are calculated for all candidate inter prediction modes.

[0217] In step S33, the motion prediction and compensation unit 76 compares the cost function values ​​for the inter-prediction modes calculated in step S32 and determines the prediction mode that gives the minimum value as the optimal inter-prediction mode. The motion prediction and compensation unit 76 then supplies the prediction image generated in the optimal inter-prediction mode and its cost function value to the prediction image selection unit 77.

[0218] [Explanation of Intra Prediction Processing] Next, the intra-prediction process in step S31 of Figure 17 will be explained with reference to the flowchart in Figure 18. Note that the example in Figure 18 uses the case of a luminance signal as an example.

[0219] In step S25 of Figure 16 described above, the rate control unit 78 supplies quantization parameters for the target block. In step S41, the quantization parameter buffer 92 receives and stores the quantization parameters for the target block from the rate control unit 78.

[0220] In step S42, the predictive image generation unit 82 selects one intra-prediction mode from among the 4x4 pixel, 8x8 pixel, and 16x16 pixel intra-prediction modes. The information of the selected intra-prediction mode is stored in the predictive mode buffer 91.

[0221] The low-pass filter setting unit 93 reads intra-prediction mode information from the prediction mode buffer 91 and reads quantization parameter values ​​from the quantization parameter buffer 92. Then, in step S43, the low-pass filter setting unit 93 sets filter coefficients corresponding to the intra-prediction mode and quantization parameters from among the filter coefficients calculated for each slice and stored in the filter coefficient memory 94. The set filter coefficients are supplied to the adjacent pixel setting unit 81.

[0222] In step S44, the adjacent pixel setting unit 81 uses the set filter coefficients to perform a filter on the adjacent pixel values ​​of the target block and supplies the filtered adjacent pixel values ​​to the prediction image generation unit 82.

[0223] In step S43, the prediction image generation unit 82 uses the filtered adjacent pixel values ​​to perform intra-prediction of the target block in the intra-prediction mode selected in step S42, and generates a prediction image.

[0224] The optimal prediction mode determination unit 83 is supplied with the intra-prediction image read from the screen sorting buffer 62, the prediction image generated by the prediction image generation unit 82, and its intra-prediction mode information.

[0225] In step S46, the optimal prediction mode determination unit 83 uses the supplied information to calculate a cost function value for the intra-prediction mode that generated the predicted image. Here, the cost function value is calculated based on either the High Complexity mode or the Low Complexity mode method. These modes are defined in the JM (Joint Model), which is the reference software for the H.264 / AVC method.

[0226] In other words, in High Complexity mode, as part of step S45, provisional coding is performed for all candidate prediction modes. Then, a cost function value represented by the following equation (73) is calculated for each prediction mode, and the prediction mode that gives the minimum value is selected as the optimal prediction mode.

[0227] Cost(Mode) = D + λ・R ... (73) D is the difference (distortion) between the original image and the decoded image, R is the generated code quantity including the orthogonal transformation coefficients, and λ is the Lagrangian multiplier given as a function of the quantization parameter QP.

[0228] On the other hand, in Low Complexity mode, as part of the process in step S45, prediction images are generated for all candidate prediction modes, and header bits such as motion vector information, prediction mode information, and flag information are calculated. Then, a cost function value expressed by the following equation (74) is calculated for each prediction mode, and the prediction mode that gives the minimum value is selected as the optimal prediction mode.

[0229] Cost(Mode) = D + QPtoQuant(QP)・Header_Bit...(74) D is the difference (distortion) between the original image and the decoded image, Header_Bit is the header bit for the prediction mode, and QPtoQuant is a function given as a function of the quantization parameter QP.

[0230] In Low Complexity mode, for all prediction modes, only the prediction image is generated, and there is no need for encoding or decoding, thus reducing the amount of computation required.

[0231] In step S47, the optimal prediction mode determination unit 83 determines whether processing for all intra prediction modes has been completed. That is, in step S47, it is determined whether processing in steps S42 to S46 has been performed for all intra prediction modes for 4x4 pixels, 8x8 pixels, and 16x16 pixels.

[0232] If it is determined in step S47 that processing for all intra prediction modes has not been completed, the process returns to step S42 and the subsequent processing is repeated.

[0233] If it is determined in step S47 that processing for all intra prediction modes has been completed, the process proceeds to step S48. In step S48, the optimal prediction mode determination unit 83 determines the intra prediction mode in which the calculated cost function value is minimized as the optimal intra prediction mode.

[0234] The predicted image of the optimal intra prediction mode and the corresponding cost function value are supplied to the predicted image selection unit 77.

[0235] If the prediction image selection unit 77 selects a prediction image generated in the optimal intra prediction mode, the optimal prediction mode determination unit 83 supplies information indicating the optimal intra prediction mode to the reversible coding unit 66. The reversible coding unit 66 then encodes and adds it to the header information of the compressed image (step S23 in Figure 16 described above).

[0236] The filter coefficients calculated by the learning process and stored in the filter coefficient memory 94 are also stored in the image decoding device 151 shown in Figure 22, which will be described later. This eliminates the need to add the set filter coefficients to the header information of the compressed image and send it.

[0237] Therefore, in the case of H.264 / AVC, there are 51 possible quantization parameters and 9 possible intra-prediction modes for 4x4 or 8x8. Considering these combinations, a massive number of filter coefficients, 51 x 9 = 459, would be required. Since it is not necessary to send such a large amount of information about filter coefficients to the decoding side, processing can be achieved without increasing the overhead of coefficient information.

[0238] [Explanation of Inter-Motion Prediction Processing] Next, with reference to the flowchart in Figure 19, the inter-motion prediction process in step S32 of Figure 17 will be explained.

[0239] In step S61, the motion prediction and compensation unit 76 determines a motion vector and a reference image for each of the eight types of inter-prediction modes, each consisting of 16x16 pixels to 4x4 pixels. That is, a motion vector and a reference image are determined for each block to be processed in each inter-prediction mode.

[0240] In step S62, the motion prediction and compensation unit 76 performs motion prediction and compensation processing on the reference image based on the motion vector determined in step S61 for each of the eight types of inter prediction modes consisting of 16x16 pixels to 4x4 pixels. This motion prediction and compensation processing generates a predicted image for each inter prediction mode.

[0241] In step S63, the motion prediction and compensation unit 76 generates motion vector information to be added to the compressed image for the motion vectors determined for each of the eight types of interpretation modes consisting of 16x16 pixels to 4x4 pixels.

[0242] The generated motion vector information is also used in the cost function value calculation in the next step S64, and if the corresponding predicted image is finally selected by the predicted image selection unit 77, it is output to the reversible encoding unit 66 along with the predicted mode information and reference frame information.

[0243] In step S64, the motion prediction and compensation unit 76 calculates a cost function value shown in equation (73) or equation (74) above for each of the eight types of inter prediction modes consisting of 16 × 16 pixels to 4 × 4 pixels. The cost function value calculated here is used in determining the optimal inter prediction mode in step S34 of Figure 17 above.

[0244] Next, with reference to Figure 20, an example of a second method for calculating the optimal filter coefficients will be described, which involves an online process that sequentially calculates the optimal filter coefficients for each slice.

[0245] In this case, the filter coefficients calculated on the encoding side must be sent to the decoding side for each slice, and sending too many divided filter coefficients would lead to a decrease in encoding efficiency. Therefore, only one filter coefficient is transmitted per slice, or only one for each prediction mode for each block size, or only one for each type of prediction mode such as horizontal prediction, vertical prediction, etc.

[0246] In addition, in the case of offline processing described above, an example was explained in which the intra-prediction mode and quantization parameters are used as parameters for calculating the filter coefficients. In contrast, in the case of online processing, if there are many parameters for calculating the filter coefficients, the processing load will increase, so in the example in Figure 20, an example is explained in which only the intra-prediction mode is used as a parameter. Although the explanation is omitted, of course, it is also possible to use only the quantization parameter or to use both parameters.

[0247] [Other configuration examples of the intra prediction unit and adjacent pixel interpolation filter switching unit] Figure 20 is a block diagram showing another configuration example of the intra-prediction unit 74 and the adjacent pixel interpolation filter switching unit 75 when performing online processing to sequentially calculate the optimal filter coefficient for each slice.

[0248] In the example of Figure 20, a switch 101 is provided between the intra prediction unit 74 and the adjacent pixel interpolation filter switching unit 75. Depending on whether the switch 101 is turned on or off, the intra prediction unit 74 performs intra prediction twice, unlike in the case of Figure 14. That is, in the intra prediction unit 74, when the switch 101 is off, intra prediction as defined in H.264 / AVC is performed, and filter coefficients suitable for that intra prediction are calculated. Then, with the switch 101 on, intra prediction is performed using the filter coefficients set by the adjacent pixel interpolation filter switching unit 75 from among the calculated filter coefficients.

[0249] The intra prediction unit 74 in Figure 20 is composed of an adjacent image setting unit 111, a prediction image generation unit 112, and an optimal prediction mode determination unit 113.

[0250] The adjacent pixel interpolation filter switching unit 75 is composed of a prediction mode buffer 121, an optimal filter calculation unit 122, and a low-pass filter setting unit 123.

[0251] The adjacent image setting unit 111 is supplied with adjacent pixel values ​​for all target blocks of the target slice for intra prediction from the frame memory 72. In the case of Figure 20, the switch 73 is also omitted from the illustration. Note that in the case of intra prediction, the adjacent pixel values ​​used are the pixel values ​​that have not been deblocked by the deblocking filter 71.

[0252] When the switch 101 is in the off state, the adjacent pixel setting unit 111 applies a filter to the adjacent pixel values ​​of the target block from the frame memory 72 using a filter coefficient, only in the intra prediction mode defined in H.264 / AVC, and supplies it to the prediction image generation unit 112. That is, only in the case of the 8x8 pixel prediction mode described in equations (14) to (24) above, the filtered adjacent pixel values ​​are supplied to the prediction image generation unit 112. In all other cases, the adjacent pixel values ​​of the target block from the frame memory 72 are supplied to the prediction image generation unit 112 as is.

[0253] When switch 101 is ON, the adjacent pixel setting unit 111 receives filter coefficients from the low-pass filter setting unit 123. Therefore, the adjacent pixel setting unit 111 uses the filter coefficients set by the low-pass filter setting unit 123 to filter the adjacent pixel values ​​of the target block from the frame memory 72, and supplies the filtered adjacent pixel values ​​to the prediction image generation unit 112.

[0254] The prediction image generation unit 112 uses the adjacent pixel values ​​from the adjacent pixel setting unit 111 to perform intra-prediction for all target blocks in intra-prediction modes and generates a prediction image. The generated prediction image, along with the intra-prediction mode information, is supplied to the optimal prediction mode determination unit 113.

[0255] The optimal prediction mode determination unit 113 is supplied with the intra-prediction image read from the screen sorting buffer 62, the prediction image generated by the prediction image generation unit 112, and its intra-prediction mode information.

[0256] The optimal prediction mode determination unit 113 uses the supplied information to calculate a cost function value for the intra prediction mode that generated the prediction image, and determines the intra prediction mode that gives the minimum calculated cost function value as the optimal intra prediction mode.

[0257] When the switch 101 is in the off state, the optimal prediction mode determination unit 113 supplies information on the optimal intra prediction mode to the prediction mode buffer 121. When the switch 101 is in the on state, the optimal prediction mode determination unit 113 supplies the prediction image of the optimal intra prediction mode and the corresponding cost function value to the prediction image selection unit 77.

[0258] In addition, if the prediction image selected by the prediction image selection unit 77 is selected by the optimal prediction mode determination unit 113, the unit provides information indicating the optimal prediction mode to the reversible coding unit 66.

[0259] The prediction mode buffer 121 stores intra prediction mode information from the optimal prediction mode determination unit 113.

[0260] The optimal filter calculation unit 122 is supplied with the intra-predicted image read from the screen sorting buffer 62 and the adjacent pixel values ​​of the target block from the frame memory 72. The optimal filter calculation unit 122 reads the intra-prediction mode of each block included in the target slice from the prediction mode buffer 121. Then, using this information, the optimal filter calculation unit 122 calculates the optimal filter coefficient for each intra-prediction mode of the target slice as described above with reference to Figure 15, and supplies the calculated filter coefficient to the low-pass filter setting unit 123.

[0261] The low-pass filter setting unit 123 sets the filter coefficient for the target block from the calculated filter coefficients of the target slice, turns on the terminal of the switch 101, and supplies the set filter coefficient to the adjacent pixel setting unit 111. The low-pass filter setting unit 123 also supplies the filter coefficient of the target slice to the reversible encoding unit 66.

[0262] [Other explanations of intra-predictive processing] Next, the intra-prediction process performed by the intra-prediction unit 74 and the adjacent pixel interpolation filter switching unit 75 in Figure 20 will be explained with reference to the flowchart in Figure 21. This intra-prediction process is another example of the intra-prediction process in step S31 of Figure 17.

[0263] First, switch 101 is in the off state. The adjacent image setting unit 111 is supplied with adjacent pixel values ​​of all target blocks of the target slice for intra prediction from the frame memory 72. The adjacent pixel setting unit 111 applies a filter to the adjacent pixel values ​​of the target blocks from the frame memory 72 using filter coefficients, but only for the 8x8 pixel intra prediction mode defined in H.264 / AVC, and supplies it to the prediction image generation unit 112. In other intra prediction modes, the adjacent pixel values ​​of the target blocks from the frame memory 72 are supplied to the prediction image generation unit 112 as is.

[0264] In step S101, the prediction image generation unit 112 performs intra-prediction processing for all blocks included in the target slice. That is, the prediction image generation unit 112 uses the adjacent pixel values ​​of the target block from the adjacent pixel setting unit 111 to perform intra-prediction in each intra-prediction mode and generate a prediction image.

[0265] The optimal prediction mode determination unit 113 is supplied with the intra-prediction image read from the screen sorting buffer 62, the prediction image generated by the prediction image generation unit 112, and its intra-prediction mode information.

[0266] In step S102, the optimal prediction mode determination unit 113 uses the supplied information to calculate the cost function value of equation (73) or equation (74) described above for all intra prediction modes that generated the prediction image.

[0267] In step S103, the optimal prediction mode determination unit 113 determines the intra prediction mode that minimizes the cost function of equation (73) or equation (74) as the optimal intra prediction mode, and supplies the information of the determined intra prediction mode to the prediction mode buffer 121.

[0268] The optimal filter calculation unit 122 is supplied with the intra-prediction image read from the screen sorting buffer 62 and the adjacent pixel values ​​of the target block from the frame memory 72. The optimal filter calculation unit 122 reads the intra-prediction mode of each block included in the target slice from the prediction mode buffer 121.

[0269] Then, in step S104, the optimal filter calculation unit 122 uses this information to calculate the filter coefficient that minimizes the residual over the entire target slice for each intra prediction mode of the target slice, and this is the optimal filter coefficient. The filter coefficient calculated as described above with reference to Figure 15 is supplied to the low-pass filter setting unit 123.

[0270] The low-pass filter setting unit 123 sets a filter coefficient for the target block from the calculated filter coefficients of the target slice, turns on the terminal of the switch 101, and supplies the set filter coefficient to the adjacent pixel setting unit 111.

[0271] In step S105, the adjacent pixel setting unit 111 performs a filter operation on the adjacent pixel values ​​of the target block from the frame memory 72 using the filter coefficient set by the low-pass filter setting unit 123.

[0272] The filtered adjacent pixel values ​​are supplied to the prediction image generation unit 112. In step S106, the prediction image generation unit 112 uses the filtered adjacent pixel values ​​to perform intra-prediction again for all blocks included in the target slice and generates a prediction image. The generated prediction image is supplied to the optimal prediction mode determination unit 113 along with the intra-prediction mode information.

[0273] When the switch 101 is ON, the optimal prediction mode determination unit 113 supplies the prediction image of the optimal intra prediction mode and the corresponding cost function value to the prediction image selection unit 77.

[0274] In step S22 of Figure 16 described above, the prediction image selection unit 77 determines one of the optimal intra prediction mode and the optimal inter prediction mode to be the optimal prediction mode and supplies the prediction image selection information.

[0275] In response to this selection information of the predicted image, the optimal prediction mode determination unit 113 determines in step S107 whether or not the predicted image of the optimal intra prediction mode has been selected. If it is determined in step S107 that the predicted image of the optimal intra prediction mode has been selected, the process proceeds to step S108.

[0276] In step S108, the optimal prediction mode determination unit 113 supplies intra prediction mode information to the reversible coding unit 66. If filter coefficients have not yet been supplied for the target slice, the filter coefficients from the optimal filter calculation unit 122 are also supplied to the reversible coding unit 66.

[0277] If it is determined in step S107 that no prediction image for the optimal intra prediction mode has been selected, the intra prediction process is terminated.

[0278] Furthermore, by repeating the process described in steps S104 to S106, it is also possible to obtain even more optimized filter coefficients.

[0279] The encoded compressed image is transmitted via a predetermined transmission path and decoded by an image decoding device.

[0280] [Example of image decoding device configuration] Figure 22 shows the configuration of one embodiment of an image decoding device as an image processing device to which the present invention is applied.

[0281] The image decoding device 151 is composed of a storage buffer 161, a reversible decoding unit 162, an inverse quantization unit 163, an inverse orthogonal transform unit 164, an arithmetic unit 165, a deblocking filter 166, a screen rearrangement buffer 167, a D / A conversion unit 168, a frame memory 169, a switch 170, an intra prediction unit 171, an adjacent pixel interpolation filter switching unit 172, a motion prediction and compensation unit 173, and a switch 174.

[0282] The storage buffer 161 stores the transmitted compressed image. The reversible decoding unit 162 decodes the information encoded by the reversible encoding unit 66 in Figure 1, which is supplied from the storage buffer 161, in a manner corresponding to the encoding method of the reversible encoding unit 66. The inverse quantization unit 163 inversely quantizes the image decoded by the reversible decoding unit 162 in a manner corresponding to the quantization method of the quantization unit 65 in Figure 1. The inverse orthogonal transform unit 164 inversely orthogonal transforms the output of the inverse quantization unit 163 in a manner corresponding to the orthogonal transform method of the orthogonal transform unit 64 in Figure 1.

[0283] The inverse orthogonal transformed output is added to the predicted image supplied from the switch 174 by the calculation unit 165 and decoded. The deblocking filter 166 removes block distortion from the decoded image, then supplies it to the frame memory 169 for storage and outputs it to the screen rearrangement buffer 167.

[0284] The screen reordering buffer 167 reorders the images. That is, the order of the frames that were reordered by the screen reordering buffer 62 in Figure 1 for encoding order is reordered to the original display order. The D / A conversion unit 168 performs D / A conversion on the images supplied from the screen reordering buffer 167 and outputs them to a display (not shown) for display.

[0285] The switch 170 reads the interprocessed image and the referenced image from the frame memory 169 and outputs them to the motion prediction and compensation unit 173, and also reads the image used for intraprediction from the frame memory 169 and supplies it to the intraprediction unit 171.

[0286] The intra prediction unit 171 is supplied with information indicating the intra prediction mode obtained by decoding the header information from the reversible decoding unit 162. Based on this information, the intra prediction unit 171 generates a predicted image by performing filtering and intra prediction on adjacent pixel values ​​using the filter coefficients set by the adjacent pixel interpolation filter switching unit 172, and outputs the generated predicted image to the switch 174.

[0287] The adjacent pixel interpolation filter switching unit 172 is supplied with at least one of the following from the reversible decoding unit 162: information indicating the intra prediction mode obtained by decoding the header information in accordance with the encoding by the image encoding device 51, and information on the quantization parameters. The adjacent pixel interpolation filter switching unit 172 stores filter coefficients corresponding to at least one of the quantization parameters and intra prediction mode, which are obtained by learning in the learning device 251 shown in Figure 28, which will be described later, the same as the adjacent pixel interpolation filter switching unit 75 in Figure 1.

[0288] The adjacent pixel interpolation filter switching unit 172 sets filter coefficients corresponding to at least one of the quantization parameters and intra prediction modes from the reversible decoding unit 162. The adjacent pixel interpolation filter switching unit 172 supplies the set filter coefficients to the intra prediction unit 74.

[0289] In the adjacent pixel interpolation filter switching unit 172, filter coefficients that have been learned offline in advance are stored. However, in the adjacent pixel interpolation filter switching unit 75 of Figure 1, if the filter coefficients are calculated online, those filter coefficients are transmitted, for example, for each slice. In this case, the adjacent pixel interpolation filter switching unit 172 uses the filter coefficients that have been decoded by the reversible decoding unit 162.

[0290] The motion prediction and compensation unit 173 is supplied with information obtained by decoding the header information (prediction mode information, motion vector information, and reference frame information) from the reversible decoding unit 162. If information indicating the inter-prediction mode is supplied, the motion prediction and compensation unit 173 performs motion prediction and compensation processing on the image based on the motion vector information and reference frame information to generate a predicted image. The motion prediction and compensation unit 173 outputs the predicted image generated by the inter-prediction mode to the switch 174.

[0291] The switch 174 selects a predicted image generated by the motion prediction and compensation unit 173 or the intra prediction unit 171 and supplies it to the calculation unit 165.

[0292] In the image encoding device 51 of Figure 1, intra-prediction processing is performed for all intra-prediction modes in order to determine the prediction mode based on the cost function. In contrast, in the image decoding device 151, intra-prediction processing is performed only based on the information of the intra-prediction mode that is encoded and sent to the device.

[0293] [Example of configuration of the intra prediction unit and adjacent pixel interpolation filter switching unit] Figure 23 is a block diagram showing a detailed configuration example of the intra prediction unit and the adjacent pixel interpolation filter switching unit. Note that the functional blocks in Figure 23 correspond to the functional blocks in the case of offline processing of the image encoding device 51 shown in Figure 14.

[0294] In the example shown in Figure 23, the intra prediction unit 171 is composed of a prediction image generation unit 181 and an adjacent pixel setting unit 182. The adjacent pixel interpolation filter switching unit 172 is composed of a prediction mode buffer 191, a quantization parameter buffer 192, and a low-pass filter setting unit 193. The low-pass filter setting unit 193 has a built-in filter coefficient memory 194.

[0295] The prediction image generation unit 181 is supplied with intra-prediction mode information from the reversible decoding unit 162 and filtered adjacent pixel values ​​from the adjacent pixel setting unit 182. The prediction image generation unit 181 uses the supplied adjacent pixel values ​​to perform intra-prediction in the intra-prediction mode from the reversible decoding unit 162, generates a prediction image, and supplies the generated prediction image to the switch 174.

[0296] The adjacent pixel setting unit 182 is supplied with adjacent pixel values ​​of the target block for intra prediction from the frame memory 169. In the case of Figure 23, the switch 170 is not shown, but in reality, the adjacent pixel values ​​are supplied from the frame memory 169 to the adjacent image setting unit 182 via the switch 170.

[0297] The adjacent pixel setting unit 182 uses the filter coefficients set by the low-pass filter setting unit 193 to apply a filter to the adjacent pixel values ​​of the target block from the frame memory 169, and supplies the filtered adjacent pixel values ​​to the prediction image generation unit 181.

[0298] The prediction mode buffer 191 stores intra-prediction mode information from the reversible decoding unit 162. The quantization parameter buffer 192 stores quantization parameters from the reversible decoding unit 162.

[0299] The low-pass filter setting unit 193 reads intra-prediction mode information of the target block from the prediction mode buffer 191 and reads quantization parameters for the target block from the quantization parameter buffer 192. The low-pass filter setting unit 193 sets filter coefficients corresponding to this information from among the filter coefficients stored in the built-in filter coefficient memory 194 and supplies the set filter coefficients to the adjacent pixel setting unit 182.

[0300] The filter coefficient memory 194 stores filter coefficients corresponding to the quantization parameters and intra-prediction modes, which are obtained by learning in the learning device 251 shown in Figure 28, which will be described later, similar to the filter coefficient memory 94 in Figure 14.

[0301] These filter coefficients are calculated and stored for each slice, for example, as described above with reference to Figure 15. In the filter coefficient memory 194, the filter coefficients are also stored as n-bit values ​​(where n is an integer), depending on the register length of the processor.

[0302] [Explanation of the decoding process of the image decoding device] Next, the decoding process performed by the image decoding device 151 will be explained with reference to the flowchart in Figure 24.

[0303] In step S131, the storage buffer 161 stores the transmitted image. In step S132, the reversible decoding unit 162 decodes the compressed image supplied from the storage buffer 161. That is, the I picture, P picture, and B picture encoded by the reversible encoding unit 66 in Figure 1 are decoded.

[0304] At this time, motion vector information, reference frame information, prediction mode information (information indicating intra-prediction mode or inter-prediction mode), quantization parameter information, and flag information are also decoded.

[0305] In other words, if the prediction mode information is intra-prediction mode information, the prediction mode information is supplied to the intra-prediction unit 171 and the adjacent pixel interpolation filter switching unit 172. Also, if the quantization parameter information is decoded, this is also supplied to the adjacent pixel interpolation filter switching unit 172. If the prediction mode information is inter-prediction mode information, the prediction mode information and the corresponding motion vector information and reference frame information are supplied to the motion prediction and compensation unit 173.

[0306] In step S133, the inverse quantization unit 163 inversely quantizes the conversion coefficients decoded by the reversible decoding unit 162 using characteristics corresponding to the characteristics of the quantization unit 65 in Figure 1. In step S134, the inverse orthogonal transformation unit 164 inversely orthogonal transforms the conversion coefficients inversely quantized by the inverse quantization unit 163 using characteristics corresponding to the characteristics of the orthogonal transformation unit 64 in Figure 1. As a result, the difference information corresponding to the input of the orthogonal transformation unit 64 (output of the calculation unit 63) in Figure 1 is decoded.

[0307] In step S135, the calculation unit 165 adds the predicted image, which was selected in the processing of step S141 described later and input via switch 174, to the difference information. This decodes the original image. In step S136, the deblocking filter 166 filters the image output from the calculation unit 165. This removes block distortion. In step S137, the frame memory 169 stores the filtered image.

[0308] In step S138, the intra prediction unit 171 and the motion prediction / compensation unit 173 each perform image prediction processing in accordance with the prediction mode information supplied from the reversible decoding unit 162.

[0309] In other words, when intra prediction mode information is supplied from the reversible decoding unit 162, the intra prediction unit 171 performs intra prediction processing for the intra prediction mode. At this time, the intra prediction unit 171 performs filtering and intra prediction processing for adjacent pixels using the filter coefficients set by the adjacent pixel interpolation filter switching unit 172.

[0310] Details of the prediction process in step S138 will be described later with reference to Figure 25, but as a result of this process, the predicted image generated by the intra prediction unit 171 or the predicted image generated by the motion prediction / compensation unit 173 is supplied to the switch 174.

[0311] In step S139, the switch 174 selects a predicted image. That is, a predicted image generated by the intra prediction unit 171 or a predicted image generated by the motion prediction and compensation unit 173 is supplied. Therefore, the supplied predicted image is selected and supplied to the calculation unit 165, and as described above, in step S134 it is added with the output of the inverse orthogonal transform unit 164.

[0312] In step S140, the screen reordering buffer 167 performs the reordering. That is, the order of the frames that have been reordered for encoding by the screen reordering buffer 62 of the image encoding device 51 is rearranged to the original display order.

[0313] In step S141, the D / A conversion unit 168 performs D / A conversion on the image from the screen rearrangement buffer 167. This image is output to a display (not shown) and the image is displayed.

[0314] [Explanation of prediction processing] Next, the prediction process in step S138 of Figure 24 will be explained with reference to the flowchart in Figure 25.

[0315] In step S171, the predictive image generation unit 181 determines whether or not the target block is intra-encoded. When intra-predictive mode information is supplied to the predictive image generation unit 181 from the reversible decoding unit 162, the predictive image generation unit 181 determines in step S171 that the target block is intra-encoded, and the process proceeds to step S172.

[0316] In step S172, the predictive image generation unit 181 receives and acquires intra predictive mode information from the reversible decoding unit 162. At this time, intra predictive mode information is also supplied to and stored in the predictive mode buffer 191.

[0317] When the reversible decoding unit 162 supplies quantization parameter information to the quantization parameter buffer 192, the quantization parameter buffer 192 acquires and stores the quantization parameters in step S173.

[0318] The low-pass filter setting unit 193 reads intra-prediction mode information for the target block from the prediction mode buffer 191 and reads quantization parameters for the target block from the quantization parameter buffer 192. In step S174, the low-pass filter setting unit 193 sets filter coefficients for adjacent pixels corresponding to this information from among the filter coefficients for each slice stored in the built-in filter coefficient memory 194. The set filter coefficients are supplied to the adjacent pixel setting unit 182.

[0319] In step S175, the adjacent pixel setting unit 182 uses the filter coefficient set by the low-pass filter setting unit 193 to apply a filter to the adjacent pixel values ​​of the target block from the frame memory 169, and supplies the filtered adjacent pixel values ​​to the prediction image generation unit 181.

[0320] The prediction image generation unit 181 uses the adjacent pixel values ​​supplied from the adjacent pixel setting unit 182 to perform intra-prediction in the intra-prediction mode acquired in step S172 and generates a prediction image. The generated prediction image is supplied to the switch 174.

[0321] On the other hand, if it is determined in step S171 that the data is not intra-encoded, the process proceeds to step S177.

[0322] If the image to be processed is an image to be interprocessed, the reversible decoding unit 162 supplies interpretation mode information, reference frame information, and motion vector information to the motion prediction and compensation unit 173. In step S177, the motion prediction and compensation unit 173 acquires the interpretation mode information, reference frame information, motion vector information, etc. from the reversible decoding unit 162.

[0323] Then, in step S178, the motion prediction and compensation unit 173 performs inter-motion prediction. That is, if the image to be processed is an image to be inter-predicted, the necessary image is read from the frame memory 169 and supplied to the motion prediction and compensation unit 173 via the switch 170. In step S177, the motion prediction and compensation unit 173 performs motion prediction in inter-prediction mode based on the motion vector acquired in step S176 and generates a predicted image. The generated predicted image is output to the switch 174.

[0324] [Other configuration examples of the intra prediction unit and adjacent pixel interpolation filter switching unit] Figure 26 is a block diagram showing a detailed configuration example of the intra-prediction unit and the adjacent pixel interpolation filter switching unit. Note that the functional blocks in Figure 26 correspond to the functional blocks in the case of online processing of the image encoding device 51 shown in Figure 20.

[0325] In the example of Figure 26, the intra prediction unit 171 is composed of the prediction image generation unit 181 and the adjacent pixel setting unit 182 shown in Figure 23. The adjacent pixel interpolation filter switching unit 172 is composed of the prediction mode buffer 191, the interpolation filter buffer 201 and the low-pass filter setting unit 202 shown in Figure 23. In the example of Figure 26, the parts corresponding to those in Figure 23 are denoted by corresponding reference numerals and perform basically the same processing, so their explanation is omitted.

[0326] In the case of Figure 26, the filter coefficients calculated for the target slice are encoded and sent from the image encoding device 51. Therefore, the reversible decoding unit 162 decodes them together with other information and supplies them to the interpolation filter buffer 201 of the adjacent pixel interpolation filter switching unit 172.

[0327] The interpolation filter buffer 201 obtains and stores the filter coefficients of the target slice from the reversible decoding unit 162.

[0328] The low-pass filter setting unit 202 reads the intra-prediction mode information of the target block from the prediction mode buffer 191. The low-pass filter setting unit 202 reads the filter coefficients corresponding to the read intra-prediction mode from the filter coefficients of the target slice stored in the interpolation filter buffer 201 and sets them as the filter coefficients of the target block. The set filter coefficients are supplied to the adjacent pixel setting unit 182.

[0329] [Other explanations of prediction processing] Next, referring to the flowchart in Figure 27, the prediction process in the case of the intra-prediction unit 171 and adjacent pixel interpolation filter switching unit 172 in Figure 26 will be explained. Note that this intra-prediction process is another example of the prediction process in step S138 of Figure 24. Also, the processes in steps S181, S182, and S185 to S188 in Figure 27 are basically the same as the processes in steps S171, S172, and S175 to S178 in Figure 25, so a detailed explanation will be omitted.

[0330] In step S181, the predictive image generation unit 181 determines whether or not the target block is intra-encoded. When intra-predictive mode information is supplied to the predictive image generation unit 181 from the reversible decoding unit 162, the predictive image generation unit 181 determines in step 181 that the target block is intra-encoded, and the process proceeds to step S182.

[0331] In step S182, the predictive image generation unit 181 receives and acquires intra predictive mode information from the reversible decoding unit 162. At this time, intra predictive mode information is also supplied to and stored in the predictive mode buffer 191.

[0332] When the information of the filter coefficients of the target slice is supplied from the reversible decoding unit 162 to the interpolation filter buffer 201, the interpolation filter buffer 201 acquires and stores the filter coefficients of the target slice in step S183. These filter coefficients are supplied for each slice.

[0333] The low-pass filter setting unit 202 reads the intra-prediction mode information of the target block from the prediction mode buffer 191. In step S184, the low-pass filter setting unit 202 sets the filter coefficients for adjacent pixels that correspond to the intra-prediction mode of the target block from among the filter coefficients of the target slice stored in the interpolation filter buffer 201. The set filter coefficients are supplied to the adjacent pixel setting unit 182.

[0334] In step S185, the adjacent pixel setting unit 182 uses the filter coefficient set by the low-pass filter setting unit 202 to apply a filter to the adjacent pixel values ​​of the target block from the frame memory 169, and supplies the filtered adjacent pixel values ​​to the prediction image generation unit 181.

[0335] In step S186, the prediction image generation unit 181 uses the adjacent pixel values ​​supplied from the adjacent pixel setting unit 182 to perform intra prediction in the intra prediction mode acquired in step S172 and generates a prediction image. The generated prediction image is supplied to the switch 174.

[0336] On the other hand, if it is determined in step S181 that the data is not intra-encoded, the process proceeds to step S187.

[0337] In step S187, the motion prediction and compensation unit 173 acquires interpretation mode information, reference frame information, motion vector information, etc. from the reversible decoding unit 162.

[0338] In step S188, the motion prediction and compensation unit 173 performs inter-motion prediction. The predicted image generated by this process is output to the switch 174.

[0339] In the image encoding device 51 in Figure 1 and the image decoding device 151 in Figure 22, prior to intra-prediction, filtering is performed on adjacent pixels used for intra-prediction using filter coefficients adaptively set for the image. For example, the filter coefficients are set according to the intra-prediction mode and quantization parameters.

[0340] This allows for noise reduction according to the image and bitrate. As a result, prediction efficiency can be improved.

[0341] Figure 28 shows the configuration of one embodiment of a learning device to which the present invention is applied. In the example of Figure 28, the learning device 251 performs a learning process for filter coefficients using the training image signal.

[0342] The training image signal is a test image used to determine the filter coefficients, and may be a standard sequence used in the standardization of image compression coding, which can be obtained from www.vqeg.org. Alternatively, an input image appropriate for each application may be used. For example, if the input is a camera signal, training may be performed using a baseband signal captured using a CCD or CMOS sensor.

[0343] The learning device 251 in Figure 28 is similar to the image coding device 51 in Figure 1 in that it includes an A / D conversion unit 61, a screen sorting buffer 62, a calculation unit 63, an orthogonal conversion unit 64, a quantization unit 65, a reversible coding unit 66, a storage buffer 67, an inverse quantization unit 68, an inverse orthogonal conversion unit 69, a calculation unit 70, a deblocking filter 71, a frame memory 72, a switch 73, an intra prediction unit 74, a motion prediction and compensation unit 76, a predicted image selection unit 77, and a rate control unit 78.

[0344] Furthermore, the learning device 251 differs from the image encoding device 51 in Figure 1 in that it uses training image signals as the signals used, and that it has an adjacent pixel interpolation filter calculation unit 261 instead of an adjacent pixel interpolation filter switching unit 75.

[0345] Specifically, in the learning device 251, learning is performed using only the blocks contained in the I picture. Alternatively, in the learning device 251, learning is performed using only the blocks within the intra macroblocks contained in the B picture and P picture. The former requires less computation for learning than the latter. In the former case, the coefficients obtained from the blocks contained in the I picture may be applied only to the blocks contained in the I picture, or to the intra macroblocks contained in the B picture and P picture.

[0346] In other words, in the learning device 251, learning is performed solely by intra-prediction by the intra-prediction unit 74. Therefore, the motion prediction and compensation unit 76 is not actually functioning.

[0347] Furthermore, the adjacent pixel interpolation filter calculation unit 261 in Figure 29 is similar to the adjacent pixel interpolation filter switching unit 75 in Figure 20 in that it includes a prediction mode buffer 121, an optimal filter calculation unit 122, and a low-pass filter setting unit 123.

[0348] In contrast, the adjacent pixel interpolation filter calculation unit 261 differs from the adjacent pixel interpolation filter switching unit 75 in Figure 20 in that a filter coefficient storage unit 271 is added and quantization parameters are supplied from the rate control unit 78 to the optimal filter calculation unit 122.

[0349] To explain in more detail, in the example of Figure 29, as in the example of Figure 20, a switch 101 is provided between the intra prediction unit 74 and the adjacent pixel interpolation filter switching unit 75, and the intra prediction unit 74 performs intra prediction twice by turning the switch 101 on and off.

[0350] In other words, in the intra prediction unit 74, with the switch 101 off, intra prediction as defined in H.264 / AVC is performed, and the optimal filter coefficients for the intra prediction mode and quantization parameters are calculated for each slice. These calculated filter coefficients for each slice are stored in the filter coefficient storage unit 271. Then, with the switch 101 on, intra prediction is performed using the filter coefficients set by the adjacent pixel interpolation filter switching unit 75 from among the calculated filter coefficients for each slice.

[0351] The filter coefficients stored in this filter coefficient storage unit 271 are stored in the filter coefficient memory 94 of the image encoding device 51 in Figure 1 (Figure 14) and the filter coefficient memory 194 of the image decoding device 151 in Figure 22 (Figure 23) via a recording medium or network.

[0352] [Explanation of intra-prediction processing in the learning process] Next, referring to the flowchart in Figure 30, we will explain the intra-prediction process as one of the learning processes performed by the learning device 251 in Figure 28. The learning device 251 performs essentially the same processes as the encoding process in Figure 17, except that the prediction process in step S21 is replaced by the intra-prediction process in Figure 30.

[0353] Furthermore, steps S201 to S203 and S206 to S209 in Figure 30 perform basically the same processing as steps S101 to S103 and S105 to S108 in Figure 21, so their explanation will be omitted as it would be repetitive. That is, in step S204 in Figure 30, the optimal filter calculation unit 122 calculates the filter coefficients that minimize the residual over the entire target slice for each intra prediction mode and corresponding quantization parameter of the target slice, as the optimal filter coefficients. The calculated filter coefficients are supplied to the filter coefficient storage unit 271.

[0354] In step S205, the filter coefficient storage unit 271 stores the filter coefficients supplied from the optimal filter calculation unit 122.

[0355] The low-pass filter setting unit 123 sets a filter coefficient for the target block from among the filter coefficients of the target slice stored in the filter coefficient storage unit 271, turns on the terminal of the switch 101, and supplies the set filter coefficient to the adjacent pixel setting unit 111.

[0356] In response to this, in step S206, the set filter coefficients are used to perform filtering on the adjacent pixel values ​​of the target block from the frame memory 72.

[0357] Of course, as in the example in Figure 21, it is also possible to obtain further optimized filter coefficients by repeating the process in steps S204 to S207 described above.

[0358] As described above, in the learning device 251, the training image signal is used and processing similar to the actual encoding process is performed, and the filter coefficients calculated therein are stored in the filter coefficient storage unit 271. Therefore, the optimal filter coefficients can be determined.

[0359] The filter coefficients stored in this filter coefficient storage unit 271 are stored in the filter coefficient memory 94 of the image encoding device 51 (Figure 14) and the filter coefficient memory 194 of the image decoding device 151 (Figure 23) via a recording medium or network.

[0360] Then, in the image encoding device 51, as described above, the coefficients obtained for the blocks included in the I picture (or intra macroblocks included in the B picture and P picture) are applied only to the blocks included in the I picture. Alternatively, they are applied not only to the blocks included in the I picture, but also to the intra macroblocks included in the B picture and P picture.

[0361] This makes it possible to achieve high encoding efficiency in the image encoding device 51 in Figure 1 and the image decoding device 151 in Figure 22.

[0362] In addition, the orthogonal transform unit 64 and inverse orthogonal transform unit 69 of the image encoding device 51 in Figure 1, and the inverse orthogonal transform unit 164 of the image decoding device 151 in Figure 22 perform orthogonal and inverse orthogonal transforms as defined in H.264 / AVC. Alternatively, the orthogonal and inverse orthogonal transforms proposed in Non-Patent Literature 1 can be applied to the orthogonal transform unit 64 and inverse orthogonal transform unit 69 of the image encoding device 51 in Figure 1, and the inverse orthogonal transform unit 164 of the image decoding device 151 in Figure 22.

[0363] This makes it possible to further improve the encoding efficiency of the method proposed in Non-Patent Document 1.

[0364] In the above description, an example of performing intra prediction was explained, but the present invention can also be applied to intra prediction in the second-order prediction proposed in Non-Patent Document 2.

[0365] <2. Second Embodiment> [Other configuration examples of image encoding devices] Figure 31 shows the configuration of another embodiment of an image encoding device as an image processing device to which the present invention is applied.

[0366] The image coding device 351 is similar to the image coding device 51 in Figure 1 in that it includes an A / D conversion unit 61, a screen sorting buffer 62, a calculation unit 63, an orthogonal transformation unit 64, a quantization unit 65, a reversible coding unit 66, a storage buffer 67, an inverse quantization unit 68, an inverse orthogonal transformation unit 69, a calculation unit 70, a deblocking filter 71, a frame memory 72, a switch 73, an intra prediction unit 74, a motion prediction and compensation unit 76, a predicted image selection unit 77, and a rate control unit 78.

[0367] Furthermore, the image encoding device 351 differs from the image encoding device 51 in Figure 1 in that the adjacent pixel interpolation filter switching unit 75 is omitted, and the second-order prediction unit 361 and the adjacent pixel interpolation filter switching unit 362 are added.

[0368] In other words, in the example shown in Figure 31, the intra prediction unit 74 performs intra prediction of H.264 / AVC.

[0369] Meanwhile, the motion prediction and compensation unit 76 detects motion vectors for all candidate interprediction modes based on the interprocessing image and the reference image, performs compensation processing on the reference image based on the motion vectors, and generates a predicted image.

[0370] The motion prediction and compensation unit 76 supplies the detected motion vector information, information about the image to be interprocessed (such as the address), and the first-order residual, which is the difference between the image to be interprocessed and the generated predicted image, to the second-order prediction unit 361.

[0371] The motion prediction and compensation unit 76 determines the optimal intra prediction mode for the second-order prediction by comparing it with the second-order residuals from the second-order prediction unit 361. The motion prediction and compensation unit 76 also determines whether to encode the second-order residual or the first-order residual by comparing the second-order residual and the first-order residual. These processes are performed for all candidate intra prediction modes.

[0372] The motion prediction and compensation unit 76 then calculates cost function values ​​for all candidate inter-prediction modes. At this time, the cost function values ​​are calculated using the residuals determined for each inter-prediction mode from among the first-order residuals and second-order residuals. The motion prediction and compensation unit 76 determines the prediction mode that gives the minimum value among the calculated cost function values ​​as the optimal inter-prediction mode.

[0373] The motion prediction and compensation unit 76 supplies the predicted image (or the difference between the intersecting image and the second-order residual) generated in the optimal interpretation prediction mode, and its cost function value, to the predicted image selection unit 77. If the predicted image selection unit 77 selects a predicted image generated in the optimal interpretation prediction mode, the motion prediction and compensation unit 76 outputs information indicating the optimal interpretation prediction mode to the reversible coding unit 66.

[0374] At this time, motion vector information, reference frame information, a secondary prediction flag indicating that secondary prediction will be performed, and information on the intra prediction mode in secondary prediction are also output to the reversible coding unit 66. The reversible coding unit 66 also performs reversible coding processing such as variable length coding and arithmetic coding on the information from the motion prediction and compensation unit 76 and inserts it into the header portion of the compressed image.

[0375] The secondary prediction unit 361 reads out the adjacent pixels adjacent to the target block to be interprocessed from the frame memory 72, based on the motion vector information from the motion prediction and compensation unit 76 and the image information to be interprocessed. The secondary prediction unit 361 also reads out the adjacent reference pixels adjacent to the reference block associated with the target block by the motion vector information from the frame memory 72.

[0376] The secondary prediction unit 361 performs secondary prediction processing. Secondary prediction processing is a process that performs intraprediction between the primary residual and the difference between the target adjacent pixel and the reference adjacent pixel, and generates secondary difference information (secondary residual).

[0377] Here, the secondary prediction process will be explained with reference to Figure 32.

[0378] In the example in Figure 32, the target frame and the reference frame are shown, and the target frame shows target block A.

[0379] When a motion vector mv(mv_x,mv_y) is obtained for target block A in the reference frame and the target frame, the difference information (residual) between target block A and the block associated with target block A by the motion vector mv is calculated.

[0380] In the second-order prediction method, not only difference information about the target block A is calculated, but also difference information between the adjacent pixel group R adjacent to the target block A and the adjacent pixel group R1 which is associated with the adjacent pixel group R by motion vector mv.

[0381] In other words, the coordinates of each pixel in the adjacent pixel group R are determined from the coordinates (x,y) of the top left of the target block A. Also, the coordinates of each pixel in the adjacent pixel group R1 are determined from the coordinates (x+mv_x,y+mv_y) of the top left of the block associated with the target block A by the motion vector mv. The difference information of the adjacent pixel groups is calculated using these coordinate values.

[0382] In the quadratic prediction method, intra-prediction in the H.264 / AVC method is performed between the difference information for the target block calculated in this way and the difference information for adjacent pixels, thereby generating quadratic difference information. The generated quadratic difference information is orthogonally transformed, quantized, encoded together with the compressed image, and sent to the decoding side.

[0383] Prior to this secondary prediction, the secondary prediction unit 361 uses the filter coefficients set by the adjacent pixel interpolation filter switching unit 362 to perform a filter on the difference between the target adjacent pixel and the reference adjacent pixel used for intra prediction. Then, the secondary prediction unit 361 performs a secondary prediction using the filtered difference between the target adjacent pixel and the reference adjacent pixel, and outputs the obtained secondary difference information (secondary residual) to the motion prediction and compensation unit 76.

[0384] In other words, the secondary prediction unit 361 is configured to include the intra prediction unit 74 shown in Figure 14, etc.

[0385] The adjacent pixel interpolation filter switching unit 362 is basically configured in the same way as the adjacent pixel interpolation filter switching unit 75 in Figure 1 and performs the same processing. That is, the adjacent pixel interpolation filter switching unit 362 sets filter coefficients according to the intra prediction mode information from the second-order prediction unit 361 and the quantization parameters from the rate control unit 78, and supplies the set filter coefficients to the second-order prediction unit 361.

[0386] Note that the encoding process of the image encoding device 351 in Figure 31 differs only in the intra processing and motion prediction processing described below, and the other processes are basically the same as the encoding process of Figure 16 by the image encoding device 51 in Figure 1, so their explanation will be omitted.

[0387] In other words, in the image encoding device 351 of Figure 31, intra-prediction in the H.264 / AVC scheme is performed as intra-processing. In addition, as motion prediction processing, the filter coefficients set by the adjacent pixel interpolation filter switching unit 362 are used during the motion prediction processing to generate second-order difference information. Then, the one with better encoding efficiency is selected from the first-order difference information and the second-order difference information, and the optimal inter-prediction mode is determined by comparing the cost function values.

[0388] An image decoding device that receives and decodes the compressed image encoded by the image encoding device 351 will be described with reference to Figure 33.

[0389] [Other configuration examples of image decoding devices] Figure 33 shows the configuration of another embodiment of an image decoding device as an image processing device to which the present invention is applied.

[0390] The image decoding device 401 is similar to the image decoding device 151 in Figure 22 in that it comprises a storage buffer 161, a reversible decoding unit 162, an inverse quantization unit 163, an inverse orthogonal transform unit 164, an arithmetic unit 165, a deblocking filter 166, a screen rearrangement buffer 167, a D / A conversion unit 168, a frame memory 169, a switch 170, an intra prediction unit 171, a motion prediction and compensation unit 173, and a switch 174.

[0391] Furthermore, the image decoding device 401 differs from the image decoding device 151 in Figure 22 in that the adjacent pixel interpolation filter switching unit 172 is omitted, and the second-order prediction unit 411 and the adjacent pixel interpolation filter switching unit 412 are added.

[0392] In other words, the intra prediction unit 171 is supplied with information indicating the intra prediction mode obtained by decoding the header information from the reversible decoding unit 162. Based on this information, the intra prediction unit 171 generates a prediction image and outputs the generated prediction image to the switch 174.

[0393] The motion prediction and compensation unit 173 is supplied with information obtained by decoding the header information, including prediction mode information, motion vector information, and reference frame information, from the reversible decoding unit 162. In addition, if secondary prediction processing is applied to the target block, the motion prediction and compensation unit 173 is also supplied with a secondary prediction flag indicating that secondary prediction should be performed, and intra prediction mode information for secondary prediction, from the reversible decoding unit 162.

[0394] If the motion prediction and compensation unit 173 determines that secondary prediction processing is being applied, it controls the secondary prediction unit 411 to perform secondary prediction in the intra prediction mode indicated by the intra prediction mode information in the secondary prediction.

[0395] The motion prediction and compensation unit 173 performs motion prediction and compensation processing on the image based on motion vector information and reference frame information to generate a predicted image. That is, the predicted image of the target block is generated using the pixel values ​​of the reference block that are associated with the target block by motion vector in the reference frame. The motion prediction and compensation unit 173 then adds the generated predicted image with the predicted difference value from the secondary prediction unit 411 and outputs it to the switch 174.

[0396] The secondary prediction unit 411 performs secondary prediction using the difference between the target adjacent pixel and the reference adjacent pixel read from the frame memory 169. Prior to this secondary prediction, the secondary prediction unit 411 performs filtering on the difference between the target adjacent pixel and the reference adjacent pixel using the filter coefficient set by the adjacent pixel interpolation filter switching unit 412. Then, the secondary prediction unit 411 performs secondary prediction processing using the filtered difference between the target adjacent pixel and the reference adjacent pixel, and outputs the obtained secondary difference information (secondary residual) to the motion prediction and compensation unit 173.

[0397] In other words, the secondary prediction unit 411 is configured to include the intra prediction unit 171 shown in Figure 26.

[0398] The adjacent pixel interpolation filter switching unit 412 is configured in basically the same way as the adjacent pixel interpolation filter switching unit 172. That is, the adjacent pixel interpolation filter switching unit 412 sets filter coefficients corresponding to at least one of the quantization parameters and intra prediction modes from the reversible decoding unit 162. The adjacent pixel interpolation filter switching unit 412 supplies the set filter coefficients to the second-order prediction unit 411.

[0399] Note that the encoding process of the image decoding device 401 in Figure 33 differs only in the intra processing and motion prediction processing described below, and the other processes are basically the same as the decoding process in Figure 24 by the image decoding device 151 in Figure 22, so their explanation will be omitted.

[0400] In other words, in the image decoding device 401 of Figure 33, intra prediction in the H.264 / AVC format is performed as intra processing. In addition, as motion prediction processing, the filter coefficients set by the adjacent pixel interpolation filter switching unit 412 are used during the motion prediction processing to perform second-order prediction (intra prediction), and second-order difference information is generated.

[0401] The present invention can also be applied to intra prediction in the secondary prediction process described above.

[0402] In the above explanation, an example was described in which, prior to intra-prediction, the adjacent pixels used for intra-prediction are filtered using filter coefficients adaptively set for the image.

[0403] Incidentally, the noise contained in adjacent pixels used in the intra prediction method varies depending on the coding conditions such as the image content and quantization parameters. For this reason, for example, by applying the filtering process used in the H.264 / AVC method, there are blocks for which the coding efficiency is improved and blocks for which it is not.

[0404] Nevertheless, in the H.264 / AVC scheme, when 8x8 block-based intra prediction was performed on macroblocks, all blocks were uniformly filtered, which resulted in some blocks having reduced encoding efficiency.

[0405] Next, we will explain an example of turning on / off filtering of adjacent pixels used in intra prediction.

[0406] <3. Third Embodiment> [Other configuration examples of image encoding devices] Figure 34 shows the configuration of another embodiment of an image encoding device as an image processing device to which the present invention is applied.

[0407] The image encoding device 451 is similar to the image encoding device 51 in Figure 1 in that it includes an A / D conversion unit 61, a screen sorting buffer 62, a calculation unit 63, an orthogonal transformation unit 64, a quantization unit 65, a reversible encoding unit 66, a storage buffer 67, an inverse quantization unit 68, an inverse orthogonal transformation unit 69, a calculation unit 70, a deblocking filter 71, a frame memory 72, a switch 73, an intra prediction unit 74, a motion prediction and compensation unit 76, a predicted image selection unit 77, and a rate control unit 78.

[0408] Furthermore, the image encoding device 451 differs from the image encoding device 51 in Figure 1 in that the adjacent pixel interpolation filter switching unit 75 is replaced by the adjacent pixel interpolation filter control unit 461.

[0409] In other words, the adjacent pixel interpolation filter control unit 461 controls the filtering of adjacent pixels, which was uniformly performed for all blocks when 8x8 block-based intra prediction is performed in the H.264 / AVC system, by turning it on or off. Note that in the H.264 / AVC system, filtering was performed only for 8x8 block-based intra prediction, but in the image encoding device 451, filtering is also performed for intra 4x4 and intra 16x16.

[0410] The on / off control signal for the filter processing from the adjacent pixel interpolation filter control unit 461 is supplied to the intra prediction unit 74.

[0411] The intra prediction unit 74 performs intra prediction processing for all candidate intra prediction modes based on the intra prediction image read from the screen sorting buffer 62 and the reference image supplied from the frame memory 72, and generates a predicted image. At this time, prior to intra prediction, the intra prediction unit 74 turns the filter processing on or off according to the control signal from the adjacent pixel interpolation filter control unit 461, and intra prediction is performed, and as a result, the one with the smaller cost function value is adopted.

[0412] Furthermore, the intra prediction unit 74 generates a flag indicating whether this filtering process is on or off. This flag information is supplied to the reversible coding unit 66 along with information indicating the optimal intra prediction mode when the prediction image selection unit 77 selects a prediction image generated in the optimal intra prediction mode.

[0413] [Example of Intra Prediction Unit Configuration] Figure 35 is a block diagram showing a detailed configuration example of the intra-prediction unit 74 in Figure 34. In the example of Figure 35, the intra-prediction unit 74 is composed of a prediction image generation unit 471, a cost function value generation unit 472, and a mode and on / off determination unit 473.

[0414] The prediction image generation unit 471 is supplied with adjacent pixel values ​​of the target block for intra prediction from the frame memory 72. In the case of Figure 35, the switch 73 is not shown, but in reality, the adjacent pixel values ​​are supplied from the frame memory 72 to the prediction image generation unit 471 via the switch 73. In the case of intra prediction, the adjacent pixel values ​​used are the pixel values ​​that have not been deblocked by the deblocking filter 71.

[0415] Based on the control signal from the adjacent pixel interpolation filter control unit 461, the prediction image generation unit 471 performs filtering on adjacent pixel values, or performs intra-prediction processing for all candidate intra-prediction modes without filtering, and generates a prediction image. The on / off control by the adjacent pixel interpolation filter control unit 461 indicated by this control signal is performed in block units or macroblock units, as will be described later with reference to Figures 36 to 38.

[0416] The prediction image generation unit 471 supplies the generated prediction image pixel values ​​and the information of its intra prediction mode to the cost function value generation unit 472.

[0417] The cost function value generation unit 472 is supplied with original image pixel values ​​from the screen sorting buffer 62. The cost function value generation unit 472 uses the original image pixel values ​​and predicted image pixel values ​​to calculate the cost function value for each intra prediction mode when the filter processing is on or off. The cost function value generation unit 472 supplies the calculated cost function value, predicted image pixel values, and intra prediction mode information to the mode and on / off determination unit 473.

[0418] The mode and on / off determination unit 473 uses the cost function value from the cost function value generation unit 472 to determine the optimal intra prediction mode and whether to set the filter processing on or off, and generates on / off flag information indicating whether the filter processing is on or off.

[0419] The mode and on / off determination unit 473 supplies the predicted image pixel values ​​of the optimal intra prediction mode to the predicted image selection unit 77. If the predicted image of the optimal intra prediction mode is selected by the predicted image selection unit 77, the mode and on / off determination unit 473 supplies information indicating the optimal intra prediction mode and the corresponding on / off flag information to the reversible encoding unit 66.

[0420] The processing performed by the image encoding device 451 is the same as that performed by the image encoding device 51 in Figure 1, except for the intra prediction processing described below, so the explanation will be omitted as it would be repetitive.

[0421] Next, referring to the flowchart in Figure 36, the intra prediction process of the intra prediction unit 74 in Figure 34 when on or off control is performed on a block-by-block basis will be explained. Note that this process is another example of the intra prediction process in step S31 of Figure 17, and in the example in Figure 36, an intra 4x4 will be used as an example. Also, hereafter, the filter processing on / off will be simply referred to as filter on / off.

[0422] In step S401, the cost function value generation unit 472 generates cost function values ​​for the target block for each of the nine intra prediction modes shown in Figure 3 or Figure 4.

[0423] In other words, the prediction image generation unit 471 is supplied with adjacent pixel values ​​of the target block for intra prediction from the frame memory 72. The prediction image generation unit 471 performs intra prediction for each of the nine intra prediction modes shown in Figure 3 or Figure 4 and generates a prediction image of the target block.

[0424] At this time, the adjacent pixel interpolation filter control unit 461 supplies a control signal that does not perform filtering on adjacent pixels, and the prediction image generation unit 471 performs intra prediction without performing filtering on adjacent pixels. Note that a control signal that performs filtering on adjacent pixels may also be supplied here. However, rather than performing different controls, for example, performing filtering for Vertical but not for Horizontal among the nine modes, the same control should be applied to all nine modes to decide whether or not to perform filtering. Also, in this case, the amount of computation is reduced if filtering is not performed for all modes.

[0425] The prediction image generation unit 471 supplies the generated prediction image pixel values ​​and the intra prediction mode information to the cost function value generation unit 472. The cost function value generation unit 472 uses the original image pixel values ​​and prediction image pixel values ​​from the screen sorting buffer 62 to calculate the cost function value shown by equation (73) or equation (74) above for each intra prediction mode when the filter is off. The cost function value generation unit 472 supplies the calculated cost function value, prediction image pixel values, and intra prediction mode information to the mode and on / off determination unit 473.

[0426] In step S402, the mode and on / off determination unit 473 selects the optimal intra prediction mode for the target block using the cost function value from the cost function value generation unit 472. The selected intra prediction mode information is supplied to the adjacent pixel interpolation filter control unit 461.

[0427] In step S403, the adjacent pixel interpolation filter control unit 461 causes the cost function value generation unit 472 to generate cost function values ​​for filter on and filter off for the selected intra prediction mode. Note that since the cost function value for filter off was generated in step S401, in fact, the cost function value for filter on is generated in step S403.

[0428] In other words, the adjacent pixel interpolation filter control unit 461 supplies the prediction image generation unit 471 with a control signal to turn the filter on and information about the selected intra prediction mode. The prediction image generation unit 471 performs filtering on the adjacent pixel values ​​used in the selected intra prediction mode, performs intra prediction in the selected intra prediction mode, and generates a prediction image of the target block.

[0429] The prediction image generation unit 471 supplies the generated prediction image pixel values ​​and the information of the selected intra prediction mode to the cost function value generation unit 472. The cost function value generation unit 472 uses the original image pixel values ​​and prediction image pixel values ​​from the screen sorting buffer 62 to calculate the cost function value shown by equation (73) or equation (74) above for the selected intra prediction mode when the filter is on. The cost function value generation unit 472 supplies the calculated cost function value and prediction image pixel values ​​to the mode and on / off determination unit 473.

[0430] In step S404, the mode and on / off determination unit 473 compares the cost function values ​​of filter on and filter off for the selected intra prediction mode and determines whether the filter is on or off for the target block. That is, if the cost function value of filter on is small, it is determined that the filter is on for the target block, and if the cost function value of filter off is small, it is determined that the filter is off for the target block. The mode and on / off determination unit 473 then supplies the determined prediction image value to the prediction image selection unit 77.

[0431] In step S405, the mode and on / off determination unit 473 generates an on / off flag for the target block indicating whether it is on or off as determined in step S404. For example, if the filter is on, the value of the on / off filter is set to 1. If the filter is off, the value of the on / off filter is set to 0.

[0432] The generated on / off flag information is supplied to the reversible encoding unit 66 along with information indicating the optimal intra prediction mode when the prediction image for the intra prediction mode is selected in step S22 of Figure 16 described above. Then, in step S23 of Figure 16, this supplied information is encoded, added to the header of the compressed image, and transmitted to the decoding side.

[0433] Next, with reference to the flowchart in Figure 37, another example of the intra prediction processing of the intra prediction unit 74 in Figure 34 when on or off control is performed on a block-by-block basis will be described. In the example of Figure 37, an intra 4x4 will also be used as an example.

[0434] In step S421, the cost function value generation unit 472 generates cost function values ​​for each intra prediction mode for the target block, based on whether the filter is on or off.

[0435] In other words, the prediction image generation unit 471 is supplied with adjacent pixel values ​​of the target block for intra prediction from the frame memory 72. The prediction image generation unit 471 performs intra prediction for each of the nine intra prediction modes shown in Figure 3 or Figure 4 and generates a prediction image of the target block.

[0436] At this time, first, the adjacent pixel interpolation filter control unit 461 supplies a control signal that does not perform filtering on adjacent pixels, and the prediction image generation unit 471 performs intra-prediction for each intra-prediction mode without performing filtering on adjacent pixels and generates a prediction image. Furthermore, the adjacent pixel interpolation filter control unit 461 supplies a control signal that performs filtering on adjacent pixels, and the prediction image generation unit 471 performs filtering on adjacent pixels, performs intra-prediction for each intra-prediction mode and generates a prediction image.

[0437] The predictive image generation unit 471 supplies the information for each intra-prediction mode, filter on and off, and the corresponding predicted image pixel values ​​to the cost function value generation unit 472. The cost function value generation unit 472 uses the original image pixel values ​​and predicted image pixel values ​​from the screen sorting buffer 62 to calculate the cost function value shown by the above-described equation (73) or equation (74) for each intra-prediction mode, for both the case of filter processing off and on. The cost function value generation unit 472 supplies the calculated cost function value for each case of filter processing on and off, the predicted image pixel values, and the information for the intra-prediction mode to the mode and on / off determination unit 473.

[0438] In step S422, the mode and on / off determination unit 473 uses the cost function value from the cost function value generation unit 472 to determine whether to turn the filter on or off for each intra prediction mode for the target block.

[0439] Furthermore, in step S423, the mode and on / off determination unit 473 selects the optimal intra prediction mode for the target block from among the intra prediction modes for which the filter on or off has been determined.

[0440] In step S424, the mode and on / off determination unit 473 generates an on / off flag indicating the state (on or off) of the filter of the selected intra prediction mode. The generated on / off flag information is supplied to the reversible encoding unit 66 along with information indicating the optimal intra prediction mode when a prediction image of the intra prediction mode is selected in step S22 of Figure 16 described above. Then, in step S23 of Figure 16, this supplied information is encoded, added to the header of the compressed image, and transmitted to the decoding side.

[0441] Next, referring to the flowchart in Figure 38, the intra prediction process of the intra prediction unit 74 in Figure 34 when on or off control is performed on a macroblock basis will be explained. Note that this process is another example of the intra prediction process in step S31 of Figure 17, and in the example in Figure 38, an intra 4x4 will be used as an example.

[0442] In step S451, the adjacent pixel interpolation filter control unit 461 fixes the filter for the entire macroblock to either off or on. In this case, the adjacent pixel interpolation filter control unit 461 fixes the filter to off and supplies a control signal to turn the filter off to the predictive image generation unit 471. The filter can be fixed to either on or off, but fixing it to off allows it to be achieved with less computation.

[0443] In step S452, the intra prediction unit 74 determines an intra prediction mode for each block. That is, the prediction image generation unit 471 is supplied with adjacent pixel values ​​of the target block for intra prediction from the frame memory 72. The prediction image generation unit 471 performs intra prediction for each of the nine intra prediction modes shown in Figure 3 or Figure 4 and generates a prediction image of the target block.

[0444] At this time, the adjacent pixel interpolation filter control unit 461 supplies a control signal that does not perform filtering on adjacent pixels, and the prediction image generation unit 471 performs intra-prediction without filtering on adjacent pixels and generates a prediction image. The prediction image generation unit 471 supplies the generated prediction image pixel values ​​and the information of its intra-prediction mode to the cost function value generation unit 472.

[0445] The cost function value generation unit 472 uses the original image pixel values ​​and predicted image pixel values ​​from the screen sorting buffer 62 to calculate the cost function value shown by equation (73) or equation (74) above for each intra prediction mode when the filter is off. The cost function value generation unit 472 supplies the calculated cost function value, predicted image pixel values, and intra prediction mode information to the mode and on / off determination unit 473.

[0446] The mode and on / off determination unit 473 uses the cost function value from the cost function value generation unit 472 to determine the optimal intra prediction mode for each block. The determined intra prediction mode information is supplied to the adjacent pixel interpolation filter control unit 461.

[0447] In step S453, the adjacent pixel interpolation filter control unit 461 causes the cost function value generation unit 472 to generate cost function values ​​for filter on and off for the entire macroblock. Note that the cost function value for filter off for the optimal intra prediction mode of each block within the macroblock (i.e., the entire macroblock) is generated in step S452. Therefore, in practice, in step S453, the cost function value for filter on for the entire macroblock is generated.

[0448] In other words, the adjacent pixel interpolation filter control unit 461 supplies the prediction image generation unit 471 with a control signal to turn the filter on and information of the intra prediction mode determined for each block. The prediction image generation unit 471 performs filtering on the adjacent pixel values ​​used in the determined intra prediction mode, performs intra prediction in the determined intra prediction mode, and generates a prediction image of the target block.

[0449] The prediction image generation unit 471 supplies the generated prediction image pixel values ​​and the determined intra-prediction mode information to the cost function value generation unit 472. The cost function value generation unit 472 uses the original image pixel values ​​and prediction image pixel values ​​from the screen sorting buffer 62 to calculate the cost function value shown by equation (73) or equation (74) above for the determined intra-prediction mode when the filter processing is on. The cost function value generation unit 472 supplies the calculated cost function values ​​for the filter on and off cases, and the prediction image pixel values, to the mode and on / off determination unit 473.

[0450] In step S454, the mode and on / off determination unit 473 compares the cost function values ​​of all blocks in the macroblock in the cases of filter on and off from the cost function value generation unit 472, and determines whether to apply filter on or off to the entire macroblock.

[0451] In step S455, the mode and on / off determination unit 473 generates an on / off flag for the entire macroblock indicating whether it is on or off as determined in step S454. The generated on / off flag information is supplied to the reversible encoding unit 66 for each macroblock. This supplied information is then encoded in step S23, added to the header of the compressed image, and transmitted to the decoding side.

[0452] As described above, the on / off (on or off) control of the filter can be performed at the block level or at the macroblock level. Controlling on / off at the block level can improve the prediction accuracy of the intra prediction processing, but it increases the amount of information required to transmit flag information for each block. In contrast, when controlling at the macroblock level, the improvement in prediction accuracy is lower than when controlling at the block level, but since only one flag is needed for each macroblock, the increase in the amount of information due to flags can be reduced.

[0453] In the above explanation, an example of a luminance signal was described, but it can also be used for intra-prediction of a chrominance signal. Furthermore, the filter coefficients in the controlled filtering process are not limited to the three taps {1,2,1} / / 4 in the H.264 / AVC method, but can be applied to any coefficient with any tap length, or to the filter coefficients set in the image encoding device 51 in Figure 1.

[0454] In other words, when the filter is on, it is also possible to perform filtering using the filter coefficients set in the image encoding device 51 in Figure 1.

[0455] An image decoding device that receives and decodes the compressed image encoded by the image encoding device 451 will be described with reference to Figure 39.

[0456] [Other configuration examples of image decoding devices] Figure 39 shows the configuration of another embodiment of an image decoding device as an image processing device to which the present invention is applied.

[0457] The image decoding device 501 is similar to the image decoding device 151 in Figure 22 in that it comprises a storage buffer 161, a reversible decoding unit 162, an inverse quantization unit 163, an inverse orthogonal transform unit 164, an arithmetic unit 165, a deblocking filter 166, a screen rearrangement buffer 167, a D / A conversion unit 168, a frame memory 169, a switch 170, an intra prediction unit 171, a motion prediction and compensation unit 173, and a switch 174.

[0458] Furthermore, the image decoding device 501 differs from the image decoding device 151 in Figure 22 in that the adjacent pixel interpolation filter switching unit 172 is replaced by the adjacent pixel interpolation filter control unit 511.

[0459] In other words, the intra prediction unit 171 is supplied with information indicating the intra prediction mode obtained by decoding the header information from the reversible decoding unit 162. The intra prediction unit 171 generates a predicted image by performing intra prediction according to the intra prediction mode indicated by the information, and outputs the generated predicted image to the switch 174. At this time, prior to intra prediction, the intra prediction unit 171 performs filtering on adjacent pixel values ​​in accordance with the control signal from the adjacent pixel interpolation filter control unit 511 (or does not perform filtering).

[0460] The adjacent pixel interpolation filter control unit 511 is supplied with on / off flag information from the reversible decoding unit 162 for each macroblock or block, in accordance with the encoding by the image encoding device 451. The adjacent pixel interpolation filter control unit 511 supplies a control signal to the intra prediction unit 171 indicating whether or not to perform filtering, according to the supplied on / off flag information.

[0461] In the image encoding device 451 in Figure 34, both the case where the filter is on and the case where it is off are tested, and the one that is determined to give a higher encoding efficiency based on the cost function value is selected and intra prediction processing is performed. In contrast, in the image decoding device 501, the filter is controlled to be on or off based on the on / off flag information that is encoded and sent, and intra prediction processing is performed.

[0462] [Example of configuration of the intra prediction unit and adjacent pixel interpolation filter unit] Figure 40 is a block diagram showing a detailed configuration example of the intra prediction unit and the adjacent pixel interpolation filter control unit.

[0463] In the example of Figure 40, the intra prediction unit 171 is composed of a prediction mode buffer 521 and a prediction image generation unit 522. The adjacent pixel interpolation filter control unit 511 is composed of a flag buffer 531 and a control signal generation unit 532.

[0464] The prediction mode buffer 521 is supplied with intra-prediction mode information from the reversible decoding unit 162. The prediction image generation unit 522 is supplied with adjacent pixel values ​​of the target block for intra-prediction from the frame memory 169. In the case of Figure 40, although the switch 170 is not shown, in reality, adjacent pixel values ​​are supplied from the frame memory 169 to the prediction image generation unit 522 via the switch 170.

[0465] The prediction image generation unit 522 reads intra prediction mode information for the target block from the prediction mode buffer 521, performs intra prediction for the target block using the read intra prediction mode, and generates a prediction image. Prior to this intra prediction, the prediction image generation unit 522 filters the adjacent image pixel values ​​from the frame memory 169 in accordance with the control signal from the control signal generation unit 532.

[0466] On / off flag information is supplied to the flag buffer 531 from the reversible decoding unit 162 for each macroblock or block. The control signal generation unit 532 reads the corresponding on / off flag from the flag buffer 531, generates a control signal for each block indicating whether or not to perform filtering, and supplies the generated control signal to the prediction image generation unit 522.

[0467] The processing performed by the image decoding device 501 is the same as that performed by the image decoding device 151 in Figure 22, except for the prediction process described below, so the explanation will be omitted as it would be repetitive.

[0468] [Explanation of prediction processing] Next, the prediction process of the image decoding device 501 in Figure 39 will be explained with reference to the flowchart in Figure 41. This process is another example of the prediction process in step S138 of Figure 24.

[0469] In step S501, the predictive image generation unit 522 determines whether or not the target block is intra-encoded. Intra-predictive mode information is supplied from the reversible decoding unit 162 to the predictive mode buffer 521, and the predictive image generation unit 522 reads it out. In response, the predictive image generation unit 522 determines in step S501 that the target block is intra-encoded, and the process proceeds to step S502.

[0470] In step S502, the predictive image generation unit 522 acquires intra predictive mode information from the predictive mode buffer 521.

[0471] When on / off flag information is supplied to the flag buffer 531 from the reversible decoding unit 162, the flag buffer 531 acquires and stores the on / off flag in step S503.

[0472] The control signal generation unit 532 reads the on / off flag corresponding to the target block from the flag buffer 531 and determines in step S504 whether the on / off flag is 1 or not. If in step S504 it is determined that the on / off flag is 1, that is, that the filter processing is on, the control signal generation unit 532 supplies a control signal to the prediction image generation unit 522 to perform the filter processing.

[0473] In response to this control signal, the prediction image generation unit 522 performs filtering on adjacent pixels using filter coefficients in step S505. Then, in step S506, the prediction image generation unit 522 performs intraprediction using the filtered adjacent pixel values ​​and generates a prediction image.

[0474] On the other hand, if in step S504 it is determined that the on / off flag is not 1, that is, that the filtering process is off, the filtering process in step S505 is skipped and the process proceeds to step S506.

[0475] In step S506, the predicted image generation unit 522 performs intraprediction using adjacent image pixel values ​​from the frame memory 169 and generates a predicted image.

[0476] The predicted image generated in step S506 is supplied to the switch 174.

[0477] On the other hand, if it is determined in step S501 that the data is not intra-encoded, the process proceeds to step S507.

[0478] In step S507, the motion prediction and compensation unit 173 performs inter-motion prediction. That is, if the image to be processed is an image to be inter-predicted, the necessary image is read from the frame memory 169 and supplied to the motion prediction and compensation unit 173 via the switch 170. In step S508, the motion prediction and compensation unit 173 performs motion prediction in inter-prediction mode based on the motion vector acquired in step S507 and generates a predicted image. The generated predicted image is output to the switch 174.

[0479] As described above, the image encoding device 451 and the image decoding device 501 control whether filtering is performed on adjacent pixels used for intra prediction, and filtering is not performed on blocks where the encoding efficiency would decrease. This improves the encoding efficiency.

[0480] In the above explanation, an example of performing intra prediction was described, but referring to Figure 32, the on or off control of the filter processing can also be applied to the intra prediction in the secondary prediction described above.

[0481] <4. Fourth Embodiment> [Other configuration examples of image encoding devices] Figure 42 shows the configuration of another embodiment of an image encoding device as an image processing device to which the present invention is applied.

[0482] The image encoding device 551 is the same as the image encoding device 451 in Figure 34 in that it includes an A / D conversion unit 61, a screen sorting buffer 62, a calculation unit 63, an orthogonal transformation unit 64, a quantization unit 65, a reversible encoding unit 66, a storage buffer 67, an inverse quantization unit 68, an inverse orthogonal transformation unit 69, a calculation unit 70, a deblocking filter 71, a frame memory 72, a switch 73, an intra prediction unit 74, a motion prediction and compensation unit 76, a predicted image selection unit 77, and a rate control unit 78.

[0483] Furthermore, the image encoding device 551 differs from the image encoding device 451 in Figure 34 in that the adjacent pixel interpolation filter control unit 461 is omitted, and the second-order prediction unit 361 and the adjacent pixel interpolation filter control unit 561 shown in Figure 31 are added.

[0484] In other words, in the example shown in Figure 42, the intra prediction unit 74 performs intra prediction of H.264 / AVC.

[0485] Meanwhile, the motion prediction and compensation unit 76 detects motion vectors for all candidate interprediction modes based on the interprocessing image and the reference image, performs compensation processing on the reference image based on the motion vectors, and generates a predicted image.

[0486] The motion prediction and compensation unit 76 supplies the detected motion vector information, information about the image to be interprocessed (such as the address), and the first-order residual, which is the difference between the image to be interprocessed and the generated predicted image, to the second-order prediction unit 361.

[0487] The motion prediction and compensation unit 76 determines the optimal intra prediction mode for the second-order prediction by comparing it with the second-order residuals from the second-order prediction unit 361. The motion prediction and compensation unit 76 also determines whether to encode the second-order residual or the first-order residual by comparing the second-order residual and the first-order residual. These processes are performed for all candidate intra prediction modes.

[0488] The motion prediction and compensation unit 76 then calculates cost function values ​​for all candidate inter-prediction modes. At this time, the cost function values ​​are calculated using the residuals determined for each inter-prediction mode from among the first-order residuals and second-order residuals. The motion prediction and compensation unit 76 determines the prediction mode that gives the minimum value among the calculated cost function values ​​as the optimal inter-prediction mode.

[0489] The motion prediction and compensation unit 76 supplies the predicted image (or the difference between the intersecting image and the second-order residual) generated in the optimal interpretation prediction mode, and its cost function value, to the predicted image selection unit 77. If the predicted image selection unit 77 selects a predicted image generated in the optimal interpretation prediction mode, the motion prediction and compensation unit 76 outputs information indicating the optimal interpretation prediction mode to the reversible coding unit 66.

[0490] At this time, motion vector information, reference frame information, a secondary prediction flag indicating that secondary prediction will be performed, information on the intra prediction mode in secondary prediction, and on / of flag information in secondary prediction are also output to the reversible coding unit 66. The reversible coding unit 66 also performs reversible coding processing such as variable-length coding and arithmetic coding on the information from the motion prediction and compensation unit 76 and inserts it into the header portion of the compressed image.

[0491] The secondary prediction unit 361 reads out the adjacent pixels adjacent to the target block to be interprocessed from the frame memory 72, based on the motion vector information from the motion prediction and compensation unit 76 and the image information to be interprocessed. The secondary prediction unit 361 also reads out the adjacent reference pixels adjacent to the reference block associated with the target block by the motion vector information from the frame memory 72.

[0492] The secondary prediction unit 361 performs the secondary prediction process described above with reference to Figure 32. The secondary prediction process is a process that performs intraprediction between the primary residual and the difference between the target adjacent pixel and the reference adjacent pixel to generate secondary difference information (secondary residual).

[0493] However, prior to this secondary prediction, the secondary prediction unit 361 in Figure 42 performs (or does not perform) filtering on the difference between the target adjacent pixel and the reference adjacent pixel used for intra-prediction, in accordance with a control signal from the adjacent pixel interpolation filter control unit 561. Then, the secondary prediction unit 361 performs secondary prediction processing using the filtered (or unfiltered) difference between the target adjacent pixel and the reference adjacent pixel, and outputs the obtained secondary difference information (secondary residual) to the motion prediction and compensation unit 76. At this time, the secondary prediction unit 361 also outputs on / off flag information to the motion prediction and compensation unit 76 indicating whether or not to perform filtering.

[0494] In other words, the secondary prediction unit 361 is configured to include the intra prediction unit 74 shown in Figure 35.

[0495] The adjacent pixel interpolation filter control unit 561 is basically configured in the same way as the adjacent pixel interpolation filter control unit 461 in Figure 34 and performs the same processing. That is, the adjacent pixel interpolation filter control unit 561 supplies a control signal to the secondary prediction unit 361 that controls whether or not to perform filtering processing on a block-by-block or macroblock-by-macroblock basis.

[0496] Note that the encoding process of the image encoding device 551 in Figure 42 differs only in the intra processing and motion prediction processing described below. The other processes are basically the same as the encoding process of the image encoding device 451 in Figure 34 (i.e., the encoding process in Figure 16), so their explanation will be omitted.

[0497] In other words, in the image encoding device 551 in Figure 42, intra-prediction in the H.264 / AVC scheme is performed as intra-processing. In addition, as motion prediction processing, during motion prediction processing, filtering is controlled according to the control signal from the adjacent pixel interpolation filter control unit 561, and filtered (or unfiltered) secondary difference information is generated. Then, the one with better encoding efficiency is selected from the primary difference information and the secondary difference information, and the optimal inter-prediction mode is determined by comparing the cost function values.

[0498] An image decoding device that receives and decodes the compressed image encoded by the image encoding device 551 will be described with reference to Figure 43.

[0499] [Other configuration examples of image decoding devices] Figure 43 shows the configuration of another embodiment of an image decoding device as an image processing device to which the present invention is applied.

[0500] The image decoding device 601 is similar to the image decoding device 501 in Figure 39 in that it comprises a storage buffer 161, a reversible decoding unit 162, an inverse quantization unit 163, an inverse orthogonal transform unit 164, an arithmetic unit 165, a deblocking filter 166, a screen rearrangement buffer 167, a D / A conversion unit 168, a frame memory 169, a switch 170, an intra prediction unit 171, a motion prediction and compensation unit 173, and a switch 174.

[0501] Furthermore, the image decoding device 601 differs from the image decoding device 501 in Figure 39 in that the adjacent pixel interpolation filter control unit 511 is omitted, and the secondary prediction unit 411 and the adjacent pixel interpolation filter control unit 611 shown in Figure 33 are added.

[0502] In other words, the intra prediction unit 171 is supplied with information indicating the intra prediction mode obtained by decoding the header information from the reversible decoding unit 162. Based on this information, the intra prediction unit 171 generates a prediction image and outputs the generated prediction image to the switch 174.

[0503] The motion prediction and compensation unit 173 is supplied with information obtained by decoding the header information, including prediction mode information, motion vector information, and reference frame information, from the reversible decoding unit 162. In addition, if secondary prediction processing is applied to the target block, the motion prediction and compensation unit 173 is also supplied with a secondary prediction flag indicating that secondary prediction should be performed, and intra prediction mode information for secondary prediction, from the reversible decoding unit 162.

[0504] If the motion prediction and compensation unit 173 determines that secondary prediction processing is being applied, it controls the secondary prediction unit 411 to perform secondary prediction in the intra prediction mode indicated by the intra prediction mode information in the secondary prediction.

[0505] The motion prediction and compensation unit 173 performs motion prediction and compensation processing on the image based on motion vector information and reference frame information to generate a predicted image. That is, the predicted image of the target block is generated using the pixel values ​​of the reference block that are associated with the target block by motion vector in the reference frame. The motion prediction and compensation unit 173 then adds the generated predicted image with the predicted difference value from the secondary prediction unit 411 and outputs it to the switch 174.

[0506] The secondary prediction unit 411 performs secondary prediction using the difference between the target adjacent pixel and the reference adjacent pixel read from the frame memory 169. However, if the secondary prediction unit 411 in Figure 43 receives a control signal from the adjacent pixel interpolation filter control unit 611 to control the performance of filtering, it performs filtering on the difference between the target adjacent pixel and the reference adjacent pixel prior to this secondary prediction. Then, the secondary prediction unit 411 performs secondary prediction processing using the filtered difference between the target adjacent pixel and the reference adjacent pixel, and outputs the obtained secondary difference information (secondary residual) to the motion prediction and compensation unit 173.

[0507] If the adjacent pixel interpolation filter control unit 611 receives a control signal that controls not to perform filtering, the secondary prediction unit 411 will not perform filtering and will perform secondary prediction processing using the difference between the target adjacent pixel and the reference adjacent pixel.

[0508] In other words, the secondary prediction unit 411 is configured to include the intra prediction unit 171 shown in Figure 40.

[0509] The adjacent pixel interpolation filter control unit 611 is configured in basically the same way as the adjacent pixel interpolation filter control unit 511 in Figure 39 and performs basically the same processing. That is, the adjacent pixel interpolation filter control unit 611 is supplied with on / off flag information from the reversible decoding unit 162, which is obtained by decoding the header information. The adjacent pixel interpolation filter control unit 611 supplies a control signal to the secondary prediction unit 411 to either perform filtering of adjacent pixels or not, according to the on / off flag information.

[0510] Note that the decoding process of the image decoding device 601 in Figure 43 differs only in the intra processing and motion prediction processing described below. The other processes are basically the same as the decoding process of the image decoding device 501 in Figure 39 (i.e., the decoding process in Figure 24), so their explanation will be omitted.

[0511] In other words, in the image decoding device 601 of Figure 43, intra prediction in the H.264 / AVC format is performed as intra processing. In addition, as motion prediction processing, during motion prediction processing, filtering is performed according to the control signal from the adjacent pixel interpolation filter control unit 611, and secondary prediction (intra prediction) is performed, and secondary difference information is generated.

[0512] The on / off control of the filter process can also be applied to the intra-prediction in the secondary prediction process described above.

[0513] In the above description, we have described the case where the size of the macroblock is 16 x 16 pixels, but the present invention can also be applied to the extended macroblock size described in Patent Document 3.

[0514] Figure 44 shows an example of an expanded macroblock size. In Patent Document 3, the macroblock size is expanded to 32 x 32 pixels.

[0515] The upper section of Figure 44 shows, from left to right, macroblocks consisting of 32x32 pixels, 32x16 pixels, 16x32 pixels, and 32x32 pixels divided into 16x16 pixel blocks (partitions). The middle section of Figure 44 shows, from left to right, blocks consisting of 16x16 pixels, 16x8 pixels, 8x16 pixels, and 16x16 pixels divided into 8x8 pixel blocks. The lower section of Figure 44 shows, from left to right, blocks of 8x8 pixels, 8x4 pixels, 4x8 pixels, and 8x8 pixels divided into 4x4 pixel blocks.

[0516] In other words, a 32x32 pixel macroblock can be processed using blocks of 32x32 pixels, 32x16 pixels, 16x32 pixels, and 16x16 pixels, as shown in the upper part of Figure 44.

[0517] In addition, the 16x16 pixel block shown on the upper right can be processed using the 16x16, 16x8, 8x16, and 8x8 pixel blocks shown in the middle, similar to the H.264 / AVC format.

[0518] Furthermore, the 8x8 pixel block shown on the right side of the middle section can be processed using the 8x8, 8x4, 4x8, and 4x4 pixel blocks shown in the lower section, similar to the H.264 / AVC format.

[0519] By adopting this hierarchical structure, in the extended macroblock size, larger blocks are defined as a superset, while maintaining compatibility with the H.264 / AVC format for blocks of 16x16 pixels and below.

[0520] The setting, calculation, or on / off control of the filter coefficients of the present invention can also be applied to the extended macroblock sizes proposed above.

[0521] In the above, the H.264 / AVC encoding scheme was used, but the present invention is not limited to this, and other encoding / decoding schemes that perform prediction using adjacent pixels (for example, intra prediction or quadratic prediction) can be applied.

[0522] The present invention can be applied to image encoding and decoding devices used when receiving image information (bitstream) compressed by orthogonal transformations such as discrete cosine transform and motion compensation, such as MPEG and H.26x, via network media such as satellite broadcasting, cable television, the internet, or mobile phones. The present invention can also be applied to image encoding and decoding devices used when processing on storage media such as optical, magnetic disk, and flash memory. Furthermore, the present invention can also be applied to motion prediction and compensation devices included in such image encoding and decoding devices.

[0523] The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0524] Figure 45 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0525] In a computer, the CPU (Central Processing Unit) 801, ROM (Read Only Memory) 802, and RAM (Random Access Memory) 803 are interconnected by a bus 804.

[0526] An input / output interface 805 is further connected to the bus 804. An input unit 806, an output unit 807, a storage unit 808, a communication unit 809, and a drive 810 are connected to the input / output interface 805.

[0527] The input unit 806 consists of a keyboard, mouse, microphone, etc. The output unit 807 consists of a display, speaker, etc. The storage unit 808 consists of a hard disk, non-volatile memory, etc. The communication unit 809 consists of a network interface, etc. The drive 810 drives removable media 811 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0528] In a computer configured as described above, the CPU 801 loads, for example, a program stored in the memory unit 808 into the RAM 803 via the input / output interface 805 and the bus 804 and executes it, thereby performing the series of processes described above.

[0529] The program executed by the computer (CPU 801) can be provided by recording it on a removable medium 811, such as a package medium. The program can also be provided via a wired or wireless transmission medium, such as a local area network, the internet, or digital broadcasting.

[0530] In a computer, a program can be installed in the storage unit 808 via the input / output interface 805 by inserting the removable media 811 into the drive 810. Alternatively, a program can be received by the communication unit 809 via a wired or wireless transmission medium and installed in the storage unit 808. In addition, a program can be pre-installed in the ROM 802 or the storage unit 808.

[0531] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0532] The embodiments of the present invention are not limited to those described above, and various modifications are possible without departing from the spirit of the present invention.

[0533] For example, the image encoding devices 51, 351, 451, and 551 and the image decoding devices 151, 401, 501, and 601 described above can be applied to any electronic device. An example is described below.

[0534] Figure 46 is a block diagram showing an example of the main configuration of a television receiver using an image decoding device to which the present invention is applied.

[0535] The television receiver 1300 shown in Figure 46 includes a terrestrial tuner 1313, a video decoder 1315, a video signal processing circuit 1318, a graphics generation circuit 1319, a panel driving circuit 1320, and a display panel 1321.

[0536] The terrestrial tuner 1313 receives the terrestrial analog broadcast wave signal via the antenna, demodulates it, acquires the video signal, and supplies it to the video decoder 1315. The video decoder 1315 decodes the video signal supplied from the terrestrial tuner 1313 and supplies the resulting digital component signal to the video signal processing circuit 1318.

[0537] The video signal processing circuit 1318 performs predetermined processing, such as noise reduction, on the video data supplied from the video decoder 1315, and supplies the resulting video data to the graphic generation circuit 1319.

[0538] The graphics generation circuit 1319 generates video data of programs to be displayed on the display panel 1321, as well as image data obtained through processing based on applications supplied via the network, and supplies the generated video data and image data to the panel drive circuit 1320. The graphics generation circuit 1319 also generates video data (graphics) for displaying screens used by the user for selecting items, etc., and supplies the resulting video data to the panel drive circuit 1320, as appropriate.

[0539] The panel drive circuit 1320 drives the display panel 1321 based on the data supplied from the graphics generation circuit 1319, and displays the program's video and the various screens described above on the display panel 1321.

[0540] The display panel 1321 consists of an LCD (Liquid Crystal Display) or the like, and displays the program's video and other content according to the control of the panel drive circuit 1320.

[0541] The television receiver 1300 also includes an audio A / D (Analog / Digital) conversion circuit 1314, an audio signal processing circuit 1322, an echo cancellation / speech synthesis circuit 1323, an audio amplification circuit 1324, and a speaker 1325.

[0542] The terrestrial tuner 1313 acquires not only the video signal but also the audio signal by demodulating the received broadcast wave signal. The terrestrial tuner 1313 supplies the acquired audio signal to the audio A / D conversion circuit 1314.

[0543] The audio A / D conversion circuit 1314 performs A / D conversion processing on the audio signal supplied from the terrestrial tuner 1313 and supplies the resulting digital audio signal to the audio signal processing circuit 1322.

[0544] The audio signal processing circuit 1322 performs predetermined processing, such as noise reduction, on the audio data supplied from the audio A / D conversion circuit 1314, and supplies the resulting audio data to the echo cancellation / speech synthesis circuit 1323.

[0545] The echo cancellation / speech synthesis circuit 1323 supplies the audio data supplied from the audio signal processing circuit 1322 to the audio amplification circuit 1324.

[0546] The audio amplification circuit 1324 performs D / A conversion and amplification processing on the audio data supplied from the echo cancellation / speech synthesis circuit 1323, adjusts it to a predetermined volume, and then outputs the audio from the speaker 1325.

[0547] Furthermore, the television receiver 1300 also has a digital tuner 1316 and an MPEG decoder 1317.

[0548] The digital tuner 1316 receives broadcast wave signals of digital broadcasts (terrestrial digital broadcasting, BS (Broadcasting Satellite) / CS (Communications Satellite) digital broadcasting) via an antenna, demodulates them, obtains MPEG-TS (Moving Picture Experts Group-Transport Stream), and supplies it to the MPEG decoder 1317.

[0549] The MPEG decoder 1317 decrypts the scramble applied to the MPEG-TS supplied from the digital tuner 1316 and extracts a stream containing the data of the program to be played (viewed). The MPEG decoder 1317 decodes the audio packets that make up the extracted stream and supplies the obtained audio data to the audio signal processing circuit 1322, and also decodes the video packets that make up the stream and supplies the obtained video data to the video signal processing circuit 1318. The MPEG decoder 1317 also supplies the EPG (Electronic Program Guide) data extracted from the MPEG-TS to the CPU 1332 via a path not shown.

[0550] The television receiver 1300 uses the image decoding device 151, 401, 501, or 601 described above as the MPEG decoder 1317 for decoding the video packets in this manner. Therefore, the MPEG decoder 1317, as with the image decoding devices 151 and 401, switches the filter coefficients according to the quantization parameters and prediction mode prior to intra prediction and performs filtering of adjacent pixels. Alternatively, the MPEG decoder 1317, as with the image decoding devices 501 and 601, controls whether or not to perform filtering of adjacent pixels prior to intra prediction based on an on / off flag. This improves the coding efficiency.

[0551] The video data supplied from the MPEG decoder 1317 is subjected to predetermined processing in the video signal processing circuit 1318, similar to the case of video data supplied from the video decoder 1315. The processed video data is then superimposed as appropriate by the graphics generation circuit 1319, and supplied to the display panel 1321 via the panel driving circuit 1320, where the image is displayed.

[0552] The audio data supplied from the MPEG decoder 1317 is subjected to predetermined processing in the audio signal processing circuit 1322, similar to the case of audio data supplied from the audio A / D conversion circuit 1314. The processed audio data is then supplied to the audio amplification circuit 1324 via the echo cancellation / speech synthesis circuit 1323, where D / A conversion and amplification processing are performed. As a result, audio adjusted to a predetermined volume is output from the speaker 1325.

[0553] The television receiver 1300 also has a microphone 1326 and an A / D conversion circuit 1327.

[0554] The A / D conversion circuit 1327 receives the user's voice signal captured by the microphone 1326, which is provided on the television receiver 1300 for voice conversation. The A / D conversion circuit 1327 performs A / D conversion processing on the received voice signal and supplies the resulting digital voice data to the echo cancellation / voice synthesis circuit 1323.

[0555] The echo cancellation / speech synthesis circuit 1323 performs echo cancellation on the voice data of the user (user A) of the television receiver 1300 when the voice data of user A is supplied from the A / D conversion circuit 1327. After echo cancellation, the echo cancellation / speech synthesis circuit 1323 outputs the resulting voice data, obtained by combining it with other voice data, through the voice amplification circuit 1324 to the speaker 1325.

[0556] Furthermore, the television receiver 1300 also has an audio codec 1328, an internal bus 1329, an SDRAM (Synchronous Dynamic Random Access Memory) 1330, a flash memory 1331, a CPU 1332, a USB (Universal Serial Bus) I / F 1333, and a network I / F 1334.

[0557] The A / D conversion circuit 1327 receives the user's voice signal captured by the microphone 1326, which is provided on the television receiver 1300 for voice conversation. The A / D conversion circuit 1327 performs A / D conversion processing on the received voice signal and supplies the resulting digital voice data to the voice codec 1328.

[0558] The audio codec 1328 converts the audio data supplied from the A / D conversion circuit 1327 into data in a predetermined format for transmission over the network, and supplies it to the network interface 1334 via the internal bus 1329.

[0559] The network interface 1334 is connected to the network via a cable attached to the network terminal 1335. The network interface 1334 transmits audio data supplied from the audio codec 1328 to other devices connected to the network, for example. The network interface 1334 also receives audio data transmitted from other devices connected via the network via the network terminal 1335 and supplies it to the audio codec 1328 via the internal bus 1329.

[0560] The audio codec 1328 converts the audio data supplied from the network interface 1334 into data in a predetermined format and supplies it to the echo cancellation / speech synthesis circuit 1323.

[0561] The echo cancellation / speech synthesis circuit 1323 performs echo cancellation on the audio data supplied from the audio codec 1328, and outputs the resulting audio data, obtained by combining it with other audio data, through the audio amplification circuit 1324 to the speaker 1325.

[0562] The SDRAM 1330 stores various data necessary for the CPU 1332 to perform processing.

[0563] The flash memory 1331 stores programs executed by the CPU 1332. Programs stored in the flash memory 1331 are read by the CPU 1332 at predetermined timings, such as when the television receiver 1300 is started up. The flash memory 1331 also stores EPG data acquired via digital broadcasting, data acquired from a predetermined server via the network, etc.

[0564] For example, the flash memory 1331 stores an MPEG-TS file containing content data acquired from a predetermined server via a network under the control of the CPU 1332. The flash memory 1331 supplies the MPEG-TS file to the MPEG decoder 1317 via the internal bus 1329 under the control of the CPU 1332.

[0565] The MPEG decoder 1317 processes the MPEG-TS in the same way as the MPEG-TS supplied from the digital tuner 1316. In this way, the television receiver 1300 can receive content data consisting of video and audio via the network, decode it using the MPEG decoder 1317, and display the video or output the audio.

[0566] The television receiver 1300 also has a light receiving unit 1337 that receives infrared signals transmitted from the remote controller 1351.

[0567] The light receiving unit 1337 receives infrared light from the remote controller 1351 and outputs a control code representing the content of the user operation obtained by demodulation to the CPU 1332.

[0568] The CPU 1332 executes a program stored in the flash memory 1331 and controls the overall operation of the television receiver 1300 according to control codes supplied from the light receiving unit 1337. The CPU 1332 and the various parts of the television receiver 1300 are connected via paths not shown.

[0569] The USB I / F 1333 transmits and receives data with external devices of the television receiver 1300, which are connected via a USB cable attached to the USB terminal 1336. The network I / F 1334 connects to the network via a cable attached to the network terminal 1335 and also transmits and receives data other than audio data with various devices connected to the network.

[0570] The television receiver 1300 can improve encoding efficiency by using the image decoding device 151, 401, 501, or 601 as the MPEG decoder 1317. As a result, the television receiver 1300 can obtain and display decoded images with higher resolution and at higher speed from broadcast wave signals received via the antenna or content data acquired via the network.

[0571] Figure 47 is a block diagram showing a main configuration example of a mobile phone using an image encoding device and an image decoding device to which the present invention is applied.

[0572] The mobile phone 1400 shown in Figure 47 has a main control unit 1450, a power supply circuit unit 1451, an operation input control unit 1452, an image encoder 1453, a camera I / F unit 1454, an LCD control unit 1455, an image decoder 1456, a multiplexing / decoupling unit 1457, a recording / playback unit 1462, a modulation / demodulation circuit unit 1458, and an audio codec 1459, all of which are connected to each other via a bus 1460.

[0573] The mobile phone 1400 also has an operation key 1419, a CCD (Charge Coupled Devices) camera 1416, a liquid crystal display 1418, a memory unit 1423, a transmitting and receiving circuit unit 1463, an antenna 1414, a microphone 1421, and a speaker 1417.

[0574] When the end call and power keys are turned on by the user, the power supply circuit unit 1451 supplies power from the battery pack to each part, thereby starting up the mobile phone 1400 to an operational state.

[0575] The mobile phone 1400 performs various operations such as sending and receiving voice signals, sending and receiving emails and image data, taking pictures, or recording data in various modes such as voice call mode and data communication mode, based on the control of the main control unit 1450 which consists of a CPU, ROM and RAM, etc.

[0576] For example, in voice call mode, the mobile phone 1400 converts the voice signal collected by the microphone 1421 into digital voice data using the voice codec 1459, modulates and demodulates this into spread spectrum data using the modulation / demodulation circuit 1458, and performs digital-to-analog conversion and frequency conversion processing using the transmitting / receiving circuit 1463. The mobile phone 1400 transmits the transmission signal obtained through this conversion process to a base station (not shown) via the antenna 1414. The transmission signal (voice signal) transmitted to the base station is supplied to the other party's mobile phone via the public telephone network.

[0577] Also, for example, in voice call mode, the mobile phone 1400 amplifies the received signal received by the antenna 1414 in the transmitting / receiving circuit 1463, further performs frequency conversion processing and analog-to-digital conversion processing, performs despread spectrum processing in the modulation / demodulation circuit 1458, and converts it into an analog voice signal by the voice codec 1459. The mobile phone 1400 outputs the converted analog voice signal from the speaker 1417.

[0578] Furthermore, for example, when sending an email in data communication mode, the mobile phone 1400 receives the text data of the email entered by operating the operation key 1419 in the operation input control unit 1452. The mobile phone 1400 processes the text data in the main control unit 1450 and displays it as an image on the liquid crystal display 1418 via the LCD control unit 1455.

[0579] The mobile phone 1400 also generates email data in the main control unit 1450 based on text data and user instructions received by the operation input control unit 1452. The mobile phone 1400 performs spread spectrum processing on the email data in the modulation / demodulation circuit unit 1458, and digital-to-analog conversion and frequency conversion processing in the transmitting / receiving circuit unit 1463. The mobile phone 1400 transmits the transmission signal obtained from the conversion process to a base station (not shown) via the antenna 1414. The transmission signal (email) transmitted to the base station is supplied to a predetermined destination via the network and mail server, etc.

[0580] Also, for example, when receiving an email in data communication mode, the mobile phone 1400 receives the signal transmitted from the base station via the antenna 1414 in the transmitting / receiving circuit unit 1463, amplifies it, and then performs frequency conversion processing and analog-to-digital conversion processing. The mobile phone 1400 then performs despread spectrum processing on the received signal in the modulation / demodulation circuit unit 1458 to restore the original email data. The mobile phone 1400 displays the restored email data on the liquid crystal display 1418 via the LCD control unit 1455.

[0581] The mobile phone 1400 can also record (store) received email data in the storage unit 1423 via the recording and playback unit 1462.

[0582] This storage unit 1423 is an arbitrary rewritable storage medium. The storage unit 1423 may be, for example, a semiconductor memory such as a RAM or a built-in flash memory, a hard disk, or a removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card. Of course, it may be other than these.

[0583] Further, for example, when transmitting image data in the data communication mode, the mobile phone 1400 generates image data with the CCD camera 1416 by imaging. The CCD camera 1416 has an optical device such as a lens and an aperture and a CCD as a photoelectric conversion element, images a subject, converts the intensity of the received light into an electrical signal, and generates image data of the image of the subject. The image data is converted into encoded image data by being compression-encoded by an image encoder 1453, for example, by a predetermined encoding method such as MPEG2 or MPEG4 via the camera I / F unit 1454.

[0584] The mobile phone 1400 uses the above-described image encoding devices 51, 351, 451, or 551 as the image encoder 1453 that performs such processing. Therefore, similar to the cases of the image encoding devices 51 and 351, the image encoder 1453 sets filter coefficients according to quantization parameters and prediction modes prior to intra prediction and performs filter processing on adjacent pixels. Alternatively, similar to the cases of the image encoding devices 451 and 551, the image encoder 1453 controls whether to perform filter processing on adjacent pixels that is performed prior to intra prediction. Thereby, the encoding efficiency can be improved.

[0585] Note that at this time, the mobile phone 1400 simultaneously performs analog-to-digital conversion and further encoding of the sound collected by the microphone (mic) 1421 during imaging by the CCD camera 1416 in the audio codec 1459.

[0586] The mobile phone 1400, in the multiplexing / decompression unit 1457, multiplexes encoded image data supplied from the image encoder 1453 and digital audio data supplied from the audio codec 1459 in a predetermined manner. The mobile phone 1400 performs spread spectrum processing on the resulting multiplexed data in the modulation / demodulation circuit unit 1458, and digital-to-analog conversion processing and frequency conversion processing in the transmitting / receiving circuit unit 1463. The mobile phone 1400 transmits the transmission signal obtained by the conversion processing to a base station (not shown) via the antenna 1414. The transmission signal (image data) transmitted to the base station is supplied to the communication partner via a network or the like.

[0587] If image data is not transmitted, the mobile phone 1400 can also display the image data generated by the CCD camera 1416 on the liquid crystal display 1418 via the LCD control unit 1455, without going through the image encoder 1453.

[0588] Also, for example, in data communication mode, when receiving video data linked to a simple homepage, the mobile phone 1400 receives the signal transmitted from the base station via the antenna 1414 in the transmitting / receiving circuit unit 1463, amplifies it, and then performs frequency conversion processing and analog-to-digital conversion processing. The mobile phone 1400 performs despread spectrum processing on the received signal in the modulation / demodulation circuit unit 1458 to restore the original multiplexed data. The mobile phone 1400 separates the multiplexed data in the multiplexing / decompression unit 1457, dividing it into encoded image data and audio data.

[0589] The mobile phone 1400 generates playable video data by decoding the encoded image data in the image decoder 1456 using a decoding method corresponding to a predetermined encoding method such as MPEG2 or MPEG4, and displays this on the liquid crystal display 1418 via the LCD control unit 1455. As a result, for example, video data contained in a video file linked to a simple homepage is displayed on the liquid crystal display 1418.

[0590] The mobile phone 1400 uses the image decoding devices 151, 401, 501, or 601 described above as an image decoder 1456 that performs such processing. Accordingly, similar to the cases of the image decoding devices 151 and 401, prior to intra prediction, the image decoder 1456 switches filter coefficients according to quantization parameters and prediction modes to perform filter processing on adjacent pixels. Alternatively, similar to the cases of the image decoding devices 501 and 601, the image decoder 1456 controls whether to perform filter processing on adjacent pixels that is performed prior to intra prediction based on an on / off flag. Thereby, the coding efficiency can be improved.

[0591] At this time, the mobile phone 1400 simultaneously converts digital audio data into an analog audio signal in the audio codec 1459 and outputs this from the speaker 1417. Thereby, for example, the audio data included in the moving image file linked to the simple homepage is reproduced.

[0592] Note that, similar to the case of an e-mail, the mobile phone 1400 can also record (store) the received data linked to a simple homepage or the like in the storage unit 1423 via the recording and reproducing unit 1462.

[0593] Further, the mobile phone 1400 can analyze the two-dimensional code imaged and obtained by the CCD camera 1416 in the main control unit 1450 and acquire the information recorded in the two-dimensional code.

[0594] Furthermore, the mobile phone 1400 can communicate with an external device by infrared rays using the infrared communication unit 1481.

[0595] The mobile phone 1400 uses the image encoding devices 51, 351, 451, or 551 as an image encoder 1453, whereby the coding efficiency of the encoded data generated by encoding the image data generated in the CCD camera 1416, for example, can be improved. As a result, the mobile phone 1400 can provide encoded data (image data) with good coding efficiency to other devices.

[0596] Furthermore, the mobile phone 1400 can generate highly accurate predictive images by using the image decoding device 151, 401, 501, or 601 as the image decoder 1456. As a result, the mobile phone 1400 can obtain and display higher-resolution decoded images from, for example, video files linked to a simple homepage.

[0597] In the above description, the mobile phone 1400 has been described as using a CCD camera 1416, but instead of the CCD camera 1416, an image sensor using CMOS (Complementary Metal Oxide Semiconductor) (CMOS image sensor) may be used. In this case as well, the mobile phone 1400 can capture an image of a subject and generate image data of the subject, just as when using the CCD camera 1416.

[0598] In addition, although the above has been described as a mobile phone 1400, any device having imaging and communication functions similar to the mobile phone 1400, such as a PDA (Personal Digital Assistant), smartphone, UMPC (Ultra Mobile Personal Computer), netbook, or notebook personal computer, can be fitted with the image encoding device 51, 351, 451, or 551 and the image decoding device 151, 401, 501, or 601 in the same way as the mobile phone 1400.

[0599] Figure 48 is a block diagram showing a main configuration example of a hard disk recorder using an image encoding device and an image decoding device to which the present invention is applied.

[0600] The hard disk recorder (HDD recorder) 1500 shown in Figure 48 is a device that stores audio and video data of broadcast programs contained in broadcast wave signals (television signals) transmitted from satellite or terrestrial antennas, etc., received by a tuner, on its built-in hard disk, and provides the stored data to the user at a timing according to the user's instructions.

[0601] The hard disk recorder 1500 can, for example, extract audio data and video data from a broadcast wave signal, decode them as appropriate, and store them on its built-in hard disk. The hard disk recorder 1500 can also, for example, acquire audio data and video data from other devices via a network, decode them as appropriate, and store them on its built-in hard disk.

[0602] Furthermore, the hard disk recorder 1500 can, for example, decode audio data and video data recorded on its internal hard disk and supply them to the monitor 1560, displaying the images on the monitor 1560's screen. The hard disk recorder 1500 can also output the audio from the monitor 1560's speaker.

[0603] The hard disk recorder 1500 decodes audio and video data extracted from broadcast wave signals acquired via a tuner, or audio and video data acquired from other devices via a network, and supplies it to the monitor 1560, displaying the images on the screen of the monitor 1560. The hard disk recorder 1500 can also output the audio from the speakers of the monitor 1560.

[0604] Of course, other actions are also possible.

[0605] As shown in Figure 48, the hard disk recorder 1500 includes a receiving unit 1521, a demodulation unit 1522, a demultiplexer 1523, an audio decoder 1524, a video decoder 1525, and a recorder control unit 1526. The hard disk recorder 1500 further includes an EPG data memory 1527, a program memory 1528, a work memory 1529, a display converter 1530, an OSD (On Screen Display) control unit 1531, a display control unit 1532, a recording and playback unit 1533, a D / A converter 1534, and a communication unit 1535.

[0606] The display converter 1530 also has a video encoder 1541. The recording and playback unit 1533 has an encoder 1551 and a decoder 1552.

[0607] The receiving unit 1521 receives an infrared signal from a remote controller (not shown), converts it into an electrical signal, and outputs it to the recorder control unit 1526. The recorder control unit 1526 is configured, for example, by a microprocessor and performs various processes according to a program stored in the program memory 1528. At this time, the recorder control unit 1526 uses the work memory 1529 as needed.

[0608] The communication unit 1535 is connected to a network and performs communication processing with other devices via the network. For example, the communication unit 1535 is controlled by the recorder control unit 1526 and communicates with a tuner (not shown) and mainly outputs tuning control signals to the tuner.

[0609] The demodulation unit 1522 demodulates the signal supplied from the tuner and outputs it to the demultiplexer 1523. The demultiplexer 1523 separates the data supplied from the demodulation unit 1522 into audio data, video data, and EPG data, and outputs them to the audio decoder 1524, video decoder 1525, or recorder control unit 1526, respectively.

[0610] The audio decoder 1524 decodes the input audio data, for example, in the MPEG format, and outputs it to the recording and playback unit 1533. The video decoder 1525 decodes the input video data, for example, in the MPEG format, and outputs it to the display converter 1530. The recorder control unit 1526 supplies the input EPG data to the EPG data memory 1527 for storage.

[0611] The display converter 1530 encodes the video data supplied from the video decoder 1525 or the recorder control unit 1526 into, for example, NTSC (National Television Standards Committee) video data using the video encoder 1541, and outputs it to the recording and playback unit 1533. The display converter 1530 also converts the screen size of the video data supplied from the video decoder 1525 or the recorder control unit 1526 to a size corresponding to the size of the monitor 1560. The display converter 1530 further converts the video data with the converted screen size into NTSC video data using the video encoder 1541, converts it into an analog signal, and outputs it to the display control unit 1532.

[0612] The display control unit 1532, under the control of the recorder control unit 1526, superimposes the OSD (On Screen Display) signal output by the OSD control unit 1531 onto the video signal input from the display converter 1530, and outputs it to the display of the monitor 1560 for display.

[0613] The monitor 1560 is also supplied with audio data output by the audio decoder 1524, which has been converted into an analog signal by the D / A converter 1534. The monitor 1560 outputs this audio signal from its built-in speaker.

[0614] The recording and playback unit 1533 has a hard disk as a storage medium for recording video data, audio data, etc.

[0615] The recording and playback unit 1533 encodes, for example, the audio data supplied from the audio decoder 1524 in the MPEG format using the encoder 1551. The recording and playback unit 1533 also encodes the video data supplied from the video encoder 1541 of the display converter 1530 in the MPEG format using the encoder 1551. The recording and playback unit 1533 combines the encoded audio data and the encoded video data using a multiplexer. The recording and playback unit 1533 amplifies the combined data by channel coding and writes the data to the hard disk via the recording head.

[0616] The recording and playback unit 1533 plays back the data recorded on the hard disk via the playback head, amplifies it, and separates it into audio data and video data using a demultiplexer. The recording and playback unit 1533 decodes the audio data and video data in MPEG format using a decoder 1552. The recording and playback unit 1533 performs D / A conversion on the decoded audio data and outputs it to the speaker of the monitor 1560. The recording and playback unit 1533 also performs D / A conversion on the decoded video data and outputs it to the display of the monitor 1560.

[0617] The recorder control unit 1526 reads the latest EPG data from the EPG data memory 1527 based on user instructions indicated by an infrared signal from the remote controller received via the receiver unit 1521, and supplies it to the OSD control unit 1531. The OSD control unit 1531 generates image data corresponding to the input EPG data and outputs it to the display control unit 1532. The display control unit 1532 outputs the video data input from the OSD control unit 1531 to the display of the monitor 1560 and displays it. As a result, the EPG (Electronic Program Guide) is displayed on the display of the monitor 1560.

[0618] The hard disk recorder 1500 can also acquire various types of data, such as video data, audio data, or EPG data, supplied from other devices via a network such as the Internet.

[0619] The communication unit 1535 is controlled by the recorder control unit 1526 and acquires encoded data such as video data, audio data, and EPG data transmitted from other devices via the network, and supplies it to the recorder control unit 1526. The recorder control unit 1526, for example, supplies the acquired encoded video data and audio data to the recording and playback unit 1533 and stores it on the hard disk. At this time, the recorder control unit 1526 and the recording and playback unit 1533 may perform processing such as re-encoding as needed.

[0620] The recorder control unit 1526 decodes the encoded data of the acquired video data and audio data and supplies the resulting video data to the display converter 1530. The display converter 1530 processes the video data supplied from the recorder control unit 1526 in the same way as the video data supplied from the video decoder 1525, and supplies it to the monitor 1560 via the display control unit 1532 to display the image.

[0621] In addition, in conjunction with the image display, the recorder control unit 1526 may supply the decoded audio data to the monitor 1560 via the D / A converter 1534 and output the sound from the speaker.

[0622] Furthermore, the recorder control unit 1526 decodes the encoded data of the acquired EPG data and supplies the decoded EPG data to the EPG data memory 1527.

[0623] The hard disk recorder 1500 described above uses image decoding devices 151, 401, 501, or 601 as decoders built into the video decoder 1525, decoder 1552, and recorder control unit 1526. Therefore, the decoders built into the video decoder 1525, decoder 1552, and recorder control unit 1526, similar to the case of image decoding devices 151 and 401, switch filter coefficients according to the quantization parameters and prediction mode and perform filtering of adjacent pixels prior to intra prediction. Alternatively, the decoders built into the video decoder 1525, decoder 1552, and recorder control unit 1526, similar to the case of image decoding devices 501 and 601, control whether or not to perform filtering of adjacent pixels prior to intra prediction based on an on / off flag. This improves encoding efficiency.

[0624] Therefore, the hard disk recorder 1500 can generate highly accurate predictive images. As a result, the hard disk recorder 1500 can obtain a higher-resolution decoded image from, for example, encoded video data received via the tuner, encoded video data read from the hard disk of the recording / playback unit 1533, or encoded video data acquired via the network, and display it on the monitor 1560.

[0625] The hard disk recorder 1500 also uses an image encoding device 51, 351, 451, or 551 as the encoder 1551. Therefore, the encoder 1551 sets filter coefficients according to the quantization parameters and prediction mode and performs filtering of adjacent pixels prior to intra prediction, similar to the cases of image encoding devices 51 and 351. Alternatively, the encoder 1551 controls whether or not to perform filtering of adjacent pixels prior to intra prediction, similar to the cases of image encoding devices 451 and 551. This improves encoding efficiency.

[0626] Therefore, the hard disk recorder 1500 can improve the encoding efficiency of encoded data to be recorded on the hard disk, for example. As a result, the hard disk recorder 1500 can use the storage space of the hard disk more efficiently.

[0627] In the above description, a hard disk recorder 1500 that records video data and audio data on a hard disk has been described, but of course, any recording medium can be used. For example, even in a recorder that uses a recording medium other than a hard disk, such as flash memory, optical disc, or videotape, the image encoding devices 51, 351, 451, and 551 and the image decoding devices 151, 401, 501, and 601 can be applied in the same way as in the case of the hard disk recorder 1500 described above.

[0628] Figure 49 is a block diagram showing a main configuration example of a camera using an image decoding device and an image coding device to which the present invention is applied.

[0629] The camera 1600 shown in Figure 49 captures an image of the subject, displays the image of the subject on the LCD 1616, or records it as image data on the recording medium 1633.

[0630] The lens block 1611 directs light (i.e., the image of the subject) onto the CCD / CMOS 1612. The CCD / CMOS 1612 is an image sensor using a CCD or CMOS, which converts the intensity of the received light into an electrical signal and supplies it to the camera signal processing unit 1613.

[0631] The camera signal processing unit 1613 converts the electrical signals supplied from the CCD / CMOS 1612 into Y, Cr, and Cb color difference signals and supplies them to the image signal processing unit 1614. The image signal processing unit 1614, under the control of the controller 1621, performs predetermined image processing on the image signals supplied from the camera signal processing unit 1613, or encodes the image signals using the encoder 1641, for example, in the MPEG format. The image signal processing unit 1614 supplies the encoded data generated by encoding the image signals to the decoder 1615. Furthermore, the image signal processing unit 1614 acquires display data generated on the on-screen display (OSD) 1620 and supplies it to the decoder 1615.

[0632] In the above processing, the camera signal processing unit 1613 appropriately utilizes the DRAM (Dynamic Random Access Memory) 1618 connected via the bus 1617 and stores image data and encoded data obtained by encoding that image data in the DRAM 1618 as needed.

[0633] The decoder 1615 decodes the encoded data supplied from the image signal processing unit 1614 and supplies the resulting image data (decoded image data) to the LCD 1616. The decoder 1615 also supplies the display data supplied from the image signal processing unit 1614 to the LCD 1616. The LCD 1616 appropriately combines the image of the decoded image data supplied from the decoder 1615 with the image of the display data and displays the combined image.

[0634] The on-screen display 1620, under the control of the controller 1621, outputs display data such as menu screens and icons consisting of symbols, characters, or shapes to the image signal processing unit 1614 via the bus 1617.

[0635] The controller 1621 executes various processes based on signals indicating the content of commands given by the user using the operation unit 1622, and also controls the image signal processing unit 1614, DRAM 1618, external interface 1619, on-screen display 1620, and media drive 1623, etc. via the bus 1617. The FLASH ROM 1624 stores programs and data necessary for the controller 1621 to execute various processes.

[0636] For example, the controller 1621 can encode image data stored in the DRAM 1618 or decode encoded data stored in the DRAM 1618 on behalf of the image signal processing unit 1614 and the decoder 1615. In this case, the controller 1621 may perform the encoding and decoding process using the same encoding and decoding method as the image signal processing unit 1614 and the decoder 1615, or it may perform the encoding and decoding process using a method not supported by the image signal processing unit 1614 and the decoder 1615.

[0637] Also, for example, if the operation unit 1622 instructs the start of image printing, the controller 1621 reads image data from the DRAM 1618 and supplies it to the printer 1634 connected to the external interface 1619 via the bus 1617 to print.

[0638] Furthermore, for example, if image recording is instructed from the operation unit 1622, the controller 1621 reads encoded data from the DRAM 1618 and supplies it to the recording medium 1633 mounted on the media drive 1623 via the bus 1617 for storage.

[0639] The recording medium 1633 is any read-write removable medium, such as a magnetic disk, magneto-optical disk, optical disk, or semiconductor memory. The type of removable medium for the recording medium 1633 is, of course, arbitrary and may be a tape device, a disk, or a memory card. It may also be a contactless IC card, etc.

[0640] Alternatively, the media drive 1623 and the recording medium 1633 may be integrated and configured using a non-portable storage medium such as an internal hard disk drive or an SSD (Solid State Drive).

[0641] The external interface 1619 is configured, for example, as a USB input / output terminal, and is connected to the printer 1634 when printing images. A drive 1631 is also connected to the external interface 1619 as needed, and removable media 1632 such as magnetic disks, optical disks, or magneto-optical disks are appropriately mounted, and computer programs read from them are installed in the FLASH ROM 1624 as needed.

[0642] Furthermore, the external interface 1619 has a network interface that connects to a predetermined network such as a LAN or the Internet. The controller 1621 can, for example, read encoded data from the DRAM 1618 in accordance with instructions from the operation unit 1622 and supply it to other devices connected via the network through the external interface 1619. The controller 1621 can also acquire encoded data and image data supplied from other devices via the network through the external interface 1619 and store them in the DRAM 1618 or supply them to the image signal processing unit 1614.

[0643] The camera 1600 described above uses an image decoding device 151, 401, 501, or 601 as the decoder 1615. Therefore, the decoder 1615, as with the image decoding devices 151 and 401, switches the filter coefficients according to the quantization parameters and prediction mode and performs filtering of adjacent pixels prior to intra prediction. Alternatively, the decoder 1615, as with the image decoding devices 501 and 601, controls whether or not to perform filtering of adjacent pixels prior to intra prediction based on an on / off flag. This improves coding efficiency.

[0644] Therefore, the camera 1600 can generate highly accurate predictive images. As a result, the camera 1600 can obtain a higher-resolution decoded image from, for example, image data generated in the CCD / CMOS 1612, encoded video data read from the DRAM 1618 or recording medium 1633, or encoded video data acquired via the network, and display it on the LCD 1616.

[0645] The camera 1600 also uses an image encoding device 51, 351, 451, or 551 as the encoder 1641. Therefore, the encoder 1641 sets filter coefficients according to the quantization parameters and prediction mode and performs filtering of adjacent pixels prior to intra-prediction, similar to the case of image encoding devices 51 and 351. Alternatively, the encoder 1641 controls whether or not to perform filtering of adjacent pixels prior to intra-prediction, similar to the case of image encoding devices 451 and 551. This makes it possible to improve encoding efficiency.

[0646] Therefore, the camera 1600 can improve the encoding efficiency of encoded data to be recorded on the hard disk, for example. As a result, the camera 1600 can use the storage area of ​​the DRAM 1618 and the recording medium 1633 more efficiently.

[0647] The decoding process performed by the controller 1621 may be configured to use the decoding method of the image decoding device 151, 401, 501, or 601. Similarly, the encoding process performed by the controller 1621 may be configured to use the encoding method of the image encoding device 51, 351, 451, or 551.

[0648] The image data captured by the camera 1600 may be a moving image or a still image.

[0649] Of course, the image encoding devices 51, 351, 451, or 551 and the image decoding devices 151, 401, 501, or 601 are also applicable to devices and systems other than those described above.

[0650] 51 Image encoding device, 66 Reversible encoding unit, 74 Intra prediction unit, 75 Adjacent pixel interpolation filter switching unit, 81 Adjacent pixel setting unit, 82 Prediction image generation unit, 83 Optimal prediction mode determination unit, 91 Prediction mode buffer, 92 Quantization parameter buffer, 93 Low-pass filter setting unit, 94 Filter coefficient memory, 111 Adjacent pixel setting unit, 112 Prediction image generation unit, 113 Optimal prediction mode determination unit, 121 Prediction mode buffer, 122 Optimal filter calculation unit, 93 Low-pass filter setting unit, 151 Image decoding device, 162 Reversible decoding unit, 171 Intra prediction unit, 172 Adjacent pixel interpolation filter switching unit, 181 Prediction image generation unit, 182 Adjacent pixel setting unit, 191 Prediction mode buffer, 192 Quantization parameter buffer, 193 Low-pass filter setting unit, 194 Filter coefficient memory, 202 Low-pass filter setting unit, 251 Learning device, 261 Adjacent pixel interpolation filter calculation unit, 271 Filter coefficient storage unit, 351 Image coding device, 361 Second-order prediction unit, 362 Adjacent pixel interpolation filter switching unit, 401 Image decoding device, 411 Second-order prediction unit, 412 Adjacent pixel interpolation filter switching unit, 451 Image coding device, 461 Adjacent pixel interpolation filter control unit, 501 Image decoding device, 511 Adjacent pixel interpolation filter control unit, 551 Image coding device, 561 Adjacent pixel interpolation filter control unit, 601 Image decoding device, 611 Adjacent pixel interpolation filter control unit

Claims

[1] Decryption means for decrypting the image of the target block to be decrypted, A filter setting means sets coefficients used for filtering applied to adjacent pixels of the target block, which are used for intra-prediction of the target block, according to the encoding parameters, corresponding to the target block. An intra-prediction means that performs the filtering process on the adjacent pixels using the coefficients set by the filter setting means, and performs intra-prediction of the target block. An image processing device equipped with the following features. [2] The coding parameters include the mode of intra-prediction of the target block, or the quantization parameters of the target block. The decoding means decodes the intra-prediction mode of the target block, or the quantization parameters of the target block. The filter setting means sets the coefficients according to the mode of the intra prediction decoded by the decoding means, or the quantization parameters decoded by the decoding means. The image processing apparatus according to claim 1. [3] Further comprising filter coefficient storage means for storing the coefficient. The coefficient is calculated so that the residual between the learning block to be encoded and the prediction mode of the intra prediction of the learning block or the prediction image obtained by performing intra prediction on the learning block corresponding to the quantization parameter of the learning block in the learning image is minimized, and is stored in the filter coefficient storage means. The filter setting means sets, as the coefficient, a coefficient corresponding to the intra prediction mode of the target block or the quantization parameter of the target block from among the coefficients stored in the filter coefficient storage means. The image processing apparatus according to claim 2. [4] The filter coefficient storage means stores the coefficients as n-bit values ​​(where n is an integer) according to the register length of the processor. The image processing apparatus according to claim 3. [5] The decoding means decodes the coefficients which are calculated on the encoding side such that the residual is minimized between the predicted image obtained by performing intra-prediction on the target block using the adjacent pixels before the filtering is applied, and which are set in correspondence with the target block, the mode of intra-prediction of the target block, or the quantization parameter of the target block, The filter setting means sets the coefficients that correspond to the intra-prediction mode of the target block or the quantization parameter of the target block, from among the coefficients decoded by the decoding means, as the coefficients. The image processing apparatus according to claim 2. [6] The coefficient is composed of a filters coefficient and an offset value. The image processing apparatus according to claim 2. [7] The filter coefficients consist of three taps. The image processing apparatus according to claim 6. [8] The filter coefficients have symmetry with respect to the zero phase. The image processing apparatus according to claim 6. [9] The decoding means decodes the intra prediction mode of the target block, The filter setting means sets the intra-prediction modes such that vertical mode or horizontal mode is a first class and other modes are a second class, and sets the coefficient corresponding to the first class when the intra-prediction mode of the target block belongs to the first class, and sets the coefficient corresponding to the second class when the decoded intra-prediction mode of the target block belongs to the second class. The image processing apparatus according to claim 1. [10] The system further includes receiving means for receiving flag information indicating whether to perform the filtering process on the adjacent pixels, The filter setting means sets whether to perform the filter process on the adjacent pixels based on the flag information received by the receiving means. The image processing apparatus according to claim 1. [11] The receiving means receives the flag information on a macroblock basis, The filter setting means sets whether to perform the filter process on adjacent pixels on a macroblock basis, based on the flag information received by the receiving means. The image processing apparatus according to claim 10. [12] The receiving means receives the flag information in block units, The filter setting means sets whether to perform the filter process on adjacent pixels on a block-by-block basis, based on the flag information received by the receiving means. The image processing apparatus according to claim 10. [13] The image processing device, The image of the target block to be decoded is decoded, Corresponding to the aforementioned target block, coefficients used for filtering applied to adjacent pixels of the target block, which are used for intra-prediction of the target block, are set according to the encoding parameters. The step of applying the filtering process to the adjacent pixels using the set coefficients and performing intra-prediction of the target block. Image processing methods including [specific details omitted]. [14] A filter setting means for setting coefficients used for filtering applied to adjacent pixels of a target block, which is used for intra-prediction of the target block, according to the encoding parameters, for a target block that is subject to encoding processing, An intra-prediction means that performs the filtering process on the adjacent pixels using the coefficients set by the filter setting means, and performs intra-prediction of the target block, Encoding means for encoding the image of the target block and An image processing device equipped with the following features. [15] The coding parameters include the intra-prediction mode of the target block or the quantization parameters of the target block, The filter setting means sets the coefficient according to the intra prediction mode of the target block or the quantization parameter of the target block, The encoding means encodes the intra-prediction mode of the corresponding target block, or the quantization parameters of the target block. The image processing apparatus according to claim 14. [16] Further comprising filter coefficient storage means for storing the coefficients, The coefficients are calculated in such a way that the residual between the training image and the training block to be encoded and the predicted image obtained by performing intra-prediction on the training block in accordance with the intra-prediction mode of the training block or the quantization parameter of the training block is minimized, and are stored in the filter coefficient storage means. The filter setting means sets a coefficient from among the coefficients stored in the filter coefficient storage means that corresponds to the intra prediction mode of the target block or the quantization parameter of the target block. The image processing apparatus according to claim 15. [17] The filter coefficient calculation means further comprises a means for calculating the coefficient such that the residual is minimized between the target block and the predicted image obtained by performing intra-prediction on the target block using the adjacent pixels before the filtering is applied in correspondence with the target block and the intra-prediction mode of the target block, or the quantization parameter of the target block, The filter setting means sets, from among the coefficients calculated by the filter coefficient calculation means, a coefficient that corresponds to the intra-prediction mode of the target block or the quantization parameter of the target block. The encoding means further encodes the coefficients The image processing apparatus according to claim 15. [18] The coefficient consists of a filter coefficient and an offset value. The image processing apparatus according to claim 15. [19] The filter setting means sets whether to perform the filter processing on the adjacent pixels, The encoding means encodes flag information indicating whether to perform the filter processing set by the filter setting means. The image processing apparatus according to claim 14. [20] The image processing device, For the target block to be encoded, coefficients used for filtering applied to adjacent pixels of the target block, which are used for intra-prediction of the target block, are set according to the encoding parameters. The filtering process is applied to the adjacent pixels using the set coefficients, and intra-prediction of the target block is performed. The step of encoding the image of the target block is Includes image processing methods.

Images