Encoding device and method

Abstract

To make inconspicuous an image quality deterioration due to the difference between different prediction images when encoding is performed in one image by using the prediction images.SOLUTION: An encoder performs encoding processing for every block of an input image, and comprises: encoding means that executes the encoding processing for every block by using any one prediction mode of inter-prediction, intra-prediction, and multi-reference prediction; output means that outputs a result of the encoding processing as encoding data; and calculation means that calculates, by using a predetermined function, a first encoding cost when the block is encoded in the inter-prediction mode and a second encoding cost when the block is encoded in the intra-prediction mode. When the difference between the first encoding cost and the second encoding cost is equal to or less than a threshold, the encoding means executes the encoding processing by using the multi-reference prediction mode, and when the difference is larger than the threshold, executes the encoding processing by using the prediction mode having a smaller one of the first encoding cost and the second encoding cost.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a coding apparatus and method, and more particularly to a predictive coding technique. [Background technology]

[0002] In order to efficiently transmit, record, and play back moving images, moving image coding devices that compress and code moving image data and moving image decoding devices that decode and expand moving image data are used. Currently, the mainstream moving image coding methods are H.264 / AVC (Advanced Video Coding) and H.265 / HEVC (High Efficiency Video Coding), which use hybrid coding to reduce spatial and temporal redundancy in moving image data. Furthermore, H.266 / VVC (Versatile Video Coding) is being developed as an international standard as a next-generation moving image coding method (see Non-Patent Document 1).

[0003] In such video coding methods, one screen is divided into small block units called CUs (Coding Units), and encoding / decoding processes are performed on a block-by-block basis. Furthermore, in such video coding methods, a predicted image is typically generated based on a locally decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the predicted image from the input image is coded. Inter-frame prediction (inter-prediction) and intra-frame prediction (intra-prediction) can be selected as a method for generating the predicted image.

[0004] In inter-slicing, whether to use intra-prediction or inter-prediction for encoding is generally determined by comparing cost values ​​(encoding costs) obtained using a cost function based on the value of a prediction error, the amount of coding of the prediction error, etc. Here, since the predicted images are different between intra-prediction and inter-prediction, when intra-prediction and inter-prediction blocks are adjacent to each other within the screen of the image to be encoded, the difference in the predicted images may be visually perceived as a difference in image quality. Generally, human visual characteristics are such that changes in flat areas are easily noticed, so in flat areas of an image, the difference in image quality between intra-prediction and inter-prediction blocks caused by the difference in the predicted images described above is more visually noticeable, and may be visually perceived as a deterioration in image quality.

[0005] To solve this problem, Patent Document 1 proposes the following technology: The lower the activity indicating the flatness of each block, the less likely it is that an intra prediction mode will be selected, and the less likely it is that intra prediction and inter prediction blocks will be adjacent to each other in flat parts of an image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-094081 [Non-Patent Document 1] “Versatile Video Coding(Draft 10)”,JVET-S2001-vH,Joint Video Experts Team (JVET)of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11,2020 [Non-patent document 2] “Algorithm description for Versatile Video Coding and Test Model 10(VTM10)”,JVET-S2002-v1,Joint Video Experts Team (JVET)of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11,2020 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology described in Patent Document 1 does not completely prevent intra-prediction and inter-prediction blocks from being adjacent to each other. In other words, even if the technology described in Patent Document 1 is used, intra-prediction and inter-prediction blocks may be adjacent to each other depending on the flatness of the image. In this case, the difference in image quality between the intra-prediction and inter-prediction blocks may be visually recognized as degradation in image quality.

[0008] The present invention has been made in consideration of the above-mentioned problems, and aims to make the degradation of image quality caused by the difference between predicted images less noticeable when a single image is coded using different predicted images. [Means for solving the problem]

[0009] In order to achieve the above object, the encoding device of the present invention divides an input image into blocks and performs an encoding process for each block of the input image. The encoding device includes: encoding means for executing an encoding process for each block using one of an inter prediction mode for generating an inter predicted image by performing inter prediction on a block and encoding a difference between the block and the inter predicted image; an intra prediction mode for generating an intra predicted image by performing intra prediction on a block and encoding a difference between the block and the intra predicted image; and a multi reference prediction mode for generating a multi predicted image by combining the inter predicted image generated from the block and the intra predicted image and encoding the difference between the block and the multi predicted image; output means for outputting the result of the encoding process as encoded data; and a first encoding cost when the block is encoded in the inter prediction mode and a multi reference prediction mode for encoding the block. The aforementioned and a calculation means for calculating, by using a predetermined function, each second coding cost when coding in an intra prediction mode, wherein the coding means performs coding processing on the block using the multi-reference prediction mode when a difference between the first coding cost and the second coding cost is equal to or less than a threshold, and calculates the smaller of the first coding cost and the second coding cost for the block when the difference between the first coding cost and the second coding cost is greater than the threshold. corresponds to The prediction mode is used to perform the encoding process. [Effects of the Invention]

[0010] According to the present invention, when a single image is coded using different predicted images, deterioration in image quality caused by the difference between the predicted images can be made less noticeable. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the configuration of an image encoding device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing types of intra prediction modes (mode numbers). [Figure 3] 5 is a flowchart showing the operation of a motion compensation unit in the first embodiment. [Figure 4] 5 is a flowchart showing a method for determining weighting coefficients of a multi-reference predictor according to the first embodiment. [Figure 5] FIG. 2 is a block diagram showing an example of the internal configuration of a feature detection unit according to the first embodiment. [Figure 6] 5 is a flowchart showing a prediction mode determination method in a prediction mode selection unit according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a gamma curve during HDR recording and display. [Figure 8] FIG. 10 is a block diagram showing an example of the internal configuration of a feature detection unit according to a second embodiment. [Figure 9] 10 is a flowchart showing a prediction mode determination method in a prediction mode selection unit according to the second embodiment. [Figure 10] 10 is a flowchart showing a prediction mode determination method in a prediction mode selection unit in a modified example of the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] [First embodiment] 1 is a block diagram showing the configuration of an image coding device 100 that performs predictive coding in this embodiment. The image coding device 100 is disposed, for example, in an image processing engine housed in an imaging device. The frame buffer 101 stores an input image to be coded, and outputs image data containing pixels of the required block size to the adder 102, intra prediction unit 112, motion compensation unit 114, and feature detection unit 116.

[0014] The adder 102 generates differential image data between the image data output from the frame buffer 101 and predicted image data output from a prediction mode selection unit 117 (described later), and outputs the differential image data to the orthogonal transformation unit 103 . The orthogonal transform unit 103 performs orthogonal transform such as discrete cosine transform (DCT) or discrete sine transform (DST) on the differential image data, generates transform coefficients, and outputs them to the quantization unit 104 . The quantization unit 104 quantizes the transform coefficients output from the orthogonal transform unit 103 in accordance with the quantization step size (or quantization parameter) output from the quantization control unit 106, and outputs the quantized coefficients to the entropy coding unit 105 and the inverse quantization unit 107.

[0015] The entropy coding unit 105 performs variable-length coding on the quantized transform coefficients, motion vectors, and parameters required for decoding, and outputs the coded stream in a predetermined format. The entropy coding unit 105 also calculates the amount of code generated for each block and outputs the amount of code generated to the quantization control unit 106. The quantization control unit 106 determines the quantization step size and quantization parameter Qp for each block based on the generated code amount output from the entropy coding unit 105 and a predetermined target code amount, and outputs them to the quantization unit 104 and the prediction mode selection unit 117.

[0016] The inverse quantization unit 107 performs inverse quantization on the quantized transform coefficients output from the quantization unit 104 to generate transform coefficients, and outputs the generated transform coefficients to the inverse orthogonal transformation unit . The inverse orthogonal transform unit 108 applies the inverse transform of the orthogonal transform applied by the orthogonal transform unit 103 to the transform coefficients output from the inverse quantization unit 107 to generate differential image data and output it to the adder 109 . The adder 109 adds the differential image data generated by the inverse orthogonal transform unit 108 to the predicted image data output from the prediction mode selection unit 117 described later, generates locally decoded reference image data, and outputs it to the reference frame buffer 110 and the loop filter 111.

[0017] The loop filter 111 includes a deblocking filter, an adaptive loop filter, etc., and removes block noise from the reference image data output from the adder 109, and outputs the result to the reference frame buffer 110. As described above, the reference image data output from the adder 109 can be stored in the reference frame buffer 110 without passing through the loop filter 111. In other words, the filtering process by the loop filter 111 can be omitted. The reference frame buffer 110 stores the reference image data supplied thereto, and outputs the stored reference image data to the intra prediction unit 112 and the motion compensation unit 114 at a predetermined timing.

[0018] The intra prediction unit 112 performs intra prediction using image data of the block to be coded output from the frame buffer 101 and reference image data surrounding the block to be coded read from the reference frame buffer.

[0019] Here, the intra prediction modes that can be selected by the intra prediction unit 112 will be described with reference to FIG. 2 is a diagram showing types of intra prediction modes (mode numbers). The intra prediction unit 112 can select an intra prediction mode from planar prediction (mode number 0), DC prediction (mode number 1), and 65 types of directional prediction (mode numbers 2 to 66). Furthermore, the intra prediction unit 112 selects, from among the intra prediction modes, the intra prediction mode with the smallest cost value according to a cost function that indicates coding efficiency. Although various functions are conceivable as the cost function, in this embodiment, the following formula (1) is used as a representative one.

[0020] IntraCost=SATD+Qp×Mdcost (1) In equation (1), IntraCost is a cost value indicating the coding efficiency in the intra prediction mode, SATD is the sum of absolute differences obtained by subtracting the input image data to be coded from the reference image data and then performing a Hadamard transform on the difference data, Qp is a quantization parameter used by the quantizer 104, and Mdcost is a cost value corresponding to the code amount in the intra prediction mode.

[0021] The intra prediction unit 112 generates a predicted image for intra prediction using the intra prediction mode selected using equation 1. The intra prediction unit 112 also outputs the selected intra prediction mode, a cost value when the selected intra prediction mode is applied, and predicted image data (intra prediction image) generated by intra prediction to the multi-reference prediction unit 115 and the prediction mode selection unit 117, respectively.

[0022] The motion information memory 113 is a memory for temporarily storing motion vectors output by a motion compensation unit 114, which will be described later. The motion information memory 113 stores motion vectors of blocks located around the block to be coded, and can also store motion vectors for each block of a frame that has already been coded. The motion vector information stored in the motion information memory 113 is referenced by the motion compensation unit 114, which will be described later. The motion compensation unit 114 selects the most appropriate mode from two modes: an inter mode in which a new motion vector is searched for, and a merge mode in which the most appropriate motion vector is selected from motion vector candidates of blocks that have already been coded.

[0023] Here, the operation of the motion compensation unit 114 will be described with reference to FIG. In S301, the motion compensation unit 114 searches for an optimal motion vector in inter mode using image data of the block to be coded output from the frame buffer 101 and reference image data read from the reference frame buffer 110. More specifically, the motion compensation unit 114 searches for a motion vector with the smallest cost value according to a cost function that indicates coding efficiency. While various functions can be considered as the cost function, in this embodiment, Equation (2) will be used as a representative one.

[0024] InterCost=SATD+Qp×Mvcost (2) In equation (2), InterCost is a cost value indicating the coding efficiency in inter mode. SATD is the sum of absolute differences obtained by Hadamard transforming the difference data obtained by subtracting the input image data to be coded from the reference image data after the motion vector is reflected. Qp is a quantization parameter used by the quantizer 104, and Mvcost is a cost value corresponding to the amount of code of the motion vector. The motion compensation unit 114 temporarily stores information about the motion vector determined using equation (2) and the cost value obtained when the motion vector is applied.

[0025] In S302, the motion compensation unit 114 selects an optimal motion vector in merge mode from among multiple motion vector candidates already stored in the motion information memory 113. More specifically, the motion compensation unit 114 applies equation (2) to each of the multiple motion vector candidates and selects the motion vector with the smallest cost value. The motion compensation unit 114 also calculates an index value (merge index) indicating the motion vector candidate. The motion compensation unit 114 temporarily stores the merge index and the cost value obtained when the motion vector indicated by the merge index is applied.

[0026] In S303, the motion compensation unit 114 compares the cost value in the inter mode calculated in S301 with the cost value in the merge mode calculated in S302, and selects the mode that provides the smallest cost value.

[0027] In S304, the motion compensation unit 114 generates predicted image data for inter prediction using the motion vector in the selected mode and the reference image data read from the reference frame buffer 110. In the case of merge mode, it is possible to determine the motion vector by referring to the merge index calculated in S302.

[0028] The motion compensation unit 114 outputs a flag (merge flag) indicating whether a merge mode has been selected, information on the motion vector or merge index, the cost value in the selected mode, and the generated predicted image data for inter prediction (inter prediction image) to the multi-reference prediction unit 115 and the prediction mode selection unit 117, respectively.

[0029] The multi-reference prediction unit 115 combines the predicted image data for intra prediction output from the intra prediction unit 112 and the predicted image data for inter prediction output from the motion compensation unit 114 to generate new predicted image data.

[0030] As one method for synthesizing predicted image data for intra prediction and inter prediction, Non-Patent Document 2 discloses a technology called Combined Inter and Intra Prediction (CIIP), and the method for synthesizing predicted image data for intra prediction and inter prediction in this embodiment is also based on CIIP. Meanwhile, Non-Patent Document 2 describes, as application conditions for CIIP, that predicted images for intra prediction are those to which planar prediction is applied, and predicted images for inter prediction are those to which merge mode is applied, etc. However, the method for synthesizing predicted image data for intra prediction and inter prediction in this embodiment is not necessarily limited to these conditions, and for example, directional prediction or the like can also be applied to predicted image data for intra prediction.

[0031] CIIP blends predicted images of intra prediction and inter prediction for each pixel, thereby further reducing the difference in predicted images between blocks. Based on this, in this embodiment, CIIP is applied to blocks that exist in areas where intra prediction and inter prediction blocks are likely to be adjacent to each other and in areas where image quality differences between blocks are likely to be noticeable, such as flat areas of an image. This reduces image quality degradation caused by differences in predicted images between adjacent blocks.

[0032] The following describes in detail the method of combining predicted image data for intra prediction and predicted image data for inter prediction performed by the multi-reference predictor 115. The multi-reference predictor 115 synthesizes the predicted image data for intra prediction and inter prediction for each pixel using the following equation (3).

[0033] Pciip=((4-wt)×Pinter+wt×Pintra+2) / 4 …(3) In equation (3), Pintra is a pixel value of predicted image data for intra prediction, Pinter is a pixel value of predicted image data for inter prediction, and Pciip is a pixel value of predicted image data after synthesis. wt is a weighting coefficient indicating the synthesis ratio of pixel values ​​of intra prediction and inter prediction, and can take an integer value from 1 to 3.

[0034] Here, a method for determining the weighting coefficient wt of the multi-reference predictor 115 will be described with reference to FIG. In S401, the multi-reference prediction unit 115 initializes isIntraTop and isIntraLeft to 0 and the weighting coefficient wt to 1. isIntraTop is a flag indicating whether the prediction mode of the coded block adjacent to the coding target block in the upper direction is intra prediction. Also, isIntraLeft is a flag indicating whether the prediction mode of the coded block adjacent to the coding target block in the left direction is intra prediction. Note that the prediction modes of these coded blocks are held by the prediction mode selection unit 117, as will be described later, and the multi-reference prediction unit 115 is able to refer to this information as appropriate.

[0035] In S402, the multi-reference prediction unit 115 determines whether the prediction mode of the coded block adjacent above the current block to be coded is intra prediction. If the prediction mode of the coded block adjacent above is intra prediction (Yes in S402), the multi-reference prediction unit 115 sets isIntraTop to 1 in S403. If the prediction mode of the coded block adjacent above is not intra prediction (No in S402), the multi-reference prediction unit 115 proceeds to the process of S404.

[0036] In S404, the multi-reference prediction unit 115 determines whether the prediction mode of the coded block adjacent to the left of the current block to be coded is intra prediction. If the prediction mode of the coded block adjacent to the left is intra prediction (Yes in S404), the multi-reference prediction unit 115 sets isIntraLeft to 1 in S405. If the prediction mode of the coded block adjacent to the left is not intra prediction (No in S404), the multi-reference prediction unit 115 proceeds to the process of S406.

[0037] In S406 to S410, the multi-reference predictor 115 determines the value of the weighting factor wt according to the sum of isIntraTop and isIntraLeft. If the sum of isIntraTop and isIntraLeft is 2 (Yes in S406), the process proceeds to S407, where the multi-reference predictor 115 sets the weighting factor wt to 3. If the sum of isIntraTop and isIntraLeft is 1 (No in S406 and Yes in S408), the process proceeds to S409, where the multi-reference predictor 115 sets the weighting factor wt to 2. If the sum of isIntraTop and isIntraLeft is 0 (No in S406 and No in S408), the process proceeds to S410, where the multi-reference predictor 115 sets the weighting factor wt to 1.

[0038] The processes of S406 to S410 mean adaptively changing the synthesis ratio of pixel values ​​of predicted image data for intra prediction and pixel values ​​of predicted image data for inter prediction when synthesizing a predicted image according to the prediction modes of the blocks surrounding the block to be coded. This makes it possible to reduce image quality degradation caused by differences in predicted images between the block to be coded and its surrounding blocks when multi-reference prediction is selected as the prediction mode in a prediction mode selection unit 117 (described later).

[0039] Note that the method of combining the predicted image data for intra prediction and the predicted image data for inter prediction is not limited to the example shown in equation (3) and Fig. 4. For example, the multi-reference predictor 115 can also perform the combining using the following equation (4). Pciip=((8-wt)×Pinter+wt×Pintra+4) / 8 …(4) As another example, the weighting factor wt may be adaptively changed for each pixel according to the pixel position of each pixel in the block to be coded. Furthermore, the weighting factor wt may be adaptively changed according to the intra prediction mode (DC prediction, planar prediction, etc.) output from the intra prediction unit 112, the block size of the block to be coded, etc.

[0040] The multi-reference prediction unit 115 calculates a cost value according to a cost function indicating coding efficiency using predicted image data (multi-predicted image) synthesized using equation (3). Although various functions are conceivable as the cost function, in this embodiment, equation (5) is used as a representative one.

[0041] CiipCost=SATD+Qp×Mpcost (5) In equation (5), CiipCost is a cost value indicating the coding efficiency in multi-reference prediction. SATD is the sum of absolute differences obtained by Hadamard transforming differential data obtained by subtracting input image data to be coded from synthesized predicted image data. Qp is a quantization parameter used by the quantization unit 104, and Mpcost is a cost value corresponding to the amount of code of parameters necessary for performing multi-reference prediction. Parameters necessary for performing multi-reference prediction include the intra prediction mode output from the intra prediction unit 112, as well as a merge flag, merge index, or motion vector output from the motion compensation unit 114.

[0042] The multi-reference predictor 115 outputs the cost value calculated by equation (5) and the combined predicted image data to the prediction mode selector 117.

[0043] The feature detection unit 116 generates a flat region flag indicating whether or not the current block to be coded belongs to a flat region of the image, and outputs the flag to the prediction mode selection unit 117. Here, a method for generating the flat region flag by the feature detection unit 116 will be described with reference to FIG. 5 .

[0044] FIG. 5 is a block diagram showing an example of the internal configuration of the feature detection unit 116. The flatness determination unit 501 acquires an input image of a block to be coded, and generates a flag (flatness flag) indicating whether the block to be coded is flat or not. There are various possible methods for determining whether the block is flat or not, but in this embodiment, the variance value of the pixel values ​​of the block to be coded is used as the flatness, and if the flatness is lower than a predetermined threshold, the block is determined to be flat. In this embodiment, the flag value is set to "1" when the block is determined to be flat, and the flag value is set to "0" when the block is determined not to be flat.

[0045] The flatness flag buffer 502 temporarily stores flatness flags output from the flatness determination unit 501. The flatness flag buffer 502 also stores flatness flags for a plurality of coded blocks, and the flat region determination unit 503 (described later) can refer to this information as appropriate. In this embodiment, the flatness flag buffer 502 stores flatness flags for at least the coded blocks adjacent to the current block in the upper, upper left, and left directions.

[0046] The flat region determination unit 503 determines whether the current block to be coded belongs to a flat region of the image based on the flatness flag of the current block to be coded and the flatness flags of multiple coded blocks stored in the flatness flag buffer 502. There are various possible methods for determining whether a block belongs to a flat region, but in this embodiment, the current block to be coded is determined to belong to a flat region if the flatness flag of the current block to be coded is "1" and the flatness flags of the coded blocks adjacent to the current block above, diagonally above left, and to the left are all "1". In this embodiment, the flag value is set to "1" when the current block is determined to belong to a flat region, and to "0" when the current block is determined not to belong to a flat region.

[0047] The method for determining whether a current block to be coded belongs to a flat region is not limited to the example described with reference to FIG. 5. For example, the feature detection unit 116 may determine that a current block to be coded belongs to a flat region simply if the flatness of the current block to be coded is lower than a predetermined threshold. Alternatively, the feature detection unit 116 may determine the flatness flag for each block of a frame before starting the coding process for the frame and store the flatness flags in the flatness flag buffer 502. In this way, the flat region determination unit 503 can improve the accuracy of determining whether a current block to be coded belongs to a flat region by referring to flatness flags of blocks adjacent to the current block to be coded, as well as flatness flags of blocks located farther away.

[0048] The prediction mode selection unit 117 selects one of the prediction modes: intra prediction, inter prediction, and multi-reference prediction. Here, a method for determining a prediction mode by the prediction mode selection unit 117 will be described with reference to FIG. In S601, the prediction mode selection unit 117 determines whether the flat region flag output from the feature detection unit 116 is 0 and / or whether the quantization step size Qstep of the block to be coded acquired from the quantization control unit 106 is smaller than the threshold value Thqstep.

[0049] In the following steps S602 to S608 after S601, the prediction mode selection unit 117 prioritizes coding efficiency and operates to always select a prediction mode with the smallest cost value. On the other hand, in steps S609 to S613, the prediction mode selection unit 117 prioritizes reducing image quality degradation and does not always select a prediction mode with the smallest cost value, but operates to make image quality degradation caused by differences in predicted images between blocks less noticeable. In other words, the determination formula in S601 means that the prediction mode selection unit 117 selects whether to prioritize coding efficiency or reducing image quality degradation.

[0050] In this embodiment, if the region to which the encoding target block belongs is not flat and / or the quantization step size is small, it is determined that the degree of image quality degradation caused by differences in predicted images between blocks is small, and the processes of S602 to S608 are performed with emphasis on encoding efficiency. Conversely, if the region to which the encoding target block belongs is flat and the quantization step size is large, the processes of S609 to S613 are performed with emphasis on reducing image quality degradation. Note that the determination method in S601 is not limited to the example shown here. For example, the prediction mode selection unit 117 can also make the determination using either the flat region flag or the quantization step size.

[0051] In S601, if the flat region flag is 0 and / or the quantization step size Qstep is smaller than the threshold value Thqstep, the prediction mode selection unit 117 advances the process to S602. In S602 , the prediction mode selection unit 117 compares the cost value IntraCost output from the intra prediction unit 112 with the cost value InterCost output from the motion compensation unit 114 .

[0052] If InterCost is greater than IntraCost (Yes in S602), the prediction mode selection unit 117 compares IntraCost with the cost value CiipCost output from the multi-reference prediction unit 115 (S603). If IntraCost is greater than CiipCost (Yes in S603), the prediction mode selection unit 117 selects multi-reference prediction as the prediction mode (S604). If IntraCost is equal to or less than CiipCost (No in S603), the prediction mode selection unit 117 selects intra prediction as the prediction mode (S605).

[0053] Furthermore, in S602, if InterCost is less than or equal to IntraCost (No in S602), the prediction mode selection unit 117 compares InterCost with CiipCost (S606). If InterCost is greater than CiipCost (Yes in S606), the prediction mode selection unit 117 selects multi-reference prediction as the prediction mode (S607). If InterCost is less than or equal to CiipCost (No in S606), the prediction mode selection unit 117 selects inter prediction as the prediction mode (S608).

[0054] On the other hand, in S601, if the flat region flag output from the feature detection unit 116 is 1 and the quantization step size Qstep is equal to or greater than the threshold value Thqstep, the prediction mode selection unit 117 advances the process to S609.

[0055] In S609, the prediction mode selection unit 117 compares the value of InterCost with the value of IntraCost+α, where α is a correction value for the cost value of intra prediction that can be set arbitrarily.

[0056] Generally, in regions where the cost values ​​of intra prediction and inter prediction are comparable, intra prediction tends to cause more noticeable degradation in image quality. Therefore, in this embodiment, when the block to be coded belongs to a flat region, the cost value of intra prediction is corrected so that inter prediction is preferentially selected. As a result, cases where intra prediction and inter prediction blocks are adjacent to each other are reduced.

[0057] In this embodiment, the cost value is corrected by adding α as described above, but the cost value can also be corrected by subtracting, multiplying, or dividing an arbitrary constant by the cost value. The cost value correction method is not limited to the above-described method using α, and any method may be adopted as long as it allows inter prediction to be preferentially selected.

[0058] If InterCost is greater than IntraCost+α (Yes in S609), the prediction mode selection unit 117 selects intra prediction as the prediction mode (S610).

[0059] If InterCost is equal to or less than IntraCost+α (No in S609), the prediction mode selection unit 117 determines whether the absolute value of the difference between InterCost and IntraCost is equal to or less than a constant c (S611). Here, the determination formula in S611 will be described in detail.

[0060] As described above, the determination in S609 is intended to give priority to inter prediction. However, depending on the flatness of the image, adjacent blocks between intra prediction and inter prediction are likely to occur, and there are still regions where the determination result by S609 frequently switches. Here, in regions where the determination result by S609 frequently switches, it is expected that the values ​​of InterCost and IntraCost+α in the determination formula of S609 are close. In light of this, in this embodiment, regions where the determination result by S609 frequently switches are detected using the following formula (6).

[0061] |InterCost-(IntraCost+α)|≦β …(6) β is a constant for detecting the proximity of the value of InterCost to the value of IntraCost+α. If α is a positive constant, the following equation (7) can be obtained by modifying equation (6).

[0062] |InterCost-IntraCost|≦β+α=c …(7) That is, in this embodiment, if the absolute value of the difference between InterCost and IntraCost of the block to be coded in S611 is equal to or less than the constant c, it is determined that the block is in an area where adjacent blocks in intra prediction and inter prediction are likely to occur.

[0063] The constant c can be set arbitrarily and may be adaptively changed using a target bit rate of the image encoding device, the value of the quantization parameter Qp used in the quantization unit 104, image features, etc. For example, the value of the constant c can be changed for each block depending on the flatness calculated by the flatness determination unit 501, etc.

[0064] If the absolute value of the difference between InterCost and IntraCost is equal to or less than c (S611 Yes), the prediction mode selection unit 117 selects multi-reference prediction as the prediction mode (S612). That is, in this embodiment, multi-reference prediction is applied to blocks present in an area where adjacent blocks of intra prediction and inter prediction are likely to occur. When multi-reference prediction is applied, predicted images of intra prediction and inter prediction are blended, and the difference between predicted images between blocks becomes smaller, thereby reducing image quality degradation caused by differences in predicted images between blocks.

[0065] In multi-reference prediction, it is necessary to encode parameters related to both intra prediction and inter prediction modes (intra prediction mode, motion vector, etc.), which may result in a larger amount of code related to the parameters than in intra prediction or inter prediction. Therefore, in the determination of S611, the prediction mode selection unit 117 may also consider whether the value of the merge flag output from the motion compensation unit 114 is 1. This eliminates the need to encode parameters related to motion vectors when multi-reference prediction is selected, thereby suppressing an increase in the amount of code related to the parameters.

[0066] If the absolute value of the difference between InterCost and IntraCost is not equal to or less than the constant c (No in S611), the prediction mode selection unit 117 selects inter prediction as the prediction mode (S613).

[0067] A predicted image is generated according to the prediction mode selected as described above. Then, the difference between the image data of the block to be coded and the predicted image is coded, and the coding result is output as a coded stream (coded data).

[0068] In the example shown in Figure 6, flat areas of an image with a large quantization step size are detected as areas where block neighboring between intra prediction and inter prediction is likely to occur, and a prediction mode that emphasizes reducing image quality degradation is applied to those areas. On the other hand, there are cases where block boundaries between intra prediction and inter prediction are likely to be noticeable regardless of the flatness of the image, depending on factors such as the target bit rate of the image encoding device. In such cases, the prediction mode selection unit 117 can always detect areas where block neighboring between intra prediction and inter prediction is likely to occur, without performing the determination in S601, and apply multi-reference prediction to blocks in those areas.

[0069] Furthermore, the processing from S609 onwards may be performed on all blocks, regardless of whether the block boundary between intra-prediction and inter-prediction is conspicuous.

[0070] As described above, according to the first embodiment, when encoding is performed using different predicted images within the same image, deterioration in image quality caused by differences between the predicted images can be made less noticeable. Furthermore, when the region to which the current block belongs is flat and the quantization step size is large, making the boundary between intra-prediction and inter-prediction blocks conspicuous, the prediction mode selection unit 117 operates with an emphasis on reducing degradation in image quality. On the other hand, when the boundary between intra-prediction and inter-prediction blocks is not conspicuous, the prediction mode selection unit 117 operates with an emphasis on coding efficiency. This makes it possible to reduce image quality degradation caused by differences in predicted images between blocks more than before, without compromising coding efficiency.

[0071] [Variations] Next, a method for generating coded data in the modified example will be described with reference to FIG.

[0072] As described above with reference to Fig. 3, the motion compensation unit 114 calculates a cost value when the inter prediction mode is applied to the current block to be coded (S1001). Also, as described above with reference to Fig. 2, the intra prediction unit 112 calculates a cost value when the intra prediction mode is applied to the current block to be coded (S1002).

[0073] Then, similarly to S611, the prediction mode selection unit 117 determines whether the absolute value of the difference between the cost value of the inter prediction mode calculated in S1001 and the cost value of the intra prediction mode calculated in S1002 is equal to or less than the constant c (S1003). If the absolute value of the difference is equal to or less than the constant c (Yes in S1003), the prediction mode selection unit 117 selects the multi-reference prediction mode (S1004).

[0074] On the other hand, if the absolute value of the difference is greater than the constant c (No in S1003), the prediction mode selection unit 117 calculates a cost value when the multi-reference prediction mode is applied to the block to be coded (S1005). The prediction mode selection unit 117 compares the cost values ​​of the inter prediction mode, intra prediction mode, and multi-reference prediction mode calculated in S1001, S1002, and S1005, and selects the multi-reference prediction mode if the cost value of the multi-reference prediction mode is the smallest (Yes in S1006) (S1004).

[0075] If the cost value of the multi-reference prediction mode is not the smallest (No in S1006) and the cost value of the inter prediction mode is the smallest (Yes in S1007), the inter prediction mode is selected (S1008). If the cost value of the multi-reference prediction mode is not the smallest (No in S1006) and the cost value of the inter prediction mode is not the smallest (No in S1007), that is, if the cost value of the intra prediction mode is the smallest, the intra prediction mode is selected (S1009).

[0076] Then, if a multi-reference prediction mode is selected in S1004, the multi-reference prediction unit 115 combines the predicted image data for intra prediction output from the intra prediction unit 112 and the predicted image data for inter prediction output from the motion compensation unit 114 to generate predicted image data for multi-reference prediction (S1010). Alternatively, if an inter prediction mode is selected in S1008, the motion compensation unit 114 generates predicted image data for inter prediction (S1011), as in S304. Alternatively, if an intra prediction mode is selected in S1009, the intra prediction unit 112 generates predicted image data for intra prediction (S1012). The predicted image data generated in S1011, S1011, or S1012 is output to the adder 102 by the prediction mode selection unit 117.

[0077] Then, the adder 102 generates differential image data between the block image data to be coded output from the frame buffer and the predicted image data, and the coding process is performed on the differential image data using each unit as described above (S1013), and the coding result is output as a coded stream (S1014).

[0078] Note that, before starting the process of S1001, similarly to S601, the prediction mode selection unit 117 determines whether the flat region flag output from the feature detection unit 116 is 0 and / or whether the quantization step size Qstep of the block to be coded acquired from the quantization control unit 106 is smaller than the threshold value Thqstep, and may execute the processes from S1001 onwards if the flat region flag is 1 and / or the quantization step size Qstep is equal to or greater than the threshold value Thqstep. Alternatively, the processes from S1001 onwards may be executed if the dynamic range of the input image is a high dynamic range equal to or greater than a predetermined value.

[0079] [Second embodiment] Next, a second embodiment of the present invention will be described. The latest video coding methods such as HEVC and VVC enable high-bit (10-bit, 12-bit, etc.) HDR (High Dynamic Range) recording or video recorded in Log. Log-recorded video is video that has been shot using a gamma curve independently defined by the manufacturer of the digital camera, etc., to enable recording in a higher dynamic range, as opposed to the gamma curve defined by standards such as BT.709.

[0080] HDR- or Log-recorded video (high dynamic range images) are converted and displayed using a gamma curve set at the time of display output. Figure 7(a) shows the relationship between input luminance values ​​(actual brightness at the time of shooting) and output pixel values ​​(pixel values ​​after encoding) during HDR recording. The horizontal direction is the input luminance value, and the vertical direction is the output pixel value, with the relationship between the input and output values ​​defined by a gamma curve 701. Where the input value is small, the curve is steep in the vertical direction, with many pixel values ​​assigned relative to the range of input values. Where the input value is large, the curve is gentler, with fewer pixel values ​​assigned relative to the input value. By assigning output pixel values ​​to input luminance values ​​in this way, high dynamic range recording is possible.

[0081] Figure 7(b) also shows the relationship between input values ​​(pixel values ​​after encoding) and output luminance values ​​(brightness displayed on the display) during HDR display. The horizontal direction is the input pixel value, and the vertical direction is the output luminance value that is actually displayed. The relationship between the input and output values ​​is defined by gamma curve 702, which is the inverse of the relationship of gamma curve 701. Where the input value is small, the curve is gentle in the vertical direction, and the amount of change in displayed brightness relative to the range of input values ​​is small. On the other hand, where the input value is large, the curve is steep, and the amount of change in displayed brightness relative to the input value is large. By allocating output luminance values ​​to input pixel values ​​in this way, moving images recorded in HDR can be displayed with the correct brightness.

[0082] Thus, compared to video recorded in normal SDR (Standard Dynamic Range), video recorded in HDR or Log format exhibits a greater change in brightness displayed on a display relative to the change in pixel value after encoding. In other words, the degradation of image quality after encoding is more amplified and displayed on a display. This tendency is more pronounced in areas with high brightness values ​​(referred to as "high-brightness areas" or "highlight areas"). Therefore, the degradation of image quality caused by differences in predicted images between blocks, which is the subject of the present invention, is more visually noticeable in such areas.

[0083] In the second embodiment, such HDR recorded or Log recorded moving images (for convenience, referred to as "HDR images") are targeted for encoding, and image quality degradation occurring in highlight areas of the HDR images is reduced. Note that in the second embodiment, differences from the first embodiment will be mainly described, and descriptions of similarities to the first embodiment will be omitted.

[0084] The overall configuration of the image encoding device in the second embodiment is the same as that shown in FIG. 1, but the configuration of the feature detection unit 116 is different from that described with reference to FIG. 5 in the first embodiment. 8 is a block diagram showing an example of the internal configuration of the feature detection unit 116 in the second embodiment. As in the first embodiment, the feature detection unit 116 in the second embodiment includes a flatness determination unit 501, a flatness flag buffer 502, and a flat region determination unit 503, all of which determine whether or not the block to be coded belongs to a flat region of the image. Furthermore, the feature detection unit 116 includes a high-intensity determination unit 804, a high-intensity flag buffer 805, and a highlight region determination unit 806, all of which determine whether or not the block to be coded belongs to a highlight region of the image. The flatness determination unit 501, flatness flag buffer 502, and flat region determination unit 503 operate in the same manner as in the first embodiment, and therefore a description thereof will be omitted here.

[0085] The high-luminance determination unit 804 acquires an input image of a block to be coded, and generates a flag (high-luminance flag) indicating whether the block to be coded is high-luminance or not. There are various possible methods for determining whether the block to be coded is high-luminance or not, but in this embodiment, the average luminance value of the pixels of the block to be coded is calculated, and if the calculated average luminance value is greater than a predetermined threshold, the block is determined to be high-luminance. In this embodiment, the flag value is set to "1" when the block is determined to be high-luminance, and "0" when the block is determined not to be high-luminance.

[0086] The high-brightness flag buffer 805 temporarily stores high-brightness flags output from the high-brightness determination unit 804. The high-brightness flag buffer 805 also stores high-brightness flags for multiple coded blocks, and a highlight region determination unit 806 (described later) can refer to this information as needed. In this embodiment, the high-brightness flag buffer 805 stores high-brightness flags for at least coded blocks adjacent to the coding target block in the upper, upper-left, and left directions.

[0087] The highlight region determination unit 806 determines whether the current block to be coded belongs to a highlight region of the image based on the high-brightness flag of the current block to be coded and the high-brightness flags of multiple coded blocks stored in the high-brightness flag buffer 805. There are various possible methods for determining whether a block belongs to a highlight region, but in this embodiment, the current block to be coded is determined to belong to a highlight region if the high-brightness flag of the current block to be coded is "1" and the high-brightness flags of the coded blocks adjacent to the current block above, diagonally above to the left, and to the left are all "1." In this embodiment, the flag value is set to "1" when the current block is determined to belong to a highlight region, and "0" when the current block is determined not to belong to a highlight region.

[0088] The method for determining whether a current block to be coded belongs to a highlight region is not limited to the example described with reference to FIG. 8 . For example, the feature detection unit 116 may determine that a current block to be coded belongs to a highlight region simply if the average luminance value of the current block is greater than a predetermined threshold. Alternatively, the feature detection unit 116 may determine the high-luminance flag for each block of a frame before starting the coding process for the frame and store the high-luminance flags in the high-luminance flag buffer 805. In this way, the highlight region determination unit 806 can improve the accuracy of determining whether a current block to be coded belongs to a highlight region by referring to not only the high-luminance flags of blocks adjacent to the current block to be coded, but also the high-luminance flags of blocks located farther away.

[0089] In this way, the feature detection unit 116 generates a flat region flag and a highlight region flag, and outputs them to the prediction mode selection unit 117.

[0090] 9 is a flowchart showing a method for determining a prediction mode by the prediction mode selection unit 117 in the second embodiment. Note that the same processes as those shown in FIG. 6 are denoted by the same reference numerals, and descriptions thereof will be omitted. The process of the second embodiment shown in Fig. 9 is the same as the process of the first embodiment shown in Fig. 6, except that a process (S901) of determining whether the video to be encoded is HDR recorded or Log recorded and belongs to a highlight region is added. Note that whether the video to be encoded is HDR recorded or Log recorded is notified to the image encoding device in advance from a control source of the image encoding device, such as a CPU (not shown).

[0091] In S901, if the video to be coded is an HDR-recorded or Log-recorded video, and the highlight region flag is 1, the prediction mode selection unit 117 proceeds to S601. On the other hand, in S901, if the image to be coded is not an HDR-recorded or Log-recorded video, and / or the highlight region flag is not 1, the prediction mode selection unit 117 skips the process of S601 and proceeds to S602.

[0092] In other words, when the video to be encoded is HDR recorded or Log recorded, and the block to be encoded belongs to a highlight area or a flat area, and the quantization step size is large, the prediction mode selection unit 117 places importance on image quality degradation and operates so as to make image quality degradation caused by differences in predicted images between blocks less noticeable (S609 to S613).

[0093] On the other hand, in cases other than those mentioned above, the prediction mode selection unit 117 prioritizes coding efficiency and operates to always select a prediction mode that minimizes the cost value (S602 to S608).

[0094] As described above, according to the second embodiment, not only the flatness of the region to which the encoding target block belongs and the quantization step size are taken into consideration, but also whether the image to be encoded is a video recorded in HDR or Log, and whether the encoding target block belongs to a highlight region, and then a selection is made between prioritizing encoding efficiency and reducing image quality degradation. This makes it possible to reduce image quality degradation, which is particularly noticeable in highlight regions, more than ever before without compromising encoding efficiency when encoding an HDR image.

[0095] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0096] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0097] 101...frame buffer, 102...adder, 103...orthogonal transform unit, 104...quantization unit, 105...entropy coding unit, 106...quantization control unit, 107...inverse quantization unit, 108...inverse orthogonal transform unit, 109...adder, 110...reference frame buffer, 111...loop filter, 112...intra prediction unit, 113...motion information memory, 114...motion compensation unit, 115...multi-reference prediction unit, 116...feature detection unit, 117...prediction mode selection unit, 501...flatness determination unit, 502...flatness flag buffer, 503...flat region determination unit, 804...high luminance determination unit, 805...high luminance flag buffer, 806...highlight region determination unit ?

Claims

1. An encoding device that divides an input image into blocks and performs an encoding process for each block of the input image, an inter prediction mode in which an inter prediction image is generated by performing inter prediction on a block and a difference between the block and the inter prediction image is coded; an intra prediction mode in which an intra prediction image is generated by performing intra prediction on a block and a difference between the block and the intra prediction image is coded; A multi-reference prediction mode in which a multi-prediction image is generated by combining an inter-prediction image and an intra-prediction image generated from a block, and the difference between the block and the multi-prediction image is coded. an encoding means for executing an encoding process for each block using one of the prediction modes; an output means for outputting the result of the encoding process as encoded data; a calculation means for calculating, by using a predetermined function, a first coding cost when a block is coded in the inter prediction mode and a second coding cost when the block is coded in the intra prediction mode; The encoding device is characterized in that, when the difference between the first encoding cost and the second encoding cost is equal to or less than a threshold, the encoding means performs an encoding process on the block using the multi-reference prediction mode, and, when the difference between the first encoding cost and the second encoding cost is greater than the threshold, performs an encoding process on the block using the prediction mode corresponding to the smaller of the first encoding cost and the second encoding cost.

2. The calculation means further calculates a third coding cost when the block is coded in the multi-reference prediction mode, using a predetermined function; The encoding device according to claim 1, characterized in that, when a difference between the first encoding cost and the second encoding cost is greater than a threshold, the encoding means performs the encoding process using a prediction mode corresponding to the smallest cost among the first encoding cost, the second encoding cost, and the third encoding cost.

3. 3. The encoding device according to claim 1, wherein the block to be encoded by said encoding means is included in a flat area or a high-luminance area.

4. 4. The encoding device according to claim 1, wherein the input image is a high dynamic range image.

5. 5. The encoding device according to claim 1, wherein a quantization step used in the encoding process is greater than a predetermined value.

6. The encoding device according to claim 1 , wherein the multi-prediction image is generated by weighting and synthesizing the inter-prediction image and the intra-prediction image.

7. An encoding method for dividing an input image into blocks and performing an encoding process for each block of the input image, an inter prediction mode in which an inter prediction image is generated by performing inter prediction on a block and a difference between the block and the inter prediction image is coded; an intra prediction mode in which an intra prediction image is generated by performing intra prediction on a block and a difference between the block and the intra prediction image is coded; A multi-reference prediction mode in which a multi-prediction image is generated by combining an inter-prediction image and an intra-prediction image generated from a block, and the difference between the block and the multi-prediction image is coded. performing an encoding process for each block using one of the prediction modes; outputting a result of the encoding process as encoded data; calculating, using a predetermined function, a first coding cost when the block is coded in the inter prediction mode and a second coding cost when the block is coded in the intra prediction mode; an encoding method characterized in that, when a difference between the first encoding cost and the second encoding cost is equal to or less than a threshold, an encoding process is performed on the block using the multi-reference prediction mode, and, when a difference between the first encoding cost and the second encoding cost is greater than the threshold, an encoding process is performed using a prediction mode corresponding to the smaller of the first encoding cost and the second encoding cost.

8. A program for causing a computer to function as each of the means of the encoding device according to any one of claims 1 to 6.

9. A computer-readable storage medium storing the program according to claim 8.

Images