Encoding device and method
Patent Information
- Application Number
- IN202347060149
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-03-19
- Filing Date
- 2023-09-07
- Publication Date
- 2026-07-30
- Estimated Expiration
- 2042-01-31
AI Technical Summary
Existing image coding techniques fail to completely prevent image quality deterioration due to adjacent blocks using intra-prediction and inter-prediction, especially in flat areas, leading to noticeable quality differences.
An image coding apparatus that employs a multi-reference prediction mode to combine inter-predicted and intra-predicted images, selecting the prediction mode based on coding costs and thresholds to minimize quality differences between blocks.
This approach reduces the conspicuousness of image quality deterioration caused by using different prediction modes, enhancing overall image quality by blending prediction images and optimizing coding processes.
Abstract
Description
TECHNICAL FIELD
[0001] The present invention relates to an coding apparatus and method, andmore particularly to predictive coding technology.BACKGROUND ART
[0002] In order to efficiently transmit, record, and reproduce moving images, amoving image coding apparatus that compresses and encodes moving image dataand a moving image decoding apparatus that decodes and decompresses movingimage data are used. As moving image encoding methods, H.264 / AVC(Advanced Video Coding) and H.265 / HEVC (High Efficiency Video Coding),which utilizes hybrid coding that reduces spatial and temporal redundancy ofmoving image data, are mainly used currently. Further, as a moving imageencoding method of next generation, International standardization of H.266 / VVC(Versatile Video Coding) is in progress (see NPL 1).
[0003] In this moving image encoding method, one screen is divided into smallblocks called CU (Coding Unit), and coding / decoding processing is performed foreach block. Further, in such a video coding method, a predicted image isnormally generated based on a local decoded image obtained byencoding / decoding an input image, and a prediction error (sometimes called a"difference image" or "residual image") obtained by subtracting the predictedimage from the input image is encoded. Inter-prediction and intra-prediction canbe selected as a method for generating a predicted image.
[0004] In inter-slicing, which of intra-prediction and inter-prediction is used incoding is generally determined by comparing the cost value (coding cost)obtained using a cost function based on the prediction error value and the codeamount of the prediction error. Here, since the predicted images obtained byintra-prediction and inter-prediction are different, if blocks encoded by intraprediction and inter-prediction are adjacent to each other in the screen of an imageto be encoded, the difference between the predicted images of the blocks may bevisually recognized as an image quality difference. In general, people have avisual characteristic that they tend to notice changes in flat areas, so in flat areasof an image, the difference in image quality between blocks encoded by intraprediction and inter-prediction due to the difference in the predicted imagesdescribed above is more conspicuous, and may be perceived as deterioration ofimage quality.
[0005] In order to solve this problem, PTL 1 proposes the following technique.That is, the cost value is corrected such that the lower the activity indicating theflatness of each block, the intra-prediction mode is less likely to be selected,thereby preventing blocks encoded by intra-prediction and inter-prediction frombeing adjacent to each other in a flat portion of an image.CITATION LISTPATENT LITERATURE
[0006] PTL 1: Japanese Patent Laid-Open No. 2006-094081NON PATENT LITERATURE
[0007] NPL 1: "Versatile Video Coding (Draft 10)", JVET-S2001-vH, Joint VideoExperts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11,2020NPL 2: "Algorithm description for Versatile Video Coding and Test Mode 10(VTM10)", JVET-S2002-v1, Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 2020SUMMARY OF INVENTIONTECHNICAL PROBLEM
[0008] However, the technique described in PTL 1 does not completely preventblocks of intra-prediction and inter-prediction from being adjacent to each other.In other words, even if the technique described in PTL 1 is used, blocks of intraprediction and inter-prediction may be adjacent to each other depending on thedegree of flatness of the image. In this case, the difference in image qualitybetween blocks of intra-prediction and inter-prediction may be perceived asdeterioration in image quality.
[0009] The present invention has been made in consideration of the aboveproblems, and aims to make image quality deterioration due to differences inpredicted images inconspicuous in a case where one image is coded usingdifferent predicted images.SOLUTION TO PROBLEM
[0010] In order to achieve the above object, provided is an coding apparatus thatdivides an input image into blocks and performs coding processing on each blockof the input image, comprising: coding means for executing coding processingon each block using one of prediction modes of an inter-prediction mode thatgenerates an-inter predicted image by performing inter-prediction on a block andencodes a difference between the block and the inter-predicted image, an intraprediction mode that generates an intra-predicted image by performing intraprediction on a block and encodes a difference between the block and the intra4predicted image, and a multi-reference prediction mode that generates a multipredicted image by combining an inter-predicted image and an intra-predictedimage generated from a block, and encodes a difference between the block and themulti-predicted image; output means for outputting a result of the codingprocessing as coded data; and calculation means for calculating a first coding costin a case where the block is coded in the inter-prediction mode and a secondcoding cost in a case where the block is coded in the intra-prediction mode using apredetermined function, wherein the coding means executes the coding processingon the block using the multi-reference prediction mode in a case where adifference between the first coding cost and the second coding cost is equal to orless than a threshold, and executes the coding processing on the block using theprediction mode with a smaller one of the first coding cost and the second codingcost in a case where the difference between the first coding cost and the secondcoding cost is larger than the threshold.ADVANTAGEOUS EFFECTS OF INVENTION
[0011] According to this invention, it is possible to make image qualitydeterioration due to differences in predicted images inconspicuous in a case whereone image is coded using different predicted images.
[0012] Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with the accompanyingdrawings. Note that the same reference numerals denote the same or likecomponents throughout the accompanying drawings.BRIEF DESCRIPTION OF DRAWINGS
[0013] The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and, togetherwith the description, serve to explain principles of the invention.[FIG. 1] A block diagram showing a configuration of an image codingapparatus according to an embodiment of the present invention. [FIG. 2] A schematic diagram showing types (mode numbers) of intraprediction modes. [FIG. 3] A flowchart showing an operation of a motion compensation unitaccording to a first embodiment. [FIG. 4] A flowchart showing a determination method of weightingcoefficients in a multi-reference prediction unit according to the first embodiment. [FIG. 5] A block diagram showing an example of internal configurationof a feature detection unit according to the first embodiment. [FIG. 6] A flowchart showing a prediction mode determination method ina prediction mode selection unit according to the first embodiment. [FIG. 7A] A diagram showing an example of a gamma curve in HDRrecording. [FIG. 7B] A diagram showing an example of a gamma curve in HDRdisplay. [FIG. 8] A block diagram showing an example of internal configurationof the feature detection unit according to a second embodiment. [FIG. 9] A flowchart showing a prediction mode determination method inthe prediction mode selection unit according to the second embodiment. [FIG. 10] A flowchart showing a prediction mode determination methodin the prediction mode selection unit according to a modification of the firstembodiment.DESCRIPTION OF EMBODIMENTS
[0014] Hereinafter, embodiments will be described in detail with reference to theattached drawings. Note, the following embodiments are not intended to limitthe scope of the claimed invention. Multiple features are described in theembodiments, but limitation is not made to an invention that requires all suchfeatures, and multiple such features may be combined as appropriate.Furthermore, in the attached drawings, the same reference numerals are given tothe same or similar configurations, and redundant description thereof is omitted.
[0015] [First Embodiment] Fig. 1 is a block diagram showing the configuration of an image codingapparatus 100 that performs predictive coding according to the embodiment.The image coding apparatus 100 may be arranged, for example, in an imageprocessing engine incorporated in an image capturing apparatus. A frame buffer 101 stores an input image to be encoded, and outputsimage data of pixels of a required block size to an adder 102, intra-prediction unit112, motion compensation unit 114, and feature detection unit 116.
[0016] The adder 102 generates difference image data between the image dataoutput from the frame buffer 101 and predicted image data output from aprediction mode selection unit 117 described later, and outputs the differenceimage data to an orthogonal transformation unit 103. The orthogonal transformation unit 103 performs orthogonal transformsuch as discrete cosine transform (DCT) and discrete sine transform (DST) on thedifference image data, generates transform coefficients, and outputs the transformcoefficients to a quantization unit 104. The quantization unit 104 quantizes the transform coefficients outputfrom the orthogonal transformation unit 103 according to the quantization stepsize (or quantization parameter) output by a quantization control unit 106, andoutputs the results to an entropy coding unit 105, and an inverse quantization unit107.
[0017] The entropy coding unit 105 performs variable-length coding on thequantized transform coefficients, motion vectors, parameters necessary fordecoding processing, and the like, formats them into a predetermined format, andoutputs them as a coded stream. Also, the entropy coding unit 105 calculates agenerated code amount for each block and outputs it to the quantization controlunit 106. The quantization control unit 106 determines a quantization step size foreach block and a quantization parameter Qp from the generated code amountoutput from the entropy coding unit 105 and a preset target code amount, andoutputs them to the quantization unit 104 and the prediction mode selection unit117.
[0018] The inverse quantization unit 107 performs inverse quantization on thequantized transform coefficients output from the quantization unit 104, generatestransform coefficients, and outputs the transform coefficients to an inverseorthogonal transformation unit 108. The inverse orthogonal transformation unit 108 applies inversetransformation of the orthogonal transformation applied by the orthogonaltransformation unit 103 to the transform coefficients output from the inversequantization unit 107, generates difference image data, and outputs the differenceimage data to an adder 109. The adder 109 adds the difference image data generated by the inverseorthogonal transformation unit 108 and predicted image data output from theprediction mode selection unit 117, which will be described later, to generatelocally decoded reference image data, and outputs it to a reference frame buffer110 and a loop filter 111.
[0019] The loop filter 111 includes a deblocking filter, an adaptive loop filter,etc., removes block noise from the reference image data output from the adder109, and outputs the result to the reference frame buffer 110. Note that thereference image data output from the adder 109 may be stored in the referenceframe buffer 110 without going through the loop filter 111 as described above.In other words, the filtering process by the loop filter 111 can be omitted. The reference frame buffer 110 stores the supplied reference image data,and outputs the stored reference image data to the intra-prediction unit 112 and themotion compensation unit 114 at predetermined timings.
[0020] The intra-prediction unit 112 performs intra-prediction using the imagedata of a coding target block output from the frame buffer 101 and referenceimage data of blocks around the coding target block read out from the referenceframe buffer.
[0021] Here, intra-prediction modes selectable by the intra-prediction unit 112will be described with reference to FIG. 2. FIG. 2 is a diagram showing types of intra-prediction modes (modenumbers). The intra-prediction unit 112 can select an intra-prediction mode fromplanar prediction (mode number 0), DC prediction (mode number 1), and 65 typesof directional prediction (mode numbers 2 to 66). Further, the intra-prediction unit 112 selects the intra-prediction modewith the smallest cost value according to the cost function indicating the codingefficiency among the intra-prediction modes. Various functions are conceivableas the cost function, and the following equation (1) is used as a representativeequation in this embodiment.
[0022] IntraCost = SATD + Qp Mdcost … (1) In the equation (1), IntraCost represents a cost value that indicates thecoding efficiency in intra-prediction mode, and SATD represents a value obtainedby taking a difference between input image data to be encoded and the referenceimage data, applying Hadamard transform to obtained difference data, and takinga sum of absolute differences of the transformed difference data. Qp represents aquantization parameter used in the quantization unit 104, and Mdcost represents acost value corresponding to a code amount obtained in an intra-prediction mode.
[0023] The intra-prediction unit 112 generates a predicted image for intraprediction using the intra-prediction mode selected using the equation (1). Inaddition, the intra-prediction unit 112 outputs the selected intra-prediction mode,the cost value when the selected intra-prediction mode is applied, and thepredicted image data (intra-predicted image) generated by the intra-prediction, toa multi-reference prediction unit 115 and the prediction mode selection unit 117.
[0024] A motion information memory 113 is a memory for temporarily holdingmotion vectors output from a motion compensation unit 114, which will bedescribed later. The motion information memory 113 may also hold the motionvectors of the blocks located around the coding target block, and the motionvector of each of the blocks of already coded frame or frames. Information ofthe motion vectors held in the motion information memory 113 is referred to bythe motion compensation unit 114, which will be described later. The motion compensation unit 114 selects the optimal motion vectorfrom among two modes: an inter mode for newly searching for a motion vectorand a merge mode for selecting the optimal motion vector from motion vectorcandidates of already coded blocks.
[0025] Here, the operation of the motion compensation unit 114 will beexplained using FIG. 3. In step S301, the motion compensation unit 114 uses the image data ofthe coding target block output from the frame buffer 101 and the reference imagedata read out from the reference frame buffer 110 to search for the optimummotion vector in the inter mode. More specifically, the motion compensationunit 114 searches for the motion vector with the lowest cost value calculated usingthe cost function that indicates coding efficiency. Although various functions areconceivable as the cost function, the equation (2) is used as a representative one inthis embodiment.
[0026] InterCost = SATD + Qp Mvcost … (2) In Equation (2), InterCost represents a cost value indicating codingefficiency in the inter mode. SATD represents a value obtained by taking adifference between input image data to be encoded and the reference image dataafter motion vector is applied, applying Hadamard transform to obtaineddifference data, and taking a sum of absolute differences of the transformeddifference data. Qp is a quantization parameter used in the quantization unit 104,and Mvcost is a cost value corresponding to a code amount of motion vector. The motion compensation unit 114 temporarily holds the information ofthe motion vector determined using the equation (2) and the cost value obtainedwhen the motion vector is applied
[0027] In step S302, the motion compensation unit 114 selects the optimummotion vector in the merge mode from a plurality of motion vector candidatesalready stored in the motion information memory 113. More specifically, themotion compensation unit 114 applies the equation (2) to each of the plurality ofmotion vector candidates and selects the motion vector with the lowest cost value.The motion compensation unit 114 also calculates an index value (merge index)indicating the motion vector candidate. The motion compensation unit 114 temporarily holds the merge indexand the cost value obtained when the motion vector indicated by the merge indexis applied.
[0028] In step S303, the motion compensation unit 114 compares the cost valuein the inter mode calculated in step S301 and the cost value in the merge modecalculated in step S302, and selects the mode with the lowest cost value.
[0029] In step S304, the motion compensation unit 114 uses the motion vector inthe selected mode and the reference image data read out from the reference framebuffer 110 to generate predicted image data for inter-prediction. In the mergemode, the motion vector can be determined by referring to the merge indexcalculated in step S302.
[0030] The motion compensation unit 114 outputs a flag (merge flag) indicatingwhether or not the merge mode has been selected, information on motion vectoror merge index, the cost value in the selected mode, and the generated predictedimage data for inter-prediction (inter-predicted image ) to the multi-referenceprediction unit 115 and the prediction mode selection unit 117.
[0031] The multi-reference prediction unit 115 combines the predicted imagedata for intra-prediction output from the intra-prediction unit 112 and thepredicted image data for inter-prediction output from the motion compensationunit 114 to generates new predicted image data.
[0032] As a technique for combining predicted image data for intra-predictionand predicted image data for inter-prediction, NPL 2 discloses a technology calledCombined Inter and Intra Prediction (CIIP). The method of combining thepredicted image data for intra-prediction and predicted image data for interprediction is also based on CIIP. The method of combining the predicted imagedata by intra-prediction and predicted image data for inter-prediction is also basedon CIIP. On the other hand, NPL 2 describes, as application conditions of CIIP,that intra-predicted images are applied with PLANAR prediction, inter-predictedimages are applied with merge mode, and so forth. However, the method ofcombining the predicted image data for intra-prediction and the predicted imagedata for inter-prediction in this embodiment is not necessarily limited to theseconditions, and it is also possible to apply directional prediction, and like, to thepredicted image data for intra-prediction.
[0033] In CIIP, predicted images for intra-prediction and inter-prediction areblended pixel by pixel, so that differences in predicted images between blocks canbe further reduced. Based on this, in the present embodiment, CIIP is applied toblocks that exist in areas where intra-prediction and inter-prediction blocks arelikely to be adjacent to each other, and image quality differences between blocksare likely to be noticeable, such as flat areas of an image. This reduces imagequality degradation caused by using different prediction modes to generatepredicted images of adjacent blocks.
[0034] Next, details of a method of combining predicted image data for intraprediction and predicted image data for inter-prediction performed by the multireference prediction unit 115 will be described below. The multi-reference prediction unit 115 combines predicted image databy intra-prediction and predicted image data by inter-prediction for each pixelusing the following equation (3).
[0035] Pciip = ((4-wt) Pinter + wt Pintra + 2) / 4 …(3) In the equation (3), Pintra represents the pixel value of predicted imagedata for intra-prediction, Pinter represents the pixel value of predicted image datafor inter-prediction, and Pciip is the pixel value of predicted image data aftercombining them. wt represents a weighting coefficient indicating the ratio ofpixel values for intra-prediction and inter-prediction, and can take an integer valuefrom 1 to 3 here.
[0036] Here, a method of determining the weighting coefficient wt used in themulti-reference prediction unit 115 will be described with reference to FIG. 4. In step S401, the multi-reference prediction unit 115 initializes isIntraTopand isIntraLeft to 0, and the weighting coefficient wt to 1. isIntraTop is a flagindicating whether or not the prediction mode of the encoded block that isadjacent to the coding target block in the upward direction is intra-prediction.Also, isIntraLeft is a flag indicating whether or not the prediction mode of theencoded block adjacent to the coding target block in the left direction is intraprediction. Note that the prediction modes of these encoded blocks are held bythe prediction mode selection unit 117 as described later, and the multi-referenceprediction unit 115 can appropriately refer to this information.
[0037] In step S402, the multi-reference prediction unit 115 determines whetheror not the prediction mode of the encoded block adjacent to the coding targetblock in the upward direction is intra-prediction. If the prediction mode of theencoded block adjacent in the upward direction is intra-prediction (Yes in stepS402), the multi-reference prediction unit 115 sets isIntraTop to 1 in step S403.If the prediction mode of the encoded block adjacent in the upward direction isnot intra-prediction (No in step S402), the multi-reference prediction unit 115advances the process to step S404.
[0038] In step S404, the multi-reference prediction unit 115 determines whetheror not the prediction mode of the encoded block adjacent to the coding targetblock in the left direction is intra-prediction. If the prediction mode of theencoded block adjacent in the left direction is intra-prediction (Yes in step S404),the multi-reference prediction unit 115 sets isIntraLeft to 1 in step S405. If theprediction mode of the encoded block adjacent in the left direction is not intraprediction (No in step S404), the multi-reference prediction unit 115 advances theprocess to step S406.
[0039] In steps S406 to S410, the multi-reference prediction unit 115 determinesthe value of weighting coefficient wt according to the sum of isIntraTop andisIntraLeft. If the sum of isIntraTop and isIntraLeft is 2 (Yes in step S406), theprocess proceeds to step S407 where the multi-reference prediction unit 115 setsthe weighting coefficient wt to 3. If the sum of isIntraTop and isIntraLeft is 1(No in step S406 and Yes in step S408), the process proceeds to step S409 wherethe multi-reference prediction unit 115 sets the weighting coefficient wt to 2. Ifthe sum of isIntraTop and isIntraLeft is 0 (No in step S406 and No in step S408),the process proceeds to step S410 where the multi-reference prediction unit 115sets the weighting coefficient wt to one.
[0040] The processes of steps S406 to S410 adaptively change the combiningpercentages between the pixel values of the predicted image data for intraprediction and the pixel values of the predicted image data for inter-prediction atthe time of combining the predicted images, according to the prediction mode ofthe blocks surrounding the coding target block. As a result, in a case wheremulti-reference prediction is selected as the prediction mode in the predictionmode selection unit 117, which will be described later, image quality deteriorationdue to predicted images of the coding target block and its surrounding blocksbeing different from each other can be reduced.
[0041] Note that the method of synthesizing the predicted image data for intraprediction and the predicted image data for inter-prediction is not limited to theexample shown in the equation (3) and FIG. 4. For example, the multi-referenceprediction unit 115 may perform the synthesis using the equation (4) below. Pciip = ((8-wt) Pinter + wt Pintra + 4) / 8 …(4) As another example, the weighting coefficient wt can be adaptivelychanged for each pixel according to the pixel position of each pixel in the codingtarget block. Furthermore, the weighting coefficient wt can be adaptivelychanged according to the intra-prediction mode (DC prediction, PLANARprediction, etc.) output from the intra-prediction unit 112, the block size of thecoding target block, and the like.
[0042] The multi-reference prediction unit 115 uses predicted image data (multipredicted image) synthesized using the equation (3) to calculate a cost value basedon a cost function that indicates coding efficiency. Various functions areconceivable as the cost function, but in this embodiment, the equation (5) is usedas a representative one.
[0043] CiipCost = SATD + Qp x Mpcost …(5)In the equation (5), CiipCost represents a cost value indicating coding efficiencyin multi-reference prediction. SATD represents a value obtained by taking adifference between input image data to be encoded and the synthesized predictedimage data, applying Hadamard transform to obtained difference data, and takinga sum of absolute differences of the transformed difference data. Qp is aquantization parameter used in the quantization unit 104, and Mpcost is a costvalue corresponding to a code amount of parameters required in multi-referenceprediction. Parameters required in multi-reference prediction include an intraprediction mode output from the intra-prediction unit 112, merge flag, mergeindex, or motion vector output from the motion compensation unit 114.
[0044] The multi-reference prediction unit 115 outputs the cost value calculatedusing the equation (5) and the synthesized predicted image data to the predictionmode selection unit 117.
[0045] The feature detection unit 116 generates a flat area flag indicatingwhether the coding target block belongs to a flat area of the image and outputs itto the prediction mode selection unit 117. Here, a method of generating the flatarea flag in the feature detection unit 116 will be described with reference to FIG.5
[0046] FIG. 5 is a block diagram showing an example of the internalconfiguration of the feature detection unit 116. A flatness determination unit 501 acquires an input image of the codingtarget block and generates a flag (flatness flag) indicating whether or not thecoding target block is flat. Various methods are conceivable for determiningwhether or not the block is flat. In this embodiment, the flatness is defined as thevariance of the pixel values of the coding target block, and if the flatness is lowerthan a predetermined threshold, the block is determined to be flat. In the presentembodiment, the flag value is "1" if the block is determined to be flat, and the flagvalue is "0" if the block is determined to be not flat.
[0047] A flatness flag buffer 502 temporarily holds the flatness flag output fromthe flatness determination unit 501. The flatness flag buffer 502 also holdsflatness flags of a plurality of encoded blocks, and a flat area determination unit503, which will be described later, can appropriately refer to this information.The flatness flag buffer 502 in this embodiment holds at least the flatness flags ofthe encoded blocks adjacent to the coding target block in the upper direction, theupper left direction, and the left direction.
[0048] The flat area determination unit 503 determines whether the coding targetblock belongs to a flat area of the image based on the flatness flag of the codingtarget block and the flatness flags of a plurality of encoded blocks held in theflatness flag buffer 502. Various methods are conceivable as a method ofdetermining whether or not the coding target block belongs to a flat area. In thisembodiment, if the flag values of the flat flags of the encoded blocks adjacent inthe upper direction, the upper left direction and the left direction are all "1", it isdetermined that the coding target block belongs to a flat area. In the presentembodiment, the flag value is set to "1" if the block is determined that it belongsto a flat area, and the flag value is set to "0" if the block is determined that it doesnot belong to a flat area.
[0049] Note that the method of determining whether or not the coding targetblock belongs to a flat area is not limited to the example described using FIG. 5.For example, the feature detection unit 116 may simply assume that a codingtarget block belongs to a flat area if its flatness is lower than a predeterminedthreshold. Alternatively, the feature detection unit 116 may determine a flatnessflag for each of the blocks of a frame and store the flatness flags in the flatnessflag buffer 502 before starting the coding processing of the frame. This allowsthe flat area determination unit 503 to improve determination accuracy of whetheror not the coding target block belongs to a flat area by referring not only to the flatflags of blocks adjacent to the coding target block, but also to the flat flags ofblocks further away from the coding target block.
[0050] The prediction mode selection unit 117 selects one of the intra-prediction,inter-prediction, and multi-reference prediction. Here, the determination methodof the prediction mode in the prediction mode selection unit 117 is explainedusing FIG. 6. In step S601, the prediction mode selection unit 117 determines whetherthe flat area flag output from the feature detection unit 116 is 0 and / or whether ornot the quantization step size Qstep of the coding target block obtained from thequantization control unit 106 is smaller than the threshold Thqstep.
[0051] Thereafter, in steps S602 to S608 following S601, the prediction modeselection unit 117 puts weight on coding efficiency and always selects theprediction mode with the lowest cost value. On the other hand, in steps S609 toS613, the prediction mode selection unit 117 puts weight on reduction of imagequality degradation, and does not always select the prediction mode with thelowest cost value so that degradation of image quality due to different predictedimages between blocks is less conspicuous. That is, the determination formulaof step S601 means that the prediction mode selection unit 117 selects which oneof the coding efficiency and the reduction of image quality deterioration isemphasized.
[0052] In this embodiment, in a case where the region to which the coding targetblock belongs is not flat and / or in a case where the quantization step size is small,it is determined that the degree of image quality deterioration due to differentpredicted images between blocks is small, and the processes of steps S602 to S608are performed with emphasis on the encoding efficiency. Conversely, in a casewhere the region to which the coding target block belongs is flat and thequantization step size is large, the processes of S609 to S613 are performed withemphasis on reducing image quality deterioration. Note that the determinationmethod in step S601 is not limited to the example shown here. For example, theprediction mode selection unit 117 can perform the determination using either theflat area flag or the quantization step size.
[0053] In step S601, if the flat area flag is 0 and / or the quantization step sizeQstep is smaller than the threshold Thqstep, the prediction mode selection unit117 advances the process to step S602. In step S602, the prediction mode selection unit 117 compares the costvalue IntraCost output from the intra-prediction unit 112 and the cost valueInterCost output from the motion compensation unit 114.
[0054] If InterCost is greater than IntraCost (Yes in step S602), the predictionmode selection unit 117 compares IntraCost with the cost value CiipCost outputfrom the multi-reference prediction unit 115 (step S603). If the IntraCost isgreater than the CiipCost (Yes in step S603), the prediction mode selection unit117 selects multi-reference prediction as the prediction mode (step S604). If theIntraCost is less than or equal to the CiipCost (No in step S603), the predictionmode selection unit 117 selects intra-prediction as the prediction mode (stepS605).
[0055] Also, in step S602, if InterCost is less than or equal to IntraCost (No instep S602), the prediction mode selection unit 117 compares InterCost withCiipCost (step S606). If InterCost is greater than CiipCost (Yes in step S606),the prediction mode selection unit 117 selects multi-reference prediction as theprediction mode (step S607). If InterCost is less than or equal to CiipCost (No instep S606), the prediction mode selection unit 117 selects inter-prediction as theprediction mode (step S608).
[0056] On the other hand, in step S601, if the flat area flag output from thefeature detection unit 116 is 1 and the quantization step size Qstep is equal to orgreater than the threshold Thqstep, the prediction mode selection unit 117advances the process to step S609.
[0057] In step S609, the prediction mode selection unit 117 compares InterCostwith the values of IntraCost+α. α is a correction value for the cost value of intraprediction that can be arbitrarily set.
[0058] Generally, in a region where the cost values of intra-prediction and interprediction are about the same, image quality deterioration in intra-prediction tendsto be more conspicuous than that in inter-prediction. Therefore, in thisembodiment, if the coding target block belongs to a flat area, inter-prediction ispreferentially selected by correcting the cost value of intra-prediction. As aresult, cases in which blocks of intra-prediction and blocks of inter-prediction areadjacent to each other are reduced.
[0059] In this embodiment, the cost value is corrected by adding α as describedabove, but it is also possible to correct the cost value by subtracting, multiplying,or dividing the cost value by an arbitrary constant. Also, the method ofcorrecting the cost value is not limited to the method using α as described above,and any method may be adopted as long as the method enables inter-prediction tobe preferentially selected.
[0060] If InterCost is greater than IntraCost+α (Yes in step S609), the predictionmode selection unit 117 selects intra-prediction as the prediction mode (stepS610).
[0061] If InterCost is equal to or less than IntraCost+α (No in step S609), theprediction mode selection unit 117 determines whether the absolute value of thedifference between InterCost and IntraCost is equal to or less than a constant c(step S611). Here, the details of the determination formula used in step S611will be described.
[0062] As described above, the determination in step S609 is intended topreferentially select inter-prediction. On the other hand, depending on thedegree of flatness of the image, an area in which blocks of intra-prediction andblocks of inter-prediction often become adjacent to each other, and thedetermination result of step S609 frequently switches still exists. Here, in anarea where the determination result of step S609 frequently changes, it is assumedthat the values of InterCost and IntraCost+α are close to each other in thedetermination formula used in step S609. In consideration of the above, in thepresent embodiment, the area where the determination result in step S609frequently changes is detected using the following expression (6).
[0063] |InterCost - (IntraCost+α)| …(6)is a constant for detecting the closeness between InterCost and thevalues of IntraCost+α. Here, assuming that α is a positive constant, thefollowing expression (7) is obtained by modifying the expression (6).
[0064] |InterCost - IntraCost| + α = c …(7) That is, in the present embodiment, in a case where the absolute value ofthe difference between the InterCost and IntraCost of the coding target block isequal to or less than the constant c in step S611, it is determined that the blockexists in an area where blocks of intra-prediction and blocks of inter-predictionblock tend to be adjacent to each other.
[0065] Note that the constant c can be set arbitrarily, and may be adaptivelychanged using the target bit rate of the image coding apparatus, the value of thequantization parameter Qp used in the quantization unit 104, or the characteristicsof the image. For example, the value of the constant c can be changed for eachblock according to the flatness, and the like, calculated by the flatnessdetermination unit 501.
[0066] If the absolute value of the difference between InterCost and IntraCost isc or less (Yes in step S611), the prediction mode selection unit 117 selects multireference prediction as the prediction mode (step S612). That is, in the presentembodiment, multi-reference prediction is applied to blocks existing in an areawhere blocks of intra-prediction and blocks of inter-prediction block tends to beadjacent to each other. In a case where multi-reference prediction is used, sincea predicted image of intra-prediction and a predicted image of inter-predictionpredicted are blended and the difference in predicted images between blocksbecomes small, it is possible to reduce image quality degradation caused bypredicted images being generated using different prediction modes betweenblocks.
[0067] In multi-reference prediction, it is necessary to encode parameters relatedto prediction modes for both intra-prediction and inter-prediction (intra-predictionmode, motion vector, etc.), there is a risk that a code amount related to parameterswill be larger in multi-reference prediction than in inter-prediction. Therefore,the prediction mode selection unit 117 may further perform determination ofwhether or not the value of the merge flag output from the motion compensationunit 114 being 1 in the determination of step S611. As a result, when multireference prediction is selected, there is no need to encode parameters related tomotion vectors, and an increase in code amount related to parameters can besuppressed.
[0068] If the absolute value of the difference between InterCost and IntraCost isnot equal to or less than the constant c (No in step S611), the prediction modeselection unit 117 selects inter-prediction as the prediction mode (step S613).
[0069] A predicted image is generated according to the prediction mode selectedas described above. Then, the difference between the image data of the codingtarget block and the predicted image is encoded, and the coded result is output asa coded stream (coded data).
[0070] In the example shown in FIG. 6, if an area is in the flat part of the imageand if the quantization step size is large, the area is detected as an area whereblocks of intra-prediction and blocks of inter-prediction tend to be adjacent toeach other, and the prediction mode that prioritizes reduction of image qualitydeterioration is applied in this area. On the other hand, there are cases where theboundary between a block of intra-prediction and a block of inter-prediction isconspicuous regardless of the degree of flatness of the image, depending on thetarget bit rate of the image coding apparatus. In such a case, the prediction modeselection unit 117 may not perform the determination of step S601, and mayalways detect an area where a block of intra-prediction and a block of interprediction tend to be adjacent to each other, and apply multi-reference predictionto the blocks existing in the detected area.
[0071] In addition, regardless of whether the boundary between a block of intraprediction and a block of inter-prediction is conspicuous or not, the processingfrom step S609 onward may be performed for all blocks.
[0072] As described above, according to the first embodiment, in a case wherean image is encoded using different predicted images, it is possible to make theimage quality deterioration caused by the difference in the predicted images lessconspicuous. In addition, in a case where the boundary between a block of intraprediction and a block of inter-prediction is conspicuous, such as in a case wherean area to which the coding target block belongs is flat and the quantization stepsize is large, the prediction mode selection unit 117 operates with an emphasis onreducing image quality deterioration. On the other hand, in a case where theboundary between a block of intra-prediction and a block of inter-prediction isinconspicuous, the prediction mode selection unit 117 operates with emphasis oncoding efficiency. This makes it possible to reduce image quality deteriorationdue to differences in predicted images between blocks compared to theconventional art, without impairing coding efficiency.
[0073] [Modification] Next, a method of generating coded data in the modification will bedescribed with reference to FIG. 10.
[0074] As described above with reference to FIG. 3, the motion compensationunit 114 calculates the cost value in a case where the inter-prediction mode isapplied to the coding target block (step S1001). Further, as described above withreference to FIG. 2, the intra-prediction unit 112 calculates the cost value in a casewhere the intra-prediction mode is applied to the coding target block (step S1002).
[0075] Then, as in step S611, the prediction mode selection unit 117 determineswhether or not the absolute value of the difference between the cost value of theinter-prediction mode calculated in step S1001 and the cost value of the intraprediction mode calculated in step S1002 is a constant c or less (step S1003). Asa result of the determination, if the absolute value of the difference is the constantc or less (Yes in step S1003), the prediction mode selection unit 117 selects themulti-reference prediction mode (step S1004).
[0076] On the other hand, as a result of the determination, if the absolute valueof the difference is greater than the constant c (No in step S1003), the predictionmode selection unit 117 calculates the cost value in a case where the multireference prediction mode is applied to the coding target block (step S1005).The prediction mode selection unit 117 compares the cost values of the interprediction mode, the intra-prediction mode, and the multi-reference predictionmode calculated in steps S1001, S1002, and S1005, respectively, and if the costvalue of the multi-reference prediction mode is the lowest (Yes in step S1006), themulti-reference prediction mode is selected (step S1004).
[0077] If the cost value of the multi-reference prediction mode is not the lowest(No in step S1006) and the cost value of the inter-prediction mode is the lowest(Yes in step S1007), the inter-prediction mode is selected (step S1008). If thecost value of the multi-reference prediction mode is not the lowest (No in stepS1006) and the cost value of the inter-prediction mode is not the lowest (No instep S1007), that is, if the cost value of the intra-prediction mode is the lowest, theintra-prediction mode is selected (step S1009).
[0078] Then, if the multi-reference prediction mode is selected in step S1004,the multi-reference prediction unit 115 combines the predicted image data forintra-prediction output from the intra-prediction unit 112 and the predicted imagedata for inter-prediction output from the motion compensation unit 114, andgenerates predicted image data for multi-reference prediction (step S1010).Alternatively, if the inter-prediction mode is selected in step S1008, the motioncompensation unit 114 generates predicted image data for inter-prediction (stepS1011) as in step S304. Alternatively, if the intra-prediction mode is selected instep S1009, the intra-prediction unit 112 generates predicted image data for intraprediction (step S1012). The predicted image data generated in step S1011,S1011 or S1012 is output to the adder 102 by the prediction mode selection unit117.
[0079] Then, the adder 102 generates difference image data between the blockimage data to be encoded output from the frame buffer and the predicted imagedata, and encodes the difference image data using each unit as described above(step S1013), and the coded result is output as a coded stream (step S1014).
[0080] Before starting the processing of step S1001, similarly to step S601, theprediction mode selection unit 117 may determine whether or not the flat area flagoutput from the feature detection unit 116 is 0 and / or whether or not thequantization step size Qstep of the coding target block obtained from thequantization control unit 106 is smaller than the threshold Thqstep, and if the flatarea flag is 1 and / or the quantization step size Qstep is equal to or greater than thethreshold Thqstep, the processes after step S1001 may be executed.Alternatively, in a case where the dynamic range of the input image is a highdynamic range equal to or greater than a predetermined value, the processing fromstep S1001 onward may be executed.
[0081] [Second embodiment] Next, a second embodiment of the present invention will be described. In the latest video coding methods such as HEVC and VVC, it is possibleto encode a moving image recorded with 10-bit, 12-bit, or other multi-bit HDR(High Dynamic Range) or a Log-recorded moving image. A Log-recordedmoving image is a moving image shot using a gamma curve uniquely defined bythe manufacturer of a digital camera or the like, so that recording can beperformed in a higher dynamic range, unlike the gamma curve defined bystandards such as BT.709.
[0082] HDR-recorded or Log-recorded moving images (high dynamic rangeimages) are displayed after being converted using a set gamma curve at the timeof outputting the images to a display. FIG. 7A shows the relationship betweeninput luminance values (actual brightness at the time of shooting) and output pixelvalues (pixel value after coding) in HDR-recording. The horizontal axis showsthe input luminance values, the vertical axis shows the output pixel values, andthe relationship between the input values and the output values are defined by agamma curve 701. In a range where the input value is small, the curve is steepin the vertical direction, and many pixel values are assigned with respect to therange of the input values. On the other hand, in a range where the input value islarge, the curve becomes gentle, and the number of pixel values assigned withrespect to the range of the input values is small. By assigning the output pixelvalues to the input luminance values in this way, recording with a high dynamicrange is made possible.
[0083] FIG. 7B shows the relationship between input values (pixel value aftercoding) and output luminance values (brightness displayed on a display) in HDRdisplay. The horizontal axis shows the input pixel values, and the vertical axisshows the output luminance values that are actually displayed, and therelationship between the input values and the output values is defined by a gammacurve 702, which is the opposite relationship to the gamma curve 701. In arange where the input values are small, the curve is gentle in the vertical direction,and the amount of change in displayed brightness with respect to the range of theinput values is small. On the other hand, in a range where the input values arelarge, the curve becomes steep, and the amount of change in displayed brightnesswith respect to the range of the input values is large. By assigning output theluminance values to the input pixel values in this way, HDR-recorded movingimages can be displayed with correct brightness.
[0084] As described above, in HDR-recorded or Log-recorded moving images,the amount of change in brightness that is displayed on a display is larger withrespect to the amount of change in coded pixel values compared to movingimages recorded with Standard Dynamic Range (SDR). In other words,deterioration in image quality after coding is amplified and displayed on thedisplay. In addition, this tendency becomes more conspicuous in an area withhigh luminance values (referred to as "high-luminance area or "bright area"), andimage quality degradation due to predicted images being different between blocks,which is the problem that the present invention is to solve, is also moreconspicuous in this area.
[0085] In the second embodiment, such HDR-recorded or Log-recorded movingimages (for convenience, referred to as "HDR images") are coding target, andimage quality deterioration that occurs in a bright area of HDR images is aimed atbeing reduced. Note that in the second embodiment, differences from the firstembodiment will be mainly described, and descriptions of parts that are the sameas those in the first embodiment will be omitted.
[0086] The overall configuration of an image encoding apparatus in the secondembodiment is the same as that shown in FIG. 1, but the configuration of thefeature detection unit 116 is different from that described with reference to FIG. 5in the first embodiment. FIG. 8 is a block diagram showing an example of an internalconfiguration of the feature detection unit 116 according to the secondembodiment. As in the first embodiment, the feature detection unit 116 in thesecond embodiment includes the flatness determination unit 501, the flatness flagbuffer 502, and the flat area determination unit 503 for determining whether or nota coding target block belongs to a flat area of an image. Furthermore, the featuredetection unit 116 has a high luminance determination unit 804, a high luminanceflag buffer 805, and a high luminance area determination unit 806 for determiningwhether or not the coding target block belongs to a bright area of the image. Note that the flatness determination unit 501, flatness flag buffer 502, andflat area determination unit 503 operate in the same manner as in the firstembodiment, so their description is omitted here.
[0087] The high luminance determination unit 804 acquires an input image ofthe coding target block and generates a flag (high luminance flag) indicatingwhether or not the coding target block is of high luminance. Various methodsare conceivable for determining whether or not the luminance is high, and in thisembodiment, an average of luminance values of pixels in the coding target blockis calculated, and if the calculated average luminance value is larger than apredetermined threshold, it is determined that the luminance is high. Further, inthe present embodiment, the flag value is set to "1" if it is determined that theluminance is high, and the flag value is set to "0" if it is determined that theluminance is not high.
[0088] The high luminance flag buffer 805 temporarily holds the high luminanceflag output from the high luminance determination unit 804. The highluminance flag buffer 805 also holds high luminance flags of a plurality of codedblocks, and the high luminance area determination unit 806, which will bedescribed later, can appropriately refer to this information. The high luminanceflag buffer 805 in this embodiment holds at least the high luminance flags of thecoded blocks that are adjacent to the coding target block in the upper direction, theupper left direction, and the left direction.
[0089] The high luminance area determination unit 806 determines whether ornot the coding target block belongs to the bright area of the image based on thehigh luminance flag of the coding target block held in the high luminance flagbuffer 805 and the high luminance flags of a plurality of coded blocks. Variousmethods are conceivable for determining whether or not the coding target blockbelongs to the bright area, and in this embodiment, if the high luminance flag ofthe coding target block is "1" and if the flag values of the high luminance flags ofthe coded blocks adjacent to the coding target block in the upper direction, theupper left direction and the left direction are all "1", it is determined that thecoding target block belongs to the bright area. In the present embodiment, theflag value is "1" if it is determined that the coding target block belongs to thebright area, and the flag value is "0" if it is determined that the coding target blockdoes not belong to the bright area.
[0090] Note that the method of determining whether or not the coding targetblock belongs to the bright area is not limited to the example described using FIG.8. For example, the feature detection unit 116 may simply determine that thecoding target block belongs to a bright area if the average luminance value of thecoding target block is greater than a predetermined threshold. Alternatively, thefeature detection unit 116 may determine a high luminance flag for each block ofa certain frame and store the high luminance flags in the high luminance flagbuffer 805 before starting coding processing of the frame. With this, the highluminance area determination unit 806 can refer not only to the high brightnessflags of blocks adjacent to the coding target block, but also to the high brightnessflags of blocks farther away from the coding target block, thereby it is possible toimprove the accuracy of determination whether the coding target block belongs tothe bright area.
[0091] In this way, the feature detection unit 116 generates a flat area flag and abright area flag and outputs them to the prediction mode selection unit 117.
[0092] FIG. 9 is a flowchart showing a prediction mode determination method inthe prediction mode selection unit 117 in the second embodiment. The samereference numerals are given to the same processes as those shown in FIG. 6, andthe description thereof is omitted. The processing in the second embodiment shown in FIG. 9 is differentfrom the processing in the first embodiment shown in FIG. 6 in the point that aprocess (step S901) for determining whether or not a moving image to be encodedis a HDR-recorded or Log-recorded moving image, and whether or not a blockbelongs to a bright area is added. It should be noted that information on whetheror not the moving image to be coded is HDR-recorded or Log-recorded is sent tothe image coding apparatus in advance from a control source of the imageencoding apparatus such as a CPU (not shown).
[0093] In step S901, if the moving image to be coded is an HDR-recorded orLog-recorded moving image and if the bright area flag is 1, then the predictionmode selection unit 117 advances the process to step S601. On the other hand,in step S901, if the image to be coded is not an HDR-recorded or Log-recordedmoving image and / or if the bright area flag is not 1, then the prediction modeselection unit 117 skips the process of step S601, and advances the process to stepS602.
[0094] That is, if the moving image to be coded is an HDR-recorded or Logrecorded moving image, if the coding target block belongs to a bright area and aflat area, and if the quantization step size is large, the prediction mode selectionunit 117 puts a weight on reducing image quality deterioration and operates sothat image quality deterioration due to a difference between predicted images ofblocks becomes less conspicuous (steps S609 to S613).
[0095] On the other hand, in cases other than the above, the prediction modeselection unit 117 puts a weight on coding efficiency and operates to always selecta prediction mode with the lowest cost value (steps S602 to S608).
[0096] As described above, according to the second embodiment, not only theflatness of an area to which the coding target block belongs and the quantizationstep size, but also whether or not the image to be coded is HDR-recorded or Logrecorded moving image and whether or not the coding target block belongs to abright area are considered, and whether to prioritize coding efficiency or reductionof image quality deterioration is selected. As a result, it is possible to reduceimage quality deterioration, which is particularly conspicuous in bright areas,compared to the conventional art, without impairing the coding efficiency whencoding an HDR image.
[0097] <Other Embodiment> The present invention can be implemented by processing of supplying aprogram for implementing one or more functions of the above-describedembodiments to a system or apparatus via a network or storage medium, andcausing one or more processors in the computer of the system or apparatus to readout and execute the program. The present invention can also be implemented bya circuit (for example, an ASIC) for implementing one or more functions.
[0098] The present invention is not limited to the above embodiments andvarious changes and modifications can be made within the spirit and scope of thepresent invention. Therefore, to apprise the public of the scope of the presentinvention, the following claims are made.
[0099] This application claims the benefit of Japanese Patent Application No.2021-046238, filed March 19, 2021 and Japanese Patent Application No. 2022-003254, filed January 12, 2022 which are hereby incorporated by reference hereinin their entirety.
Claims
1. An coding apparatus that divides an input image into blocks and performs coding processing on each block of the input image, comprising: coding means configured to execute coding processing on each block using one of prediction modes of an inter-prediction mode that generates an-inter predicted image by performing inter-prediction on a block and encodes a difference between the block and the inter-predicted image, an intra-prediction mode that generates an intra-predicted image by performing intra-prediction on a block and encodes a difference between the block and the intra-predicted image, and a multi-reference prediction mode that generates a multipredicted image by combining an inter-predicted image and an intra-predicted image generated from a block, and encodes a difference between the block and the multi-predicted image; output means configured to output a result of the coding processing as coded data; and calculation means configured to calculate a first coding cost in a case where the block is coded in the inter-prediction mode and a second coding cost in a case where the block is coded in the intra-prediction mode using a predetermined function, wherein the coding means executes the coding processing on the block using the multi-reference prediction mode in a case where a difference between the first coding cost and the second coding cost is equal to or less than a threshold, and executes the coding processing on the block using the prediction mode with a smaller one of the first coding cost and the second coding cost in a case where the difference between the first coding cost and the second coding cost is larger than the threshold.
2. The coding apparatus according to claim 1, wherein the calculation means further calculates a third coding cost in a case where the block is coded in the multi-reference prediction mode using a predetermined function, and the coding means executes the coding processing on the block using the prediction mode with a smallest one of the first coding cost, the second coding cost and the third coding cost in a case where the difference between the first coding cost and the second coding cost is larger than a threshold.
3. The coding apparatus according to claim 1 or 2, wherein a coding target block of the coding means is included in a flat area or a high luminance area.
4. The coding apparatus according to any one of claims 1 to 3, wherein the input image is of high dynamic range.
5. The coding apparatus according to any one of claims 1 to 4, wherein a quantization step used in the coding processing is greater than a predetermined value.
6. The coding apparatus according to any one of claims 1 to 5, wherein the multi-predicted image is generated by weighting and combining the interpredicted image and the intra-predicted image.
7. The coding apparatus according to any one of claims 1 to 6, wherein the coding cost is calculated based on a sum of absolute differences corresponding to values obtained by taking a difference between the input image and a reference image which is decoded after adding the difference to the predicted image and performing Hadamard transform on the difference, and on a quantization parameter and a code amount.
8. An coding method that divides an input image into blocks and performs coding processing on each block of the input image, comprising: a step of executing coding processing on each block using one of prediction modes of an inter-prediction mode that generates an inter-predicted image by performing inter-prediction on a block and encodes a difference between the block and the inter-predicted image, an intra-prediction mode that generates an intra-predicted image by performing intra-prediction on a block and encodes a difference between the block and the intra-predicted image, and a multi-reference prediction mode that generates a multipredicted image by synthesizing an inter-predicted image and an intra-predicted image generated from a block, and encodes a difference between the block and the multi-predicted image; a step of outputting a result of the coding processing as coded data; and a step of calculating a first coding cost in a case where the block is coded in the inter-prediction mode and a second coding cost in a case where the block is coded in the intra-prediction mode using a predetermined function, wherein the coding processing is executed on the block using the multireference prediction mode in a case where a difference between the first coding cost and the second coding cost is equal to or less than a threshold, and the coding processing is executed on the block using the prediction mode with a smaller one of the first coding cost and the second coding cost in a case where the difference between the first coding cost and the second coding cost is larger than the threshold.
