Encoder and method
Patent Information
- Application Number
- JP2022003254
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-03-19
- Filing Date
- 2022-01-12
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-01-12
AI Technical Summary
Existing video encoding methods, such as H.264/AVC, H.265/HEVC, and H.266/VVC, fail to completely prevent intra-prediction and inter-prediction blocks from being adjacent, leading to visually noticeable image quality deterioration, especially in flat areas of an image.
An encoding device that performs encoding processing for each block using inter prediction, intra prediction, and multi-reference prediction modes, calculating encoding costs for each mode and selecting the mode that minimizes image quality deterioration by blending intra-prediction and inter-prediction images, particularly in areas where block adjacency is likely.
Reduces the visibility of image quality deterioration caused by differences in predicted images between blocks, maintaining encoding efficiency while minimizing noticeable quality differences.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an encoding apparatus and method, and more particularly to prediction encoding technology.
Background Art
[0002] In order to efficiently transmit, record, and reproduce moving images, a moving image encoding apparatus that compresses and encodes moving image data and a moving image decoding apparatus that decodes and expands are used. As moving image encoding methods, currently, H.264 / AVC (Advanced Video Coding) and H.265 / HEVC (High Efficiency Video Coding), which utilize hybrid encoding that reduces spatial and temporal redundancy of moving image data, are mainstream. Furthermore, as a next-generation moving image encoding method, international standardization of H.266 / VVC (Versatile Video Coding) is in progress (see Non-Patent Document 1).
[0003] In such a moving image encoding method, one screen is divided into small blocks called CUs (Coding Units), and encoding / decoding processing is performed in units of these blocks. Also, in such a moving image encoding method, usually, a prediction image is generated based on a local decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes also referred to as a "difference image" or "residual image") obtained by subtracting the prediction image from the input image is encoded. As methods for generating a prediction image, inter-picture prediction (inter prediction) and intra-picture prediction (intra prediction) can be selected.
[0004] In inter-slicing, the decision of whether to use intra-prediction or inter-prediction for encoding is generally made by comparing cost values (encoding costs) obtained using a cost function based on the prediction error value and the code amount of the prediction error. Here, since the predicted images differ between intra-prediction and inter-prediction, if blocks of intra-prediction and inter-prediction are adjacent to each other within the screen of the image to be encoded, the difference in the predicted images may be perceived as a difference in image quality. Generally, human vision is more sensitive to changes in flat areas, so in flat areas of the image, the difference in image quality between intra-prediction and inter-prediction blocks caused by the difference in the predicted images mentioned above may be more visually noticeable and may be perceived as a degradation of image quality.
[0005] To solve this problem, Patent Document 1 proposes the following technique: the cost value is corrected so that the lower the activity indicating the flatness of each block, the less likely the intra-prediction mode is to be selected, thereby making it less likely for intra-prediction and inter-prediction blocks to be adjacent in the flat parts of the image. [Prior art documents] [Patent Documents]
[0006] [Patent Document 1] Japanese Patent Publication No. 2006-094081 [Non-Patent Document 1] “Versatile Video Coding(Draft 10)”,JVET-S2001-vH,Joint Video Experts Team (JVET)of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11,2020 [Non-Patent Document 2] “Algorithm description for Versatile Video Coding and Test Model 10(VTM10)”,JVET-S2002-v1,Joint Video Experts Team (JVET)of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11,2020 [Overview of the project] [Problems that the invention aims to solve]
[0007] However, the technology described in Patent Document 1 does not completely prevent the adjacency of intra-prediction and inter-prediction blocks. In other words, even when using the technology described in Patent Document 1, adjacency of intra-prediction and inter-prediction blocks may occur depending on the degree of flatness of the image. In this case, the difference in image quality between the intra-prediction and inter-prediction blocks may be visually perceived as a degradation of image quality.
[0008] This invention was made in view of the above-mentioned problems, and aims to minimize the degradation of image quality caused by differences in predicted images when different predicted images are used for encoding within a single image. [Means for solving the problem]
[0009] To achieve the above objective, the encoding device of the present invention divides an input image into blocks and performs encoding processing for each block of the input image. The encoding device uses one of the following prediction modes to perform encoding processing for each block: an inter-prediction mode that generates an inter-prediction image by performing inter-prediction on a block and encodes the difference between the block and the inter-prediction image; an intra-prediction mode that generates an intra-prediction image by performing intra-prediction on a block and encodes the difference between the block and the intra-prediction image; and a multi-reference prediction mode that generates a multi-prediction image by combining the inter-prediction image and the intra-prediction image generated from the block and encodes the difference between the block and the multi-prediction image. The system comprises an encoding means for performing the encoding process, an output means for outputting the result of the encoding process as encoded data, and a calculation means for calculating a first encoding cost when the block is encoded in the inter-prediction mode and a second encoding cost when the block is encoded in the intra-prediction mode, respectively, using a predetermined function. The encoding means performs the encoding process on the block using the multi-reference prediction mode if the difference between the first encoding cost and the second encoding cost is less than or equal to a threshold, and performs the encoding process on the block using the prediction mode that is smaller than the first encoding cost and the second encoding cost if the difference between the first encoding cost and the second encoding cost is greater than the threshold. [Effects of the Invention]
[0010] According to the present invention, when different predicted images are used to encode an image within a single image, the degradation of image quality caused by the difference in the predicted images can be made less noticeable. [Brief explanation of the drawing]
[0011] [Figure 1] A block diagram showing the configuration of an image encoding device in an embodiment of the present invention. [Figure 2] A schematic diagram showing the types (mode numbers) of intra-prediction modes. [Figure 3]A flowchart illustrating the operation of the motion compensation unit in the first embodiment. [Figure 4] A flowchart illustrating the method for determining the weight coefficients of the multi-reference prediction unit in the first embodiment. [Figure 5] A block diagram showing an example of the internal configuration of the feature detection unit in the first embodiment. [Figure 6] A flowchart illustrating the method for determining the prediction mode in the prediction mode selection unit in the first embodiment. [Figure 7] This figure shows an example of a gamma curve during HDR recording and display. [Figure 8] A block diagram showing an example of the internal configuration of the feature detection unit in the second embodiment. [Figure 9] A flowchart illustrating the method for determining the prediction mode in the prediction mode selection unit of the second embodiment. [Figure 10] A flowchart showing a method for determining the prediction mode in the prediction mode selection unit in a modified example of the first embodiment. [Modes for carrying out the invention]
[0012] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.
[0013] [First Embodiment] Figure 1 is a block diagram showing the configuration of the image coding device 100 that performs predictive coding in this embodiment. The image coding device 100 is located, for example, within an image processing engine housed in an imaging device. The frame buffer 101 stores the input image to be encoded, and outputs image data including pixels of the required block size to the adder 102, the intra prediction unit 112, the motion compensation unit 114, and the feature detection unit 116.
[0014] The adder 102 generates difference image data between the image data output from the frame buffer 101 and the predicted image data output from the prediction mode selection unit 117 described later, and outputs it to the orthogonal transformation unit 103. The orthogonal transformation unit 103 performs an orthogonal transformation such as a discrete cosine transform (DCT) or a discrete sine transform (DST) on the difference image data, generates transformation coefficients, and outputs them to the quantization unit 104. The quantization unit 104 quantizes the transformation coefficients output from the orthogonal transformation unit 103 according to the quantization step size (or quantization parameter) output by the quantization control unit 106, and outputs them to the entropy encoding unit 105 and the inverse quantization unit 107.
[0015] The entropy encoding unit 105 performs variable length encoding on the quantized transformation coefficients, motion vectors, parameters necessary for decoding processing, etc., formats them in a predetermined format, and outputs them as an encoded stream. Also, the entropy encoding unit 105 calculates the amount of generated codes for each block and outputs it to the quantization control unit 106. The quantization control unit 106 determines the quantization step size and the quantization parameter Qp for each block from the amount of generated codes output from the entropy encoding unit 105 and the preset target amount of codes, and outputs them to the quantization unit 104 and the prediction mode selection unit 117.
[0016] The inverse quantization unit 107 performs inverse quantization on the quantized transformation coefficients output from the quantization unit 104, generates transformation coefficients, and outputs them to the inverse orthogonal transformation unit 108. The inverse orthogonal transformation unit 108 applies the inverse transformation of the orthogonal transformation applied by the orthogonal transformation unit 103 to the transformation coefficients output from the inverse quantization unit 107, generates difference image data, and outputs it to the adder 109. The adder 109 adds the difference image data generated by the inverse orthogonal transform unit 108 and the predicted image data output from the prediction mode selection unit 117 (described later) to generate locally decoded reference image data, which is then output to the reference frame buffer 110 and the loop filter 111.
[0017] The loop filter 111 includes a deblocking filter and an adaptive loop filter, and removes block noise from the reference image data output from the adder 109, outputting it to the reference frame buffer 110. However, as mentioned above, the reference image data output from the adder 109 can be stored in the reference frame buffer 110 without going through the loop filter 111. In other words, the filtering process by the loop filter 111 can be omitted. The reference frame buffer 110 stores the supplied reference image data and outputs the stored reference image data to the intra prediction unit 112 and the motion compensation unit 114 at predetermined timings.
[0018] The intra prediction unit 112 performs intra prediction using the image data of the block to be encoded output from the frame buffer 101 and the reference image data of the area surrounding the block to be encoded read from the reference frame buffer.
[0019] Here, with reference to Figure 2, we will explain the intra-prediction modes that can be selected by the intra-prediction unit 112. Figure 2 shows the types of intra-prediction modes (mode numbers). The intra-prediction unit 112 can select an intra-prediction mode from planar prediction (mode number 0), DC prediction (mode number 1), and 65 types of directional prediction (mode numbers 2 to 66). Furthermore, the intra-prediction unit 112 selects the intra-prediction mode with the smallest cost value among the intra-prediction modes, based on a cost function that indicates coding efficiency. Various functions can be considered as the cost function, but in this embodiment, the following equation (1) is used as a representative example.
[0020] IntraCost=SATD+Qp×Mdcost (1) In equation (1), IntraCost is a cost value indicating the coding efficiency in intra-prediction mode, SATD is the sum of absolute differences obtained by subtracting the input image data to be coded from the reference image data and then performing an Adamard transform on the difference data. Qp is the quantization parameter used in the quantization unit 104, and Mdcost is a cost value corresponding to the code amount in intra-prediction mode.
[0021] The intra prediction unit 112 generates a prediction image for intra prediction using the intra prediction mode selected using equation (1). The intra prediction unit 112 also outputs the selected intra prediction mode, the cost value when the selected intra prediction mode is applied, and the prediction image data (intra prediction image) generated by intra prediction to the multi-reference prediction unit 115 and the prediction mode selection unit 117, respectively.
[0022] The motion information memory 113 is a memory for temporarily holding motion vectors output by the motion compensation unit 114, which will be described later. The motion information memory 113 holds motion vectors of blocks located around the block to be encoded, and can also hold motion vectors for each block of a frame that has already been encoded. The motion vector information held in the motion information memory 113 is referenced by the motion compensation unit 114, which will be described later. The motion compensation unit 114 selects the optimal mode from two modes: an inter-mode that searches for a new motion vector, and a merge mode that selects the optimal motion vector from among the motion vector candidates of already encoded blocks.
[0023] Here, we will explain the operation of the motion compensation unit 114 using Figure 3. In S301, the motion compensation unit 114 searches for the optimal motion vector in intermode using the image data of the encoding target block output from the frame buffer 101 and the reference image data read from the reference frame buffer 110. More specifically, the motion compensation unit 114 searches for the motion vector with the smallest cost value according to the cost function that indicates the encoding efficiency. Various functions can be considered as the cost function, but in this embodiment, equation (2) is used as a representative example.
[0024] InterCost=SATD+Qp×Mvcost (2) In equation (2), InterCost is a cost value indicating the coding efficiency in intermode. SATD is obtained by subtracting the input image data to be coded from the reference image data after the motion vector has been reflected, and then performing an Adamard transform on the difference data, and calculating the sum of the absolute differences. Qp is the quantization parameter used in the quantization unit 104, and Mvcost is a cost value corresponding to the code amount of the motion vector. The motion compensation unit 114 temporarily stores information about the motion vector determined using equation (2) and the cost value obtained when the motion vector is applied.
[0025] In S302, the motion compensation unit 114 selects the optimal motion vector in merge mode from among multiple motion vector candidates already held in the motion information memory 113. More specifically, the motion compensation unit 114 applies equation (2) to each of the multiple motion vector candidates and selects the motion vector with the smallest cost value. The motion compensation unit 114 also calculates an index value (merge index) that represents the selected motion vector candidate. The motion compensation unit 114 temporarily stores the merge index and the cost value obtained when applying the motion vector indicated by the merge index.
[0026] In S303, the motion compensation unit 114 compares the cost value in intermode calculated in S301 with the cost value in merge mode calculated in S302 and selects the mode that yields the smallest cost value.
[0027] In S304, the motion compensation unit 114 generates predicted image data for interpretation using the motion vector in the selected mode and the reference image data read from the reference frame buffer 110. In the case of merge mode, the motion vector can be determined by referring to the merge index calculated in S302.
[0028] The motion compensation unit 114 outputs a flag indicating whether or not a merge mode has been selected (merge flag), information on the motion vector or merge index, the cost value in the selected mode, and the generated prediction image data for inter-prediction (inter-prediction image) to the multi-reference prediction unit 115 and the prediction mode selection unit 117, respectively.
[0029] The multi-reference prediction unit 115 combines the prediction image data for intra-prediction output from the intra-prediction unit 112 and the prediction image data for inter-prediction output from the motion compensation unit 114 to generate new prediction image data.
[0030] One method for combining prediction image data for intra-prediction and inter-prediction is described in Non-Patent Document 2 as a technique called Combined Inter and Intra Prediction (CIIP), and the method for combining prediction image data for intra-prediction and inter-prediction in this embodiment is also based on CIIP. On the other hand, Non-Patent Document 2 describes the application conditions of CIIP, such as the fact that the prediction image for intra-prediction is one to which PLANAR prediction is applied, and the prediction image for inter-prediction is one to which merge mode is applied. However, the method for combining prediction image data for intra-prediction and inter-prediction in this embodiment is not necessarily limited to these conditions, and for example, it is possible to apply directional prediction, etc., to the prediction image data for intra-prediction.
[0031] CIIP blends intra-prediction and inter-prediction images pixel by pixel, thereby further reducing the difference in predicted images between blocks. Based on this, in this embodiment, CIIP is applied to blocks that are likely to be adjacent to intra-prediction and inter-prediction blocks, and that are in areas where differences in image quality between blocks are easily noticeable, such as flat areas of the image. This reduces image quality degradation caused by differences in predicted images between adjacent blocks.
[0032] The following describes in detail the synthesis method used by the multi-reference prediction unit 115 to combine prediction image data for intra-prediction and prediction image data for inter-prediction. The multi-reference prediction unit 115 combines the prediction image data for intra-prediction and inter-prediction pixel by pixel using the following equation (3).
[0033] Pciip=((4-wt)×Pinter+wt×Pintra+2) / 4 …(3) In equation (3), Pintra is the pixel value of the predicted image data for intra-prediction, Pinter is the pixel value of the predicted image data for inter-prediction, and Pciip is the pixel value of the combined predicted image data. wt is a weighting coefficient that indicates the ratio of the combined pixel values of intra-prediction and inter-prediction, and can take integer values from 1 to 3.
[0034] Here, we will explain how to determine the weight coefficient wt of the multi-reference prediction unit 115 using Figure 4. In S401, the multi-reference prediction unit 115 initializes isIntraTop and isIntraLeft to 0, and the weight coefficient wt to 1. isIntraTop is a flag indicating whether the prediction mode of the encoded block adjacent to the target block above was intra prediction. isIntraLeft is a flag indicating whether the prediction mode of the encoded block adjacent to the target block to the left was intra prediction. The prediction modes of these encoded blocks are held by the prediction mode selection unit 117, as described later, and the multi-reference prediction unit 115 can refer to this information as appropriate.
[0035] In S402, the multi-reference prediction unit 115 determines whether the prediction mode of the encoded block adjacent to the encoded block in the upward direction was intra prediction. If the prediction mode of the encoded block adjacent to the upward direction was intra prediction (Yes in S402), the multi-reference prediction unit 115 sets isIntraTop to 1 in S403. If the prediction mode of the encoded block adjacent to the upward direction was not intra prediction (No in S402), the multi-reference prediction unit 115 proceeds to S404.
[0036] In S404, the multi-reference prediction unit 115 determines whether the prediction mode of the encoded block adjacent to the left of the block to be encoded was intra prediction. If the prediction mode of the encoded block adjacent to the left was intra prediction (Yes in S404), the multi-reference prediction unit 115 sets isIntraLeft to 1 in S405. If the prediction mode of the encoded block adjacent to the left was not intra prediction (No in S404), the multi-reference prediction unit 115 proceeds to S406.
[0037] In S406-S410, the multi-reference prediction unit 115 determines the value of the weight coefficient wt according to the sum of isIntraTop and isIntraLeft. If the sum of isIntraTop and isIntraLeft is 2 (Yes in S406), the unit proceeds to S407 and sets the weight coefficient wt to 3. If the sum of isIntraTop and isIntraLeft is 1 (No in S406 and Yes in S408), the unit proceeds to S409 and sets the weight coefficient wt to 2. If the sum of isIntraTop and isIntraLeft is 0 (No in S406 and No in S408), the unit proceeds to S410 and sets the weight coefficient wt to 1.
[0038] The processing in S406 to S410 means that the ratio of the pixel values of the predicted image data for intra-prediction and the predicted image data for inter-prediction during the synthesis of the predicted image is adaptively changed according to the prediction mode of the surrounding blocks of the block to be encoded. As a result, when multi-reference prediction is selected as the prediction mode in the prediction mode selection unit 117 described later, it is possible to reduce the image quality degradation caused by the difference in predicted images between the block to be encoded and the surrounding blocks.
[0039] Furthermore, the method for combining the prediction image data for intra-prediction and the prediction image data for inter-prediction is not limited to the example shown in equation (3) and Figure 4. For example, the multi-reference prediction unit 115 can also combine them using the following equation (4). Pciip=((8-wt)×Pinter+wt×Pintra+4) / 8 …(4) Another example is that the weight coefficient wt can be adaptively changed for each pixel according to its pixel position within the block to be encoded. Furthermore, the weight coefficient wt can also be adaptively changed according to the intra-prediction mode (DC prediction, PLANAR prediction, etc.) output from the intra-prediction unit 112, the block size of the block to be encoded, etc.
[0040] The multi-reference prediction unit 115 uses the predicted image data (multi-prediction image) synthesized using equation (3) to calculate a cost value using a cost function that indicates the coding efficiency. Various functions can be considered as the cost function, but in this embodiment, equation (5) is used as a representative example.
[0041] CiipCost=SATD+Qp×Mpcost (5) In equation (5), CiipCost is a cost value indicating the coding efficiency in multi-reference prediction. SATD is obtained by subtracting the input image data to be coded from the synthesized predicted image data, and then applying a Hadamard transform to the difference data, and calculating the sum of the absolute differences. Qp is the quantization parameter used in the quantization unit 104, and Mpcost is a cost value corresponding to the coding amount of the parameters necessary for performing multi-reference prediction. Parameters necessary for performing multi-reference prediction include the intra-prediction mode output from the intra-prediction unit 112, as well as the merge flag, merge index, or motion vector output from the motion compensation unit 114.
[0042] The multi-reference prediction unit 115 outputs the cost value calculated by equation (5) and the synthesized predicted image data to the prediction mode selection unit 117.
[0043] The feature detection unit 116 generates a flat region flag indicating whether or not the block to be encoded belongs to a flat region of the image, and outputs it to the prediction mode selection unit 117. Here, the method of generating the flat region flag by the feature detection unit 116 will be explained using Figure 5.
[0044] Figure 5 is a block diagram showing an example of the internal configuration of the feature detection unit 116. The flatness determination unit 501 acquires the input image of the block to be encoded and generates a flag (flatness flag) indicating whether or not the block to be encoded is flat. Various methods can be considered for determining whether or not something is flat, but in this embodiment, the variance of the pixel values of the block to be encoded is used as the flatness, and if the flatness is lower than a predetermined threshold, it is determined to be flat. In this embodiment, the flag value is set to "1" when it is determined to be flat, and to "0" when it is determined not to be flat.
[0045] The flatness flag buffer 502 temporarily holds the flatness flags output from the flatness determination unit 501. The flatness flag buffer 502 also holds the flatness flags of multiple encoded blocks, and the flatness region determination unit 503, described later, can refer to this information as appropriate. In this embodiment, the flatness flag buffer 502 holds the flatness flags of encoded blocks adjacent to the encoded block in the upward, diagonally upward-left, and left directions, respectively.
[0046] The flat region determination unit 503 determines whether a block to be encoded belongs to a flat region of the image based on the flatness flag of the block to be encoded held in the flatness flag buffer 502 and the flatness flags of multiple encoded blocks. Various methods can be considered for determining whether or not a block belongs to a flat region, but in this embodiment, a block to be encoded is determined to belong to a flat region if the flatness flag of the block to be encoded is "1", and the flatness flag values of the encoded blocks adjacent to the block in the upward, diagonally upper-left, and left directions are all "1". In this embodiment, the flag value is set to "1" when it is determined to belong to a flat region, and to "0" when it is determined not to belong to a flat region.
[0047] Note that the method for determining whether a block to be encoded belongs to a flat region is not limited to the example described with reference to Figure 5. For example, the feature detection unit 116 may simply determine that a block belongs to a flat region if its flatness is lower than a predetermined threshold. Alternatively, the feature detection unit 116 can determine the flatness flag for each block in a frame before starting the encoding process for that frame, and store those flatness flags in the flatness flag buffer 502. This allows the flat region determination unit 503 to improve the accuracy of determining whether a block to be encoded belongs to a flat region by referring not only to the flatness flags of blocks adjacent to the block to be encoded, but also to the flatness flags of blocks located further away.
[0048] The prediction mode selection unit 117 selects one of the following prediction modes: intra prediction, inter prediction, or multi-reference prediction. Here, the method for determining the prediction mode of the prediction mode selection unit 117 will be explained using Figure 6. In S601, the prediction mode selection unit 117 determines whether the flat region flag output from the feature detection unit 116 is 0, and / or whether the quantization step size Qstep of the target block to be encoded, obtained from the quantization control unit 106, is smaller than the threshold Thqstep.
[0049] In the following steps S601 through S608, the prediction mode selection unit 117 prioritizes coding efficiency and operates to always select the prediction mode that minimizes the cost value. On the other hand, in steps S609 through S613, the prediction mode selection unit 117 prioritizes reducing image quality degradation and does not always select the prediction mode that minimizes the cost value, but operates in a way that makes image quality degradation caused by differences in predicted images between blocks less noticeable. In other words, the determination expression in S601 means that the prediction mode selection unit 117 chooses whether to prioritize coding efficiency or reducing image quality degradation.
[0050] In this embodiment, if the region to which the block to be encoded belongs is not flat, and / or the quantization step size is small, it is determined that the degree of image quality degradation due to differences in predicted images between blocks is small, and processing S602 to S608 is performed with an emphasis on encoding efficiency. Conversely, if the region to which the block to be encoded belongs is flat and the quantization step size is large, processing S609 to S613 is performed with an emphasis on reducing image quality degradation. Note that the determination method in S601 is not limited to the example shown here. For example, the prediction mode selection unit 117 can also make a determination using either the flat region flag or the quantization step size.
[0051] In S601, if the flat region flag is 0 and / or the quantization step size Qstep is smaller than the threshold Thqstep, the prediction mode selection unit 117 proceeds to S602. In S602, the prediction mode selection unit 117 compares the cost value IntraCost output from the intra prediction unit 112 with the cost value InterCost output from the motion compensation unit 114.
[0052] If InterCost is greater than IntraCost (Yes in S602), the prediction mode selection unit 117 compares IntraCost with the cost value CiipCost output from the multi-reference prediction unit 115 (S603). If IntraCost is greater than CiipCost (Yes in S603), the prediction mode selection unit 117 selects multi-reference prediction as the prediction mode (S604). If IntraCost is less than or equal to CiipCost (No in S603), the prediction mode selection unit 117 selects intra-prediction as the prediction mode (S605).
[0053] Furthermore, in S602, if InterCost is less than or equal to IntraCost (No in S602), the prediction mode selection unit 117 compares InterCost and CiipCost (S606). If InterCost is greater than CiipCost (Yes in S606), the prediction mode selection unit 117 selects multi-reference prediction as the prediction mode (S607). If InterCost is less than or equal to CiipCost (No in S606), the prediction mode selection unit 117 selects inter-prediction as the prediction mode (S608).
[0054] On the other hand, in S601, if the flat region flag output from the feature detection unit 116 is 1 and the quantization step size Qstep is greater than or equal to the threshold Thqstep, the prediction mode selection unit 117 proceeds to S609.
[0055] In S609, the prediction mode selection unit 117 compares the values of InterCost and IntraCost+α. α is an arbitrarily set correction value for the intra-prediction cost value.
[0056] Generally, in regions where the cost values of intra-prediction and inter-prediction are similar, intra-prediction tends to result in more noticeable image quality degradation. Therefore, in this embodiment, when the block to be encoded belongs to a flat region, the cost value of intra-prediction is corrected so that inter-prediction is preferentially selected. As a result, the cases in which blocks adjacent to intra-prediction and inter-prediction occur are reduced.
[0057] In this embodiment, the cost value is corrected by adding α as described above, but it is also possible to correct it by subtracting, multiplying, dividing, etc., an arbitrary constant from the cost value. Furthermore, the method of correcting the cost value is not limited to using α as described above; any method that prioritizes the selection of interpretation may be adopted.
[0058] If InterCost is greater than IntraCost+α (Yes in S609), the prediction mode selection unit 117 selects intra-prediction as the prediction mode (S610).
[0059] If InterCost is less than or equal to IntraCost + α (No in S609), the prediction mode selection unit 117 determines whether the absolute value of the difference between InterCost and IntraCost is less than or equal to the constant c (S611). The determination formula in S611 will now be explained in detail.
[0060] As described above, the determination in S609 is intended to prioritize the selection of inter-prediction. On the other hand, depending on the flatness of the image, block adjacency between intra-prediction and inter-prediction is likely to occur, and there are still regions where the determination result by S609 switches frequently. In these regions where the determination result by S609 switches frequently, it is assumed that the values of InterCost and IntraCost+α in the determination formula of S609 are close. Based on this, in this embodiment, regions where the determination result by S609 switches frequently are detected by the following formula (6).
[0061] |InterCost-(IntraCost+α)|≦β …(6) β is a constant used to detect the closeness of the values of InterCost and IntraCost+α. Here, if α is a positive constant, then equation (6) can be rearranged to obtain equation (7).
[0062] |InterCost-IntraCost|≦β+α=c …(7) In other words, in this embodiment, if the absolute value of the difference between the InterCost and IntraCost of the block to be encoded in S611 is less than or equal to a constant c, it is determined that the block is in a region where block adjacency between intra-prediction and inter-prediction is likely to occur.
[0063] The constant c can be set arbitrarily and may be adaptively changed using the target bitrate of the image encoding device, the value of the quantization parameter Qp used in the quantization unit 104, or image features, etc. For example, the value of the constant c can be changed for each block according to the flatness calculated by the flatness determination unit 501.
[0064] If the absolute value of the difference between InterCost and IntraCost is less than or equal to c (S611 Yes), the prediction mode selection unit 117 selects multi-reference prediction as the prediction mode (S612). In other words, in this embodiment, multi-reference prediction is applied to blocks that are located in regions where block adjacency between intra-prediction and inter-prediction is likely to occur. When multi-reference prediction is applied, the predicted images of intra-prediction and inter-prediction are blended, and the difference between predicted images between blocks becomes smaller, thereby reducing image quality degradation caused by differences in predicted images between blocks.
[0065] In multi-reference prediction, it is necessary to encode parameters related to both intra-prediction and inter-prediction prediction modes (such as the intra-prediction mode and motion vectors), and there is a risk that the amount of encoding related to the parameters will be larger than that of intra-prediction or inter-prediction. Therefore, the prediction mode selection unit 117 may add whether or not the value of the merge flag output from the motion compensation unit 114 is 1 when making the determination in S611. This eliminates the need to encode parameters related to the motion vector when multi-reference prediction is selected, thereby suppressing the increase in the amount of encoding related to the parameters.
[0066] If the absolute value of the difference between InterCost and IntraCost is not less than or equal to the constant c (No in S611), the prediction mode selection unit 117 selects interprediction as the prediction mode (S613).
[0067] A predicted image is generated according to the prediction mode selected as described above. Then, the difference between the image data of the block to be encoded and the predicted image is encoded, and the encoding result is output as an encoded stream (encoded data).
[0068] In the example shown in Figure 6, when the image is flat and the quantization step size is large, regions where block adjacency between intra-prediction and inter-prediction is likely to occur are detected, and a prediction mode prioritizing image quality degradation reduction is applied to these regions. On the other hand, depending on the target bitrate of the image encoding device, there are cases where block boundaries between intra-prediction and inter-prediction are more noticeable, regardless of the degree of image flatness. In such cases, the prediction mode selection unit 117 does not perform the determination in S601, but always detects regions where block adjacency between intra-prediction and inter-prediction is likely to occur, and it is also possible to apply multi-reference prediction to the blocks in these regions.
[0069] Alternatively, processing from S609 onwards may be performed for all blocks, regardless of whether the block boundary between intra-prediction and inter-prediction is particularly noticeable.
[0070] As described above, according to the first embodiment, when encoding using different predicted images within the same image, the degradation of image quality caused by differences in the predicted images can be made less noticeable. Furthermore, when the region to which the encoded block belongs is flat and the quantization step size is large, making the block boundary between intra-prediction and inter-prediction noticeable, the prediction mode selection unit 117 operates with an emphasis on reducing image quality degradation. On the other hand, when the block boundary between intra-prediction and inter-prediction is not noticeable, the prediction mode selection unit 117 operates with an emphasis on encoding efficiency. This makes it possible to reduce image quality degradation caused by differences in predicted images between blocks more than before, without compromising encoding efficiency.
[0071] [Differentiation] Next, the method for generating encoded data in the modified example will be explained with reference to Figure 10.
[0072] Referring to Figure 3, as described above, the motion compensation unit 114 calculates the cost value when the inter-prediction mode is applied to the block to be encoded (S1001). Also, referring to Figure 2, as described above, the intra-prediction unit 112 calculates the cost value when the intra-prediction mode is applied to the block to be encoded (S1002).
[0073] Then, similar to S611, the prediction mode selection unit 117 determines whether the absolute value of the difference between the cost value of the inter-prediction mode calculated in S1001 and the cost value of the intra-prediction mode calculated in S1002 is less than or equal to the constant c (S1003). If the result of the determination is that the absolute value of the difference is less than or equal to the constant c (Yes in S1003), the prediction mode selection unit 117 selects the multi-reference prediction mode (S1004).
[0074] On the other hand, if the absolute value of the difference is greater than the constant c as a result of the determination (No in S1003), the prediction mode selection unit 117 calculates the cost value when the multi-reference prediction mode is applied to the block to be encoded (S1005). The prediction mode selection unit 117 compares the cost values of the inter-prediction mode, intra-prediction mode, and multi-reference prediction mode calculated in S1001, S1002, and S1005, and if the cost value of the multi-reference prediction mode is the smallest (Yes in S1006), it selects the multi-reference prediction mode (S1004).
[0075] If the cost value of the multi-reference prediction mode is not the minimum (No in S1006) and the cost value of the inter-prediction mode is the minimum (Yes in S1007), then the inter-prediction mode is selected (S1008). If the cost value of the multi-reference prediction mode is not the minimum (No in S1006) and the cost value of the inter-prediction mode is not the minimum (No in S1007), that is, if the cost value of the intra-prediction mode is the minimum, then the intra-prediction mode is selected (S1009).
[0076] Then, if the multi-reference prediction mode is selected in S1004, the multi-reference prediction unit 115 combines the prediction image data for intra-prediction output from the intra-prediction unit 112 and the prediction image data for inter-prediction output from the motion compensation unit 114 to generate prediction image data for multi-reference prediction (S1010). Alternatively, if the inter-prediction mode is selected in S1008, the motion compensation unit 114 generates prediction image data for inter-prediction, similar to S304 (S1011). Alternatively, if the intra-prediction mode is selected in S1009, the intra-prediction unit 112 generates prediction image data for intra-prediction (S1012). The prediction image data generated in S1011, S1011, or S1012 is output to the adder 102 by the prediction mode selection unit 117.
[0077] Then, the adder 102 generates difference image data between the block image data to be encoded output from the frame buffer and the predicted image data. Encoding processing is performed on the difference image data using each part as described above (S1013), and the encoding result is output as an encoded stream (S1014).
[0078] Before starting the process in S1001, similar to S601, the prediction mode selection unit 117 may determine whether the flat region flag output from the feature detection unit 116 is 0 and / or whether the quantization step size Qstep of the target block to be encoded, obtained from the quantization control unit 106, is smaller than the threshold Thqstep. If the flat region flag is 1 and / or the quantization step size Qstep is greater than or equal to the threshold Thqstep, the process from S1001 onwards may be executed. Alternatively, the process from S1001 onwards may be executed if the dynamic range of the input image is a high dynamic range of a predetermined value or higher.
[0079] [Second Embodiment] Next, a second embodiment of the present invention will be described. The latest video encoding schemes, such as HEVC and VVC, allow for the encoding of multi-bit HDR (High Dynamic Range) recordings (10-bit, 12-bit, etc.) or Log-recorded videos. Log-recorded videos are those captured using a gamma curve uniquely defined by the manufacturer of the digital camera, etc., to enable recording with a higher dynamic range, unlike the gamma curve defined in standards such as BT.709.
[0080] HDR or Log-recorded video (high dynamic range image) is converted and displayed using the gamma curve set at the time of output to the display. Figure 7(a) shows the relationship between the input brightness value (actual brightness at the time of shooting) and the output pixel value (pixel value after encoding) during HDR recording. The horizontal direction represents the input brightness value, and the vertical direction represents the output pixel value, with the relationship between the input and output values defined by the gamma curve 701. Where the input value is small, the curve is steep in the vertical direction, and a large number of pixel values are assigned to a given range of input values. On the other hand, where the input value is large, the curve becomes gentler, and fewer pixel values are assigned to the input value. In this way, by assigning output pixel values to input brightness values, high dynamic range recording is made possible.
[0081] Figure 7(b) shows the relationship between the input value (pixel value after encoding) and the output brightness value (brightness displayed on the screen) during HDR display. The horizontal direction represents the input pixel value, and the vertical direction represents the output brightness value that is actually displayed. The relationship between the input value and the output value is defined by gamma curve 702, which is the inverse relationship to gamma curve 701. Where the input value is small, the curve is gentle in the vertical direction, and the amount of change in displayed brightness relative to the input value is small. On the other hand, where the input value is large, the curve is steep, and the amount of change in displayed brightness relative to the input value is large. By assigning the output brightness value to the input pixel value in this way, HDR-recorded video can be displayed with the correct brightness.
[0082] Thus, compared to standard SDR (Standard Dynamic Range) recorded video, HDR-recorded or Log-recorded video exhibits a greater change in brightness displayed on the screen relative to the change in pixel values after encoding. In other words, the degradation of image quality after encoding is amplified and displayed more prominently on the screen. Furthermore, this tendency becomes more pronounced in areas with high brightness values (referred to as "high-brightness regions" or "highlight regions"), and the image quality degradation caused by differences in predicted images between blocks, which is the issue addressed in this invention, becomes more visually noticeable in such regions.
[0083] In the second embodiment, the target of encoding is a video image recorded in HDR or Log format (referred to as an "HDR image" for convenience), and the aim is to reduce the image quality degradation that occurs in the highlight areas of the HDR image. In the second embodiment, the differences from the first embodiment will be mainly explained, and explanations of parts that are the same as in the first embodiment will be omitted.
[0084] The overall configuration of the image encoding device in the second embodiment is the same as that shown in Figure 1, but the configuration of the feature detection unit 116 differs from that described with reference to Figure 5 in the first embodiment. Figure 8 is a block diagram showing an example of the internal configuration of the feature detection unit 116 in the second embodiment. Similar to the first embodiment, the feature detection unit 116 in the second embodiment includes a flatness determination unit 501, a flatness flag buffer 502, and a flatness region determination unit 503 for determining whether the block to be encoded belongs to a flat region of the image. Furthermore, the feature detection unit 116 includes a high-brightness determination unit 804, a high-brightness flag buffer 805, and a highlight region determination unit 806 for determining whether the block to be encoded belongs to a highlight region of the image. Note that the flatness determination unit 501, the flatness flag buffer 502, and the flatness region determination unit 503 operate in the same way as in the first embodiment, so their explanation is omitted here.
[0085] The high-brightness determination unit 804 acquires the input image of the block to be encoded and generates a flag (high-brightness flag) indicating whether or not the block to be encoded is high-brightness. Various methods can be considered for determining whether or not a block is high-brightness, but in this embodiment, the average value of the brightness values of the pixels of the block to be encoded is calculated, and if the calculated average brightness value is greater than a predetermined threshold, it is determined to be high-brightness. In this embodiment, the flag value is set to "1" when it is determined to be high-brightness, and to "0" when it is determined not to be high-brightness.
[0086] The high-brightness flag buffer 805 temporarily holds the high-brightness flag output from the high-brightness determination unit 804. The high-brightness flag buffer 805 also holds the high-brightness flags of multiple encoded blocks, and the highlight area determination unit 806, described later, can refer to this information as appropriate. In this embodiment, the high-brightness flag buffer 805 holds the high-brightness flags of encoded blocks adjacent to the encoded target block in the upward, diagonally upward-left, and left directions, respectively.
[0087] The highlight region determination unit 806 determines whether or not a block to be encoded belongs to the highlight region of an image based on the high-luminance flag of the block to be encoded held in the high-luminance flag buffer 805 and the high-luminance flags of multiple encoded blocks. There are various methods for determining whether or not a block belongs to the highlight region, but in this embodiment, a block to be encoded belongs to the highlight region if the high-luminance flag of the block to be encoded is "1", and the flag values of the high-luminance flags of the encoded blocks adjacent to the block above, diagonally to the upper left, and to the left are all "1". In this embodiment, the flag value is set to "1" when it is determined that the block belongs to the highlight region, and to "0" when it is determined that the block does not belong to the highlight region.
[0088] The method for determining whether a block to be encoded belongs to the highlight region is not limited to the example described with reference to Figure 8. For example, the feature detection unit 116 may determine that a block belongs to the highlight region simply if its average brightness value is greater than a predetermined threshold. Alternatively, the feature detection unit 116 can determine the high-brightness flags for each block in a frame before starting the encoding process for that frame, and store these high-brightness flags in the high-brightness flag buffer 805. This allows the highlight region determination unit 806 to improve the accuracy of determining whether a block to be encoded belongs to the highlight region by referring not only to the high-brightness flags of blocks adjacent to the block to be encoded, but also to the high-brightness flags of blocks located further away.
[0089] In this way, the feature detection unit 116 generates a flat region flag and a highlight region flag and outputs them to the prediction mode selection unit 117.
[0090] Figure 9 is a flowchart showing the method for determining the prediction mode of the prediction mode selection unit 117 in the second embodiment. Processes similar to those shown in Figure 6 are given the same reference numerals and their explanations are omitted. The processing in the second embodiment shown in Figure 9 is the same as the processing in the first embodiment shown in Figure 6, but with the addition of a process (S901) to determine whether the video to be encoded is an HDR-recorded or Log-recorded video and whether it belongs to the highlight region. Whether or not the video to be encoded is an HDR-recorded or Log-recorded video is notified to the image encoding device in advance from the control source of the image encoding device, such as a CPU (not shown).
[0091] In S901, if the video to be encoded is an HDR-recorded or Log-recorded video and the highlight region flag is 1, the prediction mode selection unit 117 proceeds to S601. On the other hand, in S901, if the video to be encoded is not an HDR-recorded or Log-recorded video, and / or the highlight region flag is not 1, the prediction mode selection unit 117 skips the processing in S601 and proceeds to S602.
[0092] In other words, if the video to be encoded is an HDR-recorded or Log-recorded video, and the blocks to be encoded belong to the highlight region and the flat region, and the quantization step size is large, the prediction mode selection unit 117 prioritizes image quality degradation and operates in a way that makes image quality degradation caused by differences in predicted images between blocks less noticeable (S609~S613).
[0093] On the other hand, in cases other than those described above, the prediction mode selection unit 117 prioritizes coding efficiency and operates to always select the prediction mode that minimizes the cost value (S602~S608).
[0094] As described above, according to the second embodiment, in addition to the flatness of the region to which the block to be encoded belongs and the quantization step size, the system considers whether the image to be encoded is an HDR-recorded or Log-recorded video, and whether the block to be encoded belongs to a highlight region, and selects whether to prioritize encoding efficiency or reducing image quality degradation. This makes it possible to reduce image quality degradation, which is particularly noticeable in highlight regions, more than in conventional methods, without compromising encoding efficiency when encoding HDR images.
[0095] <Other Embodiments> Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.
[0096] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]
[0097] 101...Frame buffer, 102...Adder, 103...Orthogonal transform unit, 104...Quantization unit, 105...Entropy coding unit, 106...Quantization control unit, 107...Inverse quantization unit, 108...Inverse orthogonal transform unit, 109...Adder, 110...Reference frame buffer, 111...Loop filter, 112...Intra prediction unit, 113...Motion information memory, 114...Motion compensation unit, 115...Multi-reference prediction unit, 116...Feature detection unit, 117...Prediction mode selection unit, 501...Flatness determination unit, 502...Flatness flag buffer, 503...Flatness region determination unit, 804...High brightness determination unit, 805...High brightness flag buffer, 806...Highlight region determination unit ?
Claims
1. An encoding device that divides an input image into blocks and performs an encoding process for each block of the input image, an inter prediction mode in which an inter prediction image is generated by performing inter prediction on a block, and a difference between the block and the inter prediction image is encoded; an intra prediction mode in which an intra prediction image is generated by performing intra prediction on a block, and a difference between the block and the intra prediction image is encoded; A multi-reference prediction mode in which a multi-prediction image is generated by synthesizing an inter-prediction image and an intra-prediction image generated from a block, and the difference between the block and the multi-prediction image is coded. An encoding means for executing an encoding process for each block using any one of the prediction modes; an output means for outputting a result of the encoding process as encoded data; a calculation means for calculating, by using a predetermined function, a first coding cost when a block is coded in the inter prediction mode and a second coding cost when the block is coded in the intra prediction mode; The encoding device is characterized in that, when the difference between the first encoding cost and the second encoding cost is equal to or less than a threshold, the encoding means performs an encoding process on the block using the multi-reference prediction mode, and, when the difference between the first encoding cost and the second encoding cost is greater than the threshold, performs an encoding process on the block using the prediction mode corresponding to the smaller of the first encoding cost and the second encoding cost.
2. The calculation means further calculates a third coding cost when the block is coded in the multi-reference prediction mode, using a predetermined function; The encoding device according to claim 1, characterized in that, when a difference between the first encoding cost and the second encoding cost is greater than a threshold, the encoding means performs an encoding process using a prediction mode corresponding to the smallest cost among the first encoding cost, the second encoding cost, and the third encoding cost.
3. 3. The encoding device according to claim 1, wherein the block to be encoded by said encoding means is included in a flat area or a high-luminance area.
4. 4. The encoding device according to claim 1, wherein the input image is a high dynamic range image.
5. 5. The encoding device according to claim 1, wherein a quantization step used in the encoding process is greater than a predetermined value.
6. The encoding device according to claim 1 , wherein the multi-prediction image is generated by weighting and synthesizing the inter-prediction image and the intra-prediction image.
7. 7. The encoding device according to claim 1, wherein the encoding cost is calculated based on a sum of absolute differences obtained by Hadamard transform of a difference obtained by subtracting the input image from a reference image decoded by adding the difference to the predicted image, a quantization parameter, and a code amount.
8. 1. An encoding method for dividing an input image into blocks and performing an encoding process for each block of the input image, comprising the steps of: an inter prediction mode in which an inter prediction image is generated by performing inter prediction on a block, and a difference between the block and the inter prediction image is encoded; an intra prediction mode in which an intra prediction image is generated by performing intra prediction on a block, and a difference between the block and the intra prediction image is encoded; A multi-reference prediction mode in which a multi-prediction image is generated by synthesizing an inter-prediction image and an intra-prediction image generated from a block, and the difference between the block and the multi-prediction image is coded. performing an encoding process for each block using any one of the prediction modes; outputting a result of the encoding process as encoded data; calculating, by using a predetermined function, a first coding cost when the block is coded in the inter prediction mode and a second coding cost when the block is coded in the intra prediction mode; An encoding method characterized in that, when a difference between the first encoding cost and the second encoding cost is equal to or less than a threshold, an encoding process is performed on the block using the multi-reference prediction mode, and, when a difference between the first encoding cost and the second encoding cost is greater than the threshold, an encoding process is performed using a prediction mode corresponding to the smaller of the first encoding cost and the second encoding cost.
9. A program for causing a computer to function as each of the means of the encoding device according to any one of claims 1 to 7.
10. A computer-readable storage medium storing the program according to claim 9.
