Encoding assistance information generation device and program

Abstract

To find encoding difficulty with high accuracy.SOLUTION: A coding auxiliary information generating device 1 includes a coding information extracting unit 11 that performs coding processing on an original image and extracts information on a quantization parameter and a motion vector used for coding for each coding block, and an encoding difficulty calculation unit 12 that calculates encoding difficulty for each encoding block position by weighted addition of first encoding difficulty based on the value of the quantization parameter and second encoding difficulty based on a magnitude and variance of the motion vector.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to an encoding assistance information generation device and program. [Background technology]

[0002] In encoding schemes such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding), motion compensation is performed using orthogonal transforms such as DCT (Discrete Cosine Transform) and motion vectors. For more details on the H.265 / HEVC technology, see Non-Patent Document 1, for example. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Supervised by Sakae Okubo, "Impress Standard Textbook Series H.265 / HEVC Textbook," Impress Japan Co., Ltd., October 21, 2013. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] Generally, to quantitatively detect the encoding difficulty within a frame image, one can consider methods such as examining the PSNR (Peak Signal to Noise Ratio) of the encoded image for each small image region, relative to the image before encoding. However, a challenge has been that encoding quality degradation and significant block distortion, which occur in encoding schemes such as H.265 / HEVC and H.266 / VVC, do not necessarily occur in image regions with low PSNR. A method to solve this problem.

[0005] In view of these circumstances, the object of the present invention is to provide an encoding assistance information generation device that can determine the encoding difficulty with high accuracy.

Means for Solving the Problem

[0008] above To solve the above problems, an encoding auxiliary information generation device according to an embodiment performs an encoding process on an original image, and for each encoding block, an encoding information extraction unit that extracts information on quantization parameters and motion vectors used in the encoding, and while maintaining phase information for the original image Time direction and performs frequency band decomposition in the spatial direction The spatiotemporal power spectrum elemental components for each spatiotemporal frequency band and a frequency decomposition unit that generates frequency band components, a first encoding difficulty based on the value of the quantization parameter, a second encoding difficulty based on the magnitude and variance of the motion vector, and the Spacetime a third encoding difficulty based on the magnitude in the high frequency band of the frequency band component, and an encoding difficulty calculation unit that calculates the encoding difficulty for each pixel position by weighted addition.

[0010] Furthermore, in an encoding auxiliary information generation device according to an embodiment, the encoding difficulty calculation unit may calculate the third encoding difficulty by The aforementioned spatiotemporal frequency band component weighted addition with the encoding difficulty in the Furthermore, based on the size in the spatial high-frequency band. temporal low frequency band D PL and , the spatiotemporal frequency band components the encoding difficulty in the Furthermore, based on the size of the spatial high-frequency bandwidth temporal high frequency band D PH and. Furthermore, in a coding assistance information generation device according to one embodiment, the coding difficulty calculation unit determines the coding difficulty D PH The weight of the encoding difficulty D PL It can be made stronger than the weight of [the specified weight].

[0011] Furthermore, in an encoding auxiliary information generation device according to an embodiment, the encoding difficulty calculation unit may detect a region indicating pixel positions where the encoding difficulty exceeds a threshold as an encoding difficult region.

[0012] Also, a program according to an embodiment causes a computer to function as the above encoding auxiliary information generation device.

Advantages of the Invention

[0013] According to the present invention, the encoding difficulty can be determined with high accuracy, which can be used to suppress encoding quality degradation and encoding quality failure. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram showing an example configuration of an encoding assistance information generation device according to the first embodiment. [Figure 2] This is a block diagram showing an example configuration of an encoding assistance information generation device according to the second embodiment. [Figure 3] This figure shows the results of decomposing an 8K resolution original image into third-order wavelet packets in the spatial direction. [Figure 4] This figure shows the results of decomposing an 8K resolution original image into third-order wavelet packets in the spatial direction and first-order wavelet packets in the temporal direction. [Modes for carrying out the invention]

[0015] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0016] (First Embodiment) Figure 1 is a block diagram showing an example configuration of an encoding assistance information generation device 1 according to a first embodiment of the present invention. The encoding assistance information generation device 1 shown in Figure 1 comprises an encoding information extraction unit 11 and an encoding difficulty calculation unit 12.

[0017] The encoding assistance information generation device 1 takes a source image as input and calculates the encoding difficulty of the source image (input image). The encoding assistance information generation device 1 can be used, for example, in the process of generating assistance information to suppress image quality degradation and image quality failure in image encoding.

[0018] The encoding information extraction unit 11 performs encoding processing on the original image and extracts the encoding information used for encoding for each encoding block. The encoding information extraction unit 11 then outputs the encoding information to the encoding difficulty calculation unit 12. In this example, the encoding processing is performed using VVC / H.266, and the encoding information extracted includes the division information of the encoding block (i.e., the coding unit CU), the quantization parameter QP (Quantization Parameter) for each CU, and the motion vector information. The division information of the CU is the size information and position information of the CU.

[0019] The coding difficulty calculation unit 12 receives coding information from the coding information extraction unit 11 and calculates the coding difficulty for each coding block position from the value of the quantization parameter QP and the magnitude and variance of the motion vector. For example, as described below, the coding difficulty for each coding block position is calculated by weighting and adding a first coding difficulty based on the value of the quantization parameter QP and a second coding difficulty based on the magnitude and variance of the motion vector. The coding difficulty calculation unit 12 may also detect coding difficulty regions. The coding difficulty calculation unit 12 then outputs the coding difficulty region and coding difficulty to the outside as coding assistance information. Details of each calculation method are shown below.

[0020] <Encoding difficulty D for quantization parameters> Q > For the quantization parameter QP, a larger value indicates a higher coding difficulty. The coding difficulty D for QP at the CU position (CUx, CUy) is determined. Q (CUx,CUy) can be calculated, for example, by the following equation (1), where q is the QP value of the CU position (CUx,CUy).

[0021]

number

[0022] <Coding difficulty D for motion vectors> V > Regarding the motion vector, it is determined that the larger its magnitude and variance, the more difficult it is to encode. The encoding difficulty D for the motion vector at the CU position (CUx, CUy) V (CUx, CUy) is calculated, for example, by the following formula (5), where v is the magnitude of the motion vector at the CU position (CUx, CUy) and σ is the variance of the motion vector.

[0023]

Number

[0024] <Total encoding difficulty D> The total encoding difficulty D(CUx, CUy) at the CU position (CUx, CUy) is calculated, for example, by the following formula (3) as the weighted sum of formulas (1) and (2). α and β in the formula are the respective weighting coefficients of formulas (1) and (2), which may be determined in advance or experimentally obtained according to the encoding / decoding device.

[0025]

Number

[0026] <Encoding difficult region R(CUx, CUy)> The encoding difficulty calculation unit 12 detects, as the encoding difficult region R(CUx, CUy), the region indicating the encoding block where the encoding difficulty D(CUx, CUy) exceeds the threshold T H as shown in the following formula (4). The threshold T H may be determined in advance or experimentally obtained.

[0027]

Number

[0028] The encoding difficulty calculation unit 12 performs all the processes of formulas (1) to (4) for all (CUx, CUy) and outputs the final encoding difficult region R(CUx, CUy) and the encoding difficulty D(CUx, CUy).

[0029] As described above, the coding assistance information generator 1 generates a first coding difficulty D based on the value of the quantization parameter. Q And, the second coding difficulty D based on the magnitude and variance of the motion vector. V By weighting and adding these together, the coding difficulty D for each coding block position can be calculated, making it possible to determine the coding difficulty with high accuracy. In addition, it is also possible to determine the coding difficulty region with high accuracy. In the coding difficulty region, coding quality degradation occurs due to coding quality deterioration and a large amount of block distortion, which may lead to a significant decrease in subjective image quality. However, by detecting the coding difficulty and coding difficulty region using the coding assistance information generation device 1, it is possible to help suppress coding quality degradation.

[0030] (Second embodiment) Next, a coding assistance information generation device according to the second embodiment will be described. Figure 2 is a block diagram showing an example configuration of the coding assistance information generation device 2 according to the second embodiment. The coding assistance information generation device 2 shown in Figure 2 comprises a coding information extraction unit 11, a coding difficulty calculation unit 12', and a frequency decomposition unit 13. The coding assistance information generation device 2 according to the second embodiment differs from the coding assistance information generation device 1 according to the first embodiment in that it further comprises a frequency decomposition unit 13 and replaces the coding difficulty calculation unit 12 with a coding difficulty calculation unit 12'.

[0031] The encoding information extraction unit 11, as in the first embodiment, performs encoding processing on the original image and extracts the encoding information used for encoding for each encoding block. The encoding information extraction unit 11 then outputs the encoding information to the encoding difficulty calculation unit 12.

[0032] The frequency decomposition unit 13 comprises a spatial frequency band decomposition unit 14 and a time frequency band decomposition unit 15. In this embodiment, the frequency decomposition unit 13 is described as comprising a time frequency band decomposition unit 15, but a configuration without a time frequency band decomposition unit 15 is also possible.

[0033] The spatial frequency band decomposition unit 14 performs spatial frequency band decomposition of the original image while retaining phase information (i.e., without decimation), and generates elemental components for each spatial frequency band (hereinafter referred to as "spatial frequency band components"). Each element exhibits a power spectrum. The spatial frequency band decomposition unit 14 then outputs the spatial frequency band components to the temporal frequency band decomposition unit 15. In this embodiment, wavelet decomposition is performed as frequency band decomposition. The wavelet filter and the number of spatial decomposition layers can be arbitrarily set by the user.

[0034] Figure 3 shows the result of decomposing an 8K resolution original image into 3rd-order wavelet packets in the spatial direction, resulting in spatial frequency bands of 1K × 0.5K. To perform high-precision multi-resolution decomposition of the original image, it is desirable to use wavelet filters with linear phase properties, relatively long tap lengths, and steep cutoff characteristics (e.g., CDF (Cohen-Daubechies-Feauveau) 9 / 7, Biorthogonal (6,8), etc.). Furthermore, the power between each band is assumed to satisfy Parseval's identity.

[0035] Furthermore, in this embodiment, the spatial frequency band decomposition unit 14 performs spatial frequency band decomposition using wavelet packet decomposition without decimation. Therefore, although the size of each frequency band in Figure 2 is 1K × 0.5K, the number of spatial elements within each frequency band is 8K × 4K.

[0036] The time-frequency band decomposition unit 15 performs time-direction frequency band decomposition on the spatial frequency band components generated by the spatial frequency band decomposition unit 14, generating power spectral element components for each spatiotemporal frequency band (hereinafter referred to as "spatiotemporal frequency band components"). The time-frequency band decomposition unit 15 then outputs the spatiotemporal frequency band components to the coding difficulty calculation unit 12. In this embodiment, wavelet decomposition is performed as frequency band decomposition. The wavelet filter and the number of time-decomposition levels can be arbitrarily set by the user.

[0037] Figure 4 shows the results of decomposing an 8K resolution source image into third-order wavelet packets in the spatial direction and first-order wavelet packets in the temporal direction. When a spatial frequency band component of 60 frames / second is decomposed into first-order wavelet packets, it is decomposed into a temporal frequency band of 1 to 30 frames / second and 31 to 60 frames / second. Here, the wavelet filter is the same as that of the spatial frequency band decomposition unit 14. Also, when wavelet packet decomposition is performed without decimation, the number of elements in the temporal direction within each frequency band is 60. The number of elements in the spatiotemporal direction within each frequency band is 8K × 4K × 60.

[0038] The coding difficulty calculation unit 12' receives coding information from the coding information extraction unit 11 and spatiotemporal frequency band components from the time-frequency band decomposition unit 15. It calculates the coding difficulty for each pixel position (spatial phase position) from the value of the quantization parameter QP, the magnitude and variance of the motion vector, and the magnitude of the high-frequency band of the spatiotemporal frequency band component. For example, as described below, the coding difficulty for each pixel position is calculated by weighting and adding a first coding difficulty based on the value of the quantization parameter QP, a second coding difficulty based on the magnitude and variance of the motion vector, and a third coding difficulty based on the magnitude of the high-frequency band of the frequency band component. The coding difficulty calculation unit 12' may also detect coding difficulty regions. The coding difficulty calculation unit 12' then outputs the coding difficulty region and coding difficulty to the outside as coding assistance information. Details of each calculation method are shown below.

[0039] <Encoding difficulty D for quantization parameters> Q > For the quantization parameter QP, a larger value indicates greater coding difficulty. The coding difficulty D is determined by the QP value at the spatial phase position (x,y). Q (x,y) can be calculated, for example, by the following equation (5), where q is the QP value of the CU to which (x,y) belongs.

[0040]

number

[0041] <Coding difficulty D for motion vectors> V > For motion vectors, the greater their magnitude and variance, the more difficult they are to encode. Encoding difficulty D for motion vectors at spatial phase position (x,y). V (x,y) can be calculated, for example, by equation (6) below, where v is the magnitude of the motion vector of the CU to which (x,y) belongs, and σ is the variance of the motion vector.

[0042]

number

[0043] <Coding difficulty D for spatiotemporal frequency band components> P > Regarding spatiotemporal frequency band components, the greater the power of the high-frequency band components, the more difficult the encoding is determined to be. Furthermore, the weights can be changed in the spatial and temporal directions; for example, the temporal weight can be made stronger. That is, the encoding difficulty calculation unit 12' calculates the encoding difficulty D of the time-low frequency band. PL The coding difficulty (based on the magnitude of the spatiotemporal frequency band components in the low temporal frequency band and the high spatial frequency band) is determined, and the coding difficulty D of the high temporal frequency band is calculated. PH (time space The coding difficulty (D) for the spatiotemporal frequency band components is calculated by determining the coding difficulty (based on the size of the spatiotemporal high-frequency band and spatiotemporal high-frequency band) of the frequency band components, and then weighting and adding these together. P You may calculate this.

[0044] Time-based low-frequency band (t L The spatial phase position (x,y,t) of ) L Encoding difficulty D in ) PL (x,y,t L ) is the spatial frequency band HH n (x,y,t L The power component of ) is p n (x,y,t LIn the case of ), it is calculated, for example, by the following equation (7). HH is the diagonal component, and n is the number that identifies the frequency band. Here, as shown in Figure 3, we assume that n is assigned from 1 to 16 for each 2K × 1K frequency band. In equation (7), the denominator is p for n=1 to 16. n (x,y,t L ) is the sum, but for example, p from n=5 to 16 n (x,y,t L The sum of the above may also be used. Note that the method of selecting the spatial high-frequency band is not limited to this. For example, the coding difficulty may be determined based on the magnitude of the spatial frequency band components not only in the diagonal direction but also in the horizontal and vertical directions.

[0045]

number

[0046] Similarly, time-high frequency band (t H The spatial phase position (x,y,t) of ) H Encoding difficulty D in ) PH (x,y,t H ) is the spatial frequency band HH n (x,y,t H The power of p n (x,y,t H In the case of ), it is calculated using, for example, the following formula (8).

[0047]

number

[0048] Based on the above, the coding difficulty for each element component in each spatiotemporal frequency band is calculated, for example, in equation (9). In this equation, τ is the time-direction weighting coefficient, which may be predetermined or determined experimentally. For example, if the time-direction weighting is stronger than the spatial-direction weighting, τ > 1 is used.

[0049]

number

[0050] <Total encoding difficulty: D> The total coding difficulty D(x,y) at (x,y) is calculated as a weighted sum of equations (7), (8), and (9), for example, by equation (10). In this equation, α, β, and γ are the respective weight coefficients of equations (7), (8), and (9), and may be predetermined or determined experimentally depending on the coding / decoding device.

[0051]

number

[0052] <Difficult to encode region R(x,y)> The coding difficulty calculation unit 12' calculates the coding difficulty D(x,y) when the threshold T is set as shown in equation (11) below. H The region indicating a pixel position exceeding the threshold T is detected as the hard-to-encode region R(x,y). H This can be determined in advance or found through experimentation.

[0053]

number

[0054] The coding difficulty calculation unit 12' performs the processing of equations (5) to (11) for all (x,y) and outputs the final coding difficulty region R(x,y) and coding difficulty D(x,y).

[0055] As described above, the coding assistance information generator 2 generates a first coding difficulty D based on the value of the quantization parameter. Q And, the second coding difficulty D based on the magnitude and variance of the motion vector. V The third coding difficulty D is based on the size of the high-frequency band of the frequency band components. P By weighting and adding these values ​​together, the coding difficulty D for each pixel position is calculated. This makes it possible to determine the coding difficulty with even greater accuracy on a pixel-by-pixel basis.

[0056] (program) To function as the above-described encoding assistance information generation devices 1 and 2, it is also possible to use computers capable of executing program instructions. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc., for executing the required task.

[0057] A computer comprises a processor, a memory unit, an input unit, an output unit, and a communication interface. The processor may be a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc., and may be composed of multiple processors of the same or different types. The processor controls each of the above components and performs various calculations by reading and executing programs from the memory unit. At least a part of these processes may be implemented in hardware. The input unit is an input interface that receives user input operations and acquires information based on user operations, such as a pointing device, keyboard, or mouse. The output unit is an output interface that outputs information, such as a display or speaker. The communication interface is an interface for communicating with external devices, such as a LAN (Local Area Network) interface.

[0058] The program may be recorded on a computer-readable recording medium. Using such a medium, the program can be installed on the computer. The recording medium on which the program is recorded may be a non-transitory recording medium. Non-transitory recording media are not particularly limited, but may include, for example, CD-ROMs, DVD-ROMs, or USB (Universal Serial Bus) memory. Alternatively, the program may be downloaded from an external device via a network.

[0059] For example, a program to function as an encoding assistance information generation device 1 causes the computer to perform the following steps: perform encoding processing on the original image and extract information on the quantization parameters and motion vectors used for encoding for each encoding block; and calculate the encoding difficulty for each encoding block position by weighting and adding a first encoding difficulty based on the value of the quantization parameters and a second encoding difficulty based on the magnitude and variance of the motion vectors.

[0060] Furthermore, the coding assistance information generation devices 1 and 2 may be composed of one or more semiconductor chips, and the semiconductor chip may be equipped with a CPU that executes a program describing the processing content that realizes each function of the coding assistance information generation device 1.

[0061] Although the embodiments described above are representative examples, it will be apparent to those skilled in the art that many modifications and substitutions are possible within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the embodiments described above, and various modifications or changes are possible without departing from the scope of the claims. For example, it is possible to integrate multiple component blocks shown in the configuration diagram of the embodiments, or to divide a single component block. [Explanation of Symbols]

[0062] 1,2 Encoding auxiliary information generation device 11 Encoded information extraction unit 12,12' Encoding difficulty calculation unit 13. Frequency Resolution Section 14. Spatial frequency band resolution section 15-hour frequency band resolution section

Claims

1. An encoding information extraction unit performs encoding processing on the original image and extracts information on the quantization parameters and motion vectors used for encoding for each encoding block. A frequency decomposition unit performs frequency band decomposition in the time and spatial directions on the original image while retaining phase information, and generates spatiotemporal frequency band components, which are power spectral element components for each spatiotemporal frequency band. A coding difficulty calculation unit calculates the coding difficulty for each pixel position by weighting and adding a first coding difficulty based on the value of the quantization parameter, a second coding difficulty based on the magnitude and variance of the motion vector, and a third coding difficulty based on the magnitude of the spatiotemporal frequency band component in the high-frequency band. A device for generating encoding auxiliary information, equipped with the following features.

2. The coding difficulty calculation unit calculates the third coding difficulty by weighting and adding a coding difficulty DPL based on the magnitude of the spatiotemporal frequency band component in the time low frequency band and the spatial high frequency band, and a coding difficulty DPH based on the magnitude of the spatiotemporal frequency band component in the time high frequency band and the spatial high frequency band, the coding assistance information generation device according to claim 1.

3. The coding difficulty calculation unit makes the weight of the coding difficulty DPH stronger than the weight of the coding difficulty DPL, the coding assistance information generation device according to claim 2.

4. The encoding assistance information generation device according to any one of claims 1 to 3, wherein the encoding difficulty calculation unit detects a region indicating a pixel position where the encoding difficulty exceeds a threshold as an encoding difficulty region.

5. A program for causing a computer to function as an encoding assistance information generation device according to any one of claims 1 to 4.

Images