Video encoding device and program

Abstract

To reduce display delay when a video is played back.SOLUTION: A video encoding device 1 includes an encoding unit 10 that encodes input video in a plurality of encoding modes to generate encoded data, and a cost optimization unit 40 that determines an optimum encoding mode for optimizing the cost in the encoding process of the encoding unit 10 among a plurality of encoding modes, and the cost optimization unit 40 determines the optimum encoding mode according to the coding amount of the input video when encoded in a predetermined encoding mode, the encoding distortion, and the decoding cost indicating the complexity of decoding.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video encoding device and a program. [Background technology]

[0002] In recent years, 360° video, which can display a panoramic view of the surroundings, has become increasingly popular. Users can view 360° video by adjusting their viewpoint to the direction they want to focus on. Viewing 360° video using a head-mounted display (HMD) in a viewing environment similar to the human field of view (approximately 200 degrees horizontally and 130 degrees vertically) provides a more realistic and immersive experience than viewing on a conventional 2D monitor. The popularity of 360° video has also led to an increasing demand for compression technology. The latest video coding method, Versatile Video Coding (VVC), employs specialized coding tools for 360° video, such as the Equirectangular Projection (ERP) format shown in Figure 2 of Non-Patent Document 1 and the Cubemap Projection (CMP) format shown in Figure 55 of Non-Patent Document 2.

[0003] These image formats project spherical 3D images showing a 360-degree landscape in both the vertical and horizontal directions centered on a certain point onto a two-dimensional surface. The ERP format projects the sphere shown in Figure 5 onto a two-dimensional image with an aspect ratio of 2:1 (equivalent to equirectangular projection on a world map). The CMP format, as shown in Figure 6, projects the sphere onto six faces (PX, PY, PZ, NX, NY, NZ) of a cube, arranging each face. By treating each frame of 360° video as a two-dimensional image in this way, it can be encoded in the same way as conventional two-dimensional video. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Yan Ye, and Jill Boyce, “Algorithm descriptions of projection format conversion and video quality metrics in 360Lib Version 12,” JVET-T2004, Oct. 2020. [Non-patent document 2] Jianle Chen, Yan Ye, and Seung Hwan Kim “Algorithm descriptions for Versatile Video Coding and Test Model 11,” JVET-T2002, Oct. 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] While viewing 360° video on an HMD provides a sense of immersion, it can also cause simulator sickness, which can be uncomfortable. Research is ongoing into the various factors that cause simulator sickness, and one of the causes is thought to be the delay in the video displayed on the HMD in response to user operations such as viewpoint movement.

[0006] In view of the above circumstances, an object of the present invention is to provide a video encoding device and a program that can reduce display delays when playing back video. [Means for solving the problem]

[0007] In order to solve the above problem, a video encoding device according to one embodiment is a video encoding device that encodes an input video, and includes: an encoding unit that encodes the input video using a plurality of encoding modes to generate encoded data; and a cost optimization unit that determines an optimal encoding mode from the plurality of encoding modes that optimizes the cost of the encoding process of the encoding unit, and the cost optimization unit is configured to optimize the amount of code of the input video when encoded in a predetermined encoding mode. R , encoding distortion D, revengeDecoding cost indicates the complexity of decoding C D , the decoding cost C D Contribution w of and Lagrange multiplier λ Depending on The coding mode that minimizes the cost J expressed by equation (2) below is The optimal coding mode as Make a decision.

[0009] Furthermore, in one embodiment, the cost optimization unit may change the value of the contribution w depending on the position within the frame of the input video, and determine, as the optimal coding mode, a coding mode in which the code amount R is equal to or less than a target code amount depending on the position within the frame.

[0010] Furthermore, in one embodiment, the cost optimization unit may reduce the allocation of the code amount R per unit area and the contribution w as the area becomes farther away from a predetermined area.

[0011] In order to solve the above problem, a program according to the present invention causes a computer to function as the video encoding device. [Effects of the Invention]

[0012] According to the present invention, it is possible to reduce display delay when playing back video, and if the input video is 360 video, it is possible to reduce simulator sickness that occurs when watching 360 video. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a video encoding device according to an embodiment. [Figure 2] 2 is a block diagram illustrating an example of the configuration of an encoding unit of a video encoding device according to an embodiment. FIG. [Figure 3] 1 is a flowchart illustrating a video encoding method in a video encoding device according to an embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of region division in the ERP format. [Figure 5] FIG. 1 is a diagram showing a three-dimensional XYZ coordinate system in the ERP format. [Figure 6] FIG. 10 is a diagram showing the definition of coordinates in the CMP format. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, one embodiment will be described in detail with reference to the drawings.

[0015] 1 is a block diagram showing an example of the configuration of a video encoding device according to this embodiment. The video encoding device 1 shown in FIG. 1 includes a coding mode candidate derivation unit 30, an encoding unit 10, a cost optimization unit 40, and a coding result storage unit 50.

[0016] The video encoding device 1 encodes input video (e.g., 360° video) and outputs a bitstream, which is the encoding result, to the outside. In video encoding methods such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), the input video is divided into the largest block unit called a Coding Tree Unit (CTU), encoding processing is performed for each CTU, and the division shape of Coding Units (CUs) within the CTU, the prediction mode for each CU, the transform mode, etc. are sequentially determined. Note that the input video is not limited to 360° video.

[0017] The coding mode candidate derivation unit 30 determines one or more coding mode candidates for the CTU to be processed and its internal CUs, based on input parameters such as a bit rate and a GOP (Group Of Picture) structure, and the coding modes selected so far. The coding mode is a combination of a coding tool and parameters (such as a DC prediction mode for intra prediction). The coding mode candidate derivation unit 30 can apply any known method, and for example, determines (prunes) a small number of candidate coding modes from a large number of coding modes, similar to the internal processing of VVC reference software VTM (VVC Test Model). The coding mode candidate derivation unit 30 then outputs the determined coding mode candidates to the coding unit 10 and the cost optimization unit 40.

[0018] The encoding unit 10 performs encoding processing on the input video by applying the input parameters and the encoding mode candidates determined by the encoding mode candidate derivation unit 30. Then, the encoding unit 10 outputs the encoding result to the cost optimization unit 40 and the encoding result storage unit 50, and also outputs the locally decoded image to the cost optimization unit 40. Details of the encoding unit 10 will be described later.

[0019] The cost optimization unit 40 determines, from among a plurality of coding modes, a coding mode (hereinafter referred to as an "optimal coding mode") that optimizes the cost in the coding process of the coding unit 10. Then, the cost optimization unit 40 outputs the optimal coding mode to the coding mode candidate derivation unit 30 and the coding result storage unit 50.

[0020] Video coding methods such as HEVC and VVC have achieved efficient video compression by adopting various new coding tools, making it important to select the optimal coding mode from the vast number of coding modes.To select this optimal coding mode, RD (Rate-Distortion) optimization has traditionally been performed to minimize the cost J defined by equation (1).Here, D represents coding distortion, R represents the code amount, and λ represents the Lagrange multiplier.

[0021]

number

[0022] In order to reduce display delay when playing back input video, the cost optimization unit 40 determines the optimal coding mode for each image block according to the amount of code of the input video when coded in a predetermined coding mode, coding distortion, and decoding cost indicating the complexity when decoding. Specifically, the cost J is calculated using equation (2) instead of the conventional equation (1), and the optimal coding mode that minimizes the cost J is determined. In equation (2), C D denotes the decoding cost for each coding mode, and w is the decoding cost C D indicates the contribution (positive constant) of

[0023]

number

[0024] Decryption cost C D is a value experimentally determined in advance that indicates the complexity of the decoding process for each coding mode, and may be a value corresponding to the calculation time (time required for the decoding process) or a value corresponding to the amount of calculation processing. For example, in VVC, when bidirectional prediction is used, techniques such as DMVC (Decoder-side Motion Vector Refinement) and BDOF (Bi-Directional Optical Flow) can be used to correct motion vector candidates on the decoding device (decoder) side according to the positional relationship of the reference frames, the CU size, etc. By using these techniques, prediction accuracy is improved and coding efficiency is improved (the code amount R and coding distortion D in equation (2) are reduced), but the decoding cost C D In general, as the code amount R increases, the decoding cost C increases mainly due to the increase in the entropy decoding process of the decoder. D is on the rise.

[0025] Specifically, as shown in FIG. 1, the cost optimization unit 40 includes a decoding cost determination unit 41, a code amount calculation unit 42, a coding distortion calculation unit 43, and a coding mode determination unit 44.

[0026] The decoding cost determination unit 41 determines the decoding cost C of each coding mode candidate. D Then, the decoding cost determination unit 41 determines the decoding cost C D to the coding mode determination unit 44. Note that the decoding cost C D are stored in advance in a storage unit in association with each encoding mode. The storage unit may be included in the decoding cost determination unit 41 or may be provided outside the decoding cost determination unit 41.

[0027] The code amount calculation unit calculates the code amount R of the encoding result input from the encoding unit 10. Then, the code amount calculation unit outputs the code amount R to the encoding mode determination unit .

[0028] The coding distortion calculation unit 43 compares the input video with the locally decoded image input from the encoding unit 10 to calculate the coding distortion D. Then, the coding distortion calculation unit 43 outputs the coding distortion D to the coding mode determination unit 44. An evaluation value representing the coding distortion D is, for example, the sum of squared errors (SSE) or the sum of absolute differences (SAD).

[0029] The coding mode determination unit 44 calculates the cost J shown in the above formula (2) and determines the optimal coding mode that minimizes the cost J. Then, the coding mode determination unit 44 outputs the determined optimal coding mode to the coding mode candidate derivation unit 30 and the coding result storage unit 50.

[0030] The encoding result storage unit 50 outputs the encoding result, which is encoded by the encoding unit 10 using the optimal encoding mode determined by the encoding mode determination unit 44, to the outside of the video encoding device 1. The encoding result is decoded by a decoding device (not shown).

[0031] (encoding section) Next, details of the encoding unit 10 will be described. Fig. 2 is a block diagram showing an example configuration of the encoding unit 10. The encoding unit 10 shown in Fig. 2 includes a block dividing unit 11, a subtraction unit 12, a transformation unit 13, a quantization unit 14, an inverse quantization unit 15, an inverse transformation unit 16, an addition unit 17, a storage unit 18, a prediction unit 19, and an entropy encoding unit 20.

[0032] The block division unit 11 divides an input video frame into block units such as CTU and CU for encoding processing, thereby generating block images, and outputs the block images to the subtraction unit 12 and the prediction unit 19 .

[0033] The subtraction unit 12 subtracts each pixel value of the predicted block image input from the prediction unit 19 described later from each pixel value of the block image input from the block division unit 11 to generate a residual block image indicating the difference between the block image and the predicted block image, and outputs the residual block image to the conversion unit 13.

[0034] The transform unit 13 performs a transform process such as an orthogonal transform on the residual block image input from the subtraction unit 12 to calculate transform coefficients, and outputs the transform coefficients to the quantization unit 14 .

[0035] The quantization unit 14 generates quantized coefficients by dividing the transform coefficients input from the transform unit 13 by the quantization step and quantizing them, and outputs the quantized coefficients to the inverse quantization unit 15 and the entropy coding unit 20. The quantization unit 14 reduces the amount of data.

[0036] The inverse quantization unit 15 restores the transform coefficients by multiplying the quantization coefficients input from the quantization unit 14 by the quantization step, and outputs the restored transform coefficients to the inverse transform unit 16 .

[0037] The inverse transform unit 16 performs inverse transform processing (processing to undo the transform performed by the transform unit 13) on the transform coefficients input from the inverse quantization unit 15 to restore a residual block image, and outputs the restored residual block image to the addition unit 17. For example, if the transform unit 13 performs a discrete cosine transform, the inverse transform unit 16 performs an inverse discrete cosine transform.

[0038] The adder 17 adds the residual block image input from the inverse transformer 16 and the predicted image input from the predictor 19, and outputs the result as a locally decoded image to the memory 18 and the coding distortion calculator 43 of the cost optimizer 40.

[0039] In this way, the encoding unit 10 multiplies the quantization coefficients by the quantization step to restore the transform coefficients, performs inverse transform processing on the transform coefficients to restore the residual block image, and adds the residual block image to an intra-predicted image or a motion-compensated predicted image to generate a locally decoded image. Note that the encoding unit 10 may perform post-processing, such as filtering using a deblocking filter, on the locally decoded image output by the adder 17 before outputting it to the storage unit 18.

[0040] The prediction unit 19 performs intra prediction (intra-frame prediction) or inter prediction (inter-frame prediction, motion compensated prediction). In intra prediction, an intra predicted image is generated by intra predicting a locally decoded image stored in the storage unit 18 according to an intra prediction mode. In inter prediction, a motion compensated predicted image is generated by motion compensated predicting a locally decoded image stored in the storage unit 18 according to a motion vector. The prediction unit 19 switches between the intra predicted image and the motion compensated predicted image to generate a predicted block image, and outputs the image to the subtraction unit 12 and the addition unit 17. The prediction unit 19 outputs prediction parameters (intra prediction mode and motion vector information) used in the prediction process to the entropy coding unit 20.

[0041] The entropy coding unit 20 performs entropy coding on the quantization coefficients input from the quantization unit 14, and parameters such as block size information, transformation information, and prediction parameters used in the coding process, compresses the data, and generates a bit stream as the coding result, which is output to the code amount calculation unit 42 of the cost optimization unit 40 and the coding result storage unit 50.

[0042] Next, a description will be given of the video encoding method of the video encoding device 1. Fig. 3 is a flowchart showing the video encoding method of the video encoding device 1.

[0043] In step S101, the coding mode candidate derivation unit 30 determines coding mode candidates. In step S102, the coding unit 10 performs coding processing by applying the coding mode candidates. In step S103, the cost optimization unit 40 determines, from among multiple coding modes, an optimal coding mode that minimizes the cost J expressed by the above equation (2). In step S104, the coding result storage unit 50 outputs the coding result coded in the optimal coding mode.

[0044] In this way, the video encoding device 1 obtains the decoding cost C, which indicates the complexity of decoding, in addition to the code amount R and coding distortion D of the input video when encoded in a predetermined encoding mode. D The optimal encoding mode is determined taking into consideration the above factors. Therefore, according to the present invention, it is possible to reduce the display delay when playing back video. This also makes it possible to reduce simulator sickness that can occur when the input video is a 360 video.

[0045] (Variation) Next, a modified example of the video encoding device 1 will be described. In this modified example, the processing of the encoding mode determination unit 44 differs from that of the above-described video encoding device 1. The encoding mode determination unit 44 according to the modified example sets a different target code amount depending on the position within the frame of the input video. Then, taking into account the human viewing angle, the value of the contribution w in the above formula (2) is changed depending on the position within the frame of the input video, and the encoding mode that minimizes the cost J is determined as the optimal encoding mode among the encoding modes in which the code amount R is equal to or less than the target code amount depending on the position within the frame of the input video.

[0046] The horizontal viewing angle of a human being is about 60 degrees (30 degrees left and right), which is the range in which both eyes can see and distinguish colors, and the range in which both eyes can see is about 120 degrees (60 degrees left and right). The vertical viewing angle is about 40 degrees (20 degrees up and down), which is the range in which both eyes can see and distinguish colors, and the range in which both eyes can see is about 130 degrees (60 degrees up and 70 degrees down). Therefore, by gradually changing the code amount R and the contribution w in these regions centered on the point of interest, the input video can be coded efficiently. Therefore, the coding mode determination unit 44 according to the modified example divides the region of the input video, and allocates the code amount R per unit area and the decoding cost C in proportion to the distance from a predetermined region close to the point of interest. D The contribution w of may be reduced.

[0047] FIG. 4 shows an example of region division in the ERP format. As shown in FIG. 5, in the ERP format, the left and right edges of an image are horizontally adjacent regions. In FIG. 4, region a indicates a range of 60 degrees horizontally and 40 degrees vertically centered on the attention point P, region b indicates a range of 120 degrees horizontally and 130 degrees vertically outside region a, and region c indicates an area outside region b. The encoding mode determination unit 44 may set the attention point P to the center of the frame as shown in FIG. 4, or may determine it according to the creator's intention. Furthermore, the attention point P may be a point that is experimentally confirmed in advance and is likely to be noticed by the user. Alternatively, when encoding in real time, the attention point P may be determined from position information of the HMD. In the example shown in FIG. 4, the encoding mode determination unit 44 allocates the code amount R per unit area and the decoding cost C to regions a, b, and c, in that order. D Reduce the contribution w of

[0048] In the CMP format, regions may be similarly divided based on the relationship between the focus point and the viewing angle, or in the case of a 3x2 frame packing configuration consisting of Right, Front, Left, Top, Back, and Bottom as shown in Figure 55 of Non-Patent Document 2, the code amount R and contribution w for each region may be determined so that, for example, Front > Right, Left > Top, Back, Bottom.

[0049] In this way, in the modified example of the video encoding device 1, by changing the values ​​of the code amount R and the contribution w depending on the position within the frame of the input video, it is possible to further reduce the degradation of encoding in the area of ​​the user's attention and to efficiently encode the input video.

[0050] (program) A computer capable of executing program instructions can also be used to function as the video encoding device 1. Here, the computer may be a general-purpose computer, a special-purpose computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the necessary tasks.

[0051] The computer includes a processor, a storage unit, an input unit, an output unit, and a communication interface. The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an SoC (System on a Chip), or the like, and may be configured with multiple processors of the same or different types. The processor reads and executes programs from the storage unit to control the above components and perform various arithmetic processing. Note that at least a portion of these processing contents may be implemented by hardware. The input unit is an input interface that accepts user input operations and acquires information based on the user operations, such as a pointing device, keyboard, or mouse. The output unit is an output interface that outputs information, such as a display or speaker. The communication interface is an interface for communicating with external devices.

[0052] The program may be recorded on a computer-readable recording medium. Using such a recording medium, the program can be installed on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. Furthermore, the program may be downloaded from an external device via a network.

[0053] For example, a program for causing a computer to function as the above-mentioned video encoding device 1 causes the computer to execute an encoding step of encoding an input video using a plurality of encoding modes to generate encoded data, and a cost optimization step of determining an optimal encoding mode from the plurality of encoding modes that optimizes the cost of the encoding process in the encoding step, wherein the cost optimization step determines the optimal encoding mode based on the amount of code of the input video when encoded using a predetermined encoding mode, encoding distortion, and a decoding cost that indicates the complexity of decoding.

[0054] Although the above-described embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and alterations are possible without departing from the scope of the claims. For example, multiple building blocks shown in the block diagrams of the embodiments can be combined into one, or one building block can be divided. [Explanation of symbols]

[0055] 1. Video Encoding Device 10 Encoding section 11 Block division section 12 Subtraction section 13 Conversion unit 14 Quantization section 15 Inverse quantization section 16 Inverse conversion unit 17 Addition section 18 Memory section 19 Prediction Department 20 Entropy coding unit 30 Coding mode candidate derivation unit 40 Cost Optimization Department 41 Decoding cost determination unit 42 Code amount calculation unit 43 Coding distortion calculation unit 44 Encoding mode determination unit 50 Encoding result storage section

Claims

1. A video encoding device that encodes an input video, comprising: an encoding unit that encodes an input video in a plurality of encoding modes to generate encoded data; a cost optimization unit that determines an optimal coding mode from among the plurality of coding modes, the optimal coding mode optimizing a cost in the coding process of the coding unit; The cost optimization unit calculates a Lagrangian multiplier λ based on a code amount R of the input video when the input video is coded in a predetermined coding mode, a coding distortion D, a decoding cost C D indicating a complexity when decoding, a contribution w of the decoding cost C D , and the Lagrangian multiplier λ. The following formula The video encoding device determines the encoding mode that minimizes the cost J expressed by the following equation as the optimal encoding mode.

2. 2. The video encoding device according to claim 1, wherein the cost optimization unit changes the value of the contribution w depending on a position within a frame of the input video, and determines, as the optimal encoding mode, an encoding mode in which the code amount R is equal to or less than a target code amount depending on the position within the frame.

3. The video encoding device according to claim 2 , wherein the cost optimization unit reduces the allocation of the code amount R per unit area and the contribution w as the area moves away from a predetermined area.

4. A program for causing a computer to function as the video encoding device according to any one of claims 1 to 3.

Images