Video encoder, and video encoding program

Abstract

To provide a video encoder capable of selecting optimal resolution, and a video encoding program.SOLUTION: A video encoder according to one embodiment has a video reducing part which reduces resolution of an input video and outputs video data of a plurality of videos different in resolution. Also, the video encoder performs parallel encoding of respective video data of a plurality of videos of different resolutions and outputs video data of the plurality of encoded videos. Moreover, the video encoder selects as an optimal resolution resolution where encoding distortion of the encoded video becomes minimum when a bit amount of the encoded video data is a target rate with respect to video data of the respective encoded videos, and has a resolution selection part outputting the encoded video data of the selected resolution.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a video encoding device and a video encoding program. [Background technology]

[0002] Versatile Video Coding (VVC), one of the standardized video compression techniques, has a reference picture resampling (RPR) function. RPR is a technique that reduces the resolution of a reference image (picture) to make it the same resolution as the image (picture) to be processed and generates a predicted image. By applying RPR when the bandwidth fluctuation of the input video is larger than a certain level, such as in streaming, it becomes possible to refer to pictures between images of different resolutions, for example, and to avoid image quality failure.

[0003] For example, the following is a technique using RPR: That is, based on the PSNR (Peak Signal to Noise Ratio) value of the first picture of each GOP (Group Of Picture) and the QP (Quantization Parameter) (initial QP) value of the first picture, the RPR scale factor (RPR_SF) (i.e., resolution) of the i-th GOP is calculated by the following formula:

[0004]

number

[0005] In this technique, the scale factor determined by Equation 1 is applied to all pictures in the i-th GOP. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Junghak Nam, Sunmi Yoo, Jaehyun Lim, “EE1-2.1: RPR encoder with multiple scale factors”, JVET-Z0065-v1, 20-29 April 2022 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology disclosed in Non-Patent Document 1 uses a predefined criterion for calculation in consideration of reducing the amount of calculation, and therefore does not necessarily select an optimal resolution. Also, the technology disclosed in Non-Patent Document 1 calculates the resolution for the first picture in the GOP by applying Equation 1, and applies the calculated resolution to all pictures in the GOP, so it does not handle cases where the optimal resolution changes on a picture-by-picture basis, and therefore does not necessarily calculate the optimal resolution for all pictures in the GOP.

[0008] Therefore, an object of the present disclosure is to provide a video encoding device and a video encoding program that are capable of selecting an optimal resolution. [Means for solving the problem]

[0009] A video encoding device according to a first aspect includes a video reduction unit that reduces the resolution of an input video and outputs video data of a plurality of videos with different resolutions. The video encoding device also includes a video encoding unit that encodes each of the video data of a plurality of videos with different resolutions in parallel and outputs the video data of the plurality of encoded videos. The video encoding device further includes a resolution selection unit that selects, for each of the video data of the encoded video, a resolution that minimizes the encoding distortion of the encoded video when the bit rate of the video data of the encoded video is set to a target rate, as an optimal resolution, and outputs the video data of the encoded video with the selected resolution.

[0010] A video encoding program according to a second aspect causes a computer to execute a process of reducing the resolution of an input video and outputting video data of a plurality of videos with different resolutions. The video encoding program also causes a computer to execute a process of encoding each of the video data of a plurality of videos with different resolutions in parallel and outputting the video data of the plurality of encoded videos. The video encoding program further causes a computer to execute a process of selecting, for each of the video data of the encoded video, a resolution at which the encoding distortion of the encoded video is minimized when the bit rate of the video data of the encoded video is set to a target rate, as an optimal resolution, and outputting the video data of the encoded video with the selected resolution. Effect of the Invention

[0011] According to the present disclosure, it is possible to provide a video encoding device and a video encoding program that are capable of selecting an optimal resolution. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a video encoding device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a video encoding unit according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of the resolution selection unit according to the first embodiment. [Figure 4]FIG. 4 is a diagram illustrating an example of operation according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing experimental results according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [First embodiment] Hereinafter, an embodiment will be described with reference to the drawings. In the description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0014] In the following, the terms "image", "frame" and "picture" are used to mean the same thing and may not be distinguished from one another.

[0015] Furthermore, the terms "image" and "image data" are sometimes used interchangeably.

[0016] Furthermore, "video" and "video data" may be used interchangeably. For example, "input video", "input video data", and "video data of input video" may be used interchangeably. Similarly, for example, "encoded video", "video data of encoded video", and "encoded video data" may be used interchangeably.

[0017] In general, a technology for adaptively changing the resolution of an image is sometimes called Adaptive Resolution Change (ARC). ARC is realized by RPR, which is one of the functions of VVC.

[0018] (Video Encoding Apparatus According to the First Embodiment) Next, an example of the configuration of a video encoding device in the first embodiment will be described.

[0019] FIG. 1 illustrates an example of the configuration of a video encoding device 100. As shown in FIG.

[0020] The video encoding device 100 receives input video data including image data of an original image. The video encoding device 100 generates video data of a plurality of videos with different resolutions from the video data, and encodes each of the plurality of video data in parallel. The video encoding device 100 then calculates a rate distortion cost shown below for each piece of encoded video data, and selects the resolution at which the cost is minimized (i.e., the encoding distortion of the encoded video is minimized when the bit amount of the encoded video data is set to a target rate) as the optimal resolution. The video encoding device 100 outputs encoded video data of the selected resolution.

[0021] In this way, the video encoding device 100 according to the first embodiment actually performs encoding processing on a plurality of video data with different resolutions and calculates the rate-distortion cost based on the encoded video data, which makes it possible to select an optimal resolution compared to the case where a predefined value is used for the calculation.

[0022] Furthermore, in the video encoding device 100 according to the first embodiment, encoding processes are performed in parallel on multiple pieces of video data with different resolutions, making it possible to select the optimal resolution without significantly increasing calculation time.

[0023] Furthermore, in the video encoding device 100 according to the first embodiment, the rate-distortion cost is calculated on a frame-by-frame basis. Therefore, in the video encoding device 100 according to the first embodiment, even when an input video having an optimal resolution that differs from frame to frame is input, it is possible to select an optimal resolution for each frame.

[0024] As shown in FIG. 1, the video encoding device 100 includes a video reduction unit 110, a video encoding unit 120, a video enlargement unit 130, and a resolution selection unit 140.

[0025] The image reducing unit 110 reduces the resolution of the input image including the original image, and outputs the image data of the reduced image. There is no limit to the number of reduced images that the image reducing unit 110 outputs, and any number of reduced images may be output. That is, the image reducing unit 110 may output image data of a plurality of reduced images that differ for each resolution. Therefore, the image reducing unit 110 may include an image reducing unit according to the resolution. For example, the image reducing unit 110 may include an image reducing unit that reduces the resolution to 1 / 2 that of the input image, an image reducing unit that reduces the resolution to 2 / 3 that of the input image, an image reducing unit that reduces the resolution to 3 / 4 that of the input image, and so on. The image reducing unit 110 may output a reduced image of one resolution, and in that case, the input image and the reduced image may constitute a plurality of image data of different resolutions. Alternatively, the image reducing unit 110 may output image data of the same resolution as the input image without changing the resolution (or by reducing the resolution to 1x that of the input image). In this way, the image reducing unit 110 may reduce the resolution of the input image and output image data of a plurality of images with different resolutions.

[0026] Note that the reduction method in the video reduction unit 110 may be any method, but in the following description, the video is reduced using a filter implemented in VTM (VVC Tet Model), which is reference software for VVC. Alternatively, the reduction method in the video reduction unit 110 may use a neural network that uses multiple input frames to output multiple output frames (or reduced frames).

[0027] The video encoding unit 120 encodes the video data of the multiple videos with different resolutions in parallel and outputs the video data of the multiple encoded videos. In the example of Fig. 1, the input video and the reduced video are treated as multiple videos with different resolutions, and the video data of the input video and the video data of the reduced video are encoded in parallel.

[0028] Fig. 2 is a diagram showing an example of the configuration of the video encoding unit 120. As shown in Fig. 2, the video encoding unit 120 has an input video encoding unit 121 and a reduced video encoding unit 122. The input video encoding unit 121 performs an encoding process on the input video, and outputs encoded data of the input encoded video (hereinafter referred to as "input encoded video data"). On the other hand, the reduced video encoding unit 122 performs an encoding process on the reduced video, and outputs encoded data of the reduced encoded video (hereinafter referred to as "reduced encoded video data"). As shown in Fig. 2, the encoding processes are performed in parallel.

[0029] In the reduced video encoding unit 122, a plurality of reduced video encoding units 122 may be included in the video encoding unit 120 according to the number of resolutions in the reduced video. For example, when a (1 / 2)-fold reduced video and a (2 / 3)-fold reduced video are generated by the video reducing unit 110, a reduced video encoding unit 122-1 for the (1 / 2)-fold reduced video and a reduced video encoding unit 122-2 for the (2 / 3)-fold reduced video may be included in the video encoding unit 120 (or the reduced video encoding unit 122). The reduced video encoding unit 122 may output one piece of reduced encoded video data corresponding to a reduced video of one resolution, or may output a plurality of pieces of reduced encoded video data corresponding to a plurality of reduced videos of different resolutions. In this way, the reduced video encoding unit 122 outputs one or a plurality of pieces of reduced encoded video data according to the number of resolutions. The number of resolutions may correspond to the number of resolutions of the reduced video generated by the video reducing unit 110.

[0030] The same video encoding method may be used for the encoding process in the input video encoding unit 121 and the reduced video encoding unit 122. Any method may be used as the video encoding method itself, but the following description will be given assuming that VTM, which is reference software for VVC, is used.

[0031] Although the encoding process may be performed in units of any number of frames, the following description assumes that the process is performed in GOP units, every 32 frames in accordance with the Random Access mode of VTM. In this case, taking intraframes into consideration, the first GOP is processed in units of 33 frames, and the subsequent GOPs are processed in units of 32 frames.

[0032] In the encoding process, the quantization parameter QP is set to a higher value as the resolution increases. In general, the higher the resolution, the higher the degree of difficulty in encoding. The quantization parameter QP is set as a setting value for adjusting the balance between rate distortion, and the higher the resolution, the higher the value is set. For example, in the following, when the quantization parameter QP for the resolution of the original image (including input video) is x, the quantization parameter QP is set to (x-4) for a reduced video with a resolution 2 / 3 times that of the input video, and the quantization parameter QP is set to (x-6) for a reduced video with a resolution 1 / 2 times that of the input video. The quantization parameter QP may be set to a higher value as the resolution increases, and may be set to a value other than the above quantization parameter QP.

[0033] Returning to FIG. 1, the image enlargement unit 130 inputs the reduced encoded image data, enlarges the reduced encoded image data to the resolution of the input image, and outputs the image data of the enlarged encoded image (hereinafter referred to as "enlarged encoded image data"). That is, the image enlargement unit 130 performs an enlargement process so that the resolution of the enlarged encoded image is the same as the resolution of the input image. The image enlargement unit 130 outputs the same number of enlarged encoded images as the number of reduced images with different resolutions. Therefore, the image enlargement unit 130 may include the same number of image enlargement units as the number of resolutions of the reduced images. For example, an image enlargement unit that enlarges a 1 / 2 reduced image to 2 times, an image enlargement unit that enlarges a 2 / 3 reduced image to 3 / 2 times, an image enlargement unit that enlarges a 3 / 4 reduced image to 4 / 3 times, and so on may be included. The image enlargement unit 130 may output one enlarged encoded image data corresponding to one reduced image of a resolution. Alternatively, the video enlargement unit 130 may output the reduced-encoded video data as is without enlarging the resolution (or by setting the resolution to one times that of the reduced-encoded video). In this case, for example, the enlargement process may be specified for the receiver side. The enlargement process may be specified in a format in which an index value is added to a sequence parameter set (SPS) that represents parameters common to a sequence, or in a format in which an index is added to a picture parameter set (PPS) that represents parameters common to multiple pictures.

[0034] In this manner, the video enlarger 130 may enlarge the resolution of the multiple encoded videos to the resolution of the input video, and output video data of the multiple enlarged encoded videos.

[0035] While any method may be used for the enlargement method in the image enlargement unit 130, the following description will be given assuming that the enlargement process is performed using a filter implemented in the VTM. The enlargement method in the image enlargement unit 130 may also use a neural network that uses multiple input frames and outputs multiple output frames.

[0036] The resolution selection unit 140 receives the input encoded video data, the reduced encoded video data, the enlarged encoded video data, and the input video data, and selects a resolution. Specifically, the resolution selection unit 140 calculates a rate-distortion cost Li shown in the following equation 2 for each resolution, and selects a resolution that minimizes the rate-distortion cost Li.

[0037]

number

[0038] Note that the coding distortion dist may be calculated using any method, but in this embodiment, a description will be given assuming that a mean squared error (MSE) for the luminance values ​​of two videos is used.

[0039] Also, any method may be used to determine the value of λ, but in this embodiment, the description will be given assuming that the value of λ used when the input video encoding unit 121 encoded the frame is used.

[0040] Equation 2 represents a cost function for each resolution. That is, Equation 2 represents a cost function (rate-distortion cost Li) based on the coding distortion (dist) of the coded video when the bit amount (Ri) of each coded video data with different resolutions (e.g., input coded video data and reduced coded video data) is set as a target rate. The resolution selection unit 140 selects the i-th resolution with the smallest rate-distortion cost Li.

[0041] Fig. 3 is a diagram illustrating an example of the configuration of the resolution selection unit 140. As illustrated in Fig. 3, the resolution selection unit 140 includes an input encoded video bit amount calculation unit 141, a reduced encoded video bit amount calculation unit 142, and a selection processing unit 143.

[0042] The input coded video bit amount calculation unit 141 receives input coded video data and calculates the amount of bits of the input coded video data. The input coded video bit amount calculation unit 141 outputs the calculated amount of bits as the input coded video bit amount.

[0043] The reduced-coded video bit amount calculation unit 142 inputs the reduced-coded video data and calculates the bit amount of the reduced-coded video data. The reduced-coded video bit amount calculation unit 142 outputs the calculated bit amount as the reduced-coded video bit amount.

[0044] The input encoded video bit amount calculation unit 141 and the reduced encoded video bit amount calculation unit 142 calculate the bit amount (Ri) of the encoded video data in Equation 2.

[0045] The selection processing unit 143 receives the input encoded video data and reduced encoded video data (i.e., multiple encoded video data with different resolutions), the input encoded video bit amount and reduced encoded video bit amount (i.e., the bit amount of multiple encoded video data), the input video data, and the enlarged encoded video data. The selection processing unit 143 then calculates a rate-distortion cost Li for each resolution using Equation 2, and selects the resolution that minimizes the rate-distortion cost Li. The selection processing unit 143 outputs the encoded video data of the selected resolution (the input encoded video data or the reduced encoded video data) as the video data of the selected video.

[0046] The video data of the selected video may be transmitted as a bitstream to a video decoding device where the video decoding process is performed.

[0047] (Operation example according to the first embodiment) Next, an operation example according to the first embodiment will be described.

[0048] FIG. 4 is a diagram illustrating an example of operation according to the first embodiment.

[0049] As shown in FIG. 4, in step S10, the video encoding device 100 starts the process.

[0050] In step S11, the video encoding device 100 receives video data of an input video including an original image.

[0051] In step S12, the video encoding device 100 loops the processes up to step S15 for each video frame. Note that the video encoding device 100 may loop the processes from step S12 to step S15 for each GOP, not just for each video frame.

[0052] In step S13, the video encoding apparatus 100 loops the process up to step S14 for each resolution.

[0053] In step S14, the video encoding device 100 performs an encoding process on the video data of the input video. Specifically, the video reducing unit 110 performs a reduction process on the video data of the input video, and outputs the video data of the reduced video. The video reducing unit 110 may output a plurality of video data with different resolutions through the reduction process. Next, the video encoding unit 120 performs an encoding process on the video data of the input video and the video data of the reduced video in parallel. The video encoding unit 120 outputs the input encoded video data and the reduced encoded video data. The video encoding unit 120 may output a plurality of encoded video data with different resolutions by outputting the input encoded video data and the reduced encoded video data. Next, the video enlargement unit 130 performs an enlargement process on the reduced encoded video data so that the resolution of the reduced encoded video becomes the same as the resolution of the input video. The video enlargement unit 130 may output a plurality of enlarged encoded videos with different resolutions.

[0054] In step S15, the video encoding device 100 selects a resolution. Specifically, the resolution selection unit 140 (the selection processing unit 143) calculates the rate-distortion cost Li for each resolution using Equation 2, and selects the resolution that minimizes the rate-distortion cost Li.

[0055] Then, the video encoding device 100 ends the series of processes. The video encoding device 100 may perform the processes from step S10 to step S16 not only for each video frame but also for each GOP, and may perform the processes from step S10 to step S16 again for the next GOP.

[0056] (Experimental Results) Next, the experimental results will be described.

[0057] Fig. 5 is a diagram showing an example of an experimental result. The experimental result shown in Fig. 5 compares two techniques, with the technique described in Non-Patent Document 1 as the "conventional technique" and the technique described in this embodiment as the "present technique."

[0058] In the experiment, the BD-Rate for the Peak Signal to Noise Ratio (PSNR) and the BD-Rate for the Multi-Scale Structural Similarity (MS-SSIM) were measured for the two methods. Both PSNR and MS-SSIM are indices that represent image quality. Specifically, PSNR represents the quality calculated based on the mean square error (MSR) between two images. MS-SSIM represents the quality that represents the change in brightness, contrast, and the change in the structure of the object shown in the image between two images in a single index. BD-Rate represents the reduction ratio of the amount of code (bit rate) generated to obtain the same quality. In Figure 5, for the "conventional technology method," the reduction ratio of the amount of code generated to obtain the same quality (PSNR or MS-SSIM) is shown for the encoded video that has been reduced in resolution by the conventional technology method, with the encoded video that has not been reduced in resolution used as an anchor. For "this method," the reduction ratio of the amount of generated code required to obtain the same quality is expressed for coded video whose resolution has been reduced using the method described in this embodiment, using coded video whose resolution has been reduced as an anchor.

[0059] The input video was Class A1, a test sequence of JVET. The quantization parameter QP used when encoding the reduced video was set to "22", "27", "32", "37", and "42", and five videos were compared in the experiment.

[0060] As shown in Fig. 5, the "present method" was able to reduce the amount of generated code compared to the "conventional technology method" in both PSNR and MS-SSIM. In other words, the "present method" was able to improve the coding efficiency compared to the "conventional technology method". This also shows that the resolution selected by the method described in this embodiment is the optimal resolution.

[0061] (Another Example 1 of the First Embodiment) In the first embodiment, the multiple encoded video data with different resolutions includes input encoded video data for the input video and reduced encoded video data for the reduced video, but this is not limiting. The multiple encoded video data with different resolutions may include reduced encoded video data without including input encoded video data. In this case, the multiple reduced encoded video data with different resolutions may become multiple encoded video data with different resolutions. In this case, the video encoding unit 120 may encode the video data of the multiple reduced videos output from the video reducing unit 110 without encoding the input video.

[0062] (Another Example 2 of the First Embodiment) In the first embodiment, an example has been described in which the bit amount of the input encoded video data and the bit amount of the reduced encoded video are calculated by the resolution selection unit 140, but this is not limiting. The bit amount of the input encoded video data and the bit amount of the reduced encoded video may be calculated by the video encoding unit 120. In this case, the video encoding unit 120 may include an input encoded video bit amount calculation unit 141 and a reduced encoded video bit amount calculation unit 142. Alternatively, the input encoded video bit amount calculation unit 141 and the reduced encoded video bit amount calculation unit 142 may be provided between the video encoding unit 120 and the resolution selection unit 140 as bit amount calculation units.

[0063] [Other embodiments] A program may be provided that causes a computer to execute each process performed by the above-mentioned video encoding device 100. The program may be recorded in a computer-readable medium. The computer-readable medium can be used to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. In addition, a circuit that executes each process performed by the above-mentioned video encoding device 100 may be integrated, and the device may be configured as a semiconductor integrated circuit (chip set, SoC). That is, the video reduction unit 110, the video encoding unit 120, the video enlargement unit 130, and the resolution selection unit 140 may be executed by one processor. In this case, the processor and the memory may be included in the video encoding device 100, and the processor may read and execute the program stored in the memory to realize the functions of the video reduction unit 110, the video encoding unit 120, the video enlargement unit 130, and the resolution selection unit 140. The processor may be a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or anything that can be realized as a control unit.

[0064] Although the embodiments have been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes can be made without departing from the scope of the invention. In addition, each operation example, each process, or each step can be combined without any contradiction. [Explanation of symbols]

[0065] 100: Video encoding device 110: Video reduction unit 120: Video encoding unit 121: Input video encoding unit 122: Reduced image encoding unit 130: Image enlargement unit 140: Resolution selection unit 141: Input coding image bit amount calculation unit 142: Reduced coding image bit amount calculation unit 143: Selection processing unit

Claims

1. an image reducing unit that reduces the resolution of an input image and outputs image data of a plurality of images having different resolutions; a video encoding unit that encodes each of the video data of the plurality of videos having different resolutions in parallel and outputs the video data of the plurality of encoded videos; a resolution selection unit that selects, for each of the encoded video images, the resolution at which encoding distortion of the encoded video images is minimized when a bit rate of the encoded video image data is a target rate, as an optimal resolution, and outputs the encoded video image data of the selected resolution. Video encoding device.

2. The resolution selection unit selects the resolution of the encoded video at which a rate-distortion cost indicating encoding distortion of the encoded video when a bit amount of video data of the encoded video is set to a target rate is minimized.

2. The video encoding device according to claim 1.

3. a video enlargement unit that enlarges a resolution of the plurality of encoded videos to a resolution of the input video and outputs video data of the plurality of enlarged encoded videos; The rate-distortion cost is [0030] where i is the i-th resolution, λ is a constant, Ri is the bit amount of video data of the encoded video at the i-th resolution, Xori is the input video, Xi is the expanded encoded video at the i-th resolution, and dist is the encoding distortion of the expanded encoded video Xi at the i-th resolution with respect to the input video Xori.

3. The video encoding device according to claim 2.

4. The plurality of pieces of video data with different resolutions include the video data of the input video and the video data output from the video reduction unit.

2. The video encoding device according to claim 1.

5. A process of reducing the resolution of an input video and outputting video data of a plurality of videos having different resolutions; A process of encoding each of the video data of the plurality of videos having different resolutions in parallel and outputting the video data of the plurality of encoded videos; selecting, for each of the encoded video images, the resolution at which encoding distortion of the encoded video images is minimized when the bit rate of the video data of the encoded video images is a target rate, as an optimal resolution, and outputting the video data of the encoded video images at the selected resolution. Video encoding program.

Images