Multilayer encoding device and program

Abstract

To properly assign a bit rate to each layer according to an input video while suppressing an encoding throughput from increasing.SOLUTION: A multilayer encoding device 1 comprises an encoding part 10 which encodes a plurality of input videos through multilayer encoding, and a control part 20 which determines an assigned bit rate of each layer of the multilayer encoding among a plurality of previously specified bit rates. The control part 20 determines, for each frame group to be encoded in each layer, an assigned bit rate of each layer based upon a result of encoding of a frame of interest at a predetermined specified bit rate.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a multi-layer encoding device and a program. [Background technology]

[0002] VVC (Versatile Video Coding), the latest video encoding method, specifies a multi-layer encoding function (specifically, the Multilayer Main 10 profile) that can efficiently encode multiple videos of different resolutions, and services using this function are expected to become widespread.

[0003] In spatial scalable coding, which is an example of multi-layer coding, low-resolution video is coded in the base layer (BL), and the difference between a predicted image (inter-layer predicted image) from the BL coded image and a high-resolution original image is coded in the enhancement layer (EL). In this way, multi-layer coding can efficiently code video at multiple resolutions by performing inter-layer prediction to suppress the generation of redundant information (i.e., information derived from low-frequency components that are common to both low- and high-resolution video).

[0004] Here, since the accuracy of inter-layer prediction determines the performance of multi-layer coding, it is desirable to code the BL at a high bit rate with high image quality and perform inter-layer prediction with high accuracy. However, no matter how high the image quality of the BL is coded, it is impossible to code high-frequency components (detailed components) that do not exist in low-resolution video. The EL codes these detailed components (and low-frequency components that could not be coded by the BL). In order to restore the fine detail that is unique to high-resolution video, it is desirable to code the EL at a high bit rate as well.

[0005] However, the bandwidth (bit rate) of the transmission path available for video transmission services is limited. Therefore, in order to realize high-quality multi-layer coding, it is necessary to allocate the available bit rate appropriately to each layer (BL, EL).

[0006] For example, if BL is coded at a high bit rate, the efficiency of inter-layer prediction will improve, but the EL bit rate for the fine components will be insufficient, and the fine details may not be restored. Conversely, if EL is coded at a high bit rate, the efficiency of inter-layer prediction will decrease, making it impossible to reduce the amount of redundant information, and the efficiency of multi-layer coding may decrease. In addition, the bit rate for low-resolution images may be insufficient, and sufficient image quality may not be obtained for low-resolution images. In order to prevent such degradation, the issue of how to appropriately allocate bits to each layer is an issue.

[0007] In Non-Patent Document 1, by comparing the coding efficiency by changing the bit rate ratio of BL and EL, it is found that the optimal EL bit rate is 5 to 15% of the total. However, Non-Patent Document 1 investigates using video that is relatively easy to code and at low bit rates of around 1 to 5 Mbps, so the obtained results may not necessarily hold for video that is difficult to code or under high bit rate conditions.

[0008] In order to solve this problem, Non-Patent Document 2 shows that the optimal allocation bit rate differs depending on the encoding difficulty by examining the optimal allocation bit rate using various videos. t The allocated bit rate in [Mbps] is as follows:

[0009] Specifically, in the case of difficult-to-encode images, there is no room to restore the high-definition components of the EL, so the BL bit rate (R BL ) (i.e., R BL =R t On the other hand, for images that are easy to encode, R is used so that the fine components can be encoded by EL. BL =min(7,0.8R t ) Note that the EL bit rate is Rt-R BL It can be found from. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] Marquant, G., Salmon-Legagneur, C., Urban, F., and de Lagrange, P., “Spatial scalability with VVC: coding performance and complexity”, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 2022, vol. 12226. doi:10.1117 / 12.2632821. [Non-Patent Document 2] Yuichi Kondo et al., "Study on the method of determining the amount of allocated code in VVC multi-layer coding," PCSJ / IMPS 2022 Symposium Proceedings, November 28, 2022 Summary of the Invention [Problem to be solved by the invention]

[0011] Since Non-Patent Document 2 proposes an optimal allocation bit rate according to the encoding difficulty, it is believed that by adaptively switching the allocation bit rate of each layer for the input video, encoding can be performed more efficiently than the method of Non-Patent Document 1. However, since the encoding difficulty of the input video cannot be known until encoding is performed, two-pass encoding may be necessary. In two-pass encoding, the input image is encoded at the above two bit rates, and the bit rate that results in higher image quality is selected.

[0012] Although this type of two-pass coding method is expected to realize appropriate bitrate allocation according to the input video, it has the problem that the amount of coding processing increases by about 2 times. In particular, the latest coding methods such as VVC have a significantly increased amount of coding processing compared to conventional coding methods, making it difficult to use all the functions when high-speed processing such as real-time processing is required.

[0013] Therefore, an object of the present invention is to provide a multi-layer encoding device and program capable of appropriately allocating a bit rate to each layer according to an input video while suppressing an increase in the amount of encoding processing. [Means for solving the problem]

[0014] A multi-layer encoding device according to a first aspect includes an encoding means for encoding a plurality of input videos by multi-layer encoding, and a control means for determining an allocation bit rate for each layer in the multi-layer encoding from among a plurality of designated bit rates that are designated in advance. The control means determines the allocation bit rate for each group of frames to be encoded in each layer based on a result of encoding a frame of interest at a predetermined designated bit rate.

[0015] A program according to the second aspect causes a computer to function as the multi-layer encoding device according to the first aspect. Effect of the Invention

[0016] According to the present invention, it is possible to provide a multi-layer encoding device and program capable of appropriately allocating a bit rate to each layer according to an input video while suppressing an increase in the amount of encoding processing. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating an example of a video transmission system according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing a configuration of a multi-layer encoding device according to an embodiment. [Diagram 3] FIG. 2 is a diagram showing a multi-layer coding structure of a GOP to be coded. [Figure 4] FIG. 1 is a diagram illustrating an example of an operation flow of a multi-layer encoding device according to an embodiment. [Diagram 5] FIG. 11 is a diagram for explaining a first modified example of the embodiment. [Figure 6]FIG. 13 is a diagram for explaining bit rates allocated for high difficulty and low difficulty according to a third modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] A multi-layer encoding device according to an embodiment will be described with reference to the drawings. The multi-layer encoding device encodes video (i.e., moving images) as represented by MPEG. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0019] (1) Overview of multi-layer coding First, an overview of multi-layer coding according to this embodiment will be described with reference to Fig. 1. Note that multi-layer coding is also called hierarchical coding or scalable coding.

[0020] The video coding method compatible with the multilayer coding may be, for example, the Multilayer Main 10 profile of VVC or the Scalable Main 10 profile of HEVC (High Efficiency Video Coding). In this embodiment, an example using the Multilayer Main 10 profile of VVC will be mainly described.

[0021] 1 is a diagram showing an example of a video transmission system according to this embodiment. The video transmission system shown in FIG.

[0022] The multi-layer encoding device 1 generates a multi-layer bitstream by encoding a plurality of input videos by multi-layer encoding. The multi-layer bitstream includes a base layer (BL) bitstream and an enhancement layer (EL) bitstream. The multi-layer encoding device 1 transmits the multi-layer bitstream to the multi-layer decoding device 2 via a transmission path. The multi-layer decoding device 2 decodes videos from the multi-layer bitstream and outputs the decoded videos.

[0023] In such a multi-layer coding technique, the multi-layer coding device 1 codes a plurality of input images to be coded in a layer structure. Specifically, the multi-layer coding device 1 first codes the lowest layer, BL, and uses the decoded image of the BL for prediction in the coding process of the higher layer, EL, thereby improving the prediction efficiency of the EL, and thus improving coding efficiency compared to the case where each image is coded independently.

[0024] In this way, the multi-layer encoding device 1 transmits the BL as a basic stream and the difference video as the EL. The multi-layer decoding device 2 can decode the BL alone, as well as the BL and EL in combination. According to the multi-layer encoding, for example, while high-resolution or high-quality video is reproduced when bitstreams of multiple layers are used for decoding, low-resolution or low-quality video can be reproduced by decoding only the bitstreams of some layers (BL).

[0025] The first application example of multi-layer coding is SNR scalability, which realizes video services of different qualities (low quality, high quality) by transmitting EL for improving quality to BL. The second application example is spatial scalability, which transmits EL (high resolution video) for improving resolution to BL (low resolution video). With spatial scalability, it is possible to efficiently transmit videos of different resolutions simultaneously, such as a set of 2K video and 4K video, or a set of 4K video and 8K video.

[0026] In this embodiment, an example of using spatial scalability by multi-layer coding will be mainly described. Specifically, the multi-layer coding device 1 codes each low-resolution image constituting a low-resolution video as a BL, and codes each low-resolution image constituting a high-resolution video as an EL using a decoded image (reconstructed image) of the low-resolution image. For example, the multi-layer coding device 1 may code a 2K resolution image as a BL, perform a prediction process of a 4K resolution image using the 2K resolution image, and code difference information between the high-resolution image and the predicted image together with the prediction information as an EL. That is, the multi-layer coding device 1 codes the difference between the high-resolution image and an inter-layer predicted image, which is a predicted image from the BL image, in the EL. By performing such inter-layer prediction, it is possible to suppress the generation of redundant information and efficiently code videos of multiple resolutions.

[0027] Here, since the accuracy of inter-layer prediction determines the performance of multi-layer coding, it is desirable to code the BL at a high bit rate with high image quality and perform inter-layer prediction with high accuracy. However, no matter how high the image quality of the BL is coded, it is impossible to code high-frequency components (detailed components) that do not exist in low-resolution video. The EL codes these detailed components (and low-frequency components that could not be coded by the BL). In order to restore the fine detail that is unique to high-resolution video, it is desirable to code the EL at a high bit rate as well.

[0028] However, the bandwidth (bit rate) of the transmission path available for video transmission services is limited. Therefore, in order to realize high-quality multi-layer coding, it is necessary to allocate the available bit rate appropriately to each layer (BL, EL).

[0029] In addition, in the video transmission system according to the present embodiment, RTP (Realtime Transport Protocol) or MMT (MPEG Media Transport) may be adopted as the transmission method. The transmission path may be an optical transmission path or a wireless transmission path. Although not particularly limited, the transmission path may be a broadcast transmission path or a communication transmission path.

[0030] (2) Configuration of multi-layer coding device Next, the configuration of the multi-layer encoding device 1 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing the configuration of the multi-layer encoding device 1 according to this embodiment.

[0031] The multi-layer encoding device 1 shown in FIG. 2 includes an encoding unit 10 and a control unit 20. The encoding unit 10 encodes a plurality of input videos by multi-layer encoding. The control unit 20 controls the encoding unit 10. Here, the control unit 20 determines the allocation bit rate of each layer in the multi-layer encoding from among a plurality of designated bit rates designated in advance. Specifically, the control unit 20 determines the allocation bit rate of each layer based on the result of encoding a target frame at a predetermined designated bit rate for each frame group to be encoded in each layer. In this embodiment, the frame group is a GOP (Group Of Pictures). The GOP may be a collection of pictures (frames) including at least one I-picture.

[0032] This allows us to determine the bit rate allocated to each layer by encoding only the frame of interest for each GOP, and does not require two-pass encoding (encoding twice) for all frames. Therefore, compared to the method using two-pass encoding, the increase in the amount of encoding processing can be suppressed.

[0033] In addition, by determining the bit rate allocated to each layer based on the encoding result of the frame of interest, it is possible to adaptively switch the bit rate allocated to each layer to the input video. Furthermore, it is possible to adaptively switch the bit rate allocated to each layer on a GOP basis.

[0034] Therefore, according to this embodiment, it is possible to provide a multi-layer encoding device 1 that is capable of appropriately allocating the bit rate of each layer according to the input video while suppressing an increase in the amount of encoding processing.

[0035] In this embodiment, the frame of interest is the first frame to be coded in each GOP of each layer. The frame of interest may be an I-picture (Intra Picture). For example, in the case of a two-layer configuration of BL and EL, in order to determine the allocated bit rate of each layer, two frames, the first frame to be coded in BL and the first frame to be coded in EL, may be coded in the GOP.

[0036] In this embodiment, the multiple input images include a low-resolution image corresponding to a first image having a predetermined resolution, and a high-resolution image corresponding to a second image having a resolution higher than the predetermined resolution (see FIG. 1). For example, the low-resolution image may be a 2K image, and the high-resolution image may be a 4K image. The encoding unit 10 uses spatial scalability to encode the low-resolution image as a first layer (BL in this embodiment) and encode the high-resolution image as a second layer (EL in this embodiment).

[0037] In this embodiment, the number of ELs is one, and the multi-layer encoding device 1 performs multi-layer encoding of two layers. However, the number of ELs is not limited to two-layer multi-layer encoding, and the encoding unit 10 may perform multi-layer encoding of three or more layers. For example, the encoding unit 10 may use spatial scalability to encode low-resolution video (e.g., 2K video) as BL, high-resolution video (e.g., 4K video) as EL1, and super-high-resolution video (e.g., 8K video) as EL2. Even in this case, the control unit 20 determines the allocation bit rate of each layer based on the result of encoding the frame of interest at a predetermined designated bit rate for each GOP of each layer (BL, EL1, EL2).

[0038] The predetermined designated bit rates include a first designated bit rate where the BL bit rate is set to a first bit rate, and a second designated bit rate where the BL bit rate is set to a second bit rate higher than the first bit rate. The predetermined designated bit rate is a designated bit rate that is preferentially applied to the encoding of the target frame, and is one of the plurality of designated bit rates.

[0039] The first designated bit rate is a designated bit rate suitable for video with high encoding difficulty because the BL bit rate is higher and the EL bit rate is lower than the second designated bit rate. Such a first designated bit rate is called a "high difficulty allocation bit rate."

[0040] In contrast, the second designated bit rate is a designated bit rate suitable for video with low encoding difficulty because the BL bit rate is lower and the EL bit rate is higher than the first designated bit rate. Such a second designated bit rate is called a "low-complexity allocated bit rate."

[0041] For example, the total bit rate (target bit rate) of the multi-layer bitstream to be transmitted is R tWhen the bit rate is set to [Mbps], the high-definition component of the EL cannot be restored with the bit rate allocated for high difficulty, so the BL bit rate R BL Allocate all bits to R BL =R t On the other hand, in the low difficulty allocation bit rate, R is set so that the fine details can be encoded by EL. BL =min(7,0.8R t ) The EL bit rate is Rt-R BL It can be found from.

[0042] In addition, the BL bit rate R for the low difficulty allocation bit rate BL is 7[Mbps] and 0.8R t This means that, based on the assumption that the required bit rate for 2K resolution video is 7 Mbps, the minimum allocated bit rate for low difficulty is guaranteed for BL.

[0043] However, these designated bit rates are merely examples and may be changed as appropriate. For example, in the case of a high difficulty level allocation bit rate, R EL = 0. For example, R EL Set the minimum value of to 100 [kbps], for example, and BL =R t -100[kbps] may be used. In addition, in the bit rate allocation for low difficulty, BL =min(7,0.8R t ) is not limited to the above. For example, the method described in Non-Patent Document 1 can be used to BL =R t × 0.85. The values ​​such as 7, 0.8, and 0.85 that appear in these formulas may be appropriately changed and applied.

[0044] In this embodiment, the designated bit rates specified in advance are two, a high difficulty allocation bit rate and a low difficulty allocation bit rate. However, the designated bit rates specified in advance are not limited to two, and may be three or more. For example, in addition to the high difficulty allocation bit rate and the low difficulty allocation bit rate, an intermediate allocation bit rate between these may be added as an "intermediate difficulty allocation bit rate."

[0045] (2.1) Encoding section Next, the encoding unit 10 according to the present embodiment will be described with reference to Fig. 2. The configuration of the encoding unit 10 itself is similar to a conventional configuration compatible with, for example, the Multilayer Main 10 profile of VVC, but an overview of the configuration will be described here.

[0046] As shown in Fig. 2, the encoding unit 10 may be capable of simultaneously encoding input images of each layer (in this embodiment, low resolution images and high resolution images). Alternatively, the encoding unit 10 shown in Fig. 2 may be provided for each layer. For example, an encoding unit 10 for low resolution images and an encoding unit 10 for high resolution images may be provided, and these two encoding units 10 may operate in cooperation to perform inter-layer prediction.

[0047] The encoding unit 10 generates a bit stream by encoding each image (frame (picture)) constituting an input video, and outputs the generated bit stream. The encoding unit 10 has a block division unit 100, a subtraction unit 110, a transform / quantization unit 120, an entropy encoding unit 130, an inverse quantization / inverse transform unit 140, a synthesis unit 150, a memory 160, and a prediction unit 170.

[0048] The block division unit 100 divides an original image in units of frames (pictures) constituting an input video (moving image) into a plurality of image blocks, and outputs the image blocks obtained by division to the subtraction unit 110. The size of the image block is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels. The shape of the image block is not limited to a square, but may be a rectangle (non-square). Such an image block may be called a coding block (CU).

[0049] The subtraction unit 110 calculates a prediction residual representing a difference (error) between the coded block output by the block division unit 100 and a predicted block obtained by predicting the coded block by the prediction unit 170. Specifically, the subtraction unit 110 calculates the prediction residual by subtracting each pixel value of the predicted block from each pixel value of the block, and outputs the calculated prediction residual to the transformation / quantization unit 120.

[0050] The transform / quantization unit 120 performs transform processing and quantization processing on a block-by-block basis. The transform / quantization unit 120 includes a transform unit 121 and a quantization unit 122.

[0051] The transform unit 121 performs a transform process on the prediction residual output by the subtraction unit 110 to calculate a transform coefficient, and outputs the calculated transform coefficient to the quantization unit 122. The transform refers to, for example, a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen Loeve transform (KLT), etc. The transform process includes a transform skip in which no transform process is performed. The transform skip also includes a transform in which a transform process is applied only horizontally, or a transform in which a transform process is applied only vertically. The transform unit 121 may also perform a secondary transform process in which a further transform process is applied to the transform coefficient obtained by the transform process. The secondary transform process may be applied only to a partial area of ​​the transform coefficient.

[0052] The quantization unit 122 quantizes the transform coefficients output by the transform unit 121 using a quantization parameter (QP) and a quantization matrix, and outputs the quantized transform coefficients, which are the quantized transform coefficients, to the entropy coding unit 130 and the inverse quantization and inverse transform unit 140. Note that the quantization parameter (QP) is a parameter commonly applied to each transform coefficient in a block, and is a parameter that determines the coarseness of quantization. The quantization matrix is ​​a matrix having quantization values ​​for quantizing each transform coefficient as elements.

[0053] The entropy coding unit 130 performs entropy coding on the quantized transform coefficients output by the quantization unit 122, compresses the data, generates a bit stream, and outputs the bit stream to the outside of the multi-layer coding device 1. For the entropy coding, Huffman codes, CABAC (Context-based Adaptive Binary Arithmetic Coding), etc. may be used. Note that the entropy coding unit 130 receives input of prediction-related information (flags and indexes) from the prediction unit 170, and also codes the input information and outputs the bit stream.

[0054] The inverse quantization and inverse transform unit 140 performs inverse quantization processing and inverse transform processing on a block-by-block basis. The inverse quantization and inverse transform unit 140 includes an inverse quantization unit 141 and an inverse transform unit 142.

[0055] The inverse quantization unit 141 performs an inverse quantization process corresponding to the quantization process performed by the quantization unit 122. Specifically, the inverse quantization unit 141 reconstructs the transform coefficients by inverse quantizing the quantized transform coefficients output by the quantization unit 122 using a quantization parameter and a quantization matrix, and outputs the reconstructed transform coefficients to the inverse transform unit 142.

[0056] The inverse transform unit 142 performs an inverse transform process corresponding to the transform process performed by the transform unit 121. For example, when the transform unit 121 performs a discrete cosine transform, the inverse transform unit 142 performs an inverse discrete cosine transform. The inverse transform unit 142 performs an inverse transform process on the transform coefficients output by the inverse quantization unit 141 to reconstruct a prediction residual, and outputs the reconstructed prediction residual, which is the reconstructed prediction residual, to the synthesis unit 150.

[0057] The synthesis unit 150 synthesizes the reconstructed prediction residual output by the inverse transform unit 142 by adding it to the predicted block output by the prediction unit 170 on a pixel-by-pixel basis. The synthesis unit 150 adds each pixel value of the reconstructed prediction residual to each pixel value of the predicted block to decode (reconstruct) the block, and outputs the reconstructed block to the memory 160.

[0058] The memory 160 stores the reconstructed blocks output by the synthesis unit 150, and accumulates the reconstructed blocks as decoded images (reconstructed images) on a frame-by-frame basis. The memory 160 outputs the stored reconstructed blocks or decoded images to the prediction unit 170. Note that a loop filter may be provided between the synthesis unit 150 and the memory 160.

[0059] The prediction unit 170 performs prediction on a block-by-block basis. The prediction unit 170 includes an inter prediction unit 171, an intra prediction unit 172, and a switching unit 173.

[0060] The inter prediction unit 171 uses the decoded image stored in the memory 160 as a reference image, calculates a motion vector by a method such as block matching, predicts a coding block to generate an inter prediction block, and outputs the generated inter prediction block to the switching unit 173. Here, the inter prediction unit 171 selects an optimal inter prediction method from inter prediction using multiple reference images (typically, bi-prediction) and inter prediction using one reference image (unidirectional prediction), and performs inter prediction using the selected inter prediction method. The inter prediction unit 171 outputs information related to the inter prediction (motion vector, etc.) to the entropy coding unit 130.

[0061] The intra prediction unit 172 generates an intra prediction block by referring to decoded pixels around a block among the decoded images stored in the memory 160, and outputs the generated intra prediction block to the switching unit 173. In general, the intra prediction unit 172 selects an intra prediction mode to be applied to a predictive coding block of intra prediction from among a plurality of intra prediction modes, and predicts the coding block of intra prediction using the selected intra prediction mode. The intra prediction unit 172 outputs information related to the selected intra prediction mode to the entropy coding unit 130.

[0062] The switching unit 173 switches between the inter predicted block output by the inter prediction unit 171 and the intra predicted block output by the intra prediction unit 172, and outputs either the predicted block to the subtraction unit 110 and the synthesis unit 150.

[0063] (2.2) Control section Next, the control unit 20 according to the present embodiment will be described with reference to FIG.

[0064] The control unit 20 determines the allocation bit rate to be applied to the current GOP based on the result of encoding a frame of interest selected from the current GOP to be encoded at a specified bit rate. In this embodiment, the control unit 20 determines the encoding difficulty of the current GOP from the encoding result of the frame of interest selected from the current GOP, and determines whether to apply the high difficulty allocation bit rate or the low difficulty allocation bit rate to the current GOP.

[0065] For example, the control unit 20 determines whether the coding difficulty of the current GOP is high or low based on the result of coding the frame of interest of the current GOP at a low difficulty allocation bit rate (a predetermined designated bit rate), and when it is determined that the coding difficulty is high, controls the coding unit 10 to switch to a high difficulty allocation bit rate and code (re-code) the current GOP. On the other hand, when it is determined that the coding difficulty of the current GOP is low, the control unit 20 controls the coding unit 10 to maintain the low difficulty allocation bit rate and code the current GOP.

[0066] In the following description of the embodiment, the low difficulty allocation bit rate is used as the predetermined designated bit rate preferentially applied when encoding the frame of interest. As a result, when the control unit 20 determines that the encoding difficulty of the current GOP is low based on the encoding result of the frame of interest, it can continue encoding the current GOP by maintaining the low difficulty allocation bit rate without re-encoding the frame of interest.

[0067] However, the predetermined designated bit rate applied preferentially may be a high difficulty allocation bit rate. For example, the control unit 20 may determine whether the coding difficulty of the current GOP is high or low based on the result of coding the target frame of the current GOP at the high difficulty allocation bit rate (predetermined designated bit rate), and when it is determined that the coding difficulty is low, control the coding unit 10 to switch to the low difficulty allocation bit rate and code (re-code) the current GOP. On the other hand, when it is determined that the coding difficulty of the current GOP is high, the control unit 20 may control the coding unit 10 to maintain the high difficulty allocation bit rate and code the current GOP.

[0068] In this way, the control unit 20 determines the allocation bit rate for each layer from among a plurality of designated bit rates (allocation bit rate for low difficulty and allocation bit rate for high difficulty) based on the encoding result of the frame of interest for each GOP.

[0069] As shown in FIG. 2, the control unit 20 includes a target frame encoding processing unit 21, a picture quality index calculation unit 22, a switching processing unit 23, and a bit rate control unit 24.

[0070] The attention frame encoding processing unit 21 controls the encoding unit 10 to encode the attention frame at a predetermined designated bit rate (in this embodiment, the allocated bit rate for low difficulty) for each layer to generate a reconstructed image of the attention frame. For example, the attention frame encoding processing unit 21 controls the encoding unit 10 to encode the attention frame that is encoded first in the GOP at the allocated bit rate for low difficulty, and obtains the reconstructed image, which is the encoding result of the attention frame, from the memory 160.

[0071] The image quality index calculation unit 22 calculates an image quality index of a reconstructed image for each layer according to a reconstructed image of a frame of interest and an original image of the frame of interest. For example, the image quality index calculation unit 22 calculates an image quality index of BL according to an image obtained by upscaling a reconstructed image of a frame of interest in BL to an EL resolution and an original image of the corresponding EL. The image quality index calculation unit 22 also calculates an image quality index of EL according to a reconstructed image of a frame of interest in EL and an original image of the corresponding EL.

[0072] In this embodiment, the peak signal-to-noise ratio (PSNR) is used as the image quality index. However, the image quality index is not limited to the PSNR, and other image quality indexes (for example, MSE, SSIM, VMAF, etc.) may be used.

[0073] The switching processing unit 23 determines whether to continue encoding the current GOP using a predetermined designated bit rate (allocated bit rate for low difficulty) or to switch to another designated bit rate (allocated bit rate for high difficulty) and encode (re-encode) the current GOP by a comparison process that compares the image quality index of each layer. The comparison process may be a process that compares the difference between the image quality index of BL and the image quality index of EL with a threshold value.

[0074] For example, when the difference between the image quality index of BL and the image quality index of EL is larger than the threshold, that is, when the image quality index of EL is improved to some extent with respect to the image quality index of BL, it means that the coding difficulty of the current GOP is low and there is room to code the high-definition component. In this case, the switching processing unit 23 determines to code the current GOP while maintaining the allocated bit rate for low difficulty.

[0075] On the other hand, if the difference between the image quality index of BL and the image quality index of EL is equal to or less than the threshold, that is, if the image quality index of EL is not significantly improved relative to the image quality index of BL, it means that the coding difficulty of the current GOP is high and there is no room to code the high-definition components. In this case, the switching processing unit 23 decides to switch to the allocation bit rate for high difficulty and code (re-code) the current GOP.

[0076] After the switching processing unit 23 determines the allocated bit rate for each layer, the bit rate control unit 24 controls the encoding unit 10 so that the bit rate (consumed bit rate) after the actual encoding process does not exceed the determined allocated bit rate. Specifically, the control unit 20 allocates a bit rate for each frame so that it does not exceed the determined allocated bit rates for BL and EL.

[0077] Here, the bit rate changes depending on the quantization parameter (QP) used in the quantization unit 122. Therefore, the bit rate control unit 24 selects a QP that maximizes image quality from among QPs that do not exceed the determined allocated bit rate as an optimal QP, and sets the optimal QP in the quantization unit 122. In general, the optimal QP is estimated from the QP and consumed bit rate in an encoded frame or encoded block.

[0078] (3) An example of the operation of a multi-layer coding device Next, an example of the operation of the multi-layer encoding device 1 according to the present embodiment will be described with reference to Fig. 3 and Fig. 4. In the following description of the embodiment, the encoding structure is set to random access, the GOP size and the intra period are both set to 8, and the target bit rate is set to Rt.

[0079] Fig. 3 is a diagram showing the multi-layer coding structure of the GOP to be coded. The lower row is BL, and the upper row is EL. As indicated by the dotted line, the EL can refer to the coded image (reconstructed image) of the BL. "I", "P", and "B" written in each frame mean I picture, P picture, and B picture, respectively. The numbers written in each frame indicate the coding order, and BL coding and EL coding are processed in order for each frame.

[0080] First, the encoding unit 10 encodes the first frame of BL, 0 After encoding, I 0 Refer to the reconstructed image of , and select the first frame of EL, P 1 is coded by inter-layer prediction. At this time, the only thing that is actually referenced is I 0 This is an image obtained by upscaling the reconstructed image to the EL resolution (EL resolution).

[0081] Similarly, the encoding unit 10 encodes the second frame of BL, I 2 After encoding, I 2 The second frame of EL, P, is 3 After that, the encoding unit 10 encodes B4, B5, and so on in that order.

[0082] Among these, the current GOP's focus frame is I 2 , P 3 In addition, I 0 , P 1 is the frame of interest in the immediately preceding GOP that has already been coded.

[0083] FIG. 4 is a diagram showing an example of an operation flow of the multi-layer encoding device 1 according to this embodiment.

[0084] In step S1, the target frame encoding processing unit 21 encodes a target frame I of the BL of the current GOP. 2 The allocation bit rate for low difficulty, specifically, the BL bit rate for low difficulty (R BL =min(7,0.8R t )) and the I generated by the encoding is controlled to be encoded by the encoding unit 10. 2 A reconstructed image is obtained.

[0085] In step S2, the image quality index calculation unit 22 calculates the I 2 From the reconstructed image of I upscaled to EL resolution and the corresponding original EL image, 2 PSNR is an index of image quality 2 Calculate.

[0086] In step S3, the attention frame encoding processing unit 21 encodes P 3 The allocation bit rate for low difficulty, specifically, the EL bit rate for low difficulty (R EL =R t -R BL ) and controls the encoding unit 10 to encode the P 3 A reconstructed image is obtained.

[0087] In step S4, the image quality index calculation unit 22 calculates the P 3 From the reconstructed image of and the corresponding original image of EL, P 3 PSNR is an index of image quality 3 Calculate.

[0088] In step S5, the switching processing unit 23 calculates the PSNR 3 The PSNR calculated in step S2 3 By subtracting ΔPSNR (=PSNR 3 -PSNR 2 ) is calculated.

[0089] In step S6, the switching processor 23 determines whether the ΔPSNR calculated in step S5 is equal to or greater than a switching index s (e.g., s=1 [dB]). The switching index s is a threshold for determining whether the image quality index of the EL is improved to a certain degree relative to the image quality index of the BL, that is, the coding difficulty of the current GOP.

[0090] If the ΔPSNR calculated in step S5 is equal to or greater than the switching index s (step S6: YES), it indicates that the encoding difficulty is low and there is room to encode the high-definition component. In this case, in step S7, the switching processing unit 23 allocates a low-difficulty bit rate (R BL =min(7,0.8R t ), R EL =R t -R BL ) is determined to be maintained, and the bit rate allocated for low difficulty is B 4 From then on (B 4 ~B 17 ) is encoded.

[0091] On the other hand, if the ΔPSNR calculated in step S5 is less than the switching index s (step S6: NO), it indicates that the encoding difficulty is high and there is no room to encode the high-definition component. In this case, in step S8, the switching processing unit 23 changes the allocated bit rate to the high-difficulty allocated bit rate (R BL =R t , R EL Then, in step S9, the switching processing unit 23 determines to switch to I 2 and P 3 At step S10, the switching processing unit 23 controls the re-encoding of B at the bit rate allocated for high difficulty. 4 From then on (B 4 ~B 17 ) is encoded.

[0092] In this operation example, the designated bit rates specified in advance are two: a high difficulty allocation bit rate and a low difficulty allocation bit rate. However, if there are three or more designated bit rates specified in advance, multiple switching indicators may be defined.

[0093] (4) Example of changes Next, a modification of the above embodiment will be described.

[0094] (4.1) First Change Example In the above embodiment, an example has been described in which the predetermined designated bit rate preferentially applied to the encoding of the target frame is fixed to the low difficulty allocation bit rate or the high difficulty allocation bit rate. However, the predetermined designated bit rate may be variably set.

[0095] 5 is a diagram for explaining a first modification of the embodiment. In this modification, the control unit 20 (target frame encoding processing unit 21) uses the allocated bit rate applied to the GOP (already encoded GOP) immediately preceding the current GOP as the predetermined designated bit rate to be preferentially applied to encoding the target frame. This allows the predetermined designated bit rate to be variably set according to the immediately preceding GOP.

[0096] For example, if the previous GOP is coded at a bit rate allocated for high difficulty, the control unit 20 controls the current GOP to code the frame of interest at the bit rate allocated for high difficulty. On the other hand, if the previous GOP is coded at a bit rate allocated for low difficulty, the control unit 20 controls the current GOP to code the frame of interest at the bit rate allocated for low difficulty.

[0097] Since the characteristics of frames that are close in time are similar, it can be said that the characteristics of the previous GOP and the characteristics of the current GOP are similar. Therefore, by encoding the frame of interest using the assigned bit rate applied to the GOP immediately before the current GOP, the frequency of re-encoding the frame of interest is reduced, and the amount of encoding processing can be reduced.

[0098] (4.2) Second Change Example In the above embodiment, an example has been described in which the control unit 20 (switching processing unit 23) determines the allocation bit rate to be applied to the current GOP based on the result of encoding a frame of interest selected from the current GOP to be encoded at a specified bit rate. However, when switching the allocation bit rate of the current GOP according to the encoding result of the frame of interest of the current GOP, it is necessary to re-encode the frame of interest.

[0099] In this modification, the control unit 20 determines the allocation bit rate to be applied to the current GOP based on the result of encoding a frame of interest selected from the GOP (encoded GOP) immediately preceding the current GOP to be encoded. For example, when a low-difficulty allocation bit rate is selected based on the result of encoding a frame of interest in the immediately preceding GOP, the control unit 20 determines to apply the low-difficulty allocation bit rate to the current GOP. On the other hand, when a high-difficulty allocation bit rate is selected based on the result of encoding a frame of interest in the immediately preceding GOP, the control unit 20 determines to apply the high-difficulty allocation bit rate to the current GOP.

[0100] As described above, the characteristics of the previous GOP and the characteristics of the current GOP are similar. Therefore, by determining the allocated bit rate of the current GOP using the frame of interest in the previous GOP, the frame of interest does not need to be re-encoded, and the amount of encoding processing can be reduced.

[0101] (4.3) Third Change In the above embodiment, the values ​​of the bit rates allocated for high difficulty and low difficulty may be set freely. For example, a method of determining the values ​​of the bit rates allocated for high difficulty and low difficulty based on the resolution of each of the BL and EL is conceivable. This makes it possible to apply the optimal bit rate allocation according to the resolution of each of the BL and EL.

[0102] FIG. 6 is a diagram for explaining the bit rates allocated for high difficulty and low difficulty according to this modified example.

[0103] In the illustrated example, when the BL resolution is 2K and the EL resolution is 4K, the control unit 20 sets the bit rate allocated for low difficulty as BL=0.85×R t and EL = 0.15 × R t and set the allocation bit rate for high difficulty to BL=R t and set EL=0.

[0104] When the BL resolution is 4K and the EL resolution is 8K, the control unit 20 sets the bit rate allocated for low difficulty as BL=0.7×R t and EL = 0.3 × R t and set the allocation bit rate for high difficulty to BL=R t and set EL=0.

[0105] When the BL resolution is 2K and the EL resolution is 8K, the control unit 20 sets the bit rate allocated for low difficulty as BL=0.5×R t and EL = 0.5 × R t and the allocation bit rate for high difficulty is set to BL=0.8×R t and EL = 0.2 × R t Set to.

[0106] (4.4) Fourth Change In the above embodiment, the frame of interest in the current GOP is I 2 and P 3 However, the frame of interest may be changed depending on the encoding conditions (for example, random access, low delay, intra period, GOP size, etc.).

[0107] Furthermore, a plurality of frames of interest may be set for each layer within one GOP. For example, the first frame to be coded and the second frame to be coded in each layer may be set as frames of interest.

[0108] (4.5) Fifth Change In the above embodiment, an example of spatial scalability that transmits EL (high-resolution video) for improving the resolution of BL (low-resolution video) by multi-layer coding has been described. However, it is not limited to spatial scalability, and SNR scalability that realizes video services of different qualities (low quality, high quality) may be realized by multi-layer coding. In this case, the resolutions of BL and EL, that is, the resolutions of multiple input videos, may be the same.

[0109] (5) Other embodiments In the above embodiment, an example has been described in which each frame group to be coded is one GOP, and the allocated bit rate can be switched for each GOP. However, each frame group to be coded may be a GOP set consisting of two or more GOPs, and the allocated bit rate may be switched for each GOP set.

[0110] Alternatively, each group of frames to be coded may be a "chunk." A chunk is a unit that groups multiple frames together, and is a unit that is divided into bit streams of a certain length when coding video. By grouping a certain number of frames to form a bit stream, the stream can be transmitted and played back more efficiently. Therefore, the GOP in the above-mentioned embodiment may be read as a chunk.

[0111] A program may be provided that causes a computer to execute the processes performed by the multi-layer encoding device 1. The program may be recorded in a computer-readable medium. The computer-readable medium can be used to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Circuits that execute the processes performed by the multi-layer encoding device 1 may be integrated to configure the multi-layer encoding device 1 as a semiconductor integrated circuit (chip set, SoC).

[0112] The terms "based on" and "depending on / in response to" used in this disclosure do not mean "based only on" or "depending only on", unless otherwise specified. The term "based on" means both "based only on" and "based at least in part on". Similarly, the term "depending on" means both "based only on" and "depending at least in part on". The terms "include", "comprise", and variations thereof do not mean including only the recited items, but may include only the recited items or may include additional items in addition to the recited items. Additionally, the term "or" used in this disclosure is not intended to be an exclusive or. Additionally, any reference to elements using designations such as "first", "second", etc., used in this disclosure is not intended to generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some manner. In this disclosure, where articles are added by translation, such as, for example, a, an, and the in English, these articles are intended to include the plural unless the context clearly indicates otherwise.

[0113] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes, etc. are possible without departing from the spirit of the invention.

[0114] (6) Additional Notes The following additional features relate to the above-described embodiment.

[0115] (Appendix 1) An encoding means (10) for encoding a plurality of input videos by multi-layer encoding; A control means (20) for determining an allocation bit rate for each layer in the multi-layer coding from among a plurality of designated bit rates designated in advance, The control means determines the assigned bit rate for each group of frames to be encoded in each layer based on a result of encoding a frame of interest at a predetermined designated bit rate. Multi-layer coding device (1).

[0116] (Appendix 2) The frame of interest is a frame to be coded first in each of the groups of frames in each layer. 2. A multi-layer encoding device as claimed in claim 1.

[0117] (Appendix 3) The control means determines the allocated bit rate to be applied to the current group of frames based on a result of encoding the frame of interest selected from the current group of frames to be encoded at the predetermined designated bit rate. 3. A multi-layer encoding device according to claim 1 or 2.

[0118] (Appendix 4) The control means Controlling, for each of the layers, to encode the frame of interest at the predetermined designated bit rate to generate a reconstructed image of the frame of interest; Calculating an image quality index for each of the layers according to the reconstructed image and a corresponding original image; A comparison process is performed to compare the image quality indices for each layer to determine whether to continue encoding the current group of frames using the specified bit rate or to switch to another specified bit rate to encode the current group of frames. 4. A multi-layer encoding device as claimed in claim 3.

[0119] (Appendix 5) The control means uses the allocated bit rate applied to the group of frames immediately preceding the current group of frames as the predetermined designated bit rate. 5. A multi-layer encoding device according to claim 3 or 4.

[0120] (Appendix 6) The control means determines the allocated bit rate to be applied to the current group of frames based on a result of encoding the frame of interest selected from a group of frames immediately before the current group of frames to be encoded at the predetermined designated bit rate. 3. A multi-layer encoding device according to claim 1 or 2.

[0121] (Appendix 7) the plurality of input images include a first image having a predetermined resolution and a second image having a resolution higher than the predetermined resolution; The encoding means encodes the first video as a first layer and encodes the second video as a second layer; the plurality of designated bit rates include a first designated bit rate in which the bit rate of the first layer is set to a first bit rate, and a second designated bit rate in which the bit rate of the first layer is set to a second bit rate higher than the first bit rate; The predetermined designated bit rate is one of the plurality of designated bit rates. 7. A multi-layer encoding device according to any one of claims 1 to 6.

[0122] (Appendix 8) The control means determines the first bit rate and the second bit rate in accordance with the resolutions of the first video and the second video, respectively. 8. A multi-layer encoding device as claimed in claim 7.

[0123] (Appendix 9) A computer is configured to function as a multi-layer encoding device according to any one of claims 1 to 8. program. [Explanation of symbols]

[0124] 1: Multi-layer coding device 2: Multi-layer decoder 10: Encoding section 20: Control section 21: Target frame encoding processing unit 22: Image quality index calculation unit 23: Switching processing section 24: Bit rate control section 100: Block division section 110: Subtraction section 120: Transformation and quantization section 121: Conversion section 122: Quantization section 130: Entropy coding unit 140: Inverse quantization and inverse transformation unit 141: Inverse quantization section 142: Inverse conversion section 150: Composite section 160: Memory 170: Prediction Department 171: Inter prediction section 172: Intra prediction unit 173: Switching section

Claims

1. An encoding means for encoding a plurality of input videos by multi-layer encoding; A control means for determining an allocation bit rate for each layer in the multi-layer coding from among a plurality of designated bit rates designated in advance, The control means determines the assigned bit rate for each group of frames to be encoded in each layer based on a result of encoding a frame of interest at a predetermined designated bit rate. A multi-layer coding device.

2. The control means determines the allocated bit rate to be applied to the current group of frames based on a result of encoding the frame of interest selected from the current group of frames to be encoded at the predetermined designated bit rate. The multi-layer encoding device according to claim 1 .

3. The control means Controlling, for each of the layers, to encode the frame of interest at the predetermined designated bit rate to generate a reconstructed image of the frame of interest; Calculating an image quality index for each of the layers according to the reconstructed image and a corresponding original image; A comparison process is performed to compare the image quality indices for each layer to determine whether to continue encoding the current group of frames using the specified bit rate or to switch to another specified bit rate to encode the current group of frames. A multi-layer encoding device according to claim 2.

4. The control means uses the allocated bit rate applied to the group of frames immediately preceding the current group of frames as the predetermined designated bit rate. A multi-layer encoding device according to claim 2.

5. The control means determines the allocated bit rate to be applied to the current group of frames based on a result of encoding the frame of interest selected from a group of frames immediately before the current group of frames to be encoded at the predetermined designated bit rate. The multi-layer encoding device according to claim 1 .

6. the plurality of input images include a first image having a predetermined resolution and a second image having a resolution higher than the predetermined resolution; The encoding means encodes the first video as a first layer and encodes the second video as a second layer; the plurality of designated bit rates include a first designated bit rate in which the bit rate of the first layer is set to a first bit rate, and a second designated bit rate in which the bit rate of the first layer is set to a second bit rate higher than the first bit rate; the predetermined designated bit rate is any one of the plurality of designated bit rates, The control means determines the first bit rate and the second bit rate in accordance with the resolutions of the first video and the second video, respectively. The multi-layer encoding device according to claim 1 .

7. A computer is caused to function as the multi-layer encoding device according to claim 1. program.

Images