Video encoding device, video decoding device, video encoding method, video decoding method, and video system

Abstract

This video encoding device 10 comprises: a prediction means 11 that uses an intra-prediction or an inter-prediction to perform a prediction process; and an encoding control means 12 that performs a control such that, on a predetermined condition that no picture be encoded prior to any preceding picture in a display order and that no picture be referred to by the picture of any lower layer, the prediction means 11, when referring to a picture of a lower layer to perform a picture encoding, uses, as a reference picture, a picture that is the closest, in the display order, to a picture that is to be encoded.

Description

[Technical Field]

[0001] The present invention relates to a video encoding device, a video decoding device, a video encoding method, a video decoding method, and a video system. [Background technology]

[0002] In a video content distribution system, for example, on the transmitting side, a video signal is coded based on the H.264 / AVC (Advanced Video Coding) standard or the HEVC (High Efficiency Video Coding) standard, and on the receiving side, the video signal is reproduced after undergoing decoding processing.

[0003] Non-Patent Document 1 introduces the concept of SOP (Structure of Pictures). When temporal hierarchical coding (temporal scalable coding) is performed, SOP is a unit that describes the coding order and reference relationships of each AU (Access Unit). Temporal scalable coding is a coding method that allows partial extraction of frames from a video of multiple frames. Note that one GOP (Group of Pictures) is made up of one or more SOPs.

[0004] Non-Patent Document 1 defines an SOP structure that is applied to video formats other than 120 / P (Progressive) and an SOP structure that is applied to the 120 / P video format.

[0005] Fig. 12 is an explanatory diagram showing an SOP structure applied to a video format other than 120 / P. Fig. 12 shows the SOP structure when reordering (picture reordering) is applied.

[0006] The SOP structure shown in Fig. 12 includes a structure of L=0, a structure of L=1, a structure of L=2, and a structure of L=3. As shown in Fig. 12, Lx (x=0, 1, 2, 3) has the following structure. Temporal ID is identification information indicating the layer of each AU. Hereinafter, the L=x structure may be referred to as an Lx structure. Furthermore, an AU whose Temporal ID is m (a natural number equal to or greater than 0) may be referred to as a picture whose Temporal ID is m.

[0007] L0 structure: An SOP structure consisting only of pictures with a Temporal ID of 0. (In other words, the SOP contains one level (layer) of pictures. In other words, L, which indicates the maximum Temporal ID, is 0.) L1 structure: An SOP structure consisting of pictures with a Temporal ID of 0 and 1. (In other words, the SOP contains two layers. You could also say that L, which indicates the maximum Temporal ID, is 1.) L2 structure: An SOP structure consisting of pictures with Temporal IDs of 0, 1, and 2. (In other words, the SOP contains three layers. You could also say that L, which indicates the maximum Temporal ID, is 2.) L3 structure: An SOP structure consisting of pictures with Temporal IDs of 0, 1, 2, and 3. (In other words, the SOP contains four layers. In other words, the L indicating the maximum Temporal ID is 3.)

[0008] Fig. 13 is an explanatory diagram showing the SOP structure applied to the 120 / P video format. Note that the SOP structure shown in Fig. 13 corresponds to time scalable coding.

[0009] The SOP structure shown in Fig. 13 includes an L0 structure, an L1 structure, an L2 structure, an L3 structure, and an L4 structure. As shown in Fig. 13, the Lx (x=0, 1, 2, 3, 4) structure is the following structure.

[0010] L0 structure: An SOP structure consisting only of pictures with a Temporal ID of 0. (In other words, the SOP contains one layer. In other words, L, which indicates the maximum Temporal ID, is 0.) L1 structure: An SOP structure consisting of pictures with a Temporal ID of 0 and M (i.e., the SOP contains two layers. You could also say that L, which indicates the maximum Temporal ID, is 1 (or M)). L2 structure: An SOP structure consisting of pictures with Temporal IDs of 0, 1, and M. (In other words, the SOP contains three layers. It can also be said that L, which indicates the maximum Temporal ID, is 2 (or M).) L3 structure: An SOP structure consisting of pictures with Temporal IDs of 0, 1, 2, and M. (In other words, the SOP contains four layers. It can also be said that L, which indicates the maximum Temporal ID, is 3 (or M).) L4 structure: An SOP structure consisting of pictures with Temporal IDs of 0, 1, 2, 3, and M. (In other words, the SOP contains five layers. It can also be said that L, which indicates the maximum Temporal ID, is 4 (or M).)

[0011] Non-Patent Document 2 discloses a video coding method called VVC (Versatile Video Coding). VVC is also known as ITU-T H.266. In VVC, the maximum size of a coding tree unit (CTU) is extended to 128×128 pixels, compared to 64×64 pixels (hereinafter simply referred to as 64×64) in the HEVC standard.

[0012] In the video coding method described in Non-Patent Document 2, each frame of a digitized video is divided into coding tree units (CTUs), and each CTU is coded.

[0013] Each CTU is divided into coding units (CUs) using a quad-tree (QT) structure or a multi-type tree (MMT) structure and then coded. In division using a quad-tree structure, a block is divided into equal parts horizontally and vertically. In division using a multi-type tree structure, a block is divided into two or three parts horizontally or vertically.

[0014] Each CU is predictively coded. Predictive coding includes intra-prediction and motion-compensated prediction. The prediction error of each CU is transform-coded based on frequency transformation. Motion-compensated prediction generates a predicted image from a reconstructed image (reference picture) that has a different display time from the frame to be coded. Hereinafter, motion-compensated prediction is also referred to as inter-prediction.

[0015] A CU coded based on motion compensation prediction is called an inter CU. A frame coded only with intra CUs is called an I frame (or I picture). A frame coded including not only intra CUs but also inter CUs is called a P frame (or P picture). A frame coded including inter CUs that use not only one reference picture but also two reference pictures simultaneously for block inter prediction is called a B frame (or B picture). Note that inter prediction using one reference picture is called unidirectional prediction, and inter prediction using two reference pictures simultaneously is called bidirectional prediction.

[0016] When compared at the same image quality, the amount of code required for encoding based on the VVC standard is expected to be reduced by 30 to 50% compared to the amount of code required for encoding based on the HEVC standard. [Prior art documents] [Non-patent literature]

[0017] [Non-Patent Document 1] "Video Coding, Audio Coding and Multiplexing Methods for Digital Broadcasting," ARIB (Association of Radio Industries and Businesses) Standard STD-B32 Version 3.3, July 3, 2015, Association of Radio Industries and Businesses [Non-patent document 2] Benjamin Bross, et al., "Versatile Video Coding (Draft 10)", JVET-S2001-v7, Joint Video Experts Team (JVET) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 19th Meeting: by teleconference, 22 June - 1 July 2020 Summary of the Invention [Problem to be solved by the invention]

[0018] 12 and 13, arrows indicate reference pictures for the current picture to be coded. That is, the arrows indicate reference relationships. For example, in the example shown in FIG. 12, in the L2 structure, the picture in the fifth AU in display order references the picture in the third AU and the picture in the seventh AU in display order. In the L3 structure, the picture in the fourth AU in display order references the picture in the third AU and the picture in the fifth AU in display order.

[0019] In this way, a picture obtained by predicting from a reference picture can be further used as a reference picture. In the hierarchical structures shown in Figures 12 and 13, pictures in multiple lower layers (layers with small Temporal ID values: shallower layers) can be referenced for prediction of a picture in a higher layer (layers with large Temporal ID values: deeper layers). Such a reference relationship reduces the prediction residual.

[0020] As described above, the amount of code obtained by encoding based on the VVC standard is reduced compared to the amount of code obtained by encoding based on the HEVC standard. When the SOP structure defined in Non-Patent Document 1 is used, the correlation between pictures increases as the layer becomes higher, but even in the L4 structure, the number of layers is five: 0, 1, 2, 3, and M. Therefore, when encoding based on the VVC standard is performed and the SOP structure is used, there is a risk that the encoding efficiency (compression efficiency) will not be as high as expected.

[0021] Furthermore, in the SOP structure applied to the 60 / P video format, as illustrated in FIG. 12, even the L3 structure has four layers: 0, 1, 2, and 3. To increase the number of layers, it is possible to use the L4 structure applied to the 120 / P video format illustrated in FIG. 13. However, this L4 structure is adapted to temporal scalable coding. Therefore, if this L4 structure is applied directly to the 60 / P video format, the gap between the picture to be coded belonging to the layer with the highest Temporal ID (L4) and the reference picture (a picture belonging to a layer with Temporal ID=0 to 3) will become large. As a result, compression efficiency will decrease.

[0022] Note that a large interval between a picture to be coded and a reference picture means that the difference in display order between the pictures is large. In other words, a large interval between a picture to be coded and a reference picture means that the pictures are far apart on the time axis. Hereinafter, the interval between pictures will be referred to as a frame interval.

[0023] The present invention aims to provide a video encoding device, a video decoding device, a video encoding method, a video decoding method, and a video system that do not reduce compression efficiency when encoding is performed using an SOP structure. [Means for solving the problem]

[0024] The video encoding device according to the present invention is based on an SOP structure including a multi-level structure. LeaveGenerate bitstreams using structures containing 16 or more pictures possible A video encoding device includes: a prediction means for performing a prediction process using intra prediction or inter prediction; and an encoding control means for controlling the prediction means to use a picture that is closest in display order to a picture to be encoded as a reference picture when encoding a picture with reference to a picture in a lower layer under predetermined conditions that a picture that is later in display order in each layer is not encoded before a picture that is earlier in display order, and that a picture in a lower layer does not refer to a picture in a higher layer. and the encoding control means controls, based on the encoding information, to use either one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure whose maximum Temporal ID value is 4, or one of a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure whose maximum Temporal ID value is 5. .

[0025] The video decoder according to the present invention uses a structure containing 16 or more pictures in an SOP structure including a multi-level structure. do Bitstream Decrypt A video decoding device, comprising: a prediction unit that performs a prediction process using intra prediction or inter prediction; each Under predetermined conditions that a picture in a layer that is later in display order is not coded before a picture in a layer that is earlier in display order, and that a picture in a lower layer does not refer to a picture in a higher layer, when a picture is coded with reference to a picture in a lower layer, the picture that is closest in display order to the picture to be coded is used as the reference picture. and further, based on the coding information, decoding a bitstream that is coded and controlled to use either one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure in which the maximum value of the Temporal ID is 4, or one of a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure in which the maximum value of the Temporal ID is 5. .

[0026] The video encoding method according to the present invention is based on an SOP structure including a multi-level structure. Leave Generate bitstreams using structures containing 16 or more pictures possible A video coding method, comprising: performing a prediction process using intra prediction or inter prediction; and controlling a picture that is closest in display order to a picture to be coded in a lower layer to be used as a reference picture when coding a picture with reference to a picture in a lower layer under predetermined conditions that a picture that is later in display order in each layer is not coded before a picture that is earlier in display order, and a picture in a lower layer does not refer to a picture in a higher layer. and based on the encoding information, control is performed to use either one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure in which the maximum value of the Temporal ID is 4, or one of a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure in which the maximum value of the Temporal ID is 5. .

[0027] The video decoding method according to the present invention uses a structure containing 16 or more pictures in an SOP structure including a multi-level structure. do Bitstream Decrypt A video decoding method, comprising: performing a prediction process using intra prediction or inter prediction; each Under predetermined conditions that a picture in a layer that is later in display order is not coded before a picture in a layer that is earlier in display order, and that a picture in a lower layer does not refer to a picture in a higher layer, when a picture is coded with reference to a picture in a lower layer, the picture that is closest in display order to the picture to be coded is used as the reference picture. and further, based on the coding information, decoding a bitstream that is coded and controlled to use either one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure in which the maximum value of the Temporal ID is 4, or one of a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure in which the maximum value of the Temporal ID is 5. .

[0028] The video encoding program according to the present invention is based on an SOP structure including a multi-level structure. Leave Generate bitstreams using structures containing 16 or more pictures possible A video encoding program causes a computer to perform a prediction process using intra-prediction or inter-prediction, and controls the computer to use a picture that is closest in display order to a picture to be encoded as a reference picture when encoding a picture with reference to a picture in a lower layer under predetermined conditions that a picture that is later in display order in each layer is not encoded before a picture that is earlier in display order, and a picture in a lower layer does not refer to a picture in a higher layer. and based on the encoding information, control is made to use either one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure in which the maximum value of the Temporal ID is 4, or one of a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure in which the maximum value of the Temporal ID is 5. .

[0029] The video decoding program according to the present invention uses a structure containing 16 or more pictures in an SOP structure including a multi-level structure. do Bitstream Decrypt a video decoding program for causing a computer to perform a prediction process using intra prediction or inter prediction; each Under the predetermined condition that a picture in a layer that is later in display order is not coded before a picture in a layer that is earlier in display order, and a picture in a lower layer does not refer to a picture in a higher layer, when a picture is coded with reference to a picture in a lower layer, the picture that is closest in display order to the picture to be coded is used as the reference picture. and further, based on the coding information, decoding a bitstream that is controlled to use one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure whose maximum Temporal ID value is 4, or a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure whose maximum Temporal ID value is 5. .

[0030] A video system according to the present invention includes the above-described video encoding device, an audio encoding unit that encodes an audio signal, and a multiplexing unit that multiplexes the bit stream from the video encoding device and the bit stream from the audio encoding unit and outputs the multiplexed bit stream. [Effects of the Invention]

[0031] According to the present invention, when encoding is performed using the SOP structure, it is possible to prevent a decrease in compression efficiency. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 10 is an explanatory diagram showing an SOP structure applied to a video format other than 120 / P in an embodiment. [Figure 2] FIG. 10 is an explanatory diagram showing the L5 structure in the SOP structure applied to the 120 / P video format in the embodiment. [Figure 3] 1 is a block diagram showing an example of the configuration of an embodiment of a video encoding device; [Figure 4] 10 is a flowchart showing the operation of the video encoding device. [Figure 5] FIG. 1 is a block diagram illustrating an example of the configuration of an embodiment of a video decoding device. [Figure 6] 10 is a flowchart illustrating an operation of the video decoding device. [Figure 7] FIG. 1 is a block diagram illustrating an example of a video system. [Figure 8] FIG. 10 is a block diagram showing another example of a video system. [Figure 9] FIG. 1 is a block diagram illustrating an example of a computer having a CPU. [Figure 10] 1 is a block diagram showing the main parts of a video encoding device; [Figure 11] FIG. 2 is a block diagram showing the main parts of a video decoding device. [Figure 12] FIG. 10 is an explanatory diagram showing an SOP structure applied to a video format other than 120 / P. [Figure 13] FIG. 1 is an explanatory diagram showing the L5 structure in the SOP structure applied to the 120 / P video format. DETAILED DESCRIPTION OF THE INVENTION

[0033] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0034] Fig. 1 is an explanatory diagram showing an SOP structure applied to a video format other than 120 / P in an embodiment. In the SOP structure shown in Fig. 1, the arrangement and reference relationships of pictures in each layer in the L0 structure, L1 structure, L2 structure, and L3 structure are the same as those shown in Fig. 12. Note that in the embodiment, a 60P video format to which temporal scalable coding is not applied is taken as an example of a video format other than 120 / P.

[0035] In the SOP structure shown in FIG. 1, an L4 structure is added to the L0, L1, L2, and L3 structures shown in FIG. 12. The L4 structure consists of 16 pictures. The arrangement and reference relationships of pictures in each layer in the L4 structure shown in FIG. 1 differ from those shown in FIG. 13. In the L4 structure shown in FIG. 13, to achieve temporal scalable coding, each picture arranged in the layer (L4) with the highest Temporal ID is coded in odd consecutive coding order (which is also the decoding order). However, in the L4 structure shown in FIG. 1, each picture is arranged without such constraints.

[0036] In the L4 structure shown in FIG. 1, under the conditions that the coding order is not reversed from the display order in each layer (a picture in a later display order is not coded before a picture in an earlier display order), and a picture in a lower layer (a layer with a smaller Temporal ID value) does not refer to a picture in a higher layer (a layer with a larger Temporal ID value), a picture coded with reference to a picture in a lower layer uses the picture closest to the picture to be coded in display order as the reference picture. Note that not reversing the coding order from the display order means that B aThe picture of b For the picture of (a < b), it must be the picture with an earlier display order. Also, the reference picture needs to be encoded earlier than the picture to be encoded.

[0037] For example, regarding the layer with a Temporal ID of 4, the picture shown by B5 with a display order of 2 is encoded by referring to the picture shown by B3 with a display order of 1 and the picture shown by B2 with a display order of 3. The pictures shown by B3 and B2 are the closest in display order to the picture shown by B5 in the lower layer (in this example, multiple layers with Temporal IDs from 0 to 3).

[0038] Also, regarding the layer with a Temporal ID of 3, the picture shown by B6 with a display order of 5 is encoded by referring to the picture shown by B2 with a display order of 3 and the picture shown by B1 with a display order of 7. The pictures shown by B2 with a display order of 3 and B1 with a display order of 7 are the closest in display order to the picture shown by B6 in the lower layer (in this example, multiple layers with Temporal IDs from 0 to 2).

[0039] Note that the encoding order in the L4 structure is not limited to the encoding order shown in FIG. 1. Under the condition that the encoding order in each layer does not reverse the display order and the pictures in the lower layer do not refer to the pictures in the upper layer, if it is possible to use as the reference picture the picture that is closest in display order to the picture to be encoded and is encoded by referring to the pictures in the lower layer, other encoding orders may be used.

[0040] In this embodiment, since the picture that is closest in display order to the picture to be encoded is used as the reference picture, compared with the L4 structure in the SOP structure applied to the 120 / P video format shown in FIG. 13, the frame interval between the picture to be encoded and the reference picture becomes smaller. As a result, the compression efficiency of the picture is higher compared to the case where the L4 structure shown in FIG. 13 is used.

[0041] FIG. 2 is an explanatory diagram showing the L5 structure in the SOP structure applied to the 120 / P video format. The L5 structure consists of 32 pictures. In this embodiment, the SOP structure applied to the 120 / P video format is also a structure that realizes temporal scalable coding. Although not shown in FIG. 2, the L0 structure, L1 structure, L2 structure, and L4 structure are the same as those shown in FIG. 13. That is, in this embodiment, the L5 structure is added to the L0 structure, L1 structure, L2 structure, L3 structure, and L4 structure shown in FIG. 13.

[0042] In addition, in the L5 structure shown in Fig. 2, the coding order and reference relationships of pictures in the layers with Temporal IDs = 0 to 4 are the same as those in the L4 structure shown in Fig. 1. However, since the L5 structure shown in Fig. 2 is made up of 32 pictures, the numbers indicating the decoding order (which is also the coding order) in Fig. 2 are twice the numbers indicating the decoding order in Fig. 1.

[0043] When the L5 structure in the SOP structure applied to the 120 / P video format is used, the compression efficiency of pictures belonging to the base layer (in this case, pictures in the hierarchical layers with Temporal ID=0 to 4) is increased, just as the compression efficiency of each layer (pictures in the hierarchical layers with Temporal ID=0 to 4) is increased when the L4 structure in the embodiment shown in Figure 1 is used.

[0044] The coding order in the L5 structure is not limited to the coding order shown in Fig. 2. For layers with Temporal IDs 0 to 4, other coding orders may be used as long as, under certain conditions, a picture to be coded with reference to a picture in a lower layer can use the picture closest to the picture to be coded in display order as the reference picture. The certain conditions are that the coding order is not reversed from the display order, and that pictures in a lower layer do not refer to pictures in a higher layer.

[0045] Furthermore, by adding the L4 structure to the SOP structure applied to the 60 / P video format and the L5 structure to the SOP structure applied to the 120 / P video format, the picture compression efficiency is improved. This is because, in general, a large amount of code is allocated to the 0th picture in decoding order (often an I-picture or a P-picture) which is frequently referenced. In structures including higher hierarchical layers (L4 in the 60 / P video format, LM in the 120 / P video format), the 0th picture in decoding order occurs relatively less frequently.

[0046] Embodiment 1. Fig. 3 is a block diagram showing an example of the configuration of an embodiment of a video encoding device. The video encoding device shown in Fig. 3 includes a rearrangement unit 101, a subtractor 102, a transformer / quantizer 103, a dequantizer / inverse transformer 104, an entropy encoder 105, an adder 106, a buffer 107, a predictor 108, an encoding control unit 109, and a multiplexer 110. Note that although the video encoding device shown in Fig. 1 performs encoding processing based on the VVC standard, it may also perform encoding processing based on other standards, such as the HEVC standard, the H.264 / AVC standard, or the MPEG (Moving Picture Experts Group)-2 standard.

[0047] The reordering unit 101 is a memory that stores each image (picture) in a video signal that is input in display order. In this embodiment, the pictures are stored in the order of input. That is, the pictures that are input in display order are stored in the memory in ascending order of addresses. However, the pictures that are input in display order may also be stored in the reordering unit 101 in coding order. That is, the pictures that are input may also be stored in the memory in ascending order of addresses in coding order.

[0048] Whichever memory storage method (display order or coding order) is adopted, the pictures are read out in coding order from the rearrangement unit 101. Hereinafter, a picture may be referred to as an input video signal.

[0049] The rearrangement unit 101 is used when encoding is performed using the SOP structure. When the SOP structure is not used, each picture in the input video signal is supplied to the subtractor 102 as is.

[0050] The subtractor 102 subtracts the prediction signal from the input video signal (specifically, pixel values) read from the rearrangement unit 101 to generate a prediction error signal. The prediction error signal is also called a prediction residual or a prediction residual signal.

[0051] The transformer / quantizer 103 frequency-transforms the prediction error signal. The transformer / quantizer 103 further quantizes the frequency-transformed prediction error signal (transform coefficients). Hereinafter, the quantized transform coefficients are referred to as transformed and quantized values.

[0052] The entropy encoder 105 entropy-encodes the prediction parameters and the transformed and quantized values. The prediction parameters are information related to the prediction of CTUs (Coding Tree Units) and blocks, such as a prediction mode (intra prediction, inter prediction), an intra prediction block size, an intra prediction direction, an inter prediction block size, and a motion vector.

[0053] The multiplexer 110 multiplexes the entropy-encoded data supplied from the entropy encoder 105 and data (encoding information, etc.) from the encoding control unit 109, and outputs the multiplexed data as a bit stream.

[0054] The predictor 108 generates a prediction signal for the input video signal. The predictor 108 generates the prediction signal based on intra prediction or inter prediction. That is, the predictor 108 generates a prediction signal for each block (unit) that is a unit of encoding using either intra prediction or inter prediction.

[0055] The inverse quantization / inverse transformer 104 inversely quantizes the transformed and quantized values to restore transform coefficients. Furthermore, the inverse quantization / inverse transformer 104 performs inverse frequency transform on the inversely quantized transform coefficients to restore a prediction error signal. The adder 106 adds the restored prediction error signal and the prediction signal to generate a reconstructed image. The reconstructed image is supplied to a buffer 107. The buffer 107 stores the reconstructed image. The buffer 107 corresponds to a block memory for storing reference blocks for intra prediction and a frame memory for storing reference pictures for inter prediction.

[0056] The encoding control unit 109 inputs encoding information from outside the video encoding device. The encoding information includes the encoding method to be used (VVC standard, HEVC standard, H.264 / AVC standard, MPEG-2), test sequence information (60 / P, 120 / P, etc.), whether scalable encoding is possible, etc. The encoding control unit 109 controls each block in the video encoding device based on the encoding information.

[0057] Next, the operation of the video encoding device when encoding is performed using the SOP structure will be described with reference to the flowchart of FIG.

[0058] First, the rearrangement unit 101 stores each picture in a video signal input in display order (step S101).

[0059] The rearrangement unit 101 outputs pictures to the subtractor 102 in the order of coding in accordance with an instruction from the coding control unit 109 (step S102).

[0060] When compliance with the 60 / P video format is specified externally, the encoding control unit 109 controls the reordering unit 101 to read pictures in the decoding order (which is also the encoding order) shown in Fig. 1. That is, when the reordering unit 101 stores each picture in display order, the encoding control unit 109 controls the reordering unit 101 to output the picture to be encoded from the address in memory from which it is read. When the reordering unit 101 stores each picture in coding order, the encoding control unit 109 instructs the reordering unit 101 to store the picture at the memory address according to the coding order when the picture is written to the reordering unit 101.

[0061] When it is externally specified that the picture format should conform to the 120 / P video format, the encoding control unit 109 controls the pictures to be read from the rearrangement unit 101 in the decoding order shown in FIG. 12 (in the case of the L0 to L4 structure) or the decoding order shown in FIG. 2 (in the case of the L5 structure).

[0062] For example, the encoding control unit 109 can determine which of the L0 to L4 structures (in the case of 60 / P) or which of the L0 to L5 structures (in the case of 120 / P) to use depending on the situation of a video scene. For example, the encoding control unit 109 determines to use an Lx structure with a small value of x for images (pictures) constituting a video of a scene in which the entire screen does not move much, and to use an Lx structure with a large value of x for images constituting a video of a scene in which the entire screen moves quickly. In this case, the encoding control unit 109 includes a function for detecting the degree of movement in the image in advance.

[0063] The predictor 108 generates a prediction signal for the input video signal based on intra-prediction or inter-prediction (step S103), and the subtractor 102 generates a prediction error signal (step S103).

[0064] The encoding control unit 109 instructs the predictor 108 to perform encoding in accordance with the picture reference relationship shown in Figure 1 (in the case of 60 / P) or the picture reference relationship shown in Figures 12 and 2 (in the case of 120 / P).

[0065] The transformer / quantizer 103 performs frequency transform on the prediction error signal to generate transform coefficients (step S104). Furthermore, the transformer / quantizer 103 quantizes the transform coefficients using a quantization step width to generate transformed and quantized values (step S105). The transformed and quantized values are input to the inverse quantizer / inverse transformer 104 and the entropy encoder 105.

[0066] The inverse quantization / inverse transformer 104 inverse quantizes the transformed and quantized values and performs inverse frequency transform on the transformed and quantized values to restore the transform coefficients (step S106). The entropy encoder 105 entropy codes (e.g., arithmetic codes) the transformed and quantized values to generate entropy-coded data (step S107).

[0067] The processes of steps S102 to S107 are executed for all pictures that make up the SOP (step S108).

[0068] The multiplexer 110 multiplexes the entropy-encoded data supplied from the entropy encoder 105 and the data (encoding information, etc.) supplied from the encoding control unit 109, and outputs the multiplexed data as a bit stream.

[0069] In this embodiment, when an L4 structure in an SOP structure applied to a video format other than 120 / P is used, the encoding control unit 109 controls the predictor 108 to use the picture closest to the picture to be encoded in display order as the reference picture when encoding a picture with reference to a picture in a lower layer, under the conditions that the encoding order in each layer in the L4 structure is not reversed from the display order and pictures in a lower layer do not refer to pictures in a higher layer. Such control improves the compression efficiency of each picture in the SOP. Note that, in this embodiment, to achieve such control, the encoding control unit 109 causes the reordering unit 101 to output pictures in the encoding order shown in FIG. 1.

[0070] Furthermore, when the L5 structure in the SOP structure applied to the 120 / P video format is used, the encoding control unit 109 controls the predictor 108 so that, under the conditions that the coding order in the layers of Temporal ID=0 to 4 in the L5 structure is not reversed from the display order and pictures in lower layers do not refer to pictures in higher layers, pictures that are coded with reference to pictures in lower layers use as a reference picture the picture that is closest to the picture to be coded in display order. Such control improves the compression efficiency of pictures belonging to the base layer (in this case, pictures in layers of Temporal ID=0 to 4). Note that, in this embodiment, to achieve such control, the encoding control unit 109 causes the reordering unit 101 to output pictures in the decoding order shown in FIG. 12 (in the case of the L0 to L4 structures) or the decoding order shown in FIG. 2 (in the case of the L5 structure).

[0071] Embodiment 2. Fig. 5 is a block diagram showing an example of the configuration of an embodiment of a video decoding device. The video decoding device shown in Fig. 5 includes a demultiplexer 201, an entropy decoder 202, an inverse quantization / inverse transformer 203, an adder 204, a predictor 205, a buffer 206, a decoding control unit 207, and a reordering unit 208. Note that the video decoding device shown in Fig. 5 can receive a bitstream from, for example, the video encoding device shown in Fig. 3.

[0072] The demultiplexer 201 demultiplexes the input bitstream to extract entropy-encoded data, and outputs the encoding information and the like contained in the bitstream to the decoding control unit 207.

[0073] The entropy decoder 202 entropy-decodes the entropy-encoded data. The entropy decoder 202 supplies the entropy-decoded transformed and quantized values to the inverse quantization / inverse transformer 203. The entropy decoder 202 also supplies prediction parameters included in the bitstream to the predictor 205. The entropy decoder 202 supplies the coding information included in the bitstream to the decoding control unit 207.

[0074] The inverse quantization / inverse transformer 203 inversely quantizes the transformed quantized values, and further inversely frequency transforms the inversely quantized frequency transform coefficients.

[0075] The predictor 205 generates a prediction signal for each sub-block based on the prediction parameters. The prediction error signal inversely frequency transformed by the inverse quantization / inverse transformer 203 is added by the adder 204 to the prediction signal supplied from the predictor 205, and then supplied as a reconstructed image to the buffer 206. The buffer 206 stores the reconstructed image.

[0076] The reconstructed image stored in the buffer 206 is transferred to the reordering unit 208. The reordering unit 208 is a memory that stores each image (picture) in a video signal that is input in decoding order. In this embodiment, each picture is stored in decoding order. That is, each picture that is input in decoding order is stored in memory in ascending order of addresses. However, each picture that is input in decoding order may also be stored in the reordering unit 101 in display order. That is, each picture that is input may also be stored in memory in descending order of addresses in display order.

[0077] Regardless of which memory storage method (decoding order or display order) is adopted, the pictures are read out from the reordering unit 208 in display order.

[0078] Next, the operation of the video decoder when performing decoding using the SOP structure will be described with reference to the flowchart of FIG.

[0079] The entropy decoder 202 entropy decodes the entropy-encoded data included in the bitstream (step S201).

[0080] The inverse quantization / inverse transformer 203 inversely quantizes the transformed quantized value using the quantization step width (step S202). Furthermore, the inverse quantization / inverse transformer 203 inversely frequency transforms the inversely quantized frequency transform coefficients (step S203).

[0081] The predictor 205 generates a prediction signal using the reconstructed image stored in the buffer 206 (step S204). The adder 204 adds the prediction signal supplied from the predictor 205 to the prediction error image inverse frequency transformed by the inverse quantization / inverse transformer 203 to generate a reconstructed image (step S204). The reconstructed image is stored in the buffer 206.

[0082] The reconstructed image stored in the buffer 206 is transferred to the sorting unit 208 (step S205).

[0083] The processes of steps S201 to S205 are executed for all pictures that make up the SOP (step S206).

[0084] The rearrangement unit 208 outputs each image in display order in response to an output instruction from the decoding control unit 207 (step S207).

[0085] When the encoding information specifies that the pictures comply with the 60 / P video format, the decoding control unit 207 controls the reordering unit 101 to read out the pictures in the display order shown in Fig. 1. That is, when the pictures are stored in the reordering unit 208 in decoding order, the decoding control unit 207 controls the reordering unit 208 to output the pictures from the memory addresses from which they are read (addresses at which the pictures to be read out in display order are stored). Note that when the pictures are stored in the reordering unit 101 in display order, the decoding control unit 207 instructs the reordering unit 208 to store the pictures at memory addresses corresponding to the display order when the pictures are written to the reordering unit 208.

[0086] When it is externally specified that the picture format should conform to the 120 / P video format, the decoding control unit 207 controls the pictures to be read from the rearrangement unit 208 in the display order shown in FIG. 12 (in the case of the L0 to L4 structure) or the display order shown in FIG. 2 (in the case of the L5 structure).

[0087] When the video encoding device of the first embodiment receives a bitstream based on encoded data encoded using an SOP structure, the video decoding device can reproduce video from the encoded data with high compression efficiency for each picture.

[0088] That is, the video decoding device of this embodiment is capable of receiving a bitstream from a video encoding device configured such that, when encoding a picture with reference to a picture in a lower layer, the prediction means sets the picture that is closest in display order to the picture to be encoded as the reference picture, under predetermined conditions that the encoding order is not reversed from the display order and that pictures in a lower layer do not refer to pictures in a higher layer. Upon receiving such a bitstream, the predictor 205 in the video decoding device of this embodiment is capable of setting the picture that is closest in display order to the picture to be encoded as the reference picture, under predetermined conditions that the encoding order is not reversed from the display order and that pictures in a lower layer do not refer to pictures in a higher layer.

[0089] Embodiment 3. Fig. 7 is a block diagram showing an example of a video system, in which the video encoding device 100 and the video decoding device 200 according to the above-described embodiment are connected via a wireless or wired transmission path 300.

[0090] The configuration and operation of video encoding device 100 are the same as those of the video encoding device shown in Fig. 3. The configuration and operation of video decoding device 200 are the same as those of the video decoding device shown in Fig. 5.

[0091] Embodiment 4. 8 is a block diagram showing another example of a video system, which includes an audio encoding unit 401, a video encoding unit 402, and a multiplexing unit 403.

[0092] The audio encoding unit 401 creates and outputs an audio bitstream by encoding the audio signal from data (content) including video and audio based on, for example, the MPEG-4 AAC (Advanced Audio Coding) standard or the MPEG-4 ALS (Audio Lossless Coding) standard defined in the ARIB STD-B32 standard.

[0093] The video encoding unit 402 is configured, for example, as shown in FIG. 3, and creates and outputs a video bitstream.

[0094] The multiplexing unit 403 creates and outputs a bitstream by multiplexing an audio bitstream, a video bitstream, and other information based on, for example, the ARIB STD-B32 standard.

[0095] Furthermore, each of the above embodiments can be configured by hardware, but can also be realized by a computer program.

[0096] Fig. 9 is a block diagram showing an example of a computer having a CPU (Central Processing Unit). The computer can be implemented in both the video encoding device shown in Fig. 3 and the video decoding device shown in Fig. 5. The CPU 1000 executes processing in accordance with a program (video encoding program or video decoding program) stored in a storage device 1001, thereby realizing the functions of each of the above-described embodiments.

[0097] That is, when the computer is implemented in the video encoding device shown in FIG. 3, the CPU 1000 realizes the functions of the subtractor 102, transformer / quantizer 103, inverse quantizer / inverse transformer 104, entropy encoder 105, adder 106, predictor 108, encoding control unit 109, and multiplexer 110 in the video encoding device.

[0098] When the computer is implemented in the video decoding device shown in Figure 5, the CPU 1000 realizes the functions of the demultiplexer 201, entropy decoder 202, inverse quantization / inverse transformer 203, adder 204, predictor 205, and decoding control unit 207 in the video decoding device.

[0099] The storage device 1001 is, for example, a non-transitory computer-readable medium. The non-transitory computer-readable medium includes various types of tangible storage media. Specific examples of non-transitory computer-readable media include magnetic recording media (e.g., hard disks), CD-ROMs (Compact Disc-Read Only Memory), CD-Rs (Compact Disc-Recordable), CD-R / Ws (Compact Disc-Rewritable), and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), and flash ROMs).

[0100] The program may also be stored in various types of transitory computer-readable media, to which the program is supplied, for example, via a wired or wireless communication path, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0101] The memory 1002 is realized by, for example, a random access memory (RAM), and is a storage means for temporarily storing data when the CPU 1000 executes processing. A configuration is also conceivable in which a program held in the storage device 1001 or a temporary computer-readable medium is transferred to the memory 1002, and the CPU 1000 executes processing based on the program in the memory 1002.

[0102] The memory 1002 can also be used as the rearrangement unit 101 and buffer 107 shown in Fig. 3. The memory 1002 can also be used as the buffer 206 and rearrangement unit 208 shown in Fig. 5.

[0103] Fig. 10 is a block diagram showing the main components of a video encoding device. The video encoding device 10 shown in Fig. 10 is a video encoding device that generates a bitstream using an SOP structure including a multi-level (e.g., L0 to L4, L0 to L5) structure (e.g., an L0 to L4 structure, an L0 to L5 structure), and includes: prediction means 11 (implemented by a predictor 108 in the embodiment) that performs prediction processing using intra prediction or inter prediction; and encoding control means 12 (implemented by an encoding control unit 109 in the embodiment) that controls the prediction means 11 to use a picture that is closest in display order to the picture to be encoded as a reference picture when encoding a picture with reference to a picture in a lower layer under predetermined conditions that a picture that is later in display order is not encoded before a picture that is earlier in display order and a picture in a lower layer does not refer to a picture in a higher layer.

[0104] Fig. 11 is a block diagram showing main components of a video decoding device. The video decoding device 20 shown in Fig. 11 is a video decoding device that receives a bitstream generated using an SOP structure including a multi-level (e.g., L0 to L4, L0 to L5) structure (e.g., an L0 to L4 structure, an L0 to L5 structure) and performs decoding processing, and includes prediction means 21 (implemented by a predictor 205 in the embodiment) that performs prediction processing using intra prediction or inter prediction, and the prediction means 21 operates to set the picture closest to the picture to be coded in display order as the reference picture when coding a picture with reference to a picture in a lower layer under predetermined conditions that a picture later in display order is not coded before a picture earlier in display order and a picture in a lower layer (a layer with a smaller Temporal ID value) does not reference a picture in a higher layer (a layer with a larger Temporal ID value). [Explanation of symbols]

[0105] 11 Prediction methods 12 Encoding Control Means 21 Prediction methods 10,100 Video Encoding Device 101 Sorting Section 102 Subtractor 103 Transform / Quantizer 104 Inverse Quantization / Inverse Transformer 105 Entropy Encoder 106 Adder 107 buffers 108 Predictor 109 Encoding control unit 110 Multiplexer 20,200 video decoder 201 Demultiplexer 202 Entropy Decoder 203 Inverse Quantization / Inverse Transformer 204 Adder 205 Predictor 206 buffers 207 Decoding control unit 208 Sorting Section 401 Audio Encoding Unit 402 Video Encoding Unit 403 Multiplexer 1000 CPU 1001 Storage device 1002 memory

Claims

1. A video encoding device capable of generating a bitstream using a structure including 16 or more pictures in an SOP structure including a multi-level structure, comprising: a prediction means for performing a prediction process using intra prediction or inter prediction; and encoding control means for controlling the prediction means to use a picture that is closest in display order to the picture to be encoded as a reference picture when encoding a picture with reference to a picture in a lower layer under predetermined conditions that a picture that is later in display order in each layer is not encoded before a picture that is earlier in display order and that a picture in a lower layer does not refer to a picture in a higher layer, The encoding control means controls to use either one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure whose maximum Temporal ID value is 4, or one of a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure whose maximum Temporal ID value is 5, based on the encoding information. A video encoding device characterized by:

2. When a video signal in a video format other than 120 / P is encoded and an L4 structure including 16 frames is used, the encoding control means controls the prediction means to use the picture that is closest in display order in the lower layer as a reference picture under the predetermined condition.

2. The video encoding device according to claim 1.

3. When a video signal in a video format of 120 / P is coded and an L5 structure including 32 frames is used, the coding control means controls, under the predetermined condition, in a base layer that is a layer lower than the highest layer, so that when the prediction means codes a picture by referring to a picture in a lower layer, the prediction means uses the picture that is closest in display order in the lower layer as the reference picture.

2. The video encoding device according to claim 1.

4. The prediction means performs prediction processing based on the VVC standard. The video encoding device according to any one of claims 1 to 3.

5. 1. A video decoding device for decoding a bitstream that uses a structure including 16 or more pictures in an SOP structure including a multi-level structure, comprising: a prediction means for performing a prediction process using intra-prediction or inter-prediction, Decodes a bitstream that is coded under a predetermined condition that a picture that is later in display order in each layer is not coded before a picture that is earlier in display order, and that a picture in a lower layer does not refer to a picture in a higher layer, and that when coding a picture with reference to a picture in a lower layer, the picture that is closest in display order to the picture to be coded is used as the reference picture, and that is coded based on coding information to use either a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure with a maximum Temporal ID value of 4, or a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure with a maximum Temporal ID value of 5. A video decoding device comprising:

6. 1. A video encoding method capable of generating a bitstream using a structure including 16 or more pictures in an SOP structure including a multi-level structure, comprising: performing a prediction process using intra prediction or inter prediction; under predetermined conditions that a picture that is later in display order in each layer is not coded before a picture that is earlier in display order, and that a picture in a lower layer does not refer to a picture in a higher layer, when coding a picture with reference to a picture in a lower layer in the prediction process, control is performed so that a picture that is closest in display order to the picture to be coded is used as the reference picture; Based on the encoding information, control is performed so that either one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure in which the maximum value of the Temporal ID is 4, or one of a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure in which the maximum value of the Temporal ID is 5, is used. A video encoding method comprising:

7. 1. A video decoding method for decoding a bitstream using a structure including 16 or more pictures in an SOP structure including a multi-level structure, comprising: performing a prediction process using intra prediction or inter prediction; Decodes a bitstream that is coded under a predetermined condition that a picture that is later in display order in each layer is not coded before a picture that is earlier in display order, and that a picture in a lower layer does not refer to a picture in a higher layer, and that when coding a picture with reference to a picture in a lower layer, the picture that is closest in display order to the picture to be coded is used as the reference picture, and that is coded based on coding information to use either a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure with a maximum Temporal ID value of 4, or a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure with a maximum Temporal ID value of 5. A video decoding method comprising:

8. A video encoding program capable of generating a bitstream using a structure including 16 or more pictures in an SOP structure including a multi-level structure, comprising: On the computer, performing a prediction process using intra prediction or inter prediction; under predetermined conditions that a picture that is later in display order in each layer is not coded before a picture that is earlier in display order, and that a picture in a lower layer does not refer to a picture in a higher layer, when coding a picture with reference to a picture in a lower layer in the prediction process, control is performed so that a picture that is closest in display order to the picture to be coded is used as the reference picture; Based on the encoding information, control is performed to use either one of a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure in which the maximum value of the Temporal ID is 4, or one of a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure in which the maximum value of the Temporal ID is 5. A video encoding program for

9. 1. A video decoding program for decoding a bitstream that uses a structure including 16 or more pictures in an SOP structure including a multi-level structure, comprising: On the computer, performing a prediction process using intra prediction or inter prediction; A bitstream is decoded in which, under a predetermined condition that a picture later in display order in each layer is not coded before a picture earlier in display order and that a picture in a lower layer does not refer to a picture in a higher layer, coding is controlled so that when a picture is coded with reference to a picture in a lower layer, the picture closest to the picture to be coded in display order is used as the reference picture, and further, coding is controlled based on coding information so that either a set of SOP structures including an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L4 structure whose maximum Temporal ID value is 4, or either a set of SOP structures including an L4 structure, an L3 structure, an L2 structure, an L1 structure, and an L0 structure in addition to an L5 structure whose maximum Temporal ID value is 5, is used. Video decoding program for.

10. A video encoding device according to any one of claims 1 to 4; an audio encoding unit that encodes an audio signal; a multiplexing unit that multiplexes the bit stream from the video encoding device and the bit stream from the audio encoding unit and outputs the multiplexed bit stream; A video system comprising:

Images