Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

This video encoding device includes: a decimation means for decimating pictures in a video; an encoding means for encoding a video signal resulting from the decimation of the pictures; and an auxiliary information generation means for generating auxiliary information for deriving a maximum latency picture quantity.

Description

Video Encoding Device, Video Decoding Device, Video Encoding Method, and Video Decoding Method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to efficiently transmit or record video, a video encoding device that generates an encoded representation (hereinafter referred to as a bitstream) of an input video, and a video decoding device that decodes the bitstream to generate a decoded video are used.

[0003] As standards for video encoding methods, there are H.265 / HEVC (High-Efficiency Video Coding) (see Non-Patent Document 1), H.266 / VVC (Versatile Video Coding) (see Non-Patent Document 2), etc., which are standardized by ITU-T SG16 and ISO / IEC / SC29.

[0004] In these video encoding methods, video data is managed in a hierarchical structure and encoded and decoded. The hierarchical structure consists of, for example, pictures that make up the video data, slices (or tiles) obtained by dividing a picture, coding tree units (CTUs) obtained by dividing a slice, and coding units (CUs) obtained by dividing a coding tree unit.

[0005] The input image of the CU to be processed is usually prediction-encoded based on a prediction image that is encoded and decoded in the past and generated based on the decoded image of the CU to be processed. That is, the prediction error image obtained by subtracting the prediction image from the input image is encoded and decoded. As prediction encoding, there are intra prediction (intra prediction) that uses the decoded image included in the picture at the same display time as the CU to be processed, and inter prediction (inter prediction) that uses the decoded image included in the picture at a different display time from the CU to be processed.

[0006] A CU encoded based on intra prediction is called an intra CU. A CU encoded based on inter prediction is called an inter CU.

[0007] A picture that can only contain an intraCU is called an I-picture. A picture that can contain not only an intraCU but also an interCU that uses a decoded image contained in one picture with a different display time than the target CU is called a P-picture. A picture that can contain not only an intraCU and an interCU that uses a decoded image contained in one picture with a different display time than the target CU, but also an interCU that uses decoded images contained in two pictures with different display times than the target CU is called a B-picture.

[0008] The coding of the prediction error image is performed based on frequency conversion, quantization, and entropy coding. The decoding of the prediction error image is performed based on entropy decoding, inverse quantization, and inverse frequency conversion.

[0009] Next, we will explain the parameters (DPB parameters) of the Decoded Picture Buffer (DPB) in video encoding technology.

[0010] In Non-Patent Document 2, section 7.3.4 DPB parameters syntax defines the DPB parameters dpb_max_num_reorder_pics[i] and dpb_max_latency_increase_plus1[i].

[0011] dpb_max_num_reorder_pics[i] is information about the number of reference pictures. Specifically, when Htid (an ID indicating the highest temporal sublayer to be decoded) is equal to i, dpb_max_num_reorder_pics[i] specifies the maximum number of pictures in the OLS (Output Layer Set: a set of layers where one or more layers are designated as output layers) that can precede a given picture in the decoding order and follow it in the output order.

[0012] This parameter, as defined in Non-Patent Document 3, represents the maximum number of pictures that will be rearranged (maximum number of rearranged pictures) in the Structure of Pictures (SOP).

[0013] An SOP (Access Unit) is composed of multiple AUs (Access Units) with Temporal IDs of 0 or greater, starting with an AU with Temporal ID 0. An AU is essentially a bitstream for a single picture (with the same display time). A GOP (Group of Pictures) is a re-enterable AU, composed of multiple SOPs starting with an I-picture.

[0014] dpb_max_latency_increase_plus1[i] contains information about the maximum number of images that can be skipped. Specifically, dpb_max_latency_increase_plus1[i] ≠ 0 is used to calculate the value of MaxLatencyPictures[i] (which specifies the maximum number of pictures in the OLS that can precede any picture in the OLS in output order and follow that picture in decoding order when Htid is equal to i).

[0015] Furthermore, this parameter, as defined in Non-Patent Document 3, represents the maximum increase in latency per picture from decoding to display (maximum latency increase).

[0016] Figure 27 is an explanatory diagram showing the SOP structure corresponding to L=0, L=1, L=2, and L=3 as described in Non-Patent Document 3. In Figure 27, the access unit is the AU described above. Patent Document 1 also contains a description of the above parameters.

[0017] Figure 28 is an explanatory diagram illustrating the values ​​of dpb_max_num_reorder_pics[i] and dpb_max_latency_increase_plus1[i] corresponding to each of the above SOP structures.

[0018] From the descriptions in Figures 27 and 28, it can be seen that the value obtained by adding dpb_max_num_reorder_pics[i] to dpb_max_latency_increase_plus1[i] (the value obtained by subtracting 1) represents the maximum latency per picture (MaxLatencyPictures) from the time the SOP's AU is decoded until the display based on it is made.

[0019] Furthermore, section 7.4.5 DPB parameters semantics in Non-Patent Document 2 also contains the following description.

[0020] When dpb_max_latency_increase_plus1[ i ] is not equal to 0, the value of MaxLatencyPictures[ i ] is specified as follows: MaxLatencyPictures[ i ] = dpb_max_num_reorder_pics[ i ] + dpb_max_latency_increase_plus1[ i ] - 1 When dpb_max_latency_increase_plus1[ i ] is equal to 0, no corresponding limit is expressed.

[0021] In other words, the decoder (video encoder) can start processing with a fixed delay of the maximum number of latency pictures by deriving MaxLatencyPictures from the DPB parameters dpb_max_num_reorder_pics[i] and dpb_max_latency_increase_plus1[i]. That is, in the interconnection between the encoder and decoder, the decoder is freed from having to wait until the end of the bitstream to start processing.

[0022] Next, we will explain the parameters of Picture Timing Supplemental Enhancement Information (PTSEI) in video encoding technology. Picture Timing SEI is timing information that indicates the timing of decoding and display.

[0023] PTSEI is an SEI that represents the extraction delay from the CPB (Coded Picture Buffer) for each picture and the output delay from the DPB of the decoded picture, derived based on the above DPB parameters.

[0024] Non-Patent Literature 1 defines the DPB parameters au_cpb_removal_delay_minus1 and pic_dpb_output_delay in D.2.3 Picture timing SEI message syntax. Non-Patent Literature 1 also contains the following description in D.3.3:

[0025] `au_cpb_removal_delay_minus1 plus 1` is used to calculate the number of clock ticks between the nominal CPB removal times of the access unit associated with the picture-timing SEI message and the preceding access unit in decoding order that contained a buffering period SEI message. This value is also used to calculate the earliest possible time of arrival of access unit data into the CPB for the HSS. The syntax element is a fixed length code whose length in bits is given by `au_cpb_removal_delay_length_minus1 + 1`. The syntax element is a fixed-length code, and its bit length is given by au_cpb_removal_delay_length_minus1 + 1. In other words, au_cpb_removal_delay_minus1 plus 1 is the picture's CPB extraction delay (decryption start delay).

[0026] `pic_dpb_output_delay` is used to compute the DPB output time of the picture when `SubPicHrdFlag` is equal to 0. It specifies how many clock ticks to wait after the removal of the last decoding unit in an access unit from the CPB before the decoded picture is output from the DPB. In other words, `pic_dpb_output_delay` is the output delay of the decoded picture from the DPB.

[0027] As is clear from the description of F.13.3.3 Picture output in Non-Patent Document 1, the extraction time from the DPB starting from the buffering delay time is au_cpb_removal_delay_minus1+1 + pic_dpb_output_delay.

[0028] Recommendation ITU-T H.264 "High efficiency video coding", Telecommunication Standardization Sector of ITU, April 2015; Recommendation ITU-T H.266 "Versatile video coding", Telecommunication Standardization Sector of ITU, August 2020; ARIB Standard STD-B32 version 3.11 "Video coding, audio coding and multiplexing methods in digital broadcasting", Association of Radio Industries and Businesses, July 26, 2018; S. Ren et al., "Faster R-CNN: Towards real-time object detection with region proposal networks", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 6, June 2017

[0029] The following are possible ways to utilize the video encoding described above.

[0030] The video encoder applies preprocessing to the input video before encoding it using the video encoding method described above (hereinafter referred to as internal video encoding). The video encoder also encodes control information for the preprocessing. Preprocessing may include temporal sampling, which reduces the number of pictures to be encoded by downsampling the pictures in the input video.

[0031] The video decoder decodes the bitstream using the video encoding scheme described above (hereinafter referred to as internal video decoding). The video decoder then applies post-processing to the decoded picture to obtain the final decoded image. The video decoder also decodes the control information for the post-processing. The post-processing may include temporal resampling, which interpolates the pictures that were downsampled during time sampling in the video encoder.

[0032] In video encoding schemes using Temporal sampling / Temporal resampling, the AU (Automatic Unit) of the picture downsampled by the video encoder does not exist in the bitstream output of the internal encoder (internal video encoding bitstream). Therefore, the maximum latency picture count derived from the DBP parameter of the bitstream output does not match the maximum latency picture count in a video decoder using Temporal resampling. For the same reason, the picture timing of the display derived from the PTSEI of the bitstream output does not match the picture timing in a video decoder including Temporal resampling.

[0033] Such discrepancies can lead to problems in video encoding schemes using temporal sampling / temporal resampling, where the interoperability between the video encoder and video decoder is compromised in terms of processing timing.

[0034] It is conceivable that the video encoder could set the DPB parameter of the internal encoder's bitstream output to match the maximum latency picture count in the video decoder using Temporal resampling. However, doing so would result in a mismatch between the internal video encoding and the internal video decoding. The same applies to the PTSEI parameter of the internal encoder's bitstream output.

[0035] The present invention aims to ensure interoperability between the video decoding device and the video encoding device regarding the processing timing of the video decoding device, even when time sampling / time resampling is used.

[0036] The video encoding method based on this disclosure performs picture decimation in the video, encodes the video signal after picture decimation, and generates auxiliary information for deriving the maximum latency picture count.

[0037] Another aspect of the video encoding method based on this disclosure involves decimating pictures in a video, encoding the video signal after picture decimation, and generating a picture timing SEI that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing in the video decoding device, based on the encoding.

[0038] Another embodiment of the video encoding method based on this disclosure involves decimating pictures in a video, encoding the video signal after picture decimation, and setting dummy data corresponding to the decimated pictures into the video bitstream.

[0039] The video decoding method based on this disclosure obtains the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled, and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count, decodes the video bitstream, and calculates the maximum latency picture count from the auxiliary information.

[0040] A video decoding method in another aspect of the present disclosure involves obtaining the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled, and an auxiliary information bitstream containing picture timing SEI auxiliary information that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation, based on the encoding; decoding the video bitstream; and calculating the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

[0041] Another embodiment of the video decoding method based on this disclosure acquires a video bitstream that includes encoded data based on a video signal from which pictures in the video have been downsampled and dummy data corresponding to the downsampled pictures, decodes the video bitstream to reproduce the video signal, and restores the pictures downsampled by the video encoding device from the video signal.

[0042] The video encoding device according to the present disclosure includes a thinning means for thinning pictures in a video, an encoding means for encoding a video signal after the pictures are thinned, and an auxiliary information generation means for generating auxiliary information for deriving the maximum latency picture number.

[0043] Another aspect of the video encoding device according to the present disclosure includes a thinning means for thinning pictures in a video, an encoding means for encoding a video signal after the pictures are thinned, and a control means for generating a picture timing SEI including information capable of specifying an output delay from a decoded picture buffer of a decoded picture based on encoding and information capable of specifying a delay value that increases due to an interpolation process in a video decoding device.

[0044] Another aspect of the video encoding device according to the present disclosure includes a thinning means for thinning pictures in a video, an encoding means for encoding a video signal after the pictures are thinned, and a setting means for setting dummy data corresponding to the thinned pictures in a video bit stream.

[0045] The video decoding device according to the present disclosure includes an acquisition means for acquiring the video bit stream and the auxiliary information bit stream from a bit stream in which a video bit stream including encoded data based on a video signal in which pictures in a video are thinned and an auxiliary information bit stream including auxiliary information for deriving the maximum latency picture number are multiplexed, a decoding means for decoding the video bit stream, and a calculation means for calculating the maximum latency picture number from the auxiliary information.

[0046] Another aspect of the video decoding apparatus according to the present disclosure includes an acquisition unit that acquires the video bitstream including the encoded data based on the video signal with pictures thinned out in the video and the auxiliary information bitstream in which the video bitstream and the auxiliary information bitstream including the picture timing SEI auxiliary information including information that can specify the output delay from the decoded picture buffer of the decoded picture based on encoding and information that can specify the delay value increased due to the interpolation process are multiplexed, a decoding unit that decodes the video bitstream, and a calculation unit that calculates the extraction time from the decoded picture buffer from the picture timing SEI auxiliary information.

[0047] Another aspect of the video decoding apparatus according to the present disclosure includes an acquisition unit that acquires a video bitstream including encoded data based on a video signal with pictures thinned out in the video and dummy data corresponding to the thinned-out pictures, a decoding unit that decodes the video bitstream to reproduce the video signal, and a restoration unit that restores the pictures thinned out in the video encoding apparatus from the video signal.

[0048] The video encoding program according to the present disclosure causes a computer to execute a process of thinning out pictures in the video, a process of encoding the video signal after the pictures have been thinned out, and a process of generating auxiliary information for deriving the maximum latency picture number.

[0049] Another aspect of the video encoding program according to the present disclosure causes a computer to execute a process of thinning out pictures in the video, a process of encoding the video signal after the pictures have been thinned out, and a process of generating a picture timing SEI including information that can specify the output delay from the decoded picture buffer of the decoded picture based on encoding and information that can specify the delay value increased due to the interpolation process in the video decoding apparatus.

[0050] Another embodiment of the video encoding program based on this disclosure causes a computer to perform the following processes: downsampling of pictures in a video; encoding the video signal after the picture downsampling has been performed; and setting dummy data corresponding to the downsampled pictures into the video bitstream.

[0051] The video decoding program based on this disclosure causes a computer to perform the following processes: obtain the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled, and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count; decode the video bitstream; and calculate the maximum latency picture count from the auxiliary information.

[0052] A video decoding program in another aspect of the present disclosure causes a computer to perform the following processes: acquire the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed, the video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled; and the auxiliary information bitstream containing picture timing SEI auxiliary information that includes information that can identify the output delay of the decoded picture from the decoded picture buffer based on the encoding and information that can identify the delay value that increases due to interpolation; decode the video bitstream; and calculate the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

[0053] A video decoding program in another aspect of the present disclosure causes a computer to perform the following processes: acquire a video bitstream including encoded data based on a video signal from which pictures in the video have been downsampled and dummy data corresponding to the downsampled pictures; decode the video bitstream and reproduce the video signal; and restore the pictures downsampled by the video encoding device from the video signal.

[0054] According to the present invention, even when time sampling / time resampling is used, interoperability between the video decoding device and the video encoding device can be ensured regarding the processing timing of the video decoding device.

[0055] This is a block diagram showing a video encoder and video decoder that encode and decode each picture that makes up video data. This is an explanatory diagram showing an example of time sampling processing. This is an explanatory diagram showing an example of time resampling processing. This is an explanatory diagram showing an example of an extended sequence parameter set. This is an explanatory diagram showing how the decoded video is interpolated in multiple stages. This is an explanatory diagram showing a list of increases in the maximum number of sorted pictures and the maximum latency increase. This is an explanatory diagram showing an example of additional parameters related to DPB. This is a flowchart showing an example of the operation of the video encoder. This is a flowchart showing an example of the operation of the video decoder. This is an explanatory diagram showing an example of PTSEI parameters when implicit control is performed. This is an explanatory diagram showing an example of PTSEI parameters when explicit control is performed. This is an explanatory diagram showing an example of dummy picture insertion. This is an explanatory diagram showing how the decoded video is interpolated in multiple stages. This is an explanatory diagram showing a list of increases in the maximum number of sorted pictures and the maximum latency increase. This is a block diagram showing an example of the configuration of a pre-processing unit that performs time sampling, targeting, spatial sampling, and bit depth truncation. This is a block diagram showing an example of the configuration of a post-processing unit that performs spatial resampling, retargeting, time resampling, and bit depth shifting. This is an explanatory diagram showing the targeting process. This is an explanatory diagram showing the targeting process. This is an explanatory diagram showing the targeting process. This is an explanatory diagram showing the retargeting process. This is an explanatory diagram showing an example of an extended sequence parameter set for targeting and retargeting. This is an explanatory diagram showing an example of an extended picture parameter set for targeting and retargeting. This is an explanatory diagram showing spatial sampling and spatial resampling. This is an explanatory diagram showing an example of an extended sequence parameter set for spatial sampling and spatial resampling. This is an explanatory diagram showing an example of an extended sequence parameter set for spatial sampling and spatial resampling. This is an explanatory diagram showing an example of an extended sequence parameter set for bit depth truncation and bit depth shifting. This is a block diagram showing an example of the configuration of an information processing system. This is a block diagram showing the main parts of a video encoding device. This is a block diagram showing the main parts of a video decoding device.This is an explanatory diagram illustrating the SOP structure. It is an explanatory diagram illustrating the values ​​of dpb_max_num_reorder_pics[i] and dpb_max_latency_increase_plus1[i].

[0056] The embodiments will be described below with reference to the drawings.

[0057] Embodiment 1. Figure 1 is a block diagram showing a video encoder 100 and a video decoder 200 that encode and decode each picture constituting video data.

[0058] The video encoder 100 includes a pre-processing unit 101, an internal video encoder (internal encoder) 102, a multiplexer 103, and a control unit 104.

[0059] The video decoder 200 comprises a post-processing unit 201, an internal video decoder (internal decoder) 202, a demultiplexer 203, and a control unit 204. The internal video decoder 202 has a decoded picture buffer (DPB) for storing decoded pictures.

[0060] Note that the arrows in Figure 1 simply indicate the direction of signal (data) flow, but do not exclude bidirectional flow. This is also true for other block diagrams.

[0061] [Explanation of the Encoding Side] In the video encoder 100, the pre-processing unit 101 performs pre-processing. In the video decoder 200, the post-processing unit 201 performs post-processing. As described above, pre-processing may include temporal sampling. Post-processing may include temporal resampling. Hereinafter, it will be assumed that the pre-processing unit 101 performs temporal sampling as pre-processing, and the post-processing unit 201 performs temporal resampling as post-processing.

[0062] First, let's explain time sampling and time resampling.

[0063] Time sampling reduces the number of pictures encoded by internal video encoding by downsampling the input video. Time sampling has the effect of reducing the amount of code in the bitstream. Time resampling interpolates the pictures that were downsampled by time sampling in multiple stages. Time resampling restores the time resolution that was lost due to the downsampled pictures.

[0064] The maximum number of decimated pictures, MaxTPics, based on time sampling is defined, for example, by the following equation T1.

[0065] MaxTPics = (2^(max_temporal_resampling_ratio_idx_minus1+1)) - 1 (Formula T1)

[0066] max_temporal_resampling_ratio_idx_minus1 is a parameter (syntax element) that is multiplexed into the bitstream, as will be described later.

[0067] As can be seen from equation T1, max_temporal_resampling_ratio_idx_minus1 can be expressed by the following equation T2.

[0068] max_temporal_resampling_ratio_idx_minus1 = log2(MaxTPics + 1) (Formula T2)

[0069] Figure 2A is an explanatory diagram showing an example of time sampling processing performed by the preprocessing unit 101. Figure 2B is an explanatory diagram showing an example of time resampling processing performed by the postprocessing unit 201. Note that Figures 2A and 2B illustrate the case where max_temporal_resampling_ratio_idx_minus1=1, i.e., MaxTPics=3.

[0070] As illustrated in Figure 2A, the amount of encoded data is reduced by performing internal encoding after the picture has been downsampled by time sampling. Furthermore, as illustrated in Figure 2B, the downsampled picture is restored in the video decoder 200 by time resampling, for example, multi-stage interpolation.

[0071] Parameters related to time sampling and time resampling can be multiplexed into a bitstream using the extended sequence parameter set illustrated in Figure 3.

[0072] In the example shown in Figure 3, temporal_resampling_flag=1 indicates that temporal sampling / resampling is enabled. temporal_resampling_flag=0 indicates that temporal sampling / resampling is disabled.

[0073] max_temporal_resampling_ratio_idx_minus1 is used to derive the variable TemporalResamplingRatio as follows: In equation T3, the value of max_temporal_resampling_ratio_idx_minus1 is in the range of 0 to 3.

[0074] TemporalResamplingRatio = 2(max_temporal_resampling_ratio_idx_minus1 + 1) (Formula T3)

[0075] Figure 4 is an explanatory diagram illustrating how the decoded video is interpolated in multiple stages. Figure 4 shows how the interpolation is performed in multiple stages for each value of max_temporal_resampling_ratio_idx_minus1. The multiple interpolation exemplified in Figure 4 is referred to as Pattern 1.

[0076] Furthermore, Figure 5 provides an example of the increase in the maximum number of reordered pictures (dpb_max_num_reorder_pics_increase) and the increase in the maximum latency (dpb_max_latency_increase_plus1_increase) that is achieved through multi-stage interpolation. The values ​​targeted for increase are those of dpb_max_num_reorder_pics[i] and dpb_max_latency_increase_plus1[i], as exemplified in Figure 28.

[0077] In the multi-stage interpolation of Pattern 1, the post-processing unit 201, similar to the SOP structure shown in Figure 27, does not interpolate the same time layer consecutively in order to minimize the processing delay until display when processing each picture sequentially. Furthermore, within the same time layer, the post-processing unit 201 interpolates the one that appears earlier in the display order first.

[0078] In the video encoder 100, the preprocessor 101 applies the above-described time sampling to the input video. The preprocessor 101 supplies the video to which time sampling has been applied to the internal video encoder 102. The preprocessor 101 also supplies the above-described temporal_resampling_flag and max_temporal_resampling_ratio_idx_minus1 to the control unit 104 as auxiliary information.

[0079] The internal video encoder 102 encodes the video supplied from the preprocessing unit 101 and outputs a video bitstream. The control unit 104 generates an auxiliary information bitstream for post-processing in the video decoder 200.

[0080] The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream and outputs them.

[0081] Furthermore, the control unit 104 implicitly or explicitly controls the maximum number of latency pictures, including time sampling (MaxLatencyPictures_after_TR), as follows:

[0082] When implicitly controlling the maximum number of latency pictures, including time sampling, the control unit 104 derives MaxLatencyPictures_after_TR as follows, according to max_temporal_resampling_ratio_idx_minu1, as shown in Figure 5.

[0083] MaxLatencyPictures_after_TR = MaxLatencyPictures[ max_sublayers_minus1 ] + dpb_max_num_reorder_pics_increase + dpb_max_latency_increase_plus1_increase (Equation T4) where, dpb_max_num_reorder_pics_increase = max_temporal_resampling_ratio_idx_minu1 + 1 (Equation T5) dpb_max_latency_increase_plus1_increase = 2(max_temporal_resampling_ratio_idx_minus1 + 1) (Equation T6) In other words, MaxLatencyPictures_after_TR is derived with respect to the max_temporal_resampling_ratio_idx_minu1 used. Therefore, no additional parameters related to DPB are introduced.

[0084] When explicitly controlling the maximum number of latency pictures, including time sampling, additional parameters are introduced to the DPB parameters, as illustrated in Figure 6.

[0085] In the example shown in Figure 6, dpb_max_num_reorder_pics_increase represents the maximum number of reordered pictures increased by the time resampling interpolation process in the video decoder 200. dpb_max_latency_increase_plus1_increase represents the maximum latency increase increased by the time resampling interpolation process in the video decoder 200.

[0086] Figure 7 is a flowchart showing an example of the operation of the video encoder 100. The operation of the video encoder 100 will be explained with reference to Figure 7.

[0087] In the video encoder 100, the preprocessing unit 101 performs time sampling as a preprocessing step (step S101). The internal video encoder 102 encodes the time-sampled video data supplied from the preprocessing unit 101 to generate a video bitstream (step S102).

[0088] The control unit 104 generates an auxiliary information bitstream (step S103). In this embodiment, the auxiliary information includes at least the DPB parameters and the above-mentioned temporal_resampling_flag and max_temporal_resampling_ratio_idx_minus1. If the maximum number of latency pictures, including time sampling, is to be explicitly controlled, the auxiliary information includes the above-mentioned dpb_max_num_reorder_pics_increase and dpb_max_latency_increase_plus1_increase.

[0089] The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream (step S104).

[0090] [Decoding Explanation] In the video decoder 200, the demultiplexer 203 demultiplexes the bitstream to obtain a video bitstream and an auxiliary information bitstream. The internal video decoder 202 decodes the video bitstream supplied from the demultiplexer 203 to obtain a decoded video with internal video encoding.

[0091] The control unit 204 decodes the auxiliary information bitstream supplied from the demultiplexer 203 to obtain control information for post-processing. The control unit 204 also controls MaxLatencyPictures_after_TR implicitly or explicitly, as described below.

[0092] When implicitly controlling the maximum number of latency pictures, including temporal resampling, the control unit 204 derives MaxLatencyPictures_after_TR according to max_temporal_resampling_ratio_idx_minu1 obtained from the auxiliary information bitstream, as follows:

[0093] MaxLatencyPictures_after_TR = MaxLatencyPictures[ max_sublayers_minus1 ] + max_temporal_resampling_ratio_idx_minu1 + 1 + 2(max_temporal_resampling_ratio_idx_minus1 + 1) (Formula T7)

[0094] When explicitly controlling the maximum number of latency pictures, including time resampling, the control unit 204 calculates MaxLatencyPictures_after_TR using equation T4, with the decoded values ​​of the additional DPB parameters described above, dpb_max_num_reorder_pics_increase and dpb_max_latency_increase_plus1_increase.

[0095] The post-processing unit 201 applies post-processing (in this embodiment, time resampling) based on control information supplied from the control unit 204 to the decoded video of the internal video encoding supplied from the internal video decoder 202 to obtain the final decoded video. The post-processing unit 201 controls the post-processing with a fixed delay of MaxLatencyPictures_after_TR.

[0096] Figure 8 is a flowchart showing an example of the operation of the video decoder 200. The operation of the video decoder 200 will be explained with reference to Figure 8.

[0097] In the video decoder 200, the demultiplexer 203 demultiplexes the bitstream (step S201). The internal video decoder 202 decodes the video bitstream to obtain the decoded video (step S202).

[0098] The control unit 204 decodes the auxiliary information bitstream to obtain control information for post-processing (step S203). The post-processing unit 201 executes post-processing based on the control information (step S204).

[0099] In this embodiment, the maximum number of latency pictures, including time resampling, can be controlled implicitly or explicitly by the control of the control unit 104 in the video encoder 100 and the control unit 204 in the video decoder 200. As a result, interoperability between the video encoder and the video decoder can be ensured in a video encoding scheme that uses time sampling / time resampling. Furthermore, even if the DPB parameter of the bitstream output of the internal video encoder 102 is a value that does not include time resampling processing, the maximum number of latency pictures, including time resampling, can still be controlled.

[0100] Embodiment 2. In the first embodiment, the maximum number of latency pictures in the DPB, including time resampling, can be controlled implicitly or explicitly. In the second embodiment, the decoded picture DPB output delay of the PTSET, including time resampling, is controlled implicitly or explicitly.

[0101] The configuration of the video encoder 100 and video decoder 200 in this embodiment is the same as the configuration shown in Figure 1.

[0102] [Explanation of the Encoding Side] Figure 9 is an explanatory diagram showing an example of the PTSEI parameters when implicit control is performed. When implicit control is performed, the control unit 104 in the video encoder 100 generates an auxiliary information bitstream that includes the PTSEI of pic_dpb_output_delay, a value based on the internal video encoding, as illustrated in Figure 9.

[0103] Figure 10 is an explanatory diagram showing an example of PTSEI parameters when explicit control is implemented. When explicit control is implemented, the control unit 104 generates a bitstream of auxiliary information including the PTSEI (see Figure 10) of pic_dpb_output_delay, a value based on internal video encoding, and pic_dpb_output_delay_increase, a delay value that increases due to time resampling in the video decoder 200.

[0104] The increased delay value, using I / P / B0 in Figure 4 as an example, is (dpb_max_num_reorder_pics_increase + dpb_max_latency_increase_plus1_increase).

[0105] [Decoding side explanation] When implicit control is performed, the control unit 204 in the video decoder 200 sets the extraction time from the DPB, starting from the buffering delay time, to au_cpb_removal_delay_minus1+1 + pic_dpb_output_delay + the above increasing delay value.

[0106] When explicit control is implemented, the control unit 204 calculates the withdrawal time from the DPB, starting from the buffering delay time, as au_cpb_removal_delay_minus1+1 + pic_dpb_output_delay + pic_dpb_output_delay_increase.

[0107] In this embodiment, picture timing, including time resampling, can be controlled implicitly or explicitly by the control of the control unit 104 in the video encoder 100 and the control unit 204 in the video decoder 200. As a result, interoperability between the video encoder and the video decoder can be ensured in a video encoding scheme that uses time sampling / time resampling.

[0108] Embodiment 3. In a video encoding scheme using time sampling / time resampling, the cause of impaired interoperability between the video encoder and the video decoder with respect to processing timing is as follows: In the video decoder, due to picture decimation, there is a mismatch in maximum latency and picture timing between the internally decoded video derived from the bitstream and the final output video.

[0109] To eliminate such problems, one possible approach in the video encoder is to perform picture decimation and then insert a picture consisting of dummy data corresponding to the decimated picture data. Figure 11 is an explanatory diagram showing an example of dummy picture insertion.

[0110] The configuration of the video encoder 100 and video decoder 200 in this embodiment is the same as the configuration shown in Figure 1.

[0111] In each of the embodiments described above, when time sampling is used, as illustrated in Figure 2A and on the left side of Figure 11, the encoded data of the downsampled pictures is not transmitted. In this embodiment, in the video encoder 100, the internal video encoder 102 includes dummy data of the encoded data of the downsampled pictures B(n+1,n+2), B(n+2,n+3), and B(n+3,n+4), as illustrated in Figure 2A, in the video bitstream. That is, the internal video encoder 102 generates a video bitstream that includes encoded data based on the video signal from which the pictures in the video have been downsampled, and dummy data corresponding to the downsampled pictures. The content of the dummy data is arbitrary.

[0112] Furthermore, the internal video encoder 102 reduces the amount of dummy data to less than the amount of data of the encoded data of the pictures B(n+1,n+2), B(n+2,n+3), and B(n+3,n+4) that are to be downsampled. In addition, the control unit 104 sets the Temporal ID of the dummy data according to temporal_resampling_ratio_idx_minus1 and the transmission order.

[0113] Blocks other than the internal video encoder 102 (for example, the multiplexer 103) may set dummy data.

[0114] In the video decoder 200, the internal video decoder 202 does not decode the dummy data contained in the video bitstream. The post-processing unit 201 restores the pictures that were thinned out in the video encoder 100, for example, by multi-stage interpolation. At this time, the post-processing unit 201 restores the thinned-out pictures by referring to the Temporal ID of the dummy data.

[0115] The video decoder 200 does not perform time resampling, but it can recognize the existence of pictures that should be restored by time resampling, thus eliminating the cause of mismatch between maximum latency and picture timing between the internally decoded video and the final output video. Therefore, in this embodiment as well, in a video encoding scheme that uses time sampling / time resampling, interoperability between the video encoder and the video decoder can be ensured.

[0116] Embodiment 4. In the first embodiment, the maximum latency picture count of the DPB, including time resampling, is controlled, and in the second embodiment, the decoded picture DPB output delay of the PTSET, including time resampling, is controlled.

[0117] The video encoder 100 and the video decoder 200 may be configured to perform both the control of the first embodiment and the control of the second embodiment.

[0118] Furthermore, the video encoder 100 and the video decoder 200 may be configured to perform both the control of the second embodiment and the control of the third embodiment.

[0119] Embodiment 5. In the multi-stage interpolation of time resampling pattern 1 in the first embodiment, the post-processing unit 201 does not interpolate the same time hierarchical level consecutively, and within the same time hierarchical level, it interpolates the earlier display order first.

[0120] However, if the pictures can be processed in parallel rather than sequentially, it is acceptable to prioritize and interpolate pictures at the same time level consecutively.

[0121] Figure 12 is an explanatory diagram illustrating how decoded video undergoes multi-stage interpolation when pictures of the same time hierarchical level are successively interpolated. Figure 12 illustrates the multi-stage interpolation process for cases where max_temporal_resampling_ratio_idx_minus1=2 and max_temporal_resampling_ratio_idx_minus1=3. The multi-stage interpolation exemplified in Figure 12 is referred to as Pattern 2 multi-stage interpolation.

[0122] Figure 13 shows an example of the increase in the maximum number of reordered pictures (dpb_max_num_reorder_pics_increase) and the increase in the maximum latency (dpb_max_latency_increase_plus1_increase) that is achieved through multi-stage interpolation. Figure 13 shows a list corresponding to the multi-stage interpolation exemplified in Figure 12. The values ​​of dpb_max_num_reorder_pics[i] and dpb_max_latency_increase_plus1[i] that are targeted for increase are those exemplified in Figure 28.

[0123] As shown in Figure 12, in the multi-stage interpolation of pattern 2, the post-processing unit 201 continuously interpolates pictures of the same time hierarchical level.

[0124] Embodiment 6. In each of the above embodiments, the pre-processing unit 101 in the video encoder 100 performed time sampling as pre-processing, and the post-processing unit 201 in the video decoder 200 performed time resampling as post-processing. However, the pre-processing unit 101 may also perform other processes as pre-processing. Similarly, the post-processing unit 201 may also perform other processes as post-processing.

[0125] Other preprocessing steps include Targeting, Spatial sampling, and Bit Depth truncation. Other postprocessing steps include Retargeting, Spatial resampling, and Bit Depth shift.

[0126] Those processes are as follows:

[0127] Targeting and Retargeting: The video encoder reduces the amount of encoded data by lowering the image quality of non-focus areas compared to the image quality of the region of interest (ROI) in the video. The video encoder also shrinks the focus area, which is of relatively low importance. The video decoder then enlarges (restores) the shrunken focus area.

[0128] Spatial sampling and spatial resampling: The video encoder reduces the frame rate in the time domain, and the video decoder upsamples the frame rate.

[0129] Bit depth truncation and bit depth shifting: The video encoder reduces the number of bits used to represent the image, and the video decoder restores the number of bits to its original value.

[0130] Figure 14 is a block diagram showing an example configuration of a preprocessor 101 that performs targeting, spatial sampling, and bit depth truncation in addition to time sampling. The preprocessor 101 shown in Figure 14 comprises a time sampling unit 111, a targeting unit 112, a spatial sampling unit 113, and a bit depth truncation unit 114. The output of the bit depth truncation unit 114 is supplied to the internal video encoder 102. Auxiliary information is supplied to the multiplexer 103.

[0131] Figure 15 is a block diagram showing an example configuration of a post-processing unit 201 that performs spatial resampling, retargeting, temporal resampling, and bit depth shifting. The post-processing unit 201 shown in Figure 15 comprises a spatial resampling unit 211, a retargeting unit 212, a temporal resampling unit 213, and a bit depth shifting unit 214. The spatial resampling unit 211 is supplied with a decoded video bitstream from an internal video decoder 202. Auxiliary information is supplied from a demultiplexer 203.

[0132] The configuration of the video encoder 100 and video decoder 200 in this embodiment is the same as the configuration shown in Figure 1, although the configurations of the pre-processing unit 101 and post-processing unit 201 differ.

[0133] The time sampling unit 111 in the preprocessing unit 101 performs the time sampling process in each of the above embodiments. The time resampling unit 213 in the postprocessing unit 201 performs the time resampling process in each of the above embodiments.

[0134] The targeting unit 112 in the preprocessing unit 101 derives a gaze region and its preferred magnification from the input video. Furthermore, the targeting unit 112 reduces the gaze region by each magnification and changes the color of the other areas (non-gaze regions) to, for example, gray. Note that gray is an example. Targeting can be used to reduce the amount of code in the bitstream.

[0135] The retargeting unit 212 in the post-processing unit 201 restores the gaze area to its original size. In other words, the spatial resolution of the reduced gaze area is restored.

[0136] The targeting process by the preprocessing unit 101 will be explained in more detail with reference to Figures 16A to 16C. We will use the example of the input picture shown in Figure 16A being input to the preprocessing unit 101.

[0137] The targeting unit 112 can obtain a rectangular area of ​​a machine processing task, as illustrated in Figure 16B. The machine processing task is, for example, video coding for machines (VCM) used for machine recognition. As a method for obtaining the rectangular area of ​​the machine processing task, the targeting unit 112 may, as an example, use the object detection method described in Non-Patent Literature 4. Figures 16B to 16C illustrate four gaze areas. Information indicating the position of each gaze area is included in the auxiliary information.

[0138] In the examples shown in Figures 16B to 16C, a magnification of 0.5 indicates that the area is reduced by a factor of 0.5 in both the horizontal and vertical directions. The same applies to magnifications of 0.75 and 1.00. Note that these magnification values ​​are illustrative. Furthermore, a smaller reduction ratio means a lower degree of reduction. For example, a magnification of 0.75 has a smaller reduction ratio than a magnification of 0.5.

[0139] The higher the magnification value, the more important the region. In other words, the preprocessing unit 101 increases the magnification of the areas of focus that are assumed to be important, and decreases the magnification of areas of focus that are not, thereby reducing the area of ​​focus. This reduction is achieved, for example, by downsampling the samples. Information indicating the magnification is included in the auxiliary information.

[0140] The targeting unit 112 changes the color of areas other than the gaze area (non-gaze area) to gray, for example, in order to reduce the amount of information in those areas. In Figure 16C, the gray areas are indicated by diagonal lines.

[0141] Referring to Figure 17, the retargeting process performed by the retargeting unit 212 in the post-processing unit 201 will be explained. Figure 17 illustrates four gaze regions.

[0142] The retargeting unit 212 enlarges the gaze area in the video exemplified in Figure 16C based on information indicating the magnification at the position indicated by information indicating the position of the gaze area. Enlargement is achieved, for example, by sample interpolation (upsampling). The information indicating the position of the gaze area and the information indicating the magnification are obtained from an auxiliary information bitstream obtained by demultiplexing the bitstream.

[0143] The post-processing unit 201 restores the gaze region in the original image, as illustrated in Figure 17.

[0144] The parameters for targeting and retargeting are multiplexed using an extended sequence parameter set and an extended picture parameter set, as shown in Figures 18 and 19. Figure 18 is an explanatory diagram showing an example of an extended sequence parameter set for targeting and retargeting. Figure 19 is an explanatory diagram showing an example of an extended picture parameter set for targeting and retargeting.

[0145] In the extended sequence parameter set, retargeting_flag=1 indicates that targeting is enabled, and retargeting_flag=0 indicates that targeting is disabled.

[0146] In the extended picture parameter set, num_rois indicates the number of gaze regions. roi_idc[i] indicates the magnification. For example, if its value is 0 (000), the magnification is 1.0 (100%). If its value is 1 (001), the magnification is 0.75 (75%). If its value is 2 (010), the magnification is 0.5 (50%).

[0147] roi_pos_x1[i] represents the x-coordinate of the upper left corner of the gaze region. roi_pos_y1[i] represents the y-coordinate of the upper left corner of the gaze region. roi_pos_x2[i] represents the x-coordinate of the lower right corner of the gaze region. roi_pos_y2[i] represents the y-coordinate of the lower right corner of the gaze region.

[0148] Refer to Figure 20 for a more detailed explanation of spatial sampling and spatial resampling.

[0149] The spatial sampling unit 113 in the preprocessing unit 101 derives a picture magnification preferred for a machine processing task. Object detection, as described in Non-Patent Literature 4, can be used as the machine processing task. As illustrated in Figure 20, the spatial sampling unit 113 reduces the picture by the derived magnification. As a result, the amount of code in the bitstream is reduced. The information regarding the picture magnification obtained through spatial sampling is multiplexed into the bitstream as auxiliary information.

[0150] The spatial resampling unit 211 in the post-processing unit 201 enlarges the picture to its original size, as illustrated in Figure 20, based on the magnification information of the picture demultiplexed from the bitstream. As a result, the spatial resolution of the reduced picture is restored.

[0151] The parameters for spatial sampling and spatial resampling are multiplexed using an extended sequence parameter set and an extended picture parameter set, as shown in Figures 21 and 22. Figure 21 is an explanatory diagram showing an example of an extended sequence parameter set for spatial sampling and spatial resampling. Figure 22 is an explanatory diagram showing an example of an extended picture parameter set for spatial sampling and spatial resampling.

[0152] In the extended sequence parameter set, spatial_resampling_flag=1 indicates that targeting is enabled, while spatial_resampling_flag=0 indicates that targeting is disabled.

[0153] In the extended picture parameter set, spatial_upsample_width represents the upsampling width, and spatial_upsample_height represents the upsampling height.

[0154] Next, we will explain bit depth truncation and bit depth shifting in more detail.

[0155] The bit depth truncation unit 114 in the preprocessing unit 101 discards the lower bit of the luminance component of the video. This process is equivalent to halving the luminance. This process is also equivalent to shifting the luminance component one bit to the right. Bit depth truncation reduces the amount of data in the bitstream.

[0156] The bit depth shift unit 214 in the post-processing unit 201 shifts the luminance component in the decoded video one bit to the left. This process is equivalent to doubling the luminance. This process restores the brightness of the video.

[0157] Furthermore, information regarding the number of bits of the luminance component to be discarded by the bit depth truncation unit 114 is multiplexed into the bitstream as auxiliary information. The bit depth shift unit 214 then shifts the luminance component to the left based on the bit number information demultiplexed from the bitstream. This control allows for manipulation of any desired number of bits.

[0158] The parameters for bit depth truncation and bit depth shifting are multiplexed in an extended picture parameter set, as shown in Figure 23. Figure 23 is an explanatory diagram illustrating an example of an extended sequence parameter set for bit depth truncation and bit depth shifting.

[0159] In the extended picture parameter set, bit_depth_shift_flag=1 indicates that bit depth truncation is enabled. bit_depth_shift_flag=0 indicates that bit depth truncation is disabled. bit_depth_shift_luma specifies that the luminance component (luma) is shifted to the left by this value. bit_depth_shift_chroma specifies that the brightness difference component (chroma) is shifted to the left by this value.

[0160] While the above embodiment can be implemented using hardware, it can also be realized using a computer program.

[0161] The information processing system shown in Figure 24 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Multiple processors 701 may be provided. The storage mediums 703 and 704 may be separate storage mediums or may be storage areas consisting of the same storage medium. Magnetic storage media such as hard disks can be used as storage media. Furthermore, the information processing system may include multiple processors 701.

[0162] In the information processing system, the program memory 702 stores programs (video encoding programs or video decoding programs) for realizing the functions of each block shown in the above embodiment.

[0163] The processor 701 then performs processing according to the program stored in the program memory 702, thereby realizing the functions of the video encoder 100 or video decoder 200 shown in the above embodiment.

[0164] For example, the functions of the video encoder 100 are realized when the processor 701 executes processing according to a video encoding program that realizes the functions of each block in the video encoder 100 shown in Figure 1.

[0165] Furthermore, for example, the processor 701 executes processing according to a video decoding program that realizes the functions of each block in the video decoder 200 shown in Figure 1, thereby realizing the functions of the video decoder.

[0166] Furthermore, at least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transient computer-readable medium. The program is supplied to the transient computer-readable medium, for example, via a wired communication channel or a wireless communication channel, i.e., via electrical signals, optical signals or electromagnetic waves.

[0167] Figure 25 is a block diagram showing the main parts of a video encoding device. The video encoding device 10 shown in Figure 25 (implemented by a video encoder 100 in this embodiment) comprises a decimation means 11 for decimating pictures in the video (implemented by a pre-processing unit 101 or a time sampling unit 111 in this embodiment) for encoding the video signal after picture decimation (implemented by an internal video encoder 102 in this embodiment) and an auxiliary information generation means 13 for generating auxiliary information for deriving the maximum latency picture count (implemented by a control unit 104 in this embodiment).

[0168] Furthermore, auxiliary information for deriving the maximum latency picture count is, for example, max_temporal_resampling_ratio_idx_minus1. In that case, as described above, the maximum latency picture count is controlled based on max_temporal_resampling_ratio_idx_minus1.

[0169] Auxiliary information for deriving the maximum latency picture count may be information added to the DPB (Decoded Picture Buffer) parameter that can identify the maximum number of reordered pictures and the maximum latency increase that increase due to interpolation processing in the video decoder.

[0170] Figure 26 is a block diagram showing the main part of the video decoding device. The video decoding device 20 shown in Figure 26 (implemented by a video decoder 200 in this embodiment) includes an acquisition means 21 (implemented by a demultiplexer 203 in this embodiment) that acquires the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count; a decoding means 22 (implemented by an internal video decoder 202 in this embodiment) that decodes the video bitstream; and a calculation means 23 (implemented by a control unit 204 in this embodiment) that calculates the maximum latency picture count from the auxiliary information.

[0171] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0172] (Note 1) A video encoding method that performs picture decimation in video, encodes the video signal after picture decimation, and generates auxiliary information for deriving the maximum latency picture count.

[0173] (Note 2) A video encoding method that performs picture decimation in a video, encodes the video signal after the picture decimation, and generates picture timing SEI (Supplemental Enhancement Information) that includes information that can identify the output delay from the decoded picture buffer of the decoded picture and information that can identify the delay value that increases due to interpolation processing in the video decoding device, based on the encoding.

[0174] (Note 3) A video encoding method that performs picture decimation in a video, encodes the video signal after the picture decimation has been performed, and sets dummy data corresponding to the decimated pictures into the video bitstream.

[0175] (Note 4) The video encoding method according to Note 1, which generates a picture timing SEI that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing in the video decoding device, based on the encoding.

[0176] (Note 5) The video encoding method according to Note 3, which generates a picture timing SEI that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing in the video decoding device, based on the encoding.

[0177] (Note 6) A video decoding method comprising obtaining a video bitstream and an auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures have been downsampled in the video, and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count, decoding the video bitstream, and calculating the maximum latency picture count from the auxiliary information.

[0178] (Note 7) A video decoding method that obtains the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled, and an auxiliary information bitstream containing picture timing SEI auxiliary information that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing, based on the encoding, decodes the video bitstream, and calculates the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

[0179] (Note 8) A video decoding method comprising: acquiring a video bitstream that includes encoded data based on a video signal from which pictures have been downsampled and dummy data corresponding to the downsampled pictures; decoding the video bitstream to reproduce the video signal; and restoring the pictures downsampled by the video encoding device from the video signal.

[0180] (Note 9) The video decoding method according to Note 6, wherein the auxiliary information includes picture timing SEI auxiliary information which includes information that can identify the output delay of the decoded picture from the decoded picture buffer based on encoding and information that can identify the delay value that increases due to interpolation processing, the method comprising obtaining the picture timing SEI auxiliary information from the auxiliary information bitstream and calculating the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

[0181] (Note 10) The video decoding method according to Note 8, which includes picture timing SEI auxiliary information from an auxiliary information bitstream that includes information that can identify the output delay of the decoded picture from the decoded picture buffer based on encoding and information that can identify the delay value that increases due to interpolation processing, and which obtains the picture timing SEI auxiliary information and calculates the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

[0182] (Note 11) A video encoding device comprising: a decimation means for decimating pictures in a video; an encoding means for encoding the video signal after the picture decimation has been performed; and an auxiliary information generation means for generating auxiliary information for deriving the maximum latency picture count.

[0183] (Note 12) A video encoding device comprising: a decimation means for decimating pictures in a video; an encoding means for encoding a video signal after the picture decimation has been performed; and a control means (implemented in the control unit 104 in this embodiment) for generating a picture timing SEI that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing in the video decoding device, based on the encoding.

[0184] (Note 13) A video encoding device comprising: a decimation means for decimating pictures in a video; an encoding means for encoding the video signal after the picture decimation has been performed; and a setting means (in this embodiment, implemented by an internal video encoder 102) for setting dummy data corresponding to the decimated pictures into the video bitstream.

[0185] (Note 14) A video decoding device comprising: acquisition means for acquiring a video bitstream and an auxiliary information bitstream from a bitstream which is multiplexed a video bitstream containing encoded data based on a video signal in which pictures have been downsampled in the video, and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count; decoding means for decoding the video bitstream; and calculation means for calculating the maximum latency picture count from the auxiliary information.

[0186] (Note 15) An image decoding device comprising: an acquisition means for acquiring the image bitstream and the auxiliary information bitstream from a bitstream which is multiplexed the image bitstream which includes an image bitstream which includes encoded data based on an image signal from which pictures in the image have been downsampled; an auxiliary information bitstream which includes picture timing SEI auxiliary information which includes information that can identify the output delay from the decoded picture buffer of the decoded picture based on the encoding and information that can identify the delay value that increases due to interpolation processing; a decoding means for decoding the image bitstream; and a calculation means (implemented by the control unit 204 in this embodiment) for calculating the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

[0187] (Note 16) A video decoding device comprising: an acquisition means for acquiring a video bitstream that includes encoded data based on a video signal from which pictures in the video have been downsampled and dummy data corresponding to the downsampled pictures; a decoding means for decoding the video bitstream and reproducing the video signal; and a restoration means for restoring the pictures downsampled in the video encoding device from the video signal (in this embodiment, this is implemented by a post-processing unit 201 or a time resampling unit 213).

[0188] (Note 17) A video encoding program that causes a computer to perform the following processes: a process of downsampling pictures in a video, a process of encoding the video signal after the picture downsampling has been performed, and a process of generating auxiliary information for deriving the maximum latency picture count.

[0189] (Note 18) A video encoding program for causing a computer to perform the following processes: a process of downsampling pictures in a video; a process of encoding the video signal after the picture downsampling has been performed; and a process of generating a picture timing SEI based on the encoding, which includes information that can identify the output delay from the decoded picture buffer of the decoded picture and information that can identify the delay value that increases due to interpolation processing in the video decoding device.

[0190] (Note 19) A video encoding program that causes a computer to perform the following processes: downsampling of pictures in a video, encoding the video signal after the picture downsampling has been performed, and setting dummy data corresponding to the downsampled pictures into the video bitstream.

[0191] (Note 20) A video decoding program for causing a computer to perform the following processes: obtaining a video bitstream and an auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal in which pictures have been downsampled in the video and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count; decoding the video bitstream; and calculating the maximum latency picture count from the auxiliary information.

[0192] (Note 21) A video decoding program for causing a computer to perform the following processes: acquiring the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled; and an auxiliary information bitstream containing picture timing SEI auxiliary information that includes information that can identify the output delay from the decoded picture buffer of the decoded picture based on the encoding and information that can identify the delay value that increases due to interpolation; decoding the video bitstream; and calculating the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

[0193] (Note 22) A video decoding program for causing a computer to perform the following processes: acquiring a video bitstream that includes encoded data based on a video signal from which pictures in the video have been downsampled and dummy data corresponding to the downsampled pictures; decoding the video bitstream and playing back the video signal; and restoring the pictures downsampled by the video encoding device from the video signal.

[0194] (Note 23) A storage medium for storing a bitstream generated by a video encoding method that performs picture decimation in video, encodes the video signal after the picture decimation, and generates auxiliary information for deriving the maximum latency picture count.

[0195] (Note 24) A storage medium for storing a bitstream generated by a video encoding device comprising: a decimation means for decimating pictures in a video; an encoding means for encoding the video signal after the picture decimation has been performed; and an auxiliary information generation means for generating auxiliary information for deriving the maximum latency picture count.

[0196] The configuration described in Appendix 4, which is dependent on Appendix 1 above, and some or all of Appendix 5, which is dependent on Appendix 3, can be made dependent on Appendix 11 to 13 and Appendix 17 to 19 in the same dependent relationship as Appendix 4 to 5. Also, some or all of the configuration described in Appendix 9, which is dependent on Appendix 6 above, and Appendix 10, which is dependent on Appendix 8, can be made dependent on Appendix 14 to 16 and Appendix 20 to 22 in the same dependent relationship as Appendix 9 to 10. Furthermore, not limited to Appendix 11 to 13, Appendix 17 to 19, Appendix 14 to 16 and Appendix 20 to 22, some or all of the configurations described as appendices above can be made dependent on various hardware, software, various recording means for recording software, or systems, provided that the embodiments described above are not deviated from.

[0197] Although the present invention has been described above with reference to embodiments and examples, the present invention is not limited to the above embodiments and examples. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0198] This application claims priority based on Japanese Patent Application No. 2024-168518, filed on 27 September 2024, and incorporates all of its disclosures herein.

[0199] 10 Video encoding device 11 Decimation means 12 Encoding means 13 Auxiliary information generation means 20 Video decoding device 21 Acquisition means 22 Decoding means 23 Calculation means 100 Video encoder 101 Pre-processing unit 102 Internal video encoder 103 Multiplexer 104 Control unit 111 Time sampling unit 112 Targeting unit 113 Spatial sampling unit 114 Bit depth truncation unit 200 Video decoder 201 Post-processing unit 202 Internal video decoder 203 Demultiplexer 204 Control unit 211 Spatial resampling unit 212 Retargeting unit 213 Time resampling unit 214 Bit depth shift unit 701 Processor 702 Program memory 703, 704 Storage medium

Claims

1. A video encoding method that performs picture decimation in a video, encodes the video signal after the picture decimation, and generates auxiliary information for deriving the maximum latency picture count.

2. A video encoding method that performs picture decimation in a video, encodes the video signal after the picture decimation, and generates picture timing SEI (Supplemental Enhancement Information) that includes information that can identify the output delay from the decoded picture buffer of the decoded picture and information that can identify the delay value that increases due to interpolation processing in the video decoding device, based on the encoding.

3. A video encoding method comprising: performing picture decimation in a video; encoding the video signal after the picture decimation; and setting dummy data corresponding to the decimated pictures into the video bitstream.

4. The video encoding method according to claim 1, which generates a picture timing SEI that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing in the video decoding device, based on the encoding.

5. The video encoding method according to claim 3, which generates a picture timing SEI that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing in the video decoding device, based on the encoding.

6. A video decoding method comprising obtaining a video bitstream and an auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures have been downsampled in the video, and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count, decoding the video bitstream, and calculating the maximum latency picture count from the auxiliary information.

7. A video decoding method that obtains the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled, and an auxiliary information bitstream containing picture timing SEI auxiliary information that includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing, based on the encoding, decodes the video bitstream, and calculates the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

8. A video decoding method comprising: acquiring a video bitstream that includes encoded data based on a video signal from which pictures have been downsampled and dummy data corresponding to the downsampled pictures; decoding the video bitstream to reproduce the video signal; and restoring the pictures downsampled by the video encoding device from the video signal.

9. The video decoding method according to claim 6, wherein the auxiliary information includes picture timing SEI auxiliary information that includes information that can identify the output delay of the decoded picture from the decoded picture buffer based on encoding and information that can identify the delay value that increases due to interpolation processing, the method comprising obtaining the picture timing SEI auxiliary information from the auxiliary information bitstream and calculating the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

10. The video decoding method according to claim 8, which includes picture timing SEI auxiliary information from an auxiliary information bitstream that includes information that can identify the output delay of the decoded picture from the decoded picture buffer based on encoding and information that can identify the delay value that increases due to interpolation processing, and which obtains the picture timing SEI auxiliary information and calculates the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

11. A video encoding device comprising: a decimation means for performing picture decimation in a video; an encoding means for encoding the video signal after the picture decimation has been performed; and an auxiliary information generation means for generating auxiliary information for deriving the maximum latency picture count.

12. A video encoding device comprising: a decimation means for decimating pictures in a video; an encoding means for encoding a video signal after the picture decimation has been performed; and a control means for generating a picture timing SEI based on the encoding, which includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing in the video decoding device.

13. A video encoding device comprising: a decimation means for decimating pictures in a video; an encoding means for encoding the video signal after the picture decimation has been performed; and a setting means for setting dummy data corresponding to the decimated pictures into a video bitstream.

14. A video decoding device comprising: acquisition means for acquiring a video bitstream and an auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures have been downsampled in the video, and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count; decoding means for decoding the video bitstream; and calculation means for calculating the maximum latency picture count from the auxiliary information.

15. A video decoding device comprising: an acquisition means for acquiring the video bitstream and the auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures in the video have been downsampled; and an auxiliary information bitstream containing picture timing SEI auxiliary information that includes information that can identify the output delay from the decoded picture buffer of the decoded picture based on the encoding and information that can identify the delay value that increases due to interpolation processing; a decoding means for decoding the video bitstream; and a calculation means for calculating the withdrawal time from the decoded picture buffer from the picture timing SEI auxiliary information.

16. A video decoding device comprising: acquisition means for acquiring a video bitstream including encoded data based on a video signal from which pictures have been downsampled and dummy data corresponding to the downsampled pictures; decoding means for decoding the video bitstream and reproducing the video signal; and restoration means for restoring the pictures downsampled in the video encoding device from the video signal.

17. A video encoding program that causes a computer to perform the following processes: downsampling of pictures in a video, encoding the video signal after picture downsampling, and generating auxiliary information for deriving the maximum latency picture count.

18. A video encoding program that causes a computer to perform the following processes: a process of downsampling pictures in a video; a process of encoding the video signal after the picture downsampling has been performed; and a process of generating a picture timing SEI based on the encoding, which includes information that can identify the output delay of the decoded picture from the decoded picture buffer and information that can identify the delay value that increases due to interpolation processing in the video decoding device.

19. A video encoding program that causes a computer to perform the following processes: downsampling of pictures in a video, encoding the video signal after the picture downsampling, and setting dummy data corresponding to the downsampled pictures into the video bitstream.

20. A video decoding program for causing a computer to perform the following processes: obtaining a video bitstream and an auxiliary information bitstream from a bitstream that is multiplexed with a video bitstream containing encoded data based on a video signal from which pictures have been downsampled in the video, and an auxiliary information bitstream containing auxiliary information for deriving the maximum latency picture count; decoding the video bitstream; and calculating the maximum latency picture count from the auxiliary information.

21. A storage medium for storing a bitstream generated by a video encoding method that performs picture decimation in a video, encodes the video signal after the picture decimation, and generates auxiliary information for deriving the maximum latency picture count.

Images