Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

In order to accomplish favorable postprocessing in a video encoding method in which preprocessing and postprocessing are utilized, this video encoding device includes: a preprocessing means for performing preprocessing, comprising a plurality of processes, on an input video signal; an encoding means for encoding the video signal that has been subjected to the preprocessing; and a signaling means for signaling information that can identify the processing order of processes, to be executed by a video decoding device, that correspond respectively to the plurality of processes.

Description

Video Encoding Device, Video Decoding Device, Video Encoding Method, and Video Decoding Method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to efficiently transmit or record video, a video encoding device that generates an encoded representation (hereinafter referred to as a bitstream) of an input video, and a video decoding device that decodes the bitstream to generate a decoded video are used.

[0003] As standards for video encoding methods, there are H.265 / HEVC (High-Efficiency Video Coding) (see Non-Patent Document 1), H.266 / VVC (Versatile Video Coding) (see Non-Patent Document 2), etc., which are standardized by ITU-T SG16 and ISO / IEC / SC29.

[0004] In these video encoding methods, video data is managed in a hierarchical structure and encoded and decoded. The hierarchical structure consists of, for example, pictures that make up the video data, slices (or tiles) obtained by dividing a picture, coding tree units (CTUs: Coding Tree Unit) obtained by dividing a slice, and coding units (Coding Unit: CU) obtained by dividing a coding tree unit.

[0005] The input image of the CU to be processed is usually prediction-encoded based on a prediction image generated based on a decoded image that was encoded and decoded before the CU to be processed. That is, the prediction error image obtained by subtracting the prediction image from the input image is encoded and decoded. As prediction encoding, there are intra prediction (intra prediction) using a decoded image included in a picture at the same display time as the CU to be processed, and inter prediction (inter prediction) using a decoded image included in a picture at a different display time from the CU to be processed.

[0006] A CU encoded based on intra prediction is called an intra CU. A CU encoded based on inter prediction is called an inter CU.

[0007] A picture that can only contain an intraCU is called an I-picture. A picture that can contain not only an intraCU but also an interCU that uses a decoded image contained in one picture with a different display time than the target CU is called a P-picture. A picture that can contain not only an intraCU and an interCU that uses a decoded image contained in one picture with a different display time than the target CU, but also an interCU that uses decoded images contained in two pictures with different display times than the target CU is called a B-picture.

[0008] The coding of the prediction error image is performed based on frequency conversion, quantization, and entropy coding. The decoding of the prediction error image is performed based on entropy decoding, inverse quantization, and inverse frequency conversion.

[0009] Recommendation ITU-T H.264 "High efficiency video coding", Telecommunication Standardization Sector of ITU, April 2015; Recommendation ITU-T H.266 "Versatile video coding", Telecommunication Standardization Sector of ITU, August 2020; ARIB Standard STD-B32 version 3.11 "Video coding, audio coding and multiplexing methods in digital broadcasting", Association of Radio Industries and Businesses, July 26, 2018; S. Ren et al., "Faster R-CNN: Towards real-time object detection with region proposal networks", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 6, June 2017

[0010] The following are possible ways to utilize the video encoding described above.

[0011] The video encoder applies preprocessing to the input video before encoding it using the video encoding method described above (hereinafter referred to as internal video encoding). The video encoder also encodes control information for the preprocessing. Preprocessing includes Temporal sampling, which reduces the number of pictures to encode by downsampling the pictures in the input video; Targeting, which reduces the region of interest (ROI) of a picture and grays out the rest of the picture; Spatial sampling, which reduces the size of the entire picture; and Bit depth truncation, which discards the lower bit information of the pixel values ​​of the picture. In bit depth truncation, for example, the lower bit information of the luminance component is discarded.

[0012] The video decoder decodes the bitstream using the video encoding method described above (hereinafter referred to as internal video decoding). The video decoder then applies post-processing to the decoded picture to obtain the final decoded image. The video decoder also decodes control information for post-processing. Post-processing includes temporal resampling, which interpolates the picture that was downsampled by time sampling in the video encoder; retargeting, which enlarges the gaze area that was reduced by targeting back to its original size; spatial resampling, which enlarges the picture that was reduced by time sampling back to its original size; and bit depth shift, which restores the brightness of the lower bits that were discarded by bit depth truncation.

[0013] Time resampling is the process corresponding to time sampling and is equivalent to the inverse process of time sampling. Retargeting is the process corresponding to targeting and is equivalent to the inverse process of targeting. Spatial sampling is the process corresponding to spatial sampling and is equivalent to the inverse process of spatial sampling. Bit depth shift is the process corresponding to bit depth truncation and is equivalent to the inverse process of bit depth truncation.

[0014] The present invention aims to achieve a preferred post-processing step in a video encoding scheme that uses the pre-processing and post-processing steps described above.

[0015] The video encoding method based on this disclosure applies preprocessing consisting of multiple processes to an input video signal, encodes the preprocessed video signal, and signals information that can identify the processing order of the processes executed by the video decoding device corresponding to each of the multiple processes.

[0016] The video decoding method based on this disclosure decodes a video bitstream to obtain a video signal and controls the processing order of multiple post-processing steps applied to the video signal based on the signaled information.

[0017] The video encoding device according to this disclosure includes a preprocessing means for applying preprocessing consisting of a plurality of processes to an input video signal, an encoding means for encoding the preprocessed video signal, and a signaling means for signaling information that can identify the processing order of the processes performed by a video decoding device corresponding to each of the plurality of processes.

[0018] The video decoding device according to this disclosure includes decoding means for decoding a video bitstream to obtain a video signal, and control means for controlling the processing order of a plurality of post-processing operations performed on the video signal based on signaled information.

[0019] The video encoding program based on this disclosure causes a computer to perform the following actions on an input video signal: a preprocessing step consisting of multiple processes, a process for encoding the preprocessed video signal, and a process for signaling information that can identify the processing order of the processes executed by the video decoding device corresponding to each of the multiple processes.

[0020] The video decoding program based on this disclosure causes a computer to perform the following processes: decoding a video bitstream to obtain a video signal, and controlling the processing order of multiple post-processing steps applied to the video signal based on the signaled information.

[0021] According to the present invention, in a video encoding scheme that uses pre-processing and post-processing, desirable post-processing is achieved while ensuring interoperability.

[0022] This is a block diagram of a video encoder and video decoder that encode and decode each picture constituting video data. This is a block diagram of an example configuration of the pre-processing section in a video encoder. This is a block diagram of an example configuration of the post-processing section in a video decoder. This is an explanatory diagram showing an example of time sampling processing. This is an explanatory diagram showing an example of time resampling processing. This is an explanatory diagram showing an example of an extended sequence parameter set. This is an explanatory diagram showing targeting processing. This is an explanatory diagram showing targeting processing. This is an explanatory diagram showing retargeting processing. This is an explanatory diagram showing an example of an extended sequence parameter set related to targeting and retargeting. This is an explanatory diagram showing an example of an extended picture parameter set related to targeting and retargeting. This is an explanatory diagram showing spatial sampling and spatial resampling. This is an explanatory diagram showing an example of an extended sequence parameter set related to spatial sampling and spatial resampling. This is an explanatory diagram showing an example of an extended picture parameter set related to bit depth truncation and bit depth shifting. This is a flowchart showing an example of the operation of a video encoder. This is a flowchart showing an example of the operation of a video decoder. This is an explanatory diagram showing a first comparative example of the enlargement of the entire picture and the gaze area. This is an explanatory diagram showing a second comparative example of the entire picture and the magnification of the gaze area. This is an explanatory diagram showing an example of processing in the time resampling unit. This is a block diagram showing the configuration of the post-processing unit constructed according to the processing order of the post-processing. This is a block diagram showing an example of the configuration of an information processing system. This is a block diagram showing the main part of the video encoding device. This is a block diagram showing the main part of the video decoding device.

[0023] The embodiments will be described below with reference to the drawings.

[0024] Figure 1 is a block diagram showing a video encoder 100 and a video decoder 200 that encode and decode each picture that makes up the video data.

[0025] The video encoder 100 includes a pre-processing unit 101, an internal video encoder (internal encoder) 102, a multiplexer 103, and a control unit 104.

[0026] The preprocessing unit 101 performs preprocessing. The internal video encoder 102 encodes the video signal supplied from the preprocessing unit 101 and outputs a video bitstream. The control unit 104 generates an auxiliary information bitstream containing control information for post-processing in the video decoder 200. The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream and outputs it. Therefore, the control information for post-processing is signaled to the video decoder 200.

[0027] The video decoder 200 includes a post-processing unit 201, an internal video decoder (internal decoder) 202, a demultiplexer 203, and a control unit 204.

[0028] The demultiplexer 203 demultiplexes the bitstream to obtain a video bitstream and an auxiliary information bitstream. The internal video decoder 202 decodes the video bitstream supplied from the demultiplexer 203 to obtain a decoded video using internal video encoding. The control unit 204 decodes the auxiliary information bitstream supplied from the demultiplexer 203 to obtain control information for post-processing. The post-processing unit 201 performs post-processing.

[0029] Note that the arrows in Figure 1 simply indicate the direction of signal (data) flow, but do not exclude bidirectional flow. This is also true for other block diagrams.

[0030] [Explanation of Pre-processing and Post-processing] Figure 2 is a block diagram showing an example configuration of the pre-processing unit 101 in the video encoder 100. Figure 3 is a block diagram showing an example configuration of the post-processing unit 201 in the video decoder 200.

[0031] The preprocessor 101 includes a time sampling unit 1011, a targeting unit 1012, a spatial sampling unit 1013, and a bit depth truncation unit 1014. The output of the bit depth truncation unit 1014 is supplied to the internal video encoder 102. Auxiliary information is supplied to the multiplexer 103.

[0032] The post-processing unit 201 includes a bit depth shift unit 2011, a spatial resampling unit 2012, a retargeting unit 2013, and a temporal resampling unit 2014.

[0033] The time sampling unit 1011 performs the time sampling described above. The targeting unit 1012 performs the targeting process described above. The spatial sampling unit 1013 performs the spatial sampling described above. The bit depth truncation unit 1014 performs the bit depth truncation described above.

[0034] In the video decoder 200, the bit depth shift unit 2011 performs the bit depth shift described above. The spatial resampling unit 2012 performs the spatial resampling described above. The retargeting unit 2013 performs the retargeting process described above. The time resampling unit 2014 performs the time resampling described above. The execution order of the bit depth shift, spatial resampling, retargeting process, and time resampling is controlled as described later.

[0035] Next, we will explain time sampling and time resampling in more detail.

[0036] Time sampling reduces the number of images encoded by internal video encoding by downsampling the images in the input video. This reduces the amount of code in the bitstream. Time resampling is a process that interpolates the images that were downsampled by time sampling in multiple stages. This process restores the time resolution that was lost due to the downsampled images.

[0037] The maximum number of decimated pictures, MaxTPics, based on time sampling is defined, for example, by the following equation T1.

[0038] MaxTPics = (2^(max_temporal_resampling_ratio_idx_minus1+1)) - 1 (Formula T1)

[0039] max_temporal_resampling_ratio_idx_minus1 is a parameter (syntax element) multiplexed into the bitstream as described later.

[0040] As can be seen from Equation T1, max_temporal_resampling_ratio_idx_minus1 can be expressed by the following Equation T2.

[0041] max_temporal_resampling_ratio_idx_minus1 = log2(MaxTPics + 1) (Equation T2)

[0042] FIG. 4 is an explanatory diagram showing an example of the time sampling process executed by the time sampling unit 1011. FIG. 5 is an explanatory diagram showing an example of the time resampling process executed by the time resampling unit 2014. Note that FIGS. 4 and 5 illustrate the case where max_temporal_resampling_ratio_idx_minus1 = 1, that is, the case where MaxTPics = 3.

[0043] As illustrated in FIG. 4, after pictures are thinned out by time sampling and then intra-coding is executed, the amount of encoded data is reduced. Also, as illustrated in FIG. 5, the pictures thinned out in the video encoder 100 are restored by time resampling, for example, multi-stage interpolation processing.

[0044] Parameters related to time sampling and time resampling can be multiplexed into the bitstream in the extended sequence parameter set illustrated in FIG. 6.

[0045] In the example shown in FIG. 6, temporal_resampling_flag = 1 indicates that time sampling / time resampling is effective. temporal_resampling_flag = 0 indicates that time sampling / time resampling is ineffective.

[0046] max_temporal_resampling_ratio_idx_minus1 is used to derive the variable TemporalResamplingRatio as follows. In Equation T3, the value of max_temporal_resampling_ratio_idx_minus1 ranges from 0 to 3.

[0047] TemporalResamplingRatio = 2(max_temporal_resampling_ratio_idx_minus1 + 1) (Equation T3)

[0048] The preprocessing unit 101 supplies the above-mentioned temporal_resampling_flag and max_temporal_resampling_ratio_idx_minus1 to the control unit 104 as auxiliary information.

[0049] Next, the targeting process and the retargeting process will be described in more detail.

[0050] The targeting unit 1012 reduces the fixation area at each magnification in the targeting process and changes the color of other areas (non-fixation areas) to, for example, gray. Note that gray is an example. By targeting, the effect of reducing the code amount of the bitstream can be obtained.

[0051] The retargeting unit 2013 restores the fixation area to its original size in the retargeting process. By the retargeting process, the spatial resolution of the reduced fixation area is restored.

[0052] Referring to FIGS. 7A to 7C, the targeting process by the targeting unit 1012 in the preprocessing unit 101 will be described. The case where the input picture illustrated in FIG. 7A is input to the preprocessing unit 101 will be taken as an example.

[0053] The targeting unit 1012 obtains the gaze area for the machine processing task shown in Figure 7B. As an example of the machine processing task, the targeting unit 1012 may use the object detection method described in Non-Patent Literature 4. Figures 7B to 7C illustrate four rectangular gaze areas. Information indicating the position of each gaze area is included in the auxiliary information.

[0054] In the examples shown in Figures 7B and 7C, a magnification of 0.5 indicates that the area is reduced by a factor of 0.5 in both the horizontal and vertical directions. The same applies to magnifications of 0.75 and 1.00. Note that these magnification values ​​are illustrative. Furthermore, a smaller reduction ratio means a lower degree of reduction. For example, a magnification of 0.75 has a smaller reduction ratio than a magnification of 0.5.

[0055] The higher the magnification value, the more important the region. In other words, the preprocessing unit 101 increases the magnification of the areas of focus that are assumed to be important, and decreases the magnification of areas of focus that are not, thereby reducing the area of ​​focus. This reduction is achieved, for example, by downsampling the samples. Information indicating the magnification is included in the auxiliary information.

[0056] The targeting unit 1012 changes the color of areas other than the gaze area (non-gaze area) to gray, for example, in order to reduce the amount of information in those areas. In Figure 7C, the gray areas are indicated by diagonal lines.

[0057] Referring to Figure 8, the retargeting process performed by the retargeting unit 2013 in the post-processing unit 201 will be explained. Figure 8 shows four gaze regions as examples.

[0058] The retargeting unit 2013 enlarges the gaze area in the video exemplified in Figure 7C based on information indicating the magnification at the position indicated by information indicating the position of the gaze area. Enlargement is achieved, for example, by sample interpolation (upsampling). The information indicating the position of the gaze area and the information indicating the magnification are obtained from an auxiliary information bitstream obtained by demultiplexing the bitstream.

[0059] As illustrated in Figure 8, the gaze region in the original image is restored through the processing of the retargeting unit 2013 in the post-processing unit 201.

[0060] The parameters for targeting and retargeting are multiplexed using an extended sequence parameter set and an extended picture parameter set, as shown in Figures 9 and 10. Figure 9 is an explanatory diagram showing an example of an extended sequence parameter set for targeting and retargeting. Figure 10 is an explanatory diagram showing an example of an extended picture parameter set for targeting and retargeting.

[0061] In the extended sequence parameter set, retargeting_flag=1 indicates that targeting is enabled, and retargeting_flag=0 indicates that targeting is disabled.

[0062] In the extended picture parameter set, num_rois indicates the number of gaze regions. roi_idc[i] indicates the magnification. For example, if its value is 0 (000), the magnification is 1.0 (100%). If its value is 1 (001), the magnification is 0.75 (75%). If its value is 2 (010), the magnification is 0.5 (50%).

[0063] roi_pos_x1[i] represents the x-coordinate of the upper left corner of the gaze region. roi_pos_y1[i] represents the y-coordinate of the upper left corner of the gaze region. roi_pos_x2[i] represents the x-coordinate of the lower right corner of the gaze region. roi_pos_y2[i] represents the y-coordinate of the lower right corner of the gaze region.

[0064] Next, spatial sampling and spatial resampling will be explained in more detail with reference to Figure 11.

[0065] The spatial sampling unit 1013 in the preprocessing unit 101 derives a picture magnification preferred for a machine processing task. As a machine processing task, object detection, as described in Non-Patent Literature 4, can be used. As illustrated in Figure 11, the spatial sampling unit 1013 reduces the picture by the derived magnification. As a result, the amount of code in the bitstream is reduced. The information regarding the picture magnification obtained by spatial sampling is multiplexed into the bitstream as auxiliary information.

[0066] The spatial resampling unit 2012 in the post-processing unit 201 enlarges the picture to its original size, as illustrated in Figure 11, based on the magnification information of the picture demultiplexed from the bitstream. As a result, the spatial resolution of the reduced picture is restored.

[0067] The parameters for spatial sampling and spatial resampling are multiplexed using an extended sequence parameter set and an extended picture parameter set, as shown in Figures 12 and 13. Figure 12 is an explanatory diagram showing an example of an extended sequence parameter set for spatial sampling and spatial resampling. Figure 13 is an explanatory diagram showing an example of an extended picture parameter set for spatial sampling and spatial resampling.

[0068] In the extended sequence parameter set, spatial_resampling_flag=1 indicates that targeting is enabled, while spatial_resampling_flag=0 indicates that targeting is disabled.

[0069] In the extended picture parameter set, spatial_upsample_width represents the upsampling width, and spatial_upsample_height represents the upsampling height.

[0070] Next, we will explain bit depth truncation and bit depth shifting in more detail.

[0071] The bit depth truncation unit 1014 in the preprocessing unit 101 discards the lower bit of the luminance component of the video. This process is equivalent to halving the luminance. This process is also equivalent to shifting the luminance component one bit to the right. Bit depth truncation reduces the amount of data in the bitstream.

[0072] The bit depth shift unit 2011 in the post-processing unit 201 shifts the luminance component in the decoded video one bit to the left. This process is equivalent to doubling the luminance. This process restores the brightness of the video.

[0073] Furthermore, information regarding the number of bits of the luminance component to be truncated by the bit depth truncation unit 114 is multiplexed into the bitstream as auxiliary information. The bit depth shift unit 2011 then shifts the luminance component to the left based on the bit number information demultiplexed from the bitstream. This control allows for manipulation of any number of bits.

[0074] The parameters for bit depth truncation and bit depth shifting are multiplexed in an extended picture parameter set, as shown in Figure 14. Figure 14 is an explanatory diagram illustrating an example of an extended picture parameter set for bit depth truncation and bit depth shifting.

[0075] In the extended picture parameter set, bit_depth_shift_flag=1 indicates that bit depth truncation is enabled. bit_depth_shift_flag=0 indicates that bit depth truncation is disabled. bit_depth_shift_luma specifies that the luminance component (luma) is shifted to the left by this value. bit_depth_shift_chroma specifies that the brightness difference component (chroma) is shifted to the left by this value.

[0076] [Explanation of the Encoding Side] In this embodiment, as described above, the preprocessing unit 101 supplies the internal video encoder 102 with a video signal that has undergone time sampling, targeting processing, spatial sampling, and bit depth truncation on the input video. The preprocessing unit 101 also supplies the parameters for the time sampling, targeting processing, spatial sampling, and bit depth truncation described above to the control unit 104 as auxiliary information.

[0077] Furthermore, the preprocessor 101 supplies the control unit 104 with parameters as auxiliary information that can identify the processing order of time resampling, retargeting, spatial resampling, and bit depth shift in the video decoder 200. Note that the parameters that can identify the processing order may be generated by the control unit 104.

[0078] In this embodiment, the processing order of each process constituting the post-processing is bit depth shift, retargeting, spatial resampling, and temporal resampling.

[0079] Figure 15 is a flowchart showing an example of the operation of the video encoder 100. The operation of the video encoder 100 will be explained with reference to Figure 15.

[0080] In the video encoder 100, the preprocessing unit 101 performs the preprocessing described above, namely time sampling, targeting, spatial sampling, and bit depth truncation (step S101). The internal video encoder 102 encodes the bit depth shifted video data supplied from the preprocessing unit 101 to generate a video bitstream (step S102).

[0081] The control unit 104 generates an auxiliary information bitstream (step S103). The auxiliary information bitstream includes, as auxiliary information, the parameters of each process in the preprocessing and parameters that can identify the processing order of each process constituting the postprocessing. The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream (step S104).

[0082] [Decoding Side Explanation] Hereinafter, each process performed by the post-processing unit 201 (time resampling, retargeting, spatial resampling, and bit depth shift) will be referred to as a post-subprocessing step. In this embodiment, the post-processing unit 201 performs four processes, so the processes performed by the post-processing unit 201 will be referred to as post-subprocessing steps 1 to 4. The first process performed by the post-processing unit 201 will be referred to as post-subprocessing step 1. The next process performed will be referred to as post-subprocessing step 2. The process performed after that will be referred to as post-subprocessing step 3. The last process performed will be referred to as post-subprocessing step 4. Furthermore, the blocks that perform post-subprocessing steps 1 to 4 will be referred to as post-subprocessing steps 1 to 4.

[0083] As described above, in this embodiment, the processing order of post-processing is bit depth shift, retargeting, spatial resampling, and temporal resampling. Therefore, the post-processing unit 201 positions the bit depth shift unit 2011 in the post-sub-processing unit 1. The post-processing unit 201 positions the retargeting unit 2013 in the post-sub-processing unit 2. The post-processing unit 201 positions the spatial resampling unit 2012 in the post-sub-processing unit 3. The post-processing unit 201 positions the temporal resampling unit 2014 in the post-sub-processing unit 4.

[0084] Such assignment (positioning) may also be performed by the control unit 204. The assignment of post-subprocessing 1 to 4 is performed according to information that can identify the processing order of post-processing performed by the video decoder 200 (specifically, the post-processing unit 201), which is included in the auxiliary information signaled by the video encoder 100. In other words, the processing order of multiple processes that constitute post-processing applied to the video signal is controlled based on the information signaled by the video encoder 100.

[0085] Then, the post-processing unit 201 sequentially executes post-subprocessing steps 1 to 4 on the decoded video of the internal video encoding supplied from the internal video decoder 202, using parameters based on control information supplied from the control unit 204, to obtain the final decoded video (output video).

[0086] Figure 16 is a flowchart showing an example of the operation of the video decoder 200. The operation of the video decoder 200 will be explained with reference to Figure 16.

[0087] In the video decoder 200, the demultiplexer 203 demultiplexes the bitstream, supplies the video bitstream to the internal video decoder 202, and supplies the auxiliary information bitstream to the control unit 204 (step S201). The internal video decoder 202 decodes the video bitstream to obtain the decoded video (step S202).

[0088] The control unit 204 obtains control information related to post-processing from the auxiliary information bitstream (step S203). The control unit 204 assigns the bit depth shift unit 2011, the retargeting unit 2013, the spatial resampling unit 2012, and the time resampling unit 2014 to the post-sub-processing units 1 to 4 that constitute the post-processing unit 201, according to parameters that can identify the processing order included in the control information. Each block constituting the post-processing unit 201 performs post-processing using the control information related to post-processing (step S204).

[0089] In this embodiment, the control of the control unit 104 and the control unit 204 described above enables the realization of desirable post-processing in a video encoding scheme that uses pre-processing / post-processing. Desirable post-processing is, for example, post-processing that reduces the amount of processing or improves the image quality of the video reproduced by the post-processing.

[0090] The following explains why processing load is reduced and image quality is improved.

[0091] Figure 17 is an explanatory diagram showing a first comparative example of the entire picture and the magnification of the gaze area. Figure 17 shows a post-processing unit 201 configured to perform post-processing in the following order: bit depth shift, spatial resampling, retargeting, and temporal resampling.

[0092] In the example shown in Figure 17, bit depth shifting is performed first, similar to the embodiment, but unlike the embodiment, spatial resampling is performed before the retargeting process. The entire enlarged picture becomes the target of the retargeting process. In this case, the number of pixels processed by retargeting increases.

[0093] Figure 18 is an explanatory diagram showing a second comparative example of the entire picture and the magnification of the gaze area. Figure 18 shows a post-processing unit 201 configured to perform post-processing in the following order: bit depth shift, retargeting, spatial resampling, and temporal resampling, as in this embodiment.

[0094] In the example shown in Figure 18, the picture before enlargement is the target of the retargeting process, so the number of pixels processed by retargeting does not increase.

[0095] Figure 19 is an explanatory diagram showing an example of processing in the time resampling unit.

[0096] When transmitting video signals, video encoders generally do not encode RGB signals directly. To improve compression efficiency, video encoders convert RGB signals to signals in a different color space, perform signal decimation, and then encode them.

[0097] Another color space, for example, is used. YUV 4:2:2 and YUV 4:2:0 formats can be used as signal formats for the YUV color space. Hereafter, the conversion of color space signals will be referred to as color conversion.

[0098] Regarding video encoding for machines, the sensitivity to bit depth for detecting objects is generally low. Therefore, bit depth truncation (bit depth shifting) is used to reduce the number of bits used to represent the image, particularly the Y component, by, for example, one bit. Consequently, the number of bits in the Y component (luminance component) of a video signal in YUV space is reduced by one bit. Bit depth truncation is equivalent to reducing the luminance by half.

[0099] The time resampling unit 2014 performs time resampling on signals in RGB space. Therefore, the time resampling unit 2014 includes a first conversion unit 2014A that converts a YUV signal to an RGB signal and a second conversion unit 2014B that converts an RGB signal to a YUV signal.

[0100] For conversion between the RGB color space and the YUV color space (YCbCr color space), for example, the formula defined in ITU-R Recommendation BT. 709 can be used. Furthermore, the same approach as in BT. 2020 can also be applied. Figure 19 illustrates the conversion formulas from a YUV4:2:0 signal to an RGB signal and from an RGB signal to a YUV4:2:0 signal as defined in BT. 709.

[0101] The time resampling unit 2014 performs time resampling after the color space conversion is performed by the first conversion unit 2014A. Then, the second conversion unit 2014B restores the color space to its original state.

[0102] In the video encoder 100, the internal video encoder 102 encodes the video signal after bit truncation (bit removal) has been performed by the bit depth truncation unit 1014. Therefore, if the post-processing unit in the video decoder 200 is configured such that bit depth shifting is performed after time resampling, the time resampling unit 2014 performs color conversion (conversion from YUV color space to RGB space, and conversion from RGB space to YUV color space) on the video signal containing the bit-truncate Y component. As a result, signal degradation occurs.

[0103] In this embodiment, as shown in Figure 20, bit depth shifting is performed before time resampling. Figure 20 is a block diagram showing the configuration of the post-processing unit constructed according to the processing order of post-processing in this embodiment. In this embodiment, the number of bits in the video signal is increased before color conversion (conversion from YUV color space to RGB space, and conversion from RGB space to YUV color space) is performed in the time resampling unit 2014. That is, the number of bits truncated by the video encoder 100 is restored to the original number of bits.

[0104] Since time resampling is performed with the brightness of the Y component restored, there is no signal degradation due to the color conversion included in time resampling. In other words, image quality is improved.

[0105] In this embodiment, the post-processing result of the internally decoded video in the internal video encoder 102 and the final output video are not the same, but rather the internally decoded video in the internal video encoder 102 and the output video of the internal video decoder 202 are the same. In other words, this embodiment can be interpreted as an embodiment that makes them the same.

[0106] While the above embodiment can be implemented using hardware, it can also be realized using a computer program.

[0107] The information processing system shown in Figure 21 comprises a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Multiple processors 701 may be provided. The storage mediums 703 and 704 may be separate storage mediums or may be storage areas consisting of the same storage medium. Magnetic storage media such as hard disks can be used as storage media. Furthermore, the information processing system may also comprise multiple processors 701.

[0108] In the information processing system, the program memory 702 stores programs (video encoding programs or video decoding programs) for realizing the functions of each block shown in the above embodiment.

[0109] The processor 701 then performs processing according to the program stored in the program memory 702, thereby realizing the functions of the video encoder 100 or video decoder 200 shown in the above embodiment.

[0110] For example, the processor 701 executes processing according to a video encoding program that realizes the functions of each block in the video encoder 100 shown in Figure 1, thereby realizing the functions of the video encoder 100.

[0111] Furthermore, for example, the processor 701 executes processing according to a video decoding program that realizes the functions of each block in the video decoder 200 shown in Figure 1, thereby realizing the functions of the video decoder.

[0112] Furthermore, at least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transient computer-readable medium. The program is supplied to the transient computer-readable medium, for example, via a wired communication channel or a wireless communication channel, i.e., via electrical signals, optical signals or electromagnetic waves.

[0113] Figure 22 is a block diagram showing the main parts of a video encoding device. The video encoding device 10 shown in Figure 22 (implemented by a video encoder 100 in this embodiment) comprises a preprocessing means 11 (implemented by a preprocessing unit 101 in this embodiment) that applies preprocessing consisting of multiple processes to an input video signal, an encoding means 12 (implemented by an internal video encoder 102 in this embodiment) that encodes the preprocessed video signal, and a signaling means 13 (implemented by a multiplexer 103 and a control unit 104 in this embodiment) that signals information that can identify the processing order of the processes executed by the video decoding device corresponding to each of the multiple processes.

[0114] Figure 23 is a block diagram showing the main parts of the video decoding device. The video decoding device 20 shown in Figure 23 (implemented by a video decoder 200 in this embodiment) comprises a decoding means 21 (implemented by an internal video decoder 202 in this embodiment) that decodes a video bitstream to acquire a video signal, and a control means 22 (implemented by a control unit 204 in this embodiment) that controls the processing order of a plurality of post-processing operations that are applied to the video signal based on the signaled information.

[0115] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0116] (Note 1) A video encoding method comprising: applying preprocessing consisting of multiple processes to an input video signal; encoding the preprocessed video signal; and signaling information that can identify the processing order of the processes executed by a video decoding device corresponding to each of the multiple processes.

[0117] (Note 2) The video encoding method described in Note 1, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture.

[0118] (Note 3) The video encoding method described in Note 2, which indicates that the bit depth shift is performed before the time resampling, is the information that can identify the processing order.

[0119] (Note 4) The video encoding method according to Note 2 or Note 3, wherein the information that can identify the processing order indicates that the processing is performed in the order of bit depth shift, retargeting processing, spatial resampling, and temporal resampling.

[0120] (Note 5) A video decoding method that decodes a video bitstream to obtain a video signal and controls the processing order of a plurality of post-processing operations applied to the video signal based on the signaled information.

[0121] (Note 6) The video decoding method described in Note 5, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture, and the plurality of processes are executed in a processing order based on information that allows the signaled processing order to be identified.

[0122] (Note 7) The video decoding method according to Note 6, wherein the bit depth shift is performed before the time resampling in a processing order based on information that can identify the processing order.

[0123] (Note 8) The video decoding method according to Note 6 or Note 7, wherein the bit depth shift, the retargeting process, the spatial resampling, and the temporal resampling are performed in a processing order based on information that can identify the processing order.

[0124] (Note 9) A video encoding device comprising: preprocessing means for applying preprocessing consisting of a plurality of processes to an input video signal; encoding means for encoding the preprocessed video signal; and signaling means for signaling information that can identify the processing order of the processes executed by the video decoding device corresponding to each of the plurality of processes.

[0125] (Note 10) The video encoding apparatus described in Note 9, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture.

[0126] (Note 11) A video decoding device comprising decoding means for decoding a video bitstream to obtain a video signal, and control means for controlling the processing order of a plurality of post-processing operations applied to the video signal based on signaled information.

[0127] (Note 12) The video decoding device according to Note 11, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture, and the device is further provided with post-processing means to execute the plurality of processes in a processing order based on information that can identify the signaled processing order.

[0128] (Note 13) A video encoding program for causing a computer to perform the following processes on an input video signal: a preprocessing process consisting of multiple processes; a process for encoding the preprocessed video signal; and a process for signaling information that can identify the processing order of the processes executed by a video decoding device corresponding to each of the multiple processes.

[0129] (Note 14) The video encoding program described in Note 13, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​the picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of the picture.

[0130] (Note 15) A video decoding program that causes a computer to perform the following: the process of decoding a video bitstream to obtain a video signal; and the process of controlling the processing order of a plurality of post-processing steps to be applied to the video signal based on the signaled information.

[0131] (Note 16) The plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture, and the video decoding program described in Note 15 causes the computer to execute the plurality of processes in a processing order based on information that can identify the signaled processing order.

[0132] (Note 17) A recording medium that records a bitstream generated by a video encoding method which applies preprocessing consisting of multiple processes to an input video signal, encodes the preprocessed video signal, and signals information that can identify the processing order of the processes executed by a video decoding device corresponding to each of the multiple processes.

[0133] (Note 18) A recording medium that records the bitstream described in Note 17, wherein the plurality of processes include time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture.

[0134] (Note 19) The information that can identify the processing order is a recording medium that records the bitstream described in Note 18, which indicates that the bit depth shift is performed before the time resampling.

[0135] (Note 20) The information that can identify the processing order is a recording medium that records the bitstream described in Note 18 or Note 19, indicating that the processing is performed in the order of bit depth shift, retargeting processing, spatial resampling, and temporal resampling.

[0136] The configurations described in Appendices 3 and 4, which are dependent on Appendice 2 above, and some or all of the configurations described in Appendice 4, which is dependent on Appendice 3, may be made dependent on Appendices 10 and 14 in the same dependent relationships as those for Appendices 3 and 4. Furthermore, the configurations described in Appendices 7 and 8, which are dependent on Appendice 6, and some or all of the configurations described in Appendice 8, which is dependent on Appendice 7, may be made dependent on Appendices 12 and 16 in the same dependent relationships as those for Appendices 7 and 8. Moreover, not limited to Appendices 3 and 4, and Appendices 7 and 8, some or all of the configurations described as appendices above may be made dependent on various hardware, software, various recording means for recording software, or systems, provided that they do not deviate from the embodiments described above.

[0137] Although the present invention has been described above with reference to embodiments and examples, the present invention is not limited to the above embodiments and examples. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0138] This application claims priority based on Japanese Patent Application No. 2024-174909, filed on 4 October 2024, and incorporates all of its disclosures herein.

[0139] 10 Video encoding device 11 Preprocessing means 12 Encoding means 13 Signaling means 20 Video decoding device 21 Decoding means 22 Control means 100 Video encoder 101 Preprocessing unit 102 Internal video encoder 103 Multiplexer 104 Control unit 200 Video decoder 201 Postprocessing unit 202 Internal video decoder 203 Demultiplexer 204 Control unit 701 Processor 702 Program memory 703, 704 Storage medium 1011 Time sampling unit 1012 Targeting unit 1013 Spatial sampling unit 1014 Bit depth truncation unit 2011 Bit depth shift unit 2012 Spatial resampling unit 2013 Retargeting unit 2014 Time resampling unit

Claims

1. A video encoding method comprising: applying preprocessing consisting of multiple processes to an input video signal; encoding the preprocessed video signal; and signaling information that can identify the processing order of the processes executed by a video decoding device corresponding to each of the multiple processes.

2. The video encoding method according to claim 1, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture.

3. The video encoding method according to claim 2, wherein the information that can identify the processing order indicates that the bit depth shift is performed before the time resampling.

4. The video encoding method according to claim 2 or 3, wherein the information that can identify the processing order indicates that the processing is performed in the order of bit depth shift, retargeting processing, spatial resampling, and temporal resampling.

5. A video decoding method that decodes a video bitstream to obtain a video signal and controls the processing order of multiple post-processing operations applied to the video signal based on the signaled information.

6. The video decoding method according to claim 5, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture, and the plurality of processes are executed in a processing order based on information that allows the signaled processing order to be identified.

7. The video decoding method according to claim 6, wherein the bit depth shift is performed before the time resampling in a processing order based on information that can identify the processing order.

8. The video decoding method according to claim 6 or 7, wherein the bit depth shift, the retargeting process, the spatial resampling, and the temporal resampling are performed in a processing order based on information that can identify the processing order.

9. A video encoding device comprising: a preprocessing means for applying preprocessing consisting of multiple processes to an input video signal; an encoding means for encoding the preprocessed video signal; and a signaling means for signaling information that can identify the processing order of the processes executed by a video decoding device corresponding to each of the multiple processes.

10. The video encoding apparatus according to claim 9, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture.

11. A video decoding device comprising decoding means for decoding a video bitstream to obtain a video signal, and control means for controlling the processing order of a plurality of post-processing operations applied to the video signal based on signaled information.

12. The video decoding apparatus according to claim 11, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture, and the apparatus is further provided with post-processing means to execute the plurality of processes in a processing order based on information that can identify the signaled processing order.

13. A video encoding program that causes a computer to perform the following actions on an input video signal: a preprocessing step consisting of multiple processes; a process for encoding the preprocessed video signal; and a process for signaling information that can identify the processing order of the processes executed by a video decoding device corresponding to each of the multiple processes.

14. The video encoding program according to claim 13, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture.

15. A video decoding program that causes a computer to perform the following: the process of decoding a video bitstream to obtain a video signal; and the process of controlling the processing order of a plurality of post-processing steps to be applied to the video signal based on the signaled information.

16. The video decoding program according to claim 15, wherein the plurality of processes are time resampling corresponding to time sampling to reduce the number of pictures, retargeting processing corresponding to targeting processing to reduce the gaze area of ​​a picture, spatial resampling corresponding to spatial sampling to reduce the entire picture, and bit depth shifting corresponding to bit depth truncation to shorten the pixel values ​​of a picture, and the computer is made to execute the plurality of processes in a processing order based on information that can identify the signaled processing order.

17. A recording medium that records a bitstream generated by a video encoding method which applies preprocessing consisting of multiple processes to an input video signal, encodes the preprocessed video signal, and signals information that can identify the processing order of the processes executed by a video decoding device corresponding to each of the multiple processes.

Images