Video decoding method, video encoding method, video decoding device, and video encoding device

Abstract

This video decoding device decodes a video bitstream to acquire a decoded picture, specifies a deleted partial region in the decoded picture, determines the pixel value of a pixel with which the partial region is to be padded using the pixel values of neighboring pixels neighboring the partial region in a region of interest, and pads the partial region with the pixel having the determined pixel value.

Description

Video decoding method, video encoding method, video decoding apparatus, and video encoding apparatus

[0001] The present disclosure relates to a video decoding method, a video encoding method, a video decoding apparatus, and a video encoding apparatus.

[0002] In order to efficiently transmit or record video, a video encoding apparatus that generates an encoded representation (hereinafter referred to as a bitstream) by encoding an input video, and a video decoding apparatus that decodes the bitstream to generate a decoded video are used.

[0003] As standards for video encoding methods, there are H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), etc., which are standardized by ITU-T SG16 and ISO / IEC / SC29. In addition, as recent video encoding technologies, the technology described in Non-Patent Document 1 can be cited.

[0004] In these video encoding methods, generally, irreversible processing is executed in a video encoding apparatus. By using irreversible processing, the data amount of the generated bitstream is suppressed. However, the quality of the video (decoded video) decoded by the video decoding apparatus deteriorates compared to the quality of the video (input video) input to the video encoding apparatus.

[0005] Various methods have been developed for applications for human viewing, including the above video encoding methods. Generally, the performance of these methods is evaluated from the viewpoint of how much the decoded video deteriorates with respect to the quality of the input video in terms of the quality of the video decoded from bitstreams of the same data amount generated according to the processing of each method.

[0006] In recent years, not only for human viewing, but also the scenes where video is used for machine processing are increasing. For example, as a processing executed by a machine, object detection processing for detecting a person or the like included in a video can be cited.

[0007] Development of video encoding methods intended for machine processing is underway. For example, ISO / IEC / SC29 is working on standardizing a video encoding method for machines called Video for Coding Machines (VCM).

[0008] The performance of a video encoding scheme for machines is generally evaluated by the processing performance performed by the machine when using video decoded from a bitstream of a similar amount of data. Therefore, even if the quality of the decoded video is significantly degraded compared to the quality of the input video, the encoding scheme is considered to have good performance if the degree of degradation when using the decoded video is small compared to when using the input video. In other words, the difference in quality between the input video and the decoded video is not considered a problem. For this reason, video encoding devices that perform video encoding for machines may apply pre-processing that has not been used in schemes intended for human viewing, in order to improve the performance of the encoding scheme.

[0009] One preprocessing step to improve the performance of an encoding scheme is to replace the pixel values ​​of areas outside the region of interest in the input video, i.e., non-interest regions, with arbitrary pixel values. The region of interest is the area in the video that should be emphasized (important area). Hereafter, the region of interest may be referred to as ROI (Region of Interest). Also, objects such as people that should be emphasized in the region of interest will simply be referred to as objects. Non-Patent Document 2 discloses an example of preprocessing for machine-oriented video encoding, which includes the process of replacing the pixel values ​​of non-interest regions with arbitrary pixel values. Non-Patent Document 2 also shows an example of postprocessing when encoding and decoding video for machines.

[0010] Hereinafter, the process of reducing the amount of encoded data based on the location and size of the region of interest is referred to as ROI processing. As part of ROI processing, for example, pixel values ​​in the region of non-interest are replaced with arbitrary pixel values ​​(for example, pixel values ​​corresponding to gray). Furthermore, the method disclosed in Non-Patent Document 3 utilizes a process that divides the picture to be processed into one or more rectangular regions based on the location of the region of interest and deletes the rectangular region corresponding to the region of non-interest. When this process is applied, the region of interest in the picture moves to the upper left.

[0011] Furthermore, in a method that allows referencing the H. 274 / VSEI standard, the coordinates of the region of interest detected within the picture can be transmitted as auxiliary information called annotated_regions, as defined in the H. 274 / VSEI standard.

[0012] "Algorithm description of Enhanced Compression Model 9 (ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023"Optimization of encoders and receiving systems for machine analysis of coded video content (draft 2)", JVET-AD2030, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023"Region-of-Interest-Based Video Coding for Machines", IEEE International Conference on Multimedia and Expo Workshops, 2024

[0013] As mentioned above, ROI processing is a process that reduces the amount of encoded data and thus the amount of data transmitted from the video encoder to the video decoder. One such process is the deletion of regions of non-interest in the video encoder. When this process is applied, in order to use the decoded picture in the machine processing task, the corresponding video decoder applies a process to return the regions of interest to their original positions and pads the areas deleted by the video encoder with arbitrary pixel values. The pixel values ​​used for padding are, for example, fixed values ​​that represent a specific color such as gray, or values ​​that depend on the content of the picture being processed. A single pixel value is derived by analyzing the picture being processed as a value that depends on the content of the picture. Using this pixel value is expected to improve the accuracy of the machine processing task compared to using a fixed value. However, since it is necessary to scan all the pixels to be analyzed in the picture being processed, there is a problem in that there is a waiting time for the completion of this process.

[0014] The present invention aims to solve the above problems by providing a video decoding method, a video encoding method, a video decoding device, a video encoding device, a video decoding program, and a video encoding program that can shorten processing time while further improving the accuracy of machine processing tasks.

[0015] The video decoding method based on this disclosure decodes a video bitstream to obtain a decoded picture, identifies a deleted partial region in the decoded picture, determines the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest, and pads the partial region with pixels having the determined pixel values.

[0016] The video encoding method based on this disclosure includes a function to decode a video bitstream to obtain a decoded picture, identify a deleted partial region in the decoded picture, determine the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest, and pad the partial region with pixels having the determined pixel values.

[0017] The video decoding device according to this disclosure comprises: video decoding means for decoding a video bitstream and obtaining a decoded picture; identification means for identifying a deleted partial region in the decoded picture; pixel value determination means for determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding means for padding the partial region with pixels having the determined pixel values.

[0018] The video encoding apparatus according to this disclosure is a video encoding apparatus that generates a video bitstream and comprises: video decoding means for decoding the video bitstream and obtaining a decoded picture; identification means for identifying a deleted partial region in the decoded picture; pixel value determination means for determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding means for padding the partial region with pixels having the determined pixel values.

[0019] The video decoding program based on this disclosure causes a computer to perform the following processes: decoding a video bitstream to obtain a decoded picture; identifying a deleted partial region in the decoded picture; determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding the partial region with pixels having the determined pixel values.

[0020] The video encoding program based on this disclosure causes a computer to perform the following processes: decode a video bitstream to obtain a decoded picture; identify a deleted partial region in the decoded picture; determine the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and pad the partial region with pixels having the determined pixel values.

[0021] According to the present invention, it is possible to shorten processing time while further improving the accuracy of machine processing tasks.

[0022] This is a block diagram showing embodiments of a video encoder and a video decoder. This is a block diagram showing an example configuration of the RTG processing unit in a video encoder. This is an explanatory diagram showing an example of syntax. This is an explanatory diagram showing the correspondence between index and magnification. This is a block diagram showing an example configuration of the RTG processing unit in a video decoder. This is a flowchart showing the operation of the RTG processing unit in a video decoder. This is a flowchart showing the operation of the conversion unit of the RTG processing unit in a video decoder. This is an explanatory diagram showing a reference example. This is an explanatory diagram for explaining the padding process in an embodiment. This is an explanatory diagram for explaining the padding process in a modified example. This is a block diagram showing another embodiment of a video encoder. This is a block diagram showing another embodiment of a video decoder. This is a block diagram showing yet another embodiment of a video encoder. This is a block diagram showing another example configuration of the RTG processing unit in a video decoder. This is a block diagram showing another embodiment of a video encoder. This is a block diagram showing an example configuration of an information processing system. This is a block diagram showing the main parts of a video decoding device. This is a block diagram showing the main parts of a video encoding device.

[0023] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0024] In addition to ROI processing, the following technologies can be cited as elemental technologies for pre-processing and post-processing: • Time scalability • Spatial scalability • Bit depth shift (bit depth truncation)

[0025] Temporal scalability is a technique in which the video encoder reduces the frame rate in the temporal direction, and the video decoder upsamples the frame rate. In other words, it is a technique in which the video encoder performs frame decimation, and the video decoder performs frame interpolation. Temporal frame decimation and interpolation are sometimes referred to as TR (Temporal Re-sampling).

[0026] Spatial scalability is a technique in which the video encoder reduces the spatial resolution of the video, and the video decoder increases the resolution (returning it to the original resolution). In other words, it is a technique in which the video encoder shrinks the picture, and the video decoder enlarges the picture. Hereafter, the reduction and enlargement of the picture may be expressed as SR (Spatial Re-sampling) or spatial resampling.

[0027] Specifically, in SR, a reduction process is applied to the input video on a picture-by-picture basis so that the size (width and height) of the picture to be processed is reduced. In other words, in SR, the video encoder performs spatial resampling on the picture to be processed. By reducing the number of pixels to be encoded, it becomes easier to reduce the amount of data in the bitstream. During decoding, as a corresponding post-processing step, an enlargement process to the original size is applied to the decoded video. In other words, in SR, the video decoder performs spatial reconstruction on the picture to be processed.

[0028] In general, the sensitivity of object detection to bit depth is low when it comes to video encoding for machines. Therefore, bit depth shifting is used to reduce the number of bits used to represent the image.

[0029] Figure 1 is a block diagram showing an example of a video encoder 100 and a video decoder 200 suitable for a video encoding scheme for machines. Specifically, it is a block diagram showing an example of the configuration of a video encoder 100 that performs pre-processing and a video decoder 200 that performs post-processing.

[0030] The video encoder 100 includes a time resampling unit 101, a retargeting unit (RTG processing unit) 102, a spatial resampling unit 103, a bit depth shift unit 104, an internal encoder 105, and a multiplexer 106.

[0031] Note that the arrows in Figure 1 simply indicate the direction of signal (data) flow, but do not exclude bidirectional flow. This is also true for other block diagrams.

[0032] The time resampling unit 101 performs time resampling on the input video (input picture). The time resampling performed by the time resampling unit 101 is, for example, frame decimation. The RTG processing unit 102 performs ROI processing on the frame decimated input video. The spatial resampling unit 103 applies spatial scalability to the output of the RTG processing unit 102. The spatial scalability performed by the spatial resampling unit 103 is, for example, picture reduction. The bit depth shift unit 104 applies bit depth truncation to the output of the spatial resampling unit 103. The bit depth truncation performed by the bit depth shift unit 104 is, for example, a 1-bit right shift of the pixel value.

[0033] The internal encoder 105 encodes the picture supplied from the bit depth shift unit 104 using a predetermined video encoding scheme. As the predetermined video encoding scheme, for example, an encoding scheme based on H.266 / VVC can be used. As an example, the predetermined video encoding scheme can be VVC with End2End added.

[0034] The multiplexer 106 outputs a bitstream that is a multiplexed bitstream of the video bitstream supplied from the internal encoder 105 and the bitstream of side information as auxiliary information.

[0035] The video decoder 200 includes a demultiplexer 201, an internal decoder 202, a spatial resampling unit 203, an RTG processing unit 204, a temporal resampling unit 205, and a bit depth shifting unit 206.

[0036] The demultiplexer 201 demultiplexes the bitstream input to the video decoder 200 to obtain the video bitstream and the side information bitstream. The demultiplexer 201 supplies the video bitstream to the internal decoder 202. The demultiplexer 201 supplies the side information bitstream to the RTG processing unit 204, etc.

[0037] The internal decoder 202 decodes the video bitstream and supplies the decoded picture (decoded video frame) to the spatial resampling unit 203.

[0038] The spatial resampling unit 203 applies spatial scalability to the decoded picture. The spatial scalability performed by the spatial resampling unit 203 is, for example, picture scaling. The RTG processing unit 204 performs ROI processing on the output of the spatial resampling unit 203. The temporal resampling unit 205 performs temporal resampling on the output of the RTG processing unit 204. The temporal resampling performed by the temporal resampling unit 205 is, for example, frame interpolation. The bit depth shift unit 206 reconstructs the image from the output of the temporal resampling unit 205. The bit depth shift unit 206 performs, for example, a 1-bit left shift of the pixel value.

[0039] Embodiment 1. Figure 2 is a block diagram showing an example configuration of the RTG processing unit 102 in the video encoder 100. In the example configuration shown in Figure 2, the RTG processing unit 102 includes an ROI information acquisition unit 1021, a partial region definition unit 1022, a parameter determination unit 1023, and a conversion unit 1024.

[0040] The ROI information acquisition unit 1021 acquires ROI position information based on information that allows for the identification of the ROI, which is included in the control information input to the video encoder 100 from an external source. For example, the coordinate information includes the x (horizontal direction) and y (vertical direction) coordinates of the upper left corner of the region, and the size of the region (size in the x direction and size in the y direction). The video encoder 100 may also include an object detection unit that detects objects from a picture, and the object detection unit may define the region containing an object as the ROI. As a method for detecting an object, for example, a neural network trained to detect a specific object may be used. In this case, the neural network takes, for example, the pixel values ​​of the input video picture as input and outputs the coordinates of the region determined to contain an object. The output coordinates are defined as the coordinates of the region of interest. The region of interest for each picture may be calculated only from the processing results of that picture, or it may be calculated based on the processing results of multiple pictures. If the ROI is calculated based on the processing results of multiple pictures, the ROI information acquisition unit 1021 buffers the pictures until a predetermined number of pictures are input to the RTG processing unit 204.

[0041] The sub-region definition unit 1022 sets the sub-regions as follows, for example.

[0042] The sub-region definition unit 1022 determines the division points in the picture based on the position information of the ROIs. Specifically, the sub-region definition unit 1022 defines the division points so that each ROI is divided at the boundary portion of the ROI's region. For example, the sub-region definition unit 1022 sets the coordinate values ​​of the horizontal division points to values ​​selected without overlap from the x-coordinates of the top-left and bottom-right points of each ROI. Similarly, the vertical division points are selected without overlap from the y-coordinates of the top-left and bottom-right points of each ROI. The sub-region definition unit 1022 then sets the arrays of the division point values ​​in each direction in ascending order as one-dimensional arrays Ph and Pv, respectively. Example (A) illustrates that six division points are defined in the horizontal direction and five division points are defined in the vertical direction. Note that the starting point (point with coordinate value 0) and ending point (point with coordinate value 400 or 300) are also included as division points. In this case, Ph = {0, 100, 220, 300, 360, 400} and Pv = {0, 50, 150, 250, 300}. Based on these division points, the regions of the picture to be processed are divided into a grid and defined as subregions before the forward transformation.

[0043] The sub-region definition unit 1022 determines a reduction ratio (magnification) for each ROI and determines the reduction ratio (magnification) between each division point based on the ROI reduction ratio (magnification). One method for determining the ROI magnification is to set the magnification of ROIs greater than or equal to a predetermined value to greater than 0 and less than 1.0 (0 to 1.0), and the magnification of ROIs less than a predetermined value to 1.0. Subsequently, the magnification between division points in the vertical direction (horizontal direction) is set to the maximum magnification among the magnifications of ROIs located between each division point. If no ROI exists, 0 is set. Similarly, the magnification between division points in the vertical direction is determined from the magnifications of ROIs located between each division point. The magnifications between division points in the horizontal and vertical directions are represented by the one-dimensional arrays Sh and Sv, respectively. In example (A), Sh = {0.0, 0.8, 0.0, 1.0, 0.0} and Sv = {0.0, 0.8, 1.0, 0.0}. Example (A) illustrates that the horizontal scaling factor of one ROI (the ROI containing the larger object) is set to 0.8 or 1.0, and the vertical scaling factor is set to 0.8. It also illustrates that the horizontal scaling factor of the other ROI (the ROI containing the smaller object) is set to 1.0, and the vertical scaling factor is set to 1.0.

[0044] The partial area definition unit 1022 calculates the split points after the forward transformation from the split points before the forward transformation and the magnification factors between those split points. Specifically, when expressing the split points in the horizontal direction after the forward transformation as a one-dimensional array Ph’, Ph’[0]=0, and Ph’[m]=Ph’[m-1]+(Ph[m]-Ph[m-1])*Sh[m-1]. Here, assuming the size of Ph is M and m is an integer greater than or equal to 1 and less than M. Similarly, for the split points in the vertical direction after the forward transformation as a one-dimensional array Pv’, assuming the size of Pv is N and n is an integer greater than or equal to 1 and less than N, Pv’[0]=0, and Pv’[n]=Pv’[n-1]+(Pv[n]-Pv[n-1])*Sv[n-1]. The size after the forward transformation is represented by the last elements of Ph’ and Pv’, that is, Ph’[M-1] and Pv’[N-1]. In the case of Example (A), Ph’={0, 0, 96, 96, 156, 156}, Pv’={0, 0, 80, 180, 180}, and the width and height after the forward transformation are 156 and 180 respectively. As described above, since the number of split points before and after the forward transformation is the same, the split points before and after the forward transformation can be associated with each other by elements with the same index. As described above, the size of the picture after the forward transformation is determined by the split points before the forward transformation and the magnification factors between those split points, that is, the position and magnification of the ROI. In other words, if the position or magnification of the ROI is different, the size of the picture after the forward transformation may change. As a method for aligning the sizes of the pictures in the sequence, for example, there is a method of updating the magnification factors between the split points so as to obtain a desired size. Specifically, it is updated by dividing the desired width of the picture by Ph’[M-1] and multiplying the obtained value by all the elements of Ph’. Similarly, it is updated by dividing the desired height of the picture by Pv’[N-1] and multiplying the obtained value by all the elements of Pv’. Note that the updated magnification factor is not limited to 0.0 to 1.0 or less, and may be greater than 1.0.

[0045] The sub-region definition unit 1022 includes the reduction ratio (magnification) of the ROI in the side information, along with the coordinate information of each ROI before affine transformation, i.e., the original coordinate information (corresponding to the division points shown in example (A)). The sub-region definition unit also includes information that identifies the size of the picture before and after transformation in the side information. In the example shown in Figure 2, the magnification (0.8, 0.8) (the former is the horizontal magnification, the latter is the vertical magnification) is included in the side information along with the coordinate values ​​(100, 50, 220, 150) (the first two numbers are the x and y coordinates of the upper left of the region, and the latter two numbers are the x and y coordinates of the lower right of the region). Additionally, the magnification (0.8, 1.0) is included in the side information along with the coordinate values ​​(100, 150, 220, 250). Furthermore, the magnification (1.0, 1.0) is included in the side information along with the coordinate values ​​(300, 50, 360, 250).

[0046] As mentioned above, in this embodiment, in addition to the size of the picture before and after the forward transformation, i.e., before and after the RTG processing, the x and y coordinates of the top left corner and the size of the region are included in the side information. For example, the magnification is included in the side information as syntax shown in Figure 3. As shown in Figure 4, the correspondence between the index and the magnification is defined. The syntax elements shown in Figure 3 are as follows.

[0047] retargeting_flag: A flag indicating whether or not to perform RTG processing. retargeting_image_size_len: The number of bits for retargeting_image_width and retargeting_image_size_height. retargeting_image_size_width: The width of the picture after RTG processing in the video encoder. retargeting_image_size_height: The height of the picture after RTG processing in the video encoder. retargeting_image_size_difference_flag: A flag indicating whether or not the width or height of the picture differs before and after RTG processing. retargeting_to_output_difference_len: The number of bits for retargeting_to_output_difference_width and retargeting_to_output_difference_height. retargeting_to_output_difference_width: The difference in width of the picture before and after RTG processing in the video encoder. retargeting_to_output_difference_height: The difference in height of the picture before and after RTG processing in the video encoder. retargeting_roi_flag: A flag indicating whether ROI information is displayed. If 0, the entire picture is treated as the ROI, and its magnification is set to 1.0.

[0048] roi_size_len: The number of bits of roi_pos_x[i], roi_pos_y[i], roi_size_x[i], and roi_size_y[i]. num_rois_len: The number of bits of num_rois. num_rois: The number of ROIs. roi_scaling_factor_flag: A flag indicating whether scale_factor_idx[i] is specified. If it is 0, then scale_factor_idx[i] = scale_factor_idx[i - 1]. scale_factor_idx[i]: An index indicating the magnification of the i-th ROI. For example, it is specified as shown in the table of scale_factors_idx[i] shown in FIG. 4. roi_pos_x[i]: The x coordinate of the upper left corner of the i-th ROI. roi_pos_y[i]: The y coordinate of the upper left corner of the i-th ROI. roi_size_x[i]: The width of the i-th ROI. roi_size_y[i]: The height of the i-th ROI. [[ID=!]]

[0049] Note that u(n) is an n-bit unsigned integer. When n in the syntax table is "v", the number of bits varies depending on the values of other syntax elements. These syntaxes may be set for each picture and included in the side information, or a common value may be set for multiple pictures and included in the side information. Also, for example, the units for setting the syntax elements may be changed so that the elements of the syntax set before retargeting_roi_flag are included in the side information in sequence units, and those after roi_flag are included in the side information in picture units.

[0050] Furthermore, in the set of subregions through which encoded data will be transmitted, subregions other than subregion C are regions that encompass all or part of one or more ROIs. Hereinafter, the upper right subregion C, which does not contain an object, may be referred to as a specific region. For example, the subregion definition unit 1022 replaces the pixel values ​​of the specific region with fixed values ​​(for example, pixel values ​​corresponding to gray). In other words, with respect to a subregion that does not contain an object (corresponding to a kind of non-ROI), the video encoder 100 does not delete the subregion that does not contain an object if a subregion containing an object exists in the same row or column. The specific region is then treated in the same way as an ROI.

[0051] Furthermore, the partial region definition unit 1022 sets the magnification of partial regions that do not contain objects (non-ROI) to 0.

[0052] Note that a subregion with a reduction ratio (magnification) of 1.0 means that the subregion will not be reduced. A subregion with a magnification of 0 means that the subregion will be deleted.

[0053] The parameter determination unit 1023 determines the parameters for the affine transformation for each subregion. That is, the parameter determination unit 1023 moves the subregion containing the ROI to the upper left of the picture and determines the parameters so that the subregion is reduced by the determined reduction ratio. The reduction ratio (magnification) of the subregion containing the ROI is transmitted to the video decoder 200 as side information about the parameters of the affine transformation, for example.

[0054] Then, the transformation unit 1024 performs the affine transformation operation for each sub-region based on the parameters of the affine transformation, as follows.

[0055] The transformation unit 1024 moves (translates) the subregion with a magnification of 1.0 using an affine transformation. In the example shown in Figure 2, the other ROI (the ROI containing the smaller object) is translated.

[0056] The transformation unit 1024 shrinks and moves (translates) a subregion with a magnification greater than 0 and less than 1.0 (0 to 1.0) by affine transformation. In the example shown in Figure 2, one ROI (the ROI containing the larger object) is shrunk and translated. Similarly, a subregion with a magnification greater than 1.0 is expanded and translated by affine transformation.

[0057] In this way, a picture with ROI processing applied, as shown in example (B) in Figure 2, is created.

[0058] In this embodiment, the video encoder 100 does not transmit encoded data for the picture shown in example (A), but rather transmits encoded data for the set of subregions shown in example (B) (four subregions in example (B)). That is, the video encoder 100 transmits encoded data for subregions that include ROI, but does not transmit encoded data for subregions that do not include ROI.

[0059] Furthermore, the RTG processing unit 102 in the video encoder 100 also includes in the side information the size of the set of subregions after performing ROI processing including affine transformation (see example (B) in Figure 2), that is, the size of the set of subregions to which the encoded data is transmitted (since the coordinate value of the top left of the set is 0, it can be represented by the coordinate value of the bottom right), and information representing the relationship (difference) between the coordinates of the set and the original coordinates. The video encoder 100 may also include the number of ROIs (two in example (B)) in the side information.

[0060] Figure 5 is a block diagram showing an example configuration of the RTG processing unit 204 in the video decoder 200. In the example configuration shown in Figure 5, the RTG processing unit 204 includes an ROI information acquisition unit 2041, a partial region restoration unit 2042, a parameter acquisition unit 2043, and a conversion unit 2044.

[0061] The ROI information acquisition unit 2041 acquires the ROI position information and ROI reduction ratio (magnification) from the side information obtained by demultiplexing, as well as the size of the picture before and after RTG processing in the video decoder 200.

[0062] The partial region reconstruction unit 2042 obtains division points from the size of the picture after RTG processing and the ROI position information. Referring to the example shown in Figure 5, the partial region reconstruction unit 2042 reconstructs the division points shown in example (A).

[0063] The parameter acquisition unit 2043 determines the parameters for the affine transformation from the side information. The parameter acquisition unit 2043 moves the subregion including the ROI to the original coordinate position and determines the parameters so that the subregion is enlarged by the reciprocal of the determined magnification. If the magnification is 0, the parameter acquisition unit 2043 sets the enlargement ratio to 0.

[0064] The conversion unit 2044 includes a pixel copy unit, a padding unit, and a region expansion unit. The pixel copy unit sets the pixel value of the decoded picture (the pixel value of the said region) as the pixel value of the region with a magnification of 1.0. Then, based on the ROI position information, the pixel copy unit uses an affine transformation to return the ROI to its original position (the original coordinate position before the affine transformation was applied in the video encoder 100). The pixel value of the region is the pixel value of each pixel within the region. In other words, setting a pixel value in a region means filling the region with pixels that have that pixel value, or in other words, setting pixels that have that pixel value in the region.

[0065] The padding unit determines the pixel values ​​to be set in a sub-region where the magnification in any direction is 0, and sets the determined pixel values ​​in that sub-region. In other words, the conversion unit 2044 pads the sub-regions where encoded data was not transmitted from the video encoder 100 with the determined pixel values. The pixel values ​​to be set in a sub-region with a magnification of 0 are determined, for example, as follows.

[0066] The padding unit calculates the average value of pixels adjacent to a subregion where encoded data was not transmitted (for example, the values ​​of pixels surrounding a subregion with a width of 1 pixel). When the magnification of the i-th subregion horizontally and the j-th subregion vertically is 0, the padding unit treats pixels in the pre-transformation picture where the x-coordinate is from Ph'[i]-1 to P'h[i] and the y-coordinate is from Pv'[j]-1 to Pv'[j] as adjacent pixels before transformation. The padding unit calculates the average value of adjacent pixels in all subregions where the magnification is 0. Then, the padding unit sets the pixel with the calculated average value as each pixel in the restored subregion (non-ROI). Note that pixels adjacent to a subregion where encoded data was not transmitted are pixels in the ROI. Also, pixels adjacent to a subregion where encoded data was not transmitted are adjacent pixels in the ROI adjacent to that subregion when that subregion is restored.

[0067] The region expansion unit sets the pixel value of the decoded picture (the pixel value of the sub-region) as the pixel value of the sub-region whose magnification is not 0 or 1.0. Furthermore, based on the magnification, it expands the sub-region using an affine transformation and returns the position of the sub-region whose magnification is not 0 or 1.0 to its original position (original coordinate position). If the magnification exceeds 1.0, the region expansion unit shrinks the region and returns it to its original position.

[0068] In this way, the picture is restored, and a picture based on the restored picture is output.

[0069] In this embodiment, the video encoder 100 does not transmit encoded data for sub-regions with a magnification of 0, and the video decoder 200 uses pixel values ​​calculated only from the values ​​of pixels adjacent to the sub-region for sub-regions where encoded data was not transmitted. Compared to calculating a single value for the entire picture, this reduces the waiting time for calculation and thus the processing time. Since padding can be performed independently for each sub-region, if the video decoder 200 is equipped with a processor capable of processing each sub-region in parallel, it becomes possible to reduce the processing time through parallel processing.

[0070] Figure 6 is a flowchart showing the operation of the RTG processing unit 204 in the video decoder 200.

[0071] The ROI information acquisition unit 2041 acquires ROI information from the side information (step S201). The ROI information includes the position information of the partial region and the ROI reduction ratio (magnification).

[0072] The partial region reconstruction unit 2042 reconstructs the position of the partial region based on the ROI position information, the ROI scaling factor, and information representing the relationship (difference) between the coordinates of the partial region after the affine transformation (see Example (B) in Figure 5) and the original coordinates (see Example (A) in Figure 5), which are included in the side information (step S202). Specifically, the partial region reconstruction unit 2042 reconstructs the division points as illustrated in Example (A).

[0073] The parameter acquisition unit 2043 determines the parameters for the affine transformation of each sub-region based on the side information (step S203). The parameters to be determined are the amount of displacement (translation) and the magnification factor. The magnification factor is determined based on the magnification factor included in the side information.

[0074] The padding section undergoes an affine transformation for each sub-region (step S204).

[0075] Figure 7 is a flowchart showing the operation of the conversion unit 2044 in the RTG processing unit 204.

[0076] The conversion unit 2044 moves the ROI to its position in the original picture (the picture input to the video encoder 100) by affine transformation based on the ROI's position information (step S2041).

[0077] The conversion unit 2044 calculates the pixel value to be set for each subregion with respect to the subregion with a magnification of 0, as described above, and sets it for the subregion (step S2042).

[0078] The transformation unit 2044 expands the ROI with a non-zero magnification using an affine transformation and restores the position of the ROI.

[0079] Note that the execution order of steps S2041 to S2042 is arbitrary.

[0080] In this embodiment, the video decoder 200 sets a subregion for each subregion whose magnification is set to 0 by the video encoder 100, by setting a pixel having the average value of the adjacent pixel values. However, it is also conceivable to calculate the pixel values ​​of the pixels set in each subregion all at once.

[0081] Figure 8 is an explanatory diagram showing a reference example. Figure 8 illustrates two non-ROIs (R1, R2) in a picture. The region outside the non-ROIs is an ROI. Also shown are the pixel groups (N1, N2) adjacent to each non-ROI. The two non-ROIs are regions set to a magnification of 0 in the video encoder 100. When calculating the pixel values ​​set for the two non-ROIs (R1, R2) all at once, one method is to use the average value of adjacent pixels (adjacent pixel groups) by sequentially extracting adjacent pixels of the first non-ROI (R1) and adjacent pixels of the second non-ROI (R2), and then calculating the average value of all extracted pixels (average value of the pixel group).

[0082] When using such a method, it is assumed that an extraction unit is provided in place of the padding unit described above to extract adjacent pixels of each non-ROI (pixels adjacent to the non-ROI in the ROI). Then, a padding unit is provided after the conversion unit to calculate the average value of the extracted pixels and set the calculated average value to the non-ROI (R1, R2). The extracted adjacent pixels are pixels in the ROI.

[0083] In such a configuration, padding is performed only after the process of extracting adjacent pixels for all non-ROI areas is completed, resulting in a waiting time until padding is finished. In the embodiment described above, for each sub-region (non-ROI area), the process of extracting adjacent pixels and padding are performed in the padding unit. Therefore, the above waiting time does not occur.

[0084] Figure 8 shows the restored non-ROI subregion, which means that after the non-ROI subregion is restored (however, the pixel values ​​within the subregion have not yet been restored), the pixels within the subregion are restored using the average value of adjacent pixels. However, it is also possible to first calculate the average value of adjacent pixels (pixels in the ROI that are adjacent to the non-ROI when the non-ROI is restored), then restore the non-ROI subregion, and then set the average value of adjacent pixels to the restored non-ROI.

[0085] The above embodiment and the following modifications also allow for the calculation of the average value of adjacent pixels after the non-ROI has been restored, or for the non-ROI to be restored and the average value of adjacent pixels to be set within the non-ROI after the average value of adjacent pixels has been calculated.

[0086] [Modification 1] In the above embodiment, the padding unit in the RTG processing unit 204 calculates the average value of adjacent pixels with a width of 1 pixel that are not ROI. However, the padding unit may also use pixels other than adjacent pixels with a width of 1 pixel to calculate the average value. For example, the padding unit may calculate the average value of adjacent pixels with a width of 1 pixel and pixels (groups of pixels) adjacent to adjacent pixels, and set that average value as the non-ROI. Adjacent pixels with a width of 1 pixel may be expressed as nearby pixels (groups of nearby pixels). Also, adjacent pixels with a width of 1 pixel and pixels adjacent to said adjacent pixels (which may have a width of 2 pixels or more) may be expressed together as nearby pixels. For example, if the width is 2 pixels, the nearby pixels of the i-th subregion in the horizontal direction and the j-th subregion in the vertical direction are the pixels in the picture before conversion whose x coordinates are from Ph'[i]-2 to P'h[i]+1 and whose y coordinates are from Pv'[j]-2 to Pv'[j]+1.

[0087] [Modification 2] In the first embodiment, when the padding unit in the RTG processing unit 204 performs padding on a sub-region located at the i-th position horizontally and the j-th position vertically, pixels at positions where the x-coordinate is Ph'[i]-1 to P'h[i] and the y-coordinate is Pv'[j]-1 to Pv'[j] in the pre-conversion picture are treated as adjacent pixels in the pre-conversion picture. In this case, the padding unit basically refers to pixels of up to four regions. The values ​​used for padding only need to capture the general trend of the pixels in the adjacent regions. The padding unit may refer to pixels of up to six adjacent regions in order to capture the trend more accurately. For example, pixels at positions where the x-coordinate is Ph'[i]-1 to P'h[i+1] and the y-coordinate is Pv'[j]-1 to Pv'[j+1] in the pre-conversion picture are considered adjacent pixels. In this case, although the number of pixels referenced increases, it becomes easier to capture the trend of the pixel values ​​in the adjacent regions.

[0088] Figures 9 and 10 will be used to explain the difference in processing between the first embodiment and this modified example. Figure 9 is an explanatory diagram for illustrating the padding process in the first embodiment. Figure 10 is an explanatory diagram for illustrating the padding process in this modified example (modification 2).

[0089] In Figure 9, (A) illustrates a partial region in the video decoder 200 before RTG processing. In the example shown in (A), the region of the i-th column horizontally and the j-th row vertically is deleted. (B) illustrates a partial region in the video decoder 200 after RTG processing. In the example shown in (B), the region of the i-th column horizontally and the j-th row vertically is deleted. Since the magnification of the i-th horizontal region is 0 and the magnification of the j-th vertical region is 0, Ph'[i]=Ph'[i+1] and Pv'[j]=Pv'[j+1].

[0090] Figure 9(C) illustrates the pixel range (the range of pixels indicated by the smaller rectangle) for which the padding value for the region at position (i,j) is calculated in the first embodiment. Figure 9(D) illustrates the pixel range for which the padding value for the region at position (i-1,j) is calculated in the first embodiment. As shown in Figures 9(C) and (D), in the processing method of the first embodiment, four pixels located at the four corners of adjacent regions are referenced.

[0091] In Figure 10, (A) shows an example of the pixel range for which the padding value for the region at position (i,j) is calculated in this modified example. (B) shows an example of the pixel range for which the padding value for the region at position (i-1,j) is calculated. (C) shows an example of the pixel range for which the padding value for the region at position (i,j-1) is calculated.

[0092] In this modified example, as shown in Figure 10(A), if horizontal and vertical regions are deleted from the converted picture, the same processing as in the first embodiment is performed. However, as shown in Figures 10(B) and (C), if only horizontal or vertical regions are deleted from the converted picture, the padding section refers to, for example, six regions. That is, more than four pixels are treated as adjacent pixels. Therefore, the number of samples used to calculate the average value increases, allowing for a more accurate calculation of the average value of pixels that are adjacent to more adjacent regions.

[0093] [Modification 3] By combining Modification 1 and Modification 2, the padding section may calculate the padding value from neighboring pixels in up to six adjacent regions. For example, the average value may be calculated from the pixel values ​​of pixels in the pre-transformation picture that are at x-coordinates from Ph'[i]-2 to P'h[i+1]+1 and y-coordinates from Pv'[j]-2 to Pv'[j+1]+1.

[0094] [Modification 4] Alternatively, the padding section may not calculate an average value, but instead set the pixel values ​​of adjacent pixels directly as non-ROI. Alternatively, the padding section may not calculate an average value, but instead set adjacent pixels of one pixel width, and pixels adjacent to adjacent pixels, directly as non-ROI.

[0095] [Modification 5] The padding unit may also determine the pixel value to be set in the non-ROI by performing bilinear interpolation using the adjacent pixels in the ROI located to the left of the non-ROI and the adjacent pixels in the ROI located to the right of the non-ROI. Furthermore, if enlargement is required, the padding unit may enlarge adjacent pixels, etc., and then set the pixel value of the enlarged pixels in the non-ROI, or perform bilinear interpolation after enlarging adjacent pixels, etc.

[0096] Embodiment 2. Figure 11 is a block diagram showing another embodiment of the video encoder. However, (A) in Figure 11 shows the video encoder 100 of the first embodiment. In the first embodiment, the video encoder 100 applies pre-processing to the picture to be processed. Then, the internal encoder 105 encodes the pre-processed picture to generate a bitstream.

[0097] The video encoder 110 of the second embodiment is configured as shown in Figure 11(B) as an example. That is, in addition to the part of the video encoder 100 of the first embodiment that realizes the function of generating a video bitstream (time resampling unit 101, RTG processing unit 102, spatial resampling unit 103, bit depth shift unit 104, internal encoder 105A), the video encoder 110 includes a post-processing unit 107 that has the function of executing post-processing performed by the video decoder 200. In this embodiment, the internal encoder 105A has the function of decoding the video bitstream and supplying the decoded picture to the post-processing unit 107. That is, the internal encoder 105A has the function of the internal decoder 202 in the video decoder 200 in addition to the function of the internal encoder 105 in the first embodiment.

[0098] The post-processing unit 107 includes a spatial resampling unit 1071, a retargeting unit (RTG processing unit) 1072, a time resampling unit 1073, and a bit depth shift unit 1074. The spatial resampling unit 1071, RTG processing unit 1072, time resampling unit 1073, and bit depth shift unit 1074 have the same functions as the spatial resampling unit 203, RTG processing unit 204, time resampling unit 205, and bit depth shift unit 206 in the video decoder 200. In the post-processing unit 107, the picture input to the spatial resampling unit 1071 is a reconstructed picture obtained by performing internal decoding processing in the internal encoder 105A. At this time, the reconstructed picture may be a picture obtained by applying processing similar to that of the internal decoder 202 to the bitstream, or it may be a picture obtained by applying a part of the processing of the internal decoder 202 to the picture used to generate the bitstream.

[0099] The video encoder 110 in this embodiment has a function to perform post-processing, similar to the video decoder 200, so it is possible to check the picture that will be output from the video decoder 200.

[0100] [Variation 6] The picture is shown as an example in YUV4:2:0 format, but it may also be shown in YUV4:0:0 format or YUV4:4:4 format. Furthermore, the picture may also be shown in RGB format.

[0101] Embodiment 3. Figure 12 is a block diagram showing another embodiment of the video decoder. The video decoder 210 shown in Figure 12 includes a retargeting unit (RTG processing unit) 207 that receives a bitstream as input and outputs video. The RTG processing unit 207 may receive the bitstream after processing by the internal decoder 202 in the first embodiment or by the spatial resampling unit 203. Alternatively, the RTG processing unit 207 may receive the bitstream after processing by the temporal resampling unit 205.

[0102] The RTG processing unit 207 includes an ROI information acquisition unit 2071, a partial region restoration unit 2072, a parameter acquisition unit 2073, and a conversion unit 2074.

[0103] The ROI information acquisition unit 2071 acquires the ROI position information and ROI magnification, as well as the picture size before and after RTG processing in the video decoder, from among the multiple pieces of information contained in the input bitstream.

[0104] The partial region reconstruction unit 2072 obtains division points from the size of the picture after RTG processing and the ROI position information.

[0105] The parameter acquisition unit 2073 determines the parameters for the affine transformation from the side information. The parameter acquisition unit 2073 moves the subregion including the ROI to the original coordinate position and determines the parameters so that the subregion is expanded by the reciprocal of the determined magnification. If the magnification is 0, the parameter acquisition unit 2073 sets the expansion ratio to 0. In this embodiment, the video decoder 210 accepts the video bitstream and the side information bitstream as inputs, respectively. However, the video decoder 210 may also accept a bitstream as input in which all or part of the information included in the video bitstream and the information included in the side information bitstream are combined.

[0106] The conversion unit 2074 determines the processing for the subregion according to the magnification obtained by the parameter acquisition unit 2073. Multiple processing steps are predetermined for the processing of the subregion to be performed by the conversion unit 2074. The conversion unit 2074 then determines which processing step to perform from among these steps. The conversion unit 2074 may perform only one of the multiple processing steps, or it may perform a combination of multiple processing steps. The processing for the subregion may include setting pixels included in the subregion as part of that subregion, or setting values ​​determined based on the values ​​of pixels adjacent to the subregion. The values ​​of pixels adjacent to the subregion may, for example, be the average value of multiple pixels adjacent to the subregion, but may also be the median, etc. Furthermore, pixels located to the left or above the subregion may be selected.

[0107] In this embodiment, the conversion unit 2074 may be configured to perform any or all of the processes performed by the pixel copy unit, padding unit, and area expansion unit in the first and second embodiments.

[0108] As described above, the video decoder 210 of the third embodiment restores the picture and outputs a picture based on the restored picture.

[0109] [Modification 7] The conversion unit 2074 determined the processing for the subregion based on the magnification obtained by the parameter acquisition unit 2073, but it may also determine what processing to perform on the subregion depending on whether or not it contains an ROI. For example, if information indicating that the region does not contain an ROI is obtained from the bitstream, the conversion unit 2074 performs processing to set a value obtained based on the value of the pixels adjacent to the subregion. Alternatively, if information indicating that the region contains an ROI is obtained from the bitstream, the conversion unit 2074 may perform processing to set the pixels included in the subregion into that subregion.

[0110] Furthermore, a subregion may include a region where the magnification factor used to determine the size of the region when encoding the subregion is set to 0. Also, pixels adjacent to a subregion may include pixels of subregions adjacent to the subregion included in the picture. Obtaining information about pixels adjacent to a subregion may, for example, involve calculating the average value of pixels in subregions adjacent to subregions included in multiple pictures. The padding unit may also be configured to obtain information about pixels adjacent to a subregion for each color component, based on information about the picture to be processed and information indicating the position of the subregion.

[0111] Embodiment 4. Figure 13 is a block diagram showing yet another embodiment of the video encoder. The video encoder 120 shown in Figure 13 includes, in addition to the encoding unit 150A that realizes the function of outputting the video bitstream of the video encoder 100 of the first embodiment, a post-processing unit 108 that has the function of performing post-processing performed by the video decoder 200. In this embodiment, the internal encoder included in the encoding unit 150A has the function of decoding the video bitstream and supplying the decoded picture to the post-processing unit 108. That is, the internal encoder has the function of the internal encoder 105 in the first embodiment as well as the function of the internal decoder 202 in the video decoder 200.

[0112] The post-processing unit 108 includes an ROI information acquisition unit 1081, a partial region restoration unit 1082, a parameter acquisition unit 1083, and a conversion unit 1084. The ROI information acquisition unit 1081, the partial region restoration unit 1082, the parameter acquisition unit 1083, and the conversion unit 1084 have the same functions as the ROI information acquisition unit 2071, the partial region restoration unit 2072, the parameter acquisition unit 2073, and the conversion unit 2074 included in the RTG processing unit 2074 in the video decoder 210 of the third embodiment shown in Figure 12.

[0113] The video encoder 120 of this embodiment has a function to perform post-processing, similar to the video decoder 210 of the third embodiment, so that the picture that will be output from the video decoder 210 can be confirmed.

[0114] Embodiment 5. The retargeting unit (RTG processing unit) in the video decoder may be configured as shown in Figure 14. The information acquisition unit 2091 in the RTG processing unit 209 shown in Figure 14 acquires information of the picture to be processed. The partial region setting unit 2092 acquires information indicating the location of a partial region of the picture to be processed. Based on the information of the picture to be processed and the information indicating the location of the partial region, the partial region setting unit 2092 obtains information of pixels adjacent to the partial region and includes the partial region based on the obtained pixel information in the picture. In one aspect of this embodiment, the information acquisition unit 2091 may have at least a part of the functions of the ROI information acquisition units 2041 and 2071, referring to the first embodiment (see Figure 5) and the third embodiment (see Figure 12). Also, the partial region setting unit 2092 may have at least a part of the functions of the padding unit in the conversion units 2044 and 2074.

[0115] Embodiment 6. Figure 15 is a block diagram showing another embodiment of the video encoder. The video encoder 130 shown in Figure 15 includes an encoding unit 150B (configured similarly to the encoding unit 150A) that realizes the function of outputting the video bitstream of the video encoder 100 of the first embodiment, as well as a post-processing unit 109 that has the function of performing post-processing performed by the fifth video decoder described above. In this embodiment, the internal encoder included in the encoding unit 150B has the function of decoding the video bitstream and supplying the decoded picture to the post-processing unit 109. That is, the internal encoder has the function of the internal encoder 105 in the first embodiment as well as the function of the internal decoder 202 in the video decoder 200.

[0116] The post-processing unit 109 includes an information acquisition unit 1091 and a partial region setting unit 1092. The information acquisition unit 1091 has the same function as the information acquisition unit in the fifth video decoder. The partial region setting unit 1092 has the same function as the partial region setting unit in the fifth video decoder.

[0117] The video encoder 130 of this embodiment has a function to perform post-processing, similar to the video decoder of the fifth embodiment, so that the picture to be output from the video decoder can be confirmed.

[0118] While each of the above embodiments can be implemented using hardware, it can also be realized using computer programs.

[0119] The information processing system shown in Figure 16 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Multiple processors can also be used. The storage mediums 703 and 704 may be separate storage mediums or may be storage areas consisting of the same storage medium. Magnetic storage media such as hard disks can be used as storage media.

[0120] In the information processing system, the program memory 702 stores programs (video encoding programs or video decoding programs) for realizing the functions of each block shown in the above embodiments.

[0121] The processor 701 then performs processing according to the program stored in the program memory 702, thereby realizing the functions of the video encoders 100, 110, 120, 130 and video decoders 200, 210 shown in each embodiment.

[0122] For example, the processor 701 performs processing according to a video encoding program that realizes the functions of each block in the video encoders 100, 110, 120, and 130 shown in Figures 1, 11, 13, and 15, thereby realizing the functions of the video encoders 100, 110, 120, and 130.

[0123] Furthermore, for example, the functions of the video decoders 200 and 210 are realized when the processor 701 executes processing according to a video decoding program that realizes the functions of each block in the video decoders 200 and 210 shown in Figures 1 and 12.

[0124] Furthermore, at least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transient computer-readable medium. The program is supplied to the transient computer-readable medium, for example, via a wired communication channel or a wireless communication channel, i.e., via electrical signals, optical signals or electromagnetic waves.

[0125] Figure 17 is a block diagram showing the main parts of the video decoding device. The video decoding device 20 shown in Figure 17 (in this embodiment, it is implemented as, for example, a video decoder 200) comprises: video decoding means 21 (in this embodiment, implemented as an internal decoder 202) that decodes a video bitstream and obtains a decoded picture; identification means 22 (in this embodiment, implemented as a partial area restoration unit 2042) that identifies a deleted partial area in the decoded picture; pixel value determination means 23 (in this embodiment, implemented as a padding unit) that determines the pixel values ​​of pixels to be padded in the partial area using the pixel values ​​of nearby pixels adjacent to the partial area in the area of ​​interest; and padding means 24 (in this embodiment, implemented as a padding unit) that pads the partial area with pixels having the determined pixel values.

[0126] Figure 18 is a block diagram showing the main parts of a video encoding device. The video encoding device 10 shown in Figure 18 (in this embodiment, it is implemented as, for example, a video encoder 100) comprises: a video decoding means 21 (in this embodiment, implemented as an internal decoder 202) that decodes a video bitstream and obtains a decoded picture; a identification means 22 (in this embodiment, implemented as a partial area restoration unit 2042) that identifies a deleted partial area in the decoded picture; a pixel value determination means 23 (in this embodiment, implemented as a padding unit) that determines the pixel values ​​of pixels to be padded in the partial area using the pixel values ​​of nearby pixels adjacent to the partial area in the area of ​​interest; and a padding means 24 (in this embodiment, implemented as a padding unit) that pads the partial area with pixels having the determined pixel values.

[0127] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0128] (Note 1) A video decoding method comprising: decoding a video bitstream to obtain a decoded picture; identifying a deleted partial region in the decoded picture; determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding the partial region with pixels having the determined pixel values.

[0129] (Note 2) The video decoding method according to Note 1, wherein the pixel values ​​of adjacent pixels adjacent to the subregion in the region of interest are used to calculate the pixel values ​​of pixels to be padded in the subregion.

[0130] (Note 3) The video decoding method according to Note 2, wherein the average value of the pixel values ​​of the adjacent pixel group adjacent to the subregion in the region of interest is used as the pixel value of the pixel to be padded.

[0131] (Note 4) The video decoding method according to Note 1, wherein the pixel values ​​of adjacent pixels adjacent to the subregion in the region of interest are used as the pixel values ​​of the pixels to be padded.

[0132] (Note 5) The video decoding method according to Note 1, wherein the interpolation value using neighboring pixels adjacent to the subregion in the region of interest is the pixel value of the padded pixel.

[0133] (Note 6) A video encoding method comprising the functions of decoding a video bitstream to obtain a decoded picture, identifying a deleted partial region in the decoded picture, determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest, and padding the partial region with pixels having the determined pixel values.

[0134] (Note 7) A video decoding device comprising: video decoding means for decoding a video bitstream and obtaining a decoded picture; identification means for identifying a deleted partial region in the decoded picture; pixel value determination means for determining the pixel value of a pixel to be padded in the partial region using the pixel value of a neighboring pixel adjacent to the partial region in the region of interest; and padding means for padding the partial region with a pixel having the determined pixel value.

[0135] (Note 8) The image decoding apparatus according to Note 7, wherein the pixel value determination means calculates the pixel value of a pixel to be padded in the partial region using the pixel values ​​of adjacent pixels adjacent to the partial region in the region of interest.

[0136] (Note 9) The image decoding apparatus according to Note 8, wherein the pixel value determination means is the average value of the pixel values ​​of the adjacent pixel group adjacent to the sub-region in the region of interest, and the pixel value of the pixel to be padded.

[0137] (Note 10) The image decoding device according to Note 7, wherein the pixel value determination means takes the pixel value of a neighboring pixel adjacent to the sub-region in the region of interest and uses that value as the pixel value of the pixel to be padded.

[0138] (Note 11) The image decoding device according to Note 7, wherein the pixel value determination means uses an interpolated value using a neighboring pixel adjacent to the sub-region in the region of interest as the pixel value of the pixel to be padded.

[0139] (Note 12) A video encoding device for generating a video bitstream, comprising: video decoding means for decoding the video bitstream and obtaining a decoded picture; identification means for identifying a deleted partial region in the decoded picture; pixel value determination means for determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding means for padding the partial region with pixels having the determined pixel values.

[0140] (Note 13) A video decoding program that causes a computer to perform the following processes: decoding a video bitstream to obtain a decoded picture; identifying a deleted partial region in the decoded picture; determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding the partial region with pixels having the determined pixel values.

[0141] (Note 14) The video decoding program described in Note 13, which causes a computer to perform a process in which, in the process of determining the pixel value, the average value of the adjacent pixel group adjacent to the sub-region in the region of interest is set as the pixel value of the pixel to be padded.

[0142] (Note 15) A video encoding program that causes a computer to perform the following processes: decode a video bitstream and obtain a decoded picture; identify a deleted partial region in the decoded picture; determine the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and pad the partial region with pixels having the determined pixel values.

[0143] (Note 16) A video decoding method that obtains information of a picture to be processed, obtains information indicating the location of a sub-region of the picture to be processed, determines information of pixels adjacent to the sub-region based on the information of the picture to be processed and the information indicating the location of the sub-region, and includes the sub-region based on the information of the pixels in the picture.

[0144] (Note 17) The video decoding method according to Note 16, wherein the subregion includes a region in which the magnification for determining the size of the region when encoding the subregion is set to 0.

[0145] (Note 18) The image decoding method according to Note 16, wherein the pixels adjacent to the subregion include pixels of a subregion adjacent to the subregion included in the picture.

[0146] (Note 19) The video decoding method according to any one of Notes 16 to 18, wherein obtaining information on pixels adjacent to the subregion includes obtaining the average value of pixels in subregions adjacent to subregions included in a plurality of pictures.

[0147] (Note 20) A video decoding method according to any one of Notes 16 to 19, which determines information about pixels adjacent to a partial region based on information about the picture to be processed and information indicating the position of the partial region for each color component.

[0148] (Note 21) A video encoding method that obtains information of a picture to be processed, obtains information indicating the location of a sub-region of the picture to be processed, determines information of pixels adjacent to the sub-region based on the information of the picture to be processed and the information indicating the location of the sub-region, and includes the sub-region based on the information of the pixels in the picture.

[0149] (Note 22) The video encoding method according to Note 21, wherein the subregion includes a region in which the magnification for determining the size of the region when encoding the subregion is set to 0.

[0150] (Note 23) The video encoding method according to Note 21, wherein pixels adjacent to the subregion include pixels of a subregion adjacent to the subregion included in the picture.

[0151] (Note 24) The video encoding method according to any one of Notes 21 to 23, wherein obtaining information on pixels adjacent to the subregion includes obtaining the average value of pixels in subregions adjacent to subregions included in a plurality of pictures.

[0152] (Note 25) A video encoding method according to any one of Notes 21 to 24, which determines information about pixels adjacent to a partial region based on information about the picture to be processed and information indicating the position of the partial region for each color component.

[0153] (Note 26) An image decoding device comprising: an information acquisition means for acquiring information of a picture to be processed and information indicating the location of a partial region of the picture to be processed; and a partial region setting means for determining information of pixels adjacent to the partial region based on the information of the picture to be processed and the information indicating the location of the partial region, and including the partial region based on the information of the pixels in the picture.

[0154] (Note 27) The video decoding device according to Note 26, wherein the subregion includes a region in which the magnification for determining the size of the region when encoding the subregion is set to 0.

[0155] (Note 28) The image decoding device according to Note 26, wherein the pixels adjacent to the subregion include pixels of a subregion adjacent to the subregion included in the picture.

[0156] (Note 29) The video decoding device according to any one of Notes 26 to 28, wherein obtaining information on pixels adjacent to the subregion includes obtaining the average value of pixels in subregions adjacent to subregions included in a plurality of pictures.

[0157] (Note 30) The video decoding device according to any one of Notes 26 to 29, wherein the information acquisition means obtains information of pixels adjacent to a partial region based on the information of the picture to be processed and the information indicating the position of the partial region for each color component.

[0158] (Note 31) An image encoding device comprising: an information acquisition means for acquiring information of a picture to be processed and information indicating the location of a partial region of the picture to be processed; and a partial region setting means for determining information of pixels adjacent to the partial region based on the information of the picture to be processed and the information indicating the location of the partial region, and including the partial region based on the information of the pixels in the picture.

[0159] (Note 32) The video encoding apparatus according to Note 31, wherein the sub-region includes a region in which the magnification for determining the size of the region when encoding the sub-region is set to 0.

[0160] (Note 33) The video encoding device according to Note 32, wherein the pixels adjacent to the subregion include pixels of a subregion adjacent to the subregion included in the picture.

[0161] (Note 34) The video encoding device according to any one of Notes 31 to 33, wherein obtaining information on pixels adjacent to the subregion includes obtaining the average value of pixels in subregions adjacent to subregions included in a plurality of pictures.

[0162] (Note 35) The video encoding device according to any one of Notes 31 to 34, wherein the information acquisition means obtains information of pixels adjacent to a partial region for each color component, based on information of the picture to be processed and information indicating the position of the partial region.

[0163] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0164] This application claims priority based on Japanese Patent Application No. 2024-174054, filed on 3 October 2024, and Japanese Patent Application No. 2025-121165, filed on 18 July 2025, and incorporates all of their disclosures herein.

[0165] 10 Video encoding device 11, 21 Video decoding means 12, 22 Identification means 13, 23 Pixel value determination means 14, 24 Padding means 20 Video decoding device 100, 110, 120, 130 Video encoder 101 Time resampling unit 102 RTG processing unit 103 Spatial resampling unit 104 Bit depth shift unit 105, 105A Internal encoder 106 Multiplexer 107, 108, 109 Post-processing unit 151A, 150B Encoding unit 200, 210 Video decoder 201 Demultiplexer 202 Internal decoder 203 Spatial resampling unit 204, 207, 209 RTG processing unit 205 Time resampling unit 206 Bit depth shift unit 701 Processor 702 Program memory 703, 704 Storage medium 1021 ROI information acquisition unit 1022 Partial area definition unit 1023 Parameter determination unit 1024 Conversion unit 1071 Spatial resampling unit 1072 RTG processing unit 1073 Time resampling unit 1074 Bit depth shift unit 1081 ROI information acquisition unit 1082 Partial area restoration unit 1083 Parameter acquisition unit 1084 Conversion unit 1091 Information acquisition unit 1092 Partial area setting unit 2041 ROI information acquisition unit 2042 Partial area restoration unit 2043 Parameter acquisition unit 2044 Conversion unit 2071 ROI information acquisition unit 2072 Partial area restoration unit 2073 Parameter acquisition unit 2074 Conversion unit 2091 Information acquisition unit 2092 Partial area setting unit

Claims

1. A video decoding method comprising: decoding a video bitstream to obtain a decoded picture; identifying a deleted partial region in the decoded picture; determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding the partial region with pixels having the determined pixel values.

2. The video decoding method according to claim 1, wherein the pixel values ​​of pixels to be padded in the subregion are calculated using the pixel values ​​of adjacent pixels adjacent to the subregion in the region of interest.

3. The video decoding method according to claim 2, wherein the average value of the pixel values ​​of the adjacent pixel group adjacent to the subregion in the region of interest is used as the pixel value of the pixel to be padded.

4. The video decoding method according to claim 1, wherein the pixel value of a neighboring pixel adjacent to the subregion in the region of interest is the pixel value of the pixel to be padded.

5. The video decoding method according to claim 1, wherein the interpolation value using neighboring pixels adjacent to the subregion in the region of interest is the pixel value of the padded pixels.

6. A video decoding method that obtains information about a picture to be processed, obtains information indicating the location of a sub-region of the picture to be processed, determines information about pixels adjacent to the sub-region based on the information about the picture to be processed and the information indicating the location of the sub-region, and includes the sub-region based on the information about the pixels in the picture.

7. A video encoding method comprising the functions of decoding a video bitstream to obtain a decoded picture, identifying a deleted partial region in the decoded picture, determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest, and padding the partial region with pixels having the determined pixel values.

8. A video encoding method that obtains information about a picture to be processed, obtains information indicating the location of a sub-region of the picture to be processed, determines information about pixels adjacent to the sub-region based on the information about the picture to be processed and the information indicating the location of the sub-region, and includes the sub-region based on the information about the pixels in the picture.

9. A video decoding device comprising: video decoding means for decoding a video bitstream and obtaining a decoded picture; identification means for identifying a deleted partial region in the decoded picture; pixel value determination means for determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding means for padding the partial region with pixels having the determined pixel values.

10. The image decoding apparatus according to claim 8, wherein the pixel value determination means sets the average value of the adjacent pixel group adjacent to the subregion in the region of interest to be the pixel value of the pixel to be padded.

11. An image decoding device comprising: an information acquisition means for acquiring information about a picture to be processed and information indicating the location of a partial region of the picture to be processed; and a partial region setting means for determining information about pixels adjacent to the partial region based on the information about the picture to be processed and the information indicating the location of the partial region, and including the partial region based on the information about the pixels in the picture.

12. A video encoding device for generating a video bitstream, comprising: video decoding means for decoding the video bitstream and obtaining a decoded picture; identification means for identifying a deleted partial region in the decoded picture; pixel value determination means for determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in a region of interest; and padding means for padding the partial region with pixels having the determined pixel values.

13. A video encoding device comprising: an information acquisition means for acquiring information about a picture to be processed and information indicating the location of a partial region of the picture to be processed; and a partial region setting means for determining information about pixels adjacent to the partial region based on the information about the picture to be processed and the information indicating the location of the partial region, and including the partial region based on the information about the pixels in the picture.

14. A video decoding program that causes a computer to perform the following processes: decoding a video bitstream to obtain a decoded picture; identifying a deleted partial region in the decoded picture; determining the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and padding the partial region with pixels having the determined pixel values.

15. The video decoding program according to claim 14, which causes a computer to perform a process in which it determines the pixel value, wherein the average value of the pixel values ​​of the adjacent pixel group adjacent to the subregion in the region of interest is set as the pixel value of the pixel to be padded.

16. A video encoding program that causes a computer to perform the following processes: decode a video bitstream and obtain a decoded picture; identify a deleted partial region in the decoded picture; determine the pixel values ​​of pixels to be padded in the partial region using the pixel values ​​of neighboring pixels adjacent to the partial region in the region of interest; and pad the partial region with pixels having the determined pixel values.

Images