Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

This video encoding device includes: a region of interest detection means for detecting a region of interest from an input video frame; a video encoding means for generating a video bit stream; and a decoder control means for generating an auxiliary information bit stream including information for controlling setting processing of a pixel value in a region of no interest in the video decoding device.

Description

Video encoding device, video decoding device, video encoding method, and video decoding method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to transmit or record video efficiently, a video encoding device is used to generate a coded representation (hereinafter referred to as a bitstream) of input video, and a video decoding device is used to decode the bitstream to generate decoded video.

[0003] Video coding standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), and H.266 / VVC (Versatile Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. Another recent video coding technology is the technology described in Non-Patent Document 1.

[0004] In these video encoding methods, lossy processing is generally performed in the video encoding device. By using lossy processing, the amount of data in the generated bitstream is reduced. However, the quality of the video decoded by the video decoding device (decoded video) is degraded compared to the quality of the video input to the video encoding device (input video).

[0005] In addition to the video coding methods mentioned above, various methods have been developed for human viewing. Generally, the performance of these methods is evaluated in terms of the degree to which the quality of the decoded video degrades compared to the quality of the input video, when the decoded video is decoded from bitstreams of similar data amounts generated according to the processing of each method.

[0006] In recent years, video is increasingly used not only for human viewing but also for machine processing, such as object detection, which detects people and other objects in a video.

[0007] Development of video coding methods that are designed for machine processing is underway. For example, ISO / IEC / SC29 is working on standardizing a video coding method for machines called Video for Coding Machines (VCM).

[0008] The performance of a video coding method for machines is generally evaluated by the performance of the processing performed by the machine when using video decoded from a bitstream with a similar data volume. Therefore, even if the quality of the decoded video is significantly degraded compared to the quality of the input video, the performance of the coding method is considered to be good if the degree of degradation in the performance of the processing performed by the machine when using the decoded video is small compared to when using the input video. In other words, the difference in quality between the input video and the decoded video is not considered an issue. Therefore, video coding devices that perform video coding for machines may apply pre-processing that has not been used in methods intended for human viewing in order to improve the performance of the coding method.

[0009] Pre-processing for improving the performance of an encoding method includes, for example, replacing pixel values ​​in regions other than the region of interest in an input video, i.e., regions of non-interest, with arbitrary pixel values. A region of interest is a region in a video that should be emphasized (an important region). Hereinafter, a region of interest may be referred to as a ROI (Region of Interest). Furthermore, an object in a region of interest that should be emphasized, such as a person, is simply referred to as an object. Non-Patent Document 2 discloses an example of pre-processing for machine-oriented video encoding, including a process of replacing pixel values ​​in regions of non-interest with arbitrary pixel values. Non-Patent Document 2 also shows an example of post-processing when encoding and decoding video for machines.

[0010] In addition, in a method that can refer to the H.274 / VSEI standard, it is possible to transmit the coordinates of a region of interest detected in a video frame as auxiliary information called annotated_regions, which is defined in the H.274 / VSEI standard.

[0011] "Algorithm description of Enhanced Compression Model 9 (ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023"Optimization of encoders and receiving systems for machine analysis of coded video content (draft 2)", JVET-AD2030, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023

[0012] When a video encoding method for machines is used, there is a possibility that the decoded video will be checked by a human to confirm the validity of the machine processing in applications such as surveillance. In such cases, if a pre-processing step is applied in which pixel values ​​of regions of non-interest are replaced with arbitrary pixel values, a problem arises in that visibility is impaired when the pixel values ​​of the regions of interest are close to the pixel values ​​replaced for the regions of non-interest.

[0013] Furthermore, when the decoded video is monitored on a monitor, if a certain region is determined as a region of no interest for a long period of time in the encoding device, there is a concern that monitor burn-in may occur. For example, if a video decoder uses fixed pixel values ​​(e.g., pixel values ​​corresponding to gray) for the region of no interest, there is a concern that monitor burn-in may occur if the region of no interest appears in the same position across multiple frames.

[0014] Next, configuration examples of a video encoder (video encoding device: encoder) capable of performing pre-processing including ROI processing and a video decoder (video decoding device: decoder) capable of performing post-processing including ROI processing will be described. ROI processing is a process in which the video encoder reduces the amount of encoded data by applying processing based on the position and size of a region of interest to a frame to be processed. For example, pixel values ​​in regions of non-interest are replaced with arbitrary pixel values ​​(e.g., pixel values ​​corresponding to gray). There are also processes that reduce the number of pixels to be encoded by deleting regions that include only regions of non-interest or by reducing regions that include regions of interest.

[0015] In addition to ROI processing, the following techniques can be used as elemental techniques for pre-processing and post-processing: Temporal scalability Spatial scalability Bit depth shift (bit depth truncation)

[0016] Temporal scalability is a technique in which a video encoder reduces the frame rate in the temporal direction and a video decoder upsamples the frame rate. In other words, it is a technique in which a video encoder performs frame skipping and a video decoder performs frame interpolation. Frame skipping and frame interpolation in the temporal direction are sometimes referred to as TR (Temporal Re-sampling).

[0017] Spatial scalability is a technology in which a video encoder reduces the spatial resolution of a video and a video decoder increases the resolution (returning it to the original resolution). In other words, it is a technology in which a video encoder reduces a picture and a video decoder enlarges the picture. Hereinafter, picture reduction and enlargement may be referred to as SR (Spatial Re-sampling) or spatial resampling.

[0018] Specifically, in SR, a reduction process is applied to input video on a picture-by-picture basis so as to reduce the size (width and height) of the current picture. That is, in SR, the video encoder performs spatial resampling on the current picture. By reducing the number of pixels to be coded, it becomes easier to reduce the amount of data in the bitstream. During decoding, as a corresponding post-processing, an enlargement process to the size before reduction is applied to the decoded video. That is, in SR, the video decoder performs spatial reconstruction on the current picture.

[0019] In machine-oriented video coding, the detection of objects and the like is generally insensitive to bit depth, so the number of bits used to represent the video is reduced by bit depth shifting.

[0020] FIG. 17 is a block diagram showing an example of the configuration of a video encoder that performs pre-processing and a video decoder that performs post-processing.

[0021] In the configuration example shown in Figure 17, the video encoder includes a temporal resampling unit 811 that performs temporal resampling on input video (input pictures), an RTG (retargeting) processing unit 812 that performs ROI processing on the frame-thinned input video, a spatial resampling unit 813 that applies spatial scalability to the output of the RTG processing unit 812, a bit-depth shifting unit 814 that applies bit-depth truncation to the output of the spatial resampling unit 813, and an inner encoder 815 that encodes the pictures supplied from the bit-depth shifting unit 814 using a predetermined video encoding method. The predetermined video encoding method may be, for example, an encoding method based on H.266 / VVC. As an example, the predetermined video encoding method may be VVC that takes End2End into account.

[0022] The video decoder includes an internal decoder 821 that decodes a video bitstream to obtain decoded pictures, a spatial resampling unit 822 that applies spatial scalability to the decoded pictures, an RTG processing unit 823 that performs ROI processing on the output of the spatial resampling unit 822, a temporal resampling unit 824 that performs temporal resampling on the output of the RTG processing unit 823, and a bit depth shifting unit 825 that restores video from the temporal resampling unit 824.

[0023] The present invention is an invention for solving the above-mentioned problems, and aims to provide a video encoding device, a video decoding device, a video encoding method, a video decoding method, and a program that can suppress a decrease in visibility.

[0024] A video encoding device based on the present disclosure includes a region of interest detection means for detecting a region of interest from an input video frame, a video encoding means for generating a video bitstream, and a decoder control means for generating an auxiliary information bitstream including information for controlling a process for setting pixel values ​​of regions of non-interest in a video decoding device.

[0025] A video decoding device based on the present disclosure includes a video decoding means for decoding a video bitstream to obtain a decoded video frame, a region of non-interest identification means for identifying a region of non-interest in the decoded video frame, and a region of non-interest processing control means for decoding an auxiliary information bitstream including information for controlling a setting process of pixel values ​​of the region of non-interest to obtain information for controlling the setting process.

[0026] A video encoding method according to the present disclosure detects a region of interest from an input video frame, generates a video bitstream, and generates a side information bitstream including information for controlling the process of setting pixel values ​​of regions of non-interest in a video decoding device.

[0027] A video decoding method based on the present disclosure decodes a video bitstream to obtain a decoded video frame, identifies a region of non-interest in the decoded video frame, and decodes an auxiliary information bitstream containing information for controlling a setting process of pixel values ​​of the region of non-interest to obtain information for controlling the setting process.

[0028] A video encoding program based on the present disclosure causes a computer to perform the following processes: detecting a region of interest from an input video frame; generating a video bitstream; and generating an auxiliary information bitstream including information for controlling the process of setting pixel values ​​of regions of non-interest in a video decoding device.

[0029] A video decoding program based on the present disclosure causes a computer to perform the following processes: decoding a video bitstream to obtain a decoded video frame; identifying a region of non-interest in the decoded video frame; and decoding an auxiliary information bitstream containing information for controlling a setting process of pixel values ​​of the region of non-interest to obtain information for controlling the setting process.

[0030] According to the present invention, it is possible to suppress a decrease in visibility.

[0031] 1 is a block diagram showing an embodiment of a video encoder and a video decoder of the present disclosure. FIG. 2 is a flowchart showing the operation of the video encoder of the first embodiment. FIG. 3 is a block diagram showing an embodiment of a video encoder and a video decoder of the present disclosure. FIG. 4 is a flowchart showing the operation of the video encoder of the second embodiment. FIG. 5 is a flowchart showing the operation of the video decoder of the second embodiment. FIG. 6 is a block diagram showing an embodiment of a video encoder and a video decoder of the present disclosure. FIG. 7 is a flowchart showing the operation of the video encoder of the third embodiment. FIG. 8 is a flowchart showing the operation of the video decoder of the third embodiment. FIG. 9 is a block diagram showing an embodiment of a video encoder and a video decoder of the fourth embodiment. FIG. 10 is a block diagram showing an example configuration of an RTG processing unit in a video decoder of the fourth embodiment. FIG. 11 is an explanatory diagram showing an example syntax. FIG. 12 is an explanatory diagram showing a correspondence relationship between indexes and magnifications. FIG. 13 is a block diagram showing an example configuration of an information processing system. FIG. 14 is a block diagram showing main parts of a video encoder. FIG. 15 is a block diagram showing main parts of a video decoder. FIG. 16 is a block diagram showing an example configuration of a video encoder that performs pre-processing and a video decoder that performs post-processing.

[0032] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0033] Embodiment 1. Figure 1 is a block diagram illustrating an embodiment of a video encoder and a video decoder.

[0034] The video encoder 101 shown in FIG. 1 includes a region of interest detector 1001 , a region of non-interest replacement processor 1002 , an encoder 1003 , a multiplexer 1004 , and a decoder processing controller 1005 .

[0035] The video decoder 201 includes a region of no interest replacement processing unit 2001 , a region of no interest replacement processing control unit 2002 , a region of no interest identification unit 2003 , a decoder 2004 , and a demultiplexer 2005 .

[0036] The video encoding method used by the video encoder 101 and the video decoder 201 may be any method that performs predictive encoding by referring to previously encoded frames. However, a common video encoding method is used by the video encoder and the video decoder. For example, a predictive encoding method such as H.266 / VVC may be used. Furthermore, a method using deep learning or a combination of multiple methods may be used.

[0037] 1 simply indicates the direction of signal (data) flow, but does not exclude bidirectionality. This also applies to other block diagrams.

[0038] [Description on the Encoding Side] In the video encoder 101, the region of interest detection unit 1001 extracts frame features from an input frame (input video frame). The region of interest detection unit 1001 detects a region to be determined as a region of interest based on the extracted features. The region of interest detection unit 1001 supplies a set of coordinate information indicating the coordinates of the region of interest, along with the input frame, to the region of non-interest replacement processing unit 1002. Note that detecting a region to be determined as a region of interest essentially means detecting a region of non-interest. As an example, the coordinate information includes the x (horizontal, i.e., horizontal) coordinate and y (vertical, i.e., vertical) coordinate of the top left corner of the region, and the size of the region (size in the x direction and size in the y direction).

[0039] The number of regions of interest detected by the region of interest detection unit 1001 may be plural or zero.

[0040] One method for detecting a region of interest is to use a neural network trained to detect a specific object. In this case, the neural network receives, for example, pixel values ​​of an input video frame and outputs the coordinates of a region determined to contain an object. The output coordinates are designated as the coordinates of the region of interest.

[0041] The region of interest for each frame may be calculated from only the processing result of the frame, or may be calculated based on the processing results of multiple frames. In the case where the region of interest is calculated based on the processing results of multiple frames, the region of interest detection unit 1001 buffers the frames until a predetermined number of frames have been input to the video encoder.

[0042] The region of non-interest replacement processor 1002 replaces pixel values ​​in regions other than the region of interest in the frame to be processed, i.e., the region of non-interest, with predetermined pixel values ​​based on the frame (the input frame to the video encoder 101) supplied from the region of interest detector 1001 and information about the region of interest (e.g., coordinate information). In this embodiment, the predetermined pixel values ​​are values ​​agreed upon in advance with the video decoder. Hereinafter, the process of replacing pixel values ​​in the region of non-interest with predetermined pixel values ​​may be referred to as a replacement process.

[0043] The encoder 1003 converts the frames supplied from the region of interest replacement processor 1002, to which the replacement process has been applied, into a video bitstream using a predetermined method. Any method may be used as the predetermined method. However, as described above, the predetermined method is the same as the method used by the video decoder 201. The predetermined method may be, for example, a method based on the H.266 / VVC method that combines block division, predictive coding, transform quantization, and entropy coding, or a method using a neural network.

[0044] The decoder processing control unit 1005 generates a side information bitstream based on control information input from outside to the video encoder 101. The control information is information for replacing pixel values ​​of regions of non-interest in the video decoder 201. In this embodiment and the following embodiments, the control information includes information indicating whether or not the video decoder 201 should perform a replacement process, and information regarding pixel values ​​to be replaced when the replacement process is performed in the video decoder 201. A specific example of the information indicating whether or not the video decoder 201 should perform a replacement process is, for example, a flag (e.g., 1: perform replacement process, 0: do not perform replacement process).

[0045] The information on the pixel value to be substituted is, for example, a binary representation of the pixel value to be substituted by the video decoder 201. If candidate pixel values ​​are defined in advance, the information on the pixel value to be substituted may be, for example, an identification number of the candidate.

[0046] The multiplexer 1004 generates and outputs one bitstream from the video bitstream output from the encoder 1003 and the auxiliary information bitstream output from the decoder processing control unit 1005 .

[0047] Next, the operation of the video encoder 101 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the operation of the video encoder 101.

[0048] First, the region of interest detection unit 1001 detects a region to be a region of interest from an input frame of video (step S100). The region of interest detection unit 1001 outputs information that can identify the region of interest, such as a group of coordinates of the detected region of interest, to the region of non-interest replacement processing unit 1002.

[0049] Next, the region-of-interest replacement processor 1002 performs a replacement process (step S101). That is, the region-of-interest replacement processor 1002 identifies a region of interest based on the information capable of identifying the region of interest from the region-of-interest detector 1001, and replaces pixel values ​​in the region corresponding to the region of interest with predetermined pixel values. The region-of-interest replacement processor 1002 supplies the pixel values ​​of the frame after the replacement process to the encoder 1003.

[0050] The encoder 1003 encodes the frames supplied from the region of no interest replacement processor 1002 using a predetermined method to generate a video bitstream (step S102).

[0051] The decoder processing control unit 1005 analyzes control information input from outside the video encoder 101 to obtain information indicating whether or not to perform substitution processing in the decoder and pixel values ​​after substitution if the video decoder 201 performs substitution processing (step S103). The decoder processing control unit 1005 generates a side information bitstream including the information and outputs it to the multiplexer 1004 (step S105).

[0052] The multiplexer 1004 multiplexes the video bitstream output from the encoder 1003 and the auxiliary information bitstream output from the decoder processing control unit 1005, and outputs the multiplexed bitstream (step S106). Note that the video encoder 101 may output the video bitstream and the auxiliary information bitstream separately.

[0053] [Description on the decoding side] In the video decoder 201, if a video bitstream and an auxiliary information bitstream are multiplexed, the demultiplexer 2005 demultiplexes the input bitstream (input bitstream) to obtain the video bitstream and the auxiliary information bitstream.

[0054] The decoder 2004 reconstructs frames (decoded video frames) from the video bitstream based on a decoding process corresponding to the method used by the encoder, and supplies the reconstructed frames to the region of non-interest identifying unit 2003.

[0055] The region of non-interest replacement processing control unit 2002 receives the auxiliary information bitstream and acquires control information. If the control information indicates that replacement processing is not to be performed, the region of non-interest replacement processing control unit 2002 supplies the information to the region of non-interest replacement processing unit 2001. If the control information indicates that replacement processing is to be performed, the region of non-interest replacement processing control unit 2002 supplies the pixel values ​​included in the control information to the region of non-interest replacement processing unit 2001.

[0056] The region of non-interest identifying unit 2003 detects a region having pixel values ​​(specific pixel values) previously agreed upon with the video encoder from the frame supplied from the decoder 2004. That is, the region of non-interest identifying unit 2003 identifies a region of non-interest. The region of non-interest identifying unit 2003 supplies information about the identified region of non-interest and the frame supplied from the decoder 2004 to the region of non-interest replacement processing unit 2001.

[0057] When pixel values ​​are supplied from the region of interest replacement processing control unit 2002, the region of interest replacement processing unit 2001 performs replacement processing on the region identified by the region of interest identification unit 2003 based on information about the pixel values. The region of interest replacement processing unit 2001 outputs the frame after application of replacement processing as an output frame. When control information indicating that replacement processing is not to be applied is supplied, the region of interest replacement processing unit 2001 outputs the input frame as is.

[0058] Next, the operation of the video decoder 201 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the operation of the video decoder 201.

[0059] First, the demultiplexer 2005 demultiplexes the input bitstream to obtain a video bitstream and a side information bitstream (step S200). The demultiplexer 2005 supplies the video bitstream to the decoder 2004. The demultiplexer 2005 supplies the side information bitstream to the region of no interest replacement processing control unit 2002.

[0060] The decoder 2004 decodes the video bitstream and supplies the decoded frames to the region of non-interest identifying unit 2003 (step S201).

[0061] The region of non-interest identifying unit 2003 identifies pixels from the decoded frame that have pixel values ​​after replacement by the video encoder 101 (step S202). That is, the region of non-interest identifying unit 2003 identifies a region of non-interest. The region of non-interest identifying unit 2003 supplies information capable of identifying the region of non-interest, together with the decoded frame, to the region of non-interest replacement processing unit 2001.

[0062] The region of non-interest replacement processing control unit 2002 decodes the auxiliary information bitstream (step S204). Control information is obtained through the decoding process. As described above, the control information includes information about the region of non-interest (e.g., pixel values ​​after replacement in the replacement process in the video decoder 201). The region of non-interest replacement processing control unit 2002 supplies the control information to the region of non-interest replacement processing unit 2001 (step S207).

[0063] When the control information indicates that replacement processing is to be performed, the region of interest replacement processing unit 2001 performs replacement processing on the region of interest identified by the region of interest identification unit 2003 based on pixel value information supplied from the region of interest replacement processing control unit 2002 (step S203). The region of interest replacement processing unit 2001 outputs the frame after application of replacement processing as an output frame. Note that when control information indicating that replacement processing is not to be applied is supplied from the region of interest replacement processing control unit 2002, the region of interest replacement processing unit 2001 outputs the input frame as is.

[0064] In this embodiment, in order to control the processing of the region of non-interest in the video decoder 201, the video encoder 101 transmits, in a side information bitstream, information indicating whether or not the video decoder 201 should perform a substitution processing on the region of non-interest, and information on pixel values ​​after substitution if the substitution processing is performed in the video decoder 201, to the video decoder 201. Thus, the video encoder 101 can control the substitution processing of pixel values ​​of the region of non-interest in the video decoder 201, rather than leaving the substitution processing entirely up to the video decoder 201. Being able to control the substitution processing of pixel values ​​of the region of non-interest in the video decoder 201 reduces the possibility of monitor burn-in.

[0065] In addition, the video encoder 101 notifies the video decoder 201 of the pixel values ​​after replacement that are used when the replacement process is performed in the video decoder 201, thereby reducing the processing load on the video decoder 201 related to the replacement process.

[0066] In addition, a machine that uses the output frame may be designed to perform special processing using pixel values ​​obtained by replacing pixel values ​​of regions of non-interest. If the replacement processing is left entirely to the video decoder 201, there is a concern that the replacement processing that is inconsistent with the special processing may be performed. However, in this embodiment, the video encoder 101 notifies the video decoder 201 of the replaced pixel values ​​that will be used when the replacement processing is performed in the video decoder 201, thereby preventing the replacement processing that is inconsistent with the special processing from being performed.

[0067] Embodiment 2 Figure 4 is a block diagram showing another embodiment of a video encoder and a video decoder.

[0068] In the first embodiment, the non-interest region identifying unit 2003 in the video decoder 201 identifies a non-interest region based on whether or not a pixel has a specific pixel value. If the region of interest includes a pixel with that pixel value, the non-interest region identifying unit 2003 may determine the region of interest as a non-interest region. Furthermore, the video decoder 201 is required to perform processing to identify the region of interest based on the pixel value, which increases the processing load. As described below, this embodiment reduces the possibility of determining a region of interest as a non-interest region, and also suppresses an increase in the processing load of the video decoder.

[0069] The video encoder 102 shown in FIG. 4 includes a region of interest detector 1001A, a region of non-interest replacement processor 1002, an encoder 1003, a multiplexer 1004, and a decoder processing controller 1005A.

[0070] The video decoder 202 includes a region of no interest replacement processor 2001 , a region of no interest replacement processor control unit 2002A, a region of no interest identifier 2003A, a decoder 2004 , and a demultiplexer 2005 .

[0071] In this embodiment, unlike the first embodiment, the coordinates of the region of interest are also included in the bitstream. As a result, although the amount of code increases by the amount corresponding to the transmission of the coordinates of the region of interest, the possibility of the video decoder 202 erroneously detecting a region of non-interest is reduced, and the processing load of the video decoder 202 is also reduced.

[0072] [Description of the Encoding Side] In the video encoder 102, the configurations and functions of the region of no interest replacement processor 1002, encoder 1003, and multiplexer 1004 are the same as those in the first embodiment.

[0073] In addition to the processing performed by the region of interest detection unit 1001 in the first embodiment, the region of interest detection unit 1001A performs processing to supply information capable of identifying a region of interest to the decoder processing control unit 1005A.

[0074] The decoder processing control unit 1005A generates a side information bitstream that includes the control information in the first embodiment and information about the region of interest supplied from the region of interest detection unit 1001A (in this example, information that can identify the region of interest, such as coordinate information about the region of interest). The decoder processing control unit 1005A generates the side information bitstream in accordance with, for example, annotated_regions of H.274 / VSEI. Note that the decoder processing control unit 1005A may also include coordinate information about regions of non-interest in the side information bitstream as information about the region of interest.

[0075] Next, the operation of the video encoder 102 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the operation of the video encoder 102.

[0076] The operation of the video encoder 102 is basically the same as the operation of the video encoder 101 of the first embodiment.

[0077] However, in this embodiment, when the region of interest detection unit 1001A detects a region to be considered as a region of interest from the input frame of video (step S100), it outputs information that can identify the region of interest to the region of non-interest replacement processing unit 1002 and the decoder processing control unit 1005A.

[0078] Also, in step S105A, in addition to the processing by the decoder processing control unit 1005 described above, the decoder processing control unit 1005A also performs processing to include information regarding the region of interest (in this example, coordinate information of the region of interest) in the auxiliary information bitstream.

[0079] Other processing is the same as the processing of the video encoder 101 in the first embodiment. [Description on the Decoding Side] In the video decoder 202, the configurations and functions of the region of no interest replacement processing unit 2001, the decoder 2004, and the demultiplexer 2005 are the same as those in the first embodiment.

[0080] The region of interest replacement processing control unit 2002A decodes the auxiliary information bit stream and obtains control information. If the control information indicates that replacement processing is not to be performed, the control information is supplied to the region of interest replacement processing unit 2001. If the control information indicates that replacement processing is to be performed, the pixel values ​​included in the control information are supplied to the region of interest replacement processing unit 2001. The region of interest replacement processing control unit 2002A also supplies a bit stream corresponding to the coordinates of the region of interest in the auxiliary information bit stream to the region of interest identification unit 2003A.

[0081] The region of non-interest identifying unit 2003A decodes the bitstream supplied from the region of non-interest replacement processing control unit 2002A and identifies a region of non-interest. The region of non-interest identifying unit 2003A supplies information about the identified region of non-interest and the frame supplied from the decoder 2004 to the region of non-interest replacement processing unit 2001.

[0082] Next, the operation of the video decoder 202 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the operation of the video decoder 202.

[0083] The operation of the video decoder 202 is basically the same as the operation of the video decoder 201 of the first embodiment.

[0084] However, in this embodiment, before the region of non-interest identifying unit 2003A identifies a region of non-interest in step S202A, the region of non-interest replacement processing control unit 2002A decodes the auxiliary information bit stream (step S204) and then performs step S205. In step S205, the region of non-interest replacement processing control unit 2002A acquires information about the region of interest obtained from the decoded auxiliary information bit stream and supplies the information about the region of interest to the region of non-interest identifying unit 2003A.

[0085] The region of non-interest identifying unit 2003A then decodes the bitstream supplied from the region of non-interest replacement processing control unit 2002A and identifies the region of non-interest (step S202A). In this embodiment, the region of non-interest identifying unit 2003A identifies the region of non-interest, for example, using coordinate information of the region of interest obtained from the decoded auxiliary information bitstream. The region of non-interest identifying unit 2003A supplies information about the identified region of non-interest and the frame supplied from the decoder 2004 to the region of non-interest replacement processing unit 2001.

[0086] As in the first embodiment, the region of non-interest replacement processing unit 2001 performs replacement processing on the region identified by the region of non-interest identification unit 2003A (step S203).

[0087] The other processes are the same as those of the video decoder 201 in the first embodiment.

[0088] In this embodiment, the video encoder 102 transmits information about the region of non-interest (in this example, coordinate information about the region of interest) in a side information bitstream to the video decoder 201 in order to control processing of the region of non-interest by the video decoder 202. Therefore, in addition to the effects of the first embodiment, an effect is obtained in that the possibility of the region of interest being determined to be the region of non-interest is reduced even when pixel values ​​of the region of non-interest and the region of interest are similar.

[0089] Embodiment 3 In the first embodiment, there remains a possibility that the video decoder 201 will determine that a region of interest is a region of non-interest, but in this embodiment, such a possibility is reduced.

[0090] FIG. 7 is a block diagram illustrating another embodiment of a video encoder and a video decoder.

[0091] The video encoder 103 shown in FIG. 7 includes a region of interest detector 1001, a region of non-interest replacement processor 1002B, an encoder 1003, a multiplexer 1004, a decoder processing controller 1005B, and an encoder processing controller 1006.

[0092] The video decoder 203 includes a region of no interest replacement processing unit 2001 , a region of no interest replacement processing control unit 2002 B, a region of no interest identification unit 2003 B, a decoder 2004 , and a demultiplexer 2005 .

[0093] [Description of the Encoding Side] In the video encoder 103, the configurations and functions of the region of interest detector 1001, encoder 1003, and multiplexer 1004 are the same as those in the first embodiment.

[0094] The encoder processing control unit 1006 determines pixel values ​​to replace pixel values ​​of regions of non-interest in the video encoder 103, and supplies the determined pixel values ​​to the region of non-interest replacement processing unit 1002B and the decoder processing control unit 1005B.

[0095] For example, the encoder processing control unit 1006 scans the input frame and selects pixel values ​​that are not included in the input frame as pixel values ​​of the region of non-interest after replacement in the video encoder 103. Note that the encoder processing control unit 1006 may use any value as the pixel value of the region of non-interest after replacement. The any value may be specified, for example, by control information input from outside the video encoder 103. The encoder processing control unit 1006 supplies the selected pixel values ​​to the decoder processing control unit 1005B and the encoder processing control unit 1006.

[0096] The region of non-interest replacement processing unit 1002B inputs the frame (the input frame to the video encoder 103) and pixel values ​​(in this example, the pixel values ​​of the region of non-interest after replacement in the video encoder 103) supplied from the region of interest detection unit 1001. The region of non-interest replacement processing unit 1002B replaces the pixel values ​​of the region of non-interest in the frame to be processed with pixel values ​​supplied from the encoder processing control unit 1006.

[0097] The decoder processing control unit 1005 B generates a side information bitstream based on control information input to the video encoder 103 from the outside and pixel values ​​supplied from the encoder processing control unit 1006 .

[0098] Next, the operation of the video encoder 103 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the operation of the video encoder 103.

[0099] The operation of the video encoder 103 is basically the same as the operation of the video encoder 101 of the first embodiment.

[0100] However, in this embodiment, after the encoder processing control unit 1006 determines pixel values ​​to replace the pixel values ​​of the regions of non-interest in the video encoder 103 (step S104), the region of non-interest replacement processing unit 1002B replaces the pixel values ​​of the regions of non-interest in the frame to be processed with pixel values ​​supplied from the encoder processing control unit 1006 (step S101B).

[0101] In step S105B, in addition to the processing by the decoder processing control unit 1005 described above, the decoder processing control unit 1005B also performs processing to include in the auxiliary information bitstream the pixel values ​​determined by the encoder processing control unit 1006 (pixel values ​​that have replaced the pixel values ​​of the region of non-interest in the video encoder 103).

[0102] The other processes are the same as those of the video encoder 101 in the first embodiment.

[0103] [Description on the Decoding Side] In the video decoder 203, the configurations and functions of the region of no interest replacement processor 2001, decoder 2004, and demultiplexer 2005 are the same as those in the first embodiment.

[0104] The region of interest replacement processing control unit 2002B decodes the auxiliary information bitstream and obtains control information. If the control information indicates that replacement processing is not to be performed, it supplies the information to the region of interest replacement processing unit 2001. If the control information indicates that replacement processing is to be performed, the region of interest replacement processing control unit 2002B supplies the pixel values ​​to be replaced by the video decoder 203, which are included in the control information, to the region of interest replacement processing unit 2001. The region of interest replacement processing control unit 2002B also supplies the pixel values, which are included in the control information and are used to replace pixel values ​​of regions of interest by the video encoder 103, to the region of interest identification unit 2003B.

[0105] The region of non-interest identifying unit 2003B scans pixel values ​​in the frame supplied from the decoder 2004 and detects, as a region of non-interest, a region having pixel values ​​supplied from the region of non-interest replacement processing control unit 2002B. That is, the region of non-interest identifying unit 2003B identifies a region of non-interest. The region of non-interest identifying unit 2003B supplies information identifying the region of non-interest and the frame input from the decoder 2004 to the region of non-interest replacement processing unit 2001.

[0106] Next, the operation of the video decoder 203 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the operation of the video decoder 203.

[0107] The operation of the video decoder 203 is basically the same as the operation of the video decoder 201 of the first embodiment.

[0108] However, in this embodiment, the region of non-interest replacement processing control unit 2002B extracts, from the decoded information, pixel values ​​that the video encoder 103 has replaced with pixel values ​​of the region of non-interest (step S206).The region of non-interest replacement processing control unit 2002B supplies the extracted pixel values ​​to the region of non-interest identification unit 2003B.

[0109] The region of non-interest replacement processing control unit 2002B identifies pixels having the pixel values ​​extracted by the region of non-interest replacement processing control unit 2002B from the decoded frame (step S202B). That is, the region of non-interest identification unit 2003 identifies a region of non-interest. The region of non-interest identification unit 2003B supplies information capable of identifying the region of non-interest to the region of non-interest replacement processing unit 2001, along with the decoded frame.

[0110] The other processes are the same as those of the video decoder 201 in the first embodiment.

[0111] In this embodiment, the video encoder 103 can select appropriate pixel values ​​as pixel values ​​of the region of non-interest, e.g., pixel values ​​that are not expected to occur in the region of interest in the decoded frame, thereby reducing the likelihood that the video decoder 203 will determine the region of interest as a region of non-interest.

[0112] Embodiment 4 Figure 10 is a block diagram showing yet another embodiment of a video encoder and a video decoder.

[0113] The video encoder 104 shown in FIG. 10 includes a region of interest detector 1001 , an encoder 1003 , a multiplexer 1004 , a decoder processing controller 1005 , and an RTG processor 1010 .

[0114] The video decoder 204 includes a decoder 2004, a demultiplexer 2005, and an RTG processing unit 2010.

[0115] [Description of the Encoding Side] In the video encoder 104, the configurations and functions of the region of interest detector 1001, encoder 1003, decoder processing controller 1005, and multiplexer 1004 are the same as those in the first embodiment.

[0116] However, in this embodiment, the decoder processing control unit 1005 does not perform processing to set, in the auxiliary information bitstream, information on pixel values ​​after replacement in non-ROIs (regions of non-interest) when the video decoder 201 performs ROI processing. Instead, the decoder processing control unit 1005 includes, in the auxiliary information bitstream, pixel values ​​that are used when the video decoder 204 restores partial regions for which pixel encoded data is not transmitted and that are determined by the RTG processing unit 1010.

[0117] The RTG processing unit 1010 determines the position of a partial region based on the coordinate information of the region of interest (specifically, the coordinate information of a rectangular region that is an ROI containing an object). The RTG processing unit 1010 also determines an appropriate reduction ratio (magnification factor) for each partial region.

[0118] FIG. 11 is a block diagram showing an example of the configuration of the RTG processing unit 2010 in the video decoder 204 according to the fourth embodiment, and also shows an example of a partial region.

[0119] In FIG. 11 , each of the 20 rectangles shown in example (A) corresponds to a partial region. Example (A) illustrates two ROIs, each containing an object. One ROI (the ROI containing the larger object) is divided into two partial regions. In FIG. 11 , black circles indicate division points in the picture used to generate the partial regions. The partial region restoration unit 8232 determines the division points based on the position information of the ROI. Specifically, the partial region restoration unit 8232 determines the division points so that each ROI is divided at the boundary portion of the ROI. The region between the division points is the partial region. FIG. 11 illustrates two rectangles (i.e., ROIs), each containing one object. Note that in the example shown in FIG. 11 , the larger object is further divided into two rectangles.

[0120] 11 shows an example configuration of the RTG processing unit 2010 in the video decoder 204. The RTG processing unit 1010 in the video encoder 104 determines the position of a partial region (see (A) in FIG. 11 ) based on coordinate information of a region of interest (specifically, coordinate information of a rectangular region that is an ROI including an object). As described above, a partial region is a rectangular region.

[0121] The RTG processing unit 1010 includes a partial region definition unit. Similar to the RTG processing unit 2010 in the video decoder 204, the partial region definition unit sets partial regions, for example, as follows. The RTG processing unit 2010 in the video decoder 204 will be described later.

[0122] The partial region definition unit determines division points in a picture based on the position information of the ROI. Specifically, the partial region definition unit determines division points so as to divide the picture at the boundary portions of each ROI. For example, the partial region definition unit selects coordinate values ​​of horizontal division points from the x-coordinates of the top left and bottom right points of each ROI without overlapping. Similarly, the partial region definition unit selects coordinate values ​​of vertical division points from the y-coordinates of the top left and bottom right points of each ROI without overlapping. The partial region definition unit then stores the values ​​of the division points in each direction in one-dimensional arrays Ph and Pv, respectively, in ascending order. Example (A) illustrates the case where six division points are defined in the horizontal direction and five division points are defined in the vertical direction. Note that the start point (the point with coordinate value 0) and end point (the point with coordinate value 400 or 300) of the coordinates are also included in the division points. In this case, Ph={0, 100, 220, 300, 360, 400} and Pv={0, 50, 150, 250, 300}. Regions obtained by dividing the processing target frame into a grid based on these division points are defined as partial regions before forward transformation.

[0123] The partial region definition unit determines a reduction ratio (magnification factor) for each ROI and determines the reduction ratio (magnification factor) between each division point based on the ROI reduction ratio (magnification factor). For example, the ROI magnification factor can be set to greater than 0 and less than 1.0 (0 to 1.0) for ROIs with a size equal to or greater than a predetermined value, and set to 1.0 for ROIs whose size is less than the predetermined value. The magnification factor between vertical division points (horizontal magnification factor) is then set to the maximum magnification factor among the ROIs located between those division points. If no ROI exists, it is set to 0. Similarly, the magnification factor between horizontal division points is determined from the magnification factor of the ROIs located between the division points. The magnification factors between horizontal and vertical division points are represented as one-dimensional arrays Sh and Sv, respectively. In Example (A), Sh = {0.0, 0.8, 0.0, 1.0, 0.0} and Sv = {0.0, 0.8, 1.0, 0.0}. Example (A) shows an example in which the magnification of one ROI (the ROI containing the larger object) is set to 0.8 and the magnification of the other ROI (the ROI containing the smaller object) is set to 1.0.

[0124] The partial region definition unit calculates division points after forward transformation from the division points before forward transformation and the magnifications between those division points. Specifically, when the horizontal division points after forward transformation are represented by a one-dimensional array Ph', the division points after forward transformation are calculated as Ph'[0] = 0, Ph'[m] = Ph'[m-1] + (Ph[m] - Ph[m-1]) * Sh[m-1]. Note that the size of Ph is M, where m is an integer greater than or equal to 1 and less than M. Similarly, when the vertical division points after forward transformation are represented by a one-dimensional array Pv', the size of Pv is N, where n is an integer greater than or equal to 1 and less than N, and the division points are calculated as Pv'[0] = 0, Pv'[n] = Pv'[n-1] + (Pv[n] - Pv[n-1]) * Sv[n-1]. The size after forward transformation is expressed by the last elements of Ph' and Pv', i.e., Ph'[M-1] and Pv'[N-1]. In example (A), Ph'={0, 0, 96, 96, 156, 156} and Pv'={0, 0, 80, 180, 180}, and the width and height after forward transformation are 156 and 180, respectively. As described above, the number of segmentation points before and after forward transformation is the same, so the segmentation points before and after forward transformation can be associated with elements of the same index. As described above, the size of a frame after forward transformation is determined by the segmentation points before forward transformation and the magnification between those segmentation points, i.e., the position and magnification of the ROI. In other words, if the position or magnification of the ROI differs, the size of the frame after forward transformation may change. One method for aligning the size of frames within a sequence is, for example, to update the magnification between segmentation points to achieve the desired size.

[0125] The partial region definition unit includes the ROI reduction ratio (magnification factor) in the side information (control information) along with the coordinate information of each ROI before affine transformation, i.e., original coordinate information (corresponding to the division points shown in Example (A)). The partial region definition unit also includes information specifying the frame size before and after transformation in the side information. In the example shown in FIG. 11 , the side information includes coordinate values ​​(100, 50, 220, 150) (the former two numbers are the x- and y-coordinates of the upper left corner of the region, and the latter two numbers are the x- and y-coordinates of the lower right corner of the region) as well as a magnification factor (0.8, 0.8) (the former is the horizontal magnification factor, and the latter is the vertical magnification factor). The side information also includes coordinate values ​​(300, 50, 360, 250) as well as a magnification factor (1.0, 1.0).

[0126] As described above, in this embodiment, in addition to the size of the picture before and after forward transformation, i.e., before and after RTG processing, the side information also includes the upper left x and y coordinates and the size of the region. For example, the scaling factor is always included in the bitstream as the syntax of the side information shown in Figure 12. The correspondence between the index and the scaling factor is defined as shown in Figure 13. The syntax elements shown in Figure 12 are as follows:

[0127] retargeting_flag: Flag indicating whether to perform RTG processing. retargeting_image_size_len: Number of bits for retargeting_image_width and retargeting_image_size_height. retargeting_image_size_width: Width of the picture after RTG processing in the video encoder. retargeting_image_size_height: Height of the picture after RTG processing in the video encoder. retargeting_image_size_difference_flag: Flag indicating whether the width or height of the picture differs before and after RTG processing. retargeting_to_output_difference_len: Number of bits for retargeting_to_output_difference_width and retargeting_to_output_difference_height. retargeting_to_output_difference_width: Difference in width of the picture before and after RTG processing in the video encoder. retargeting_to_output_difference_height: Difference in height of the picture before and after RTG processing in the video encoder. retargeting_roi_flag: Flag indicating whether ROI information is indicated. If 0, the entire picture is the ROI, and its magnification is 1.0.

[0128] roi_size_len: Number of bits of roi_pos_x[i], roi_pos_y[i], roi_size_x[i], roi_size_y[i]. num_rois_len: Number of bits of num_rois. num_rois: Number of ROIs. roi_scaling_factor_flag: Flag indicating whether scale_factor_idx[i] is specified. If 0, scale_factor_idx[i] = scale_factor_idx[i-1]. scale_factor_idx[i]: Index indicating the magnification of the i-th ROI. For example, it is specified as shown in the table of scale_factors_idx[i] in Figure 13. roi_pos_x[i]: X coordinate of the top left of the i-th ROI. roi_pos_y[i]: Y coordinate of the top left of the i-th ROI. roi_size_x[i]: Width of the i-th ROI. roi_size_y[i]: Height of the i-th ROI.

[0129] Note that u(n) is an n-bit unsigned integer. When n in the syntax table is "v," the number of bits varies depending on the values ​​of other syntax elements. These syntax elements may be set to a value for each frame and included in the side information, or a common value may be set for multiple frames and included in the side information. Furthermore, the unit of setting for each syntax element may be different, for example, so that syntax elements set before retargeting_roi_flag are included in the side information on a sequence-by-sequence basis, and syntax elements after roi_flag are included in the side information on a frame-by-frame basis.

[0130] Furthermore, in the set of partial regions for which encoded data is to be transmitted, partial regions other than partial region C are regions that include one or more ROIs in whole or in part. Hereinafter, partial region C, which does not include an object, may be referred to as a specific region. For example, the partial region definition unit replaces the pixel values ​​of the specific region with fixed values ​​(e.g., pixel values ​​corresponding to gray). In other words, if a partial region that does not include an object (corresponding to a type of non-ROI) exists in the same row or column as the partial region that does not include the object, the video encoder 104 does not delete the partial region that does not include the object. The specific region is then encoded in the same manner as the ROI.

[0131] A partial area with a reduction ratio (magnification) of 1.0 means that the partial area is not reduced, and a partial area with a magnification of 0 means that the partial area is deleted.

[0132] The RTG processing unit 1010 determines affine transformation parameters for each partial region from the positions of the division points and the magnification ratio between the division points. That is, the RTG processing unit 1010 determines parameters so that partial regions with non-zero horizontal and vertical magnification ratios are moved to the upper left of the picture and the partial regions are reduced at the determined reduction ratio. The RTG processing unit 1010 then performs affine transformation calculations for each partial region based on the affine transformation parameters. Once the affine transformation calculations are performed, the partial regions including the ROI are concentrated in the upper left of the picture (see example (B) in FIG. 11 ).

[0133] The RTG processing unit 1010 determines affine transformation parameters for partial regions where both the horizontal and vertical magnifications are non-zero. The scaling process in the affine transformation is achieved by interpolating pixels at sub-pel positions using, for example, bilinear interpolation. Furthermore, to align the size of frames in a sequence, pixels may be padded at the right and bottom edges of the post-affine transformation frame to achieve the desired size. This padding process is achieved, for example, by copying the right and bottom pixels.

[0134] In this embodiment, the video encoder 104 does not transmit encoded data for the picture shown in example (A), but transmits encoded data for a set of partial regions (four partial regions in example (B)) shown in example (B). That is, the video encoder 104 transmits encoded data for partial regions whose horizontal and vertical scaling factors are not 0, but does not transmit encoded data for partial regions whose horizontal or vertical scaling factor is 0.

[0135] The RTG processing unit 1010 determines pixel values ​​used when the video decoder 204 restores each partial region (partial region with a magnification factor of 0) that is not to be transmitted in the encoded data. As an example, for each partial region with a horizontal or vertical magnification factor of 0, the RTG processing unit 1010 determines a value calculated from the values ​​of pixels adjacent to the partial region (e.g., the values ​​of pixels surrounding a partial region with a width of 1 pixel) as the pixel value used when the video decoder 204 restores the partial region. Specifically, if the magnification factor of the i-th partial region in the horizontal direction and the j-th partial region in the vertical direction is 0, the pixels whose x coordinates are located at positions Ph'[i]-1 to P'h[i] and whose y coordinates are located at positions Pv'[j]-1 to Pv'[j] in the frame after forward transformation correspond to adjacent pixels after forward transformation. The RTG processing unit 1010 calculates the average value of adjacent pixels for all partial regions with a magnification factor of 0. The pixel values ​​used when the video decoder 204 restores each partial region having a horizontal or vertical magnification of 0 are not limited to the average value of the values ​​of the pixels adjacent to the partial region having a horizontal or vertical magnification of 0, but may be other statistical values ​​or may be fixed values. Furthermore, the RTG processing unit 1010 may determine pixel values ​​that are common to multiple frames.

[0136] The encoded data of the partial region with a scaling factor of 0 is not transmitted to the video decoder 204, but the pixel values ​​used to reconstruct the partial region in the video decoder 204 are transmitted to the video decoder 204 as side information, which serves as auxiliary information. The pixel values ​​can be specified for each color component, and these values ​​may be included directly in the side information. Alternatively, the video encoder 104 may assign an index to each value or combination of values ​​and include the index in the side information. Furthermore, the video encoder 104 may include a flag indicating whether pixel value information is included in the side information, and include the pixel value or the value indicated by the index of the pixel value only when the flag is 1.

[0137] [Description of Decoding Side] In the video decoder 204, the configurations and functions of the decoder 2004 and demultiplexer 2005 are the same as those in the first embodiment.

[0138] The RTG processing unit 2010 is configured, for example, as shown in Fig. 11. The RTG processing unit 2010 shown in Fig. 11 includes an ROI information acquisition unit 8231, a partial region restoration unit 8232, a parameter acquisition unit 8233, and a conversion unit 8234.

[0139] The ROI information acquisition unit 8231 acquires the position information of the ROI, the reduction ratio (magnification factor) of the ROI, and the frame sizes before and after the forward transform from the side information obtained by demultiplexing. In this embodiment, the side information also includes pixel values ​​used to reconstruct each partial region that was not transmitted from the video encoder 104. The ROI information acquisition unit 8231 also acquires the pixel values ​​used to reconstruct each partial region.

[0140] The partial region restoration unit 8232 acquires the division points before forward transformation (division points after inverse transformation), the magnification between division points, and the division points after forward transformation (division points before inverse transformation) from the information acquired from the side information in a procedure similar to that of the partial region definition unit of the RTG processing unit 1010. Referring to the example shown in Fig. 11, the partial region restoration unit 8232 restores the division points before and after forward transformation shown in example (A).

[0141] The parameter acquisition unit 8233 determines the parameters of the affine transformation from the information restored by the partial region restoration unit 8232. The parameters to be determined are the amount of movement (translation amount) and the magnification rate. The magnification rate is determined by the reciprocal of the magnification rate between the division points. If the magnification rate is 0, the magnification rate is set to 0.

[0142] The transformation unit 8234 includes a pixel copying unit, a padding unit, and a region enlargement unit. The pixel copying unit sets pixel values ​​of the decoded picture (pixel values ​​of the partial region) as pixel values ​​of the partial region with a magnification of 1.0. Then, based on the position information of the ROI, the pixel copying unit returns the position of the ROI to its original position (the position of the original coordinates before the affine transformation was performed in the video encoder 104) by affine transformation. Note that the pixel values ​​of the partial region are the pixel values ​​of each pixel within the partial region. In other words, setting a pixel value in a partial region means filling the partial region with pixels having that pixel value, or in other words, setting pixels having that pixel value in the partial region.

[0143] The padding unit sets pixel values ​​included in the side information and used when restoring the partial region to the partial region with a magnification of 0. In other words, the conversion unit 8234 pads pixel values ​​supplied from the video encoder 104 to the partial region for which no encoded data was transmitted from the video encoder 104. Furthermore, when a flag indicating whether pixel value information is included is included, the padding unit pads with a predetermined value (e.g., a value indicating gray) if the flag is 0, and pads with a value included in the side information if the flag is 1.

[0144] The area enlargement unit sets the pixel values ​​of the decoded picture (pixel values ​​of the partial area) as the pixel values ​​of the partial area whose magnification is not 0 or 1.0, and further enlarges the partial area by affine transformation based on the magnification, and returns the position of the partial area whose magnification is not 0 or 1.0 to its original position (position of the original coordinates).

[0145] In this way, the picture is restored and a frame based on the restored picture is output.

[0146] In this embodiment, the video encoder 104 does not entirely rely on the ROI processing (particularly the padding processing) to the video decoder 204, but can control the padding processing of pixel values ​​of regions of non-interest in the video decoder 204. Because the video decoder 204 can control the padding processing of pixel values ​​of regions of non-interest, the possibility of monitor burn-in can be reduced.

[0147] In addition, the pixel values ​​to be used when padding processing is performed in the video decoder 204 are notified from the video encoder 104 to the video decoder 204, thereby reducing the processing load on the video decoder 204 related to padding processing.

[0148] [Variation 1] The above-described video encoders 101, 102, and 103 internally generate auxiliary information bitstreams including control information related to replacement processing of regions of no interest in the video decoders 201, 202, and 203, and multiplex the auxiliary information bitstreams onto the video bitstream. Furthermore, the video encoder 104 internally generates auxiliary information bitstreams including control information (side information) related to padding processing of regions of no interest in the video decoder 204, and multiplexes the auxiliary information bitstreams onto the video bitstream. However, the video encoders 101, 102, 103, and 104 may be configured to supply information necessary for the processing to an external device connected to the video encoders 101, 102, 103, and 104, and the external device may perform the auxiliary information bitstream generation and multiplexing processing. Even with such a configuration, the effects of the above-described embodiments can be achieved.

[0149] Alternatively, information required for processing may be supplied to an external device connected to the video decoders 201, 202, 203, and 204, and the external device may decode the auxiliary information bitstream and replace the region of no interest. Even in such a configuration, the effects of the above-described embodiments can be obtained.

[0150] [Variation 2] In each of the above embodiments, the video encoders 101, 102, 103, and 104 and the video decoders 201, 202, 203, and 204 perform replacement or padding on a pixel-by-pixel basis. However, the video encoders 101, 102, 103, and 104 and the video decoders 201, 202, 203, and 204 can also perform replacement or padding on a block-by-block basis, which is a set of pixel values. In this case, information on the set of pixel values, rather than the pixel values, is included in the auxiliary information bitstream.

[0151] [Modification 3] Furthermore, the video encoders 101, 102, 103, and 104 can internally generate information related to the replacement process for the region of non-interest, rather than receiving the information via control information input from the outside.

[0152] [Variation 4] In the fourth embodiment, the video encoder 104 replaces a specific region that is a non-ROI with a predetermined pixel value in the RTG processing unit 1010. As described above, unlike the other embodiments, in the fourth embodiment, pixel values ​​of the non-ROI are not replaced, but similar to the first embodiment, the video decoder 204 may replace pixels of the specific region with pixel values ​​that pad the deleted region included in the bitstream after processing in the RTG processing unit 2010 is completed.

[0153] Each of the above embodiments can be configured by hardware, but can also be realized by a computer program.

[0154] The information processing system shown in Fig. 14 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing a bitstream. It is also possible to use multiple processors. The storage medium 703 and the storage medium 704 may be separate storage media or may be storage areas of the same storage medium. A magnetic storage medium such as a hard disk may be used as the storage medium.

[0155] In the information processing system, a program memory 702 stores a program (video encoding program or video decoding program) for implementing the functions of each block shown in each of the above embodiments.

[0156] The processor 701 then executes processing in accordance with the program stored in the program memory 702, thereby realizing the functions of the video encoders 101, 102, 103, and 104 and the video decoders 201, 202, 203, and 204 shown in each embodiment.

[0157] For example, the processor 701 executes processing in accordance with a video encoding program for realizing the functions of each block in the video encoders 101, 102, 103, and 104 shown in Figures 1, 4, 7, and 10, thereby realizing the functions of the video encoders 101, 102, 103, and 104.

[0158] Furthermore, for example, the processor 701 executes processing in accordance with a video decoding program for realizing the functions of each block in the video decoders 201, 202, 203, and 204 shown in Figures 1, 4, 7, and 10, thereby realizing the functions of the video decoders 201, 202, 203, and 204.

[0159] At least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transitory computer-readable medium. The program is supplied to the transitory computer-readable medium, for example, via a wired or wireless communication channel, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0160] 15 is a block diagram showing the main components of a video encoding device. The video encoding device 10 shown in Fig. 15 (implemented by video encoders 101, 102, 103, and 104 in the embodiment) includes region-of-interest detection means 11 (implemented by region-of-interest detection units 1001 and 1001A in the embodiment) for detecting a region of interest from an input video frame, video encoding means 12 (implemented by encoder 1003 in the embodiment) for generating a video bitstream, and decoder control means 13 (implemented by decoder processing control units 1005, 1005A, and 1005B in the embodiment) for generating a side information bitstream including information for controlling a process for setting pixel values ​​of regions of non-interest in the video decoding device (for example, a pixel value replacement process or a pixel value padding process, i.e., adding pixel values).

[0161] 16 is a block diagram showing the main components of a video decoding device. The video decoding device 20 shown in Fig. 16 (implemented by video decoders 201, 202, 203, and 204 in the embodiment) includes video decoding means 22 (implemented by the decoder 2004 in the embodiment) that decodes a video bitstream to obtain decoded video frames, region of interest identification means 23 (implemented by region of interest identification units 2003, 2003A, and 2003B and the RTG processing unit 2010 in the embodiment) that identifies regions of interest in the decoded video frames, and region of interest processing control means 24 (implemented by region of interest replacement processing control units 2002, 2002A, and 2002B and the RTG processing unit 2010 in the embodiment) that decodes a side information bitstream containing information for controlling a process for setting pixel values ​​of regions of interest to obtain information for controlling the setting process.

[0162] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0163] (Supplementary Note 1) A video encoding device comprising: a region of interest detection means for detecting a region of interest from an input video frame; a video encoding means for generating a video bitstream; a decoder control means for generating an auxiliary information bitstream including information for controlling a process for setting pixel values ​​of regions of non-interest in a video decoding device; and a multiplexing means for multiplexing the video bitstream and the auxiliary information bitstream.

[0164] (Supplementary Note 2) The video encoding device of Supplementary Note 1, wherein the information for controlling the setting process includes information indicating whether the video decoding device should execute the setting process and pixel values ​​to be set in the region of non-interest by the video decoding device.

[0165] (Supplementary Note 3) The video encoding device according to Supplementary Note 1 or Supplementary Note 2, wherein the decoder control means includes information capable of identifying the detected region of interest in the auxiliary information bitstream.

[0166] (Supplementary Note 4) The video encoding device of Supplementary Note 1 or Supplementary Note 2, further comprising: a region of non-interest replacement means for replacing pixel values ​​of a region of non-interest in an input video frame with predetermined pixel values, wherein the decoder control means includes the predetermined pixel values ​​in the auxiliary information bitstream.

[0167] (Supplementary Note 5) The video encoding device of Supplementary Note 1 or Supplementary Note 2, further comprising: a deletion means for deleting the region of non-interest in the input video frame; and the decoder control means including pixel values ​​set for the deleted region of non-interest in the auxiliary information bitstream.

[0168] (Supplementary Note 6) The video encoding device according to any one of Supplementary Notes 1 to 5, further comprising a multiplexing means (implemented by a multiplexer 1004 in the embodiment) for multiplexing the video bitstream and the auxiliary information bitstream.

[0169] (Supplementary Note 7) The video encoding device of Supplementary Note 6, further comprising a transmitting means (implemented by a multiplexer 1004 in the embodiment) for transmitting the video bitstream and the auxiliary information bitstream.

[0170] (Supplementary Note 8) The video encoding device according to Supplementary Note 7, wherein the transmitting means transmits the bit stream multiplexed by the multiplexing means.

[0171] (Supplementary Note 9) A video decoding device comprising: video decoding means for decoding the video bitstream to obtain a decoded video frame; region of non-interest identification means for identifying the region of non-interest in the decoded video frame; and region of non-interest replacement process control means for decoding an auxiliary information bitstream including information for controlling a process of setting pixel values ​​of the region of non-interest to obtain information for controlling the setting process.

[0172] (Supplementary Note 10) The video decoding device of Supplementary Note 9, wherein the information for controlling the setting process includes information indicating whether the video decoding device should perform the replacement process and pixel values ​​to replace pixel values ​​of the region of non-interest, and further comprising a replacement process execution means for executing the replacement process, wherein the replacement process execution means executes the replacement process when the information indicating whether the replacement process should be performed indicates that the replacement process should be performed.

[0173] (Supplementary Note 11) The video decoding device of Supplementary Note 9, wherein the information for controlling the setting process includes pixel values ​​to be set in the deleted region of non-interest, and further comprising padding means for padding the deleted region of non-interest with pixel values ​​included in the information for controlling the setting process.

[0174] (Supplementary Note 12) The video decoding device according to Supplementary Note 9 or Supplementary Note 10, wherein the non-interest region identifying means identifies a region of a predetermined pixel value as the non-interest region.

[0175] (Supplementary Note 13) The video decoding device according to Supplementary Note 9 or Supplementary Note 10, wherein the region of no interest identifying means identifies the region of no interest based on information capable of identifying the region of interest included in the auxiliary information bitstream.

[0176] (Supplementary Note 14) The video decoding device according to Supplementary Note 9 or Supplementary Note 10, wherein the region of non-interest identifying means identifies the region of non-interest based on pixel values ​​of the region of non-interest included in the side information bitstream.

[0177] (Supplementary Note 15) A video decoding device according to any one of Supplementary Note 9 to Supplementary Note 14, comprising a demultiplexing means (in an embodiment, realized by a demultiplexer 2005) that demultiplexes an input bitstream to obtain the video bitstream and the auxiliary information bitstream.

[0178] (Supplementary Note 16) A video encoding method comprising: detecting a region of interest from an input video frame; generating a video bitstream; and generating an auxiliary information bitstream including information for controlling a process of setting pixel values ​​of regions of non-interest in a video decoding device.

[0179] (Supplementary Note 17) A video decoding method comprising: decoding a video bitstream to obtain a decoded video frame; identifying a region of non-interest in the decoded video frame; and decoding an auxiliary information bitstream including information for controlling a process of setting pixel values ​​of the region of non-interest to obtain information for controlling the setting process.

[0180] (Supplementary Note 18) A video encoding program for causing a computer to execute the following processes: detecting a region of interest from an input video frame; generating a video bitstream; and generating an auxiliary information bitstream including information for controlling the process of setting pixel values ​​of regions of non-interest in a video decoding device.

[0181] (Supplementary Note 19) A video decoding program for causing a computer to execute the following processes: decoding a video bitstream to obtain a decoded video frame; identifying a region of non-interest in the decoded video frame; and decoding an auxiliary information bitstream containing information for controlling a process of setting pixel values ​​of the region of non-interest to obtain information for controlling the setting process.

[0182] (Supplementary Note 20) A storage medium for storing a bitstream generated by a video encoding device comprising: a region of interest detection means for detecting a region of interest from an input video frame; a video encoding means for generating a video bitstream; and a decoder control means for generating an auxiliary information bitstream including information for controlling a process for setting pixel values ​​of regions of non-interest in the video decoding device.

[0183] (Supplementary Note 21) The video encoding method of Supplementary Note 16, wherein the information for controlling the substitution process includes information indicating whether the video decoding device should perform the substitution process or not, and pixel values ​​that are to replace pixel values ​​of the region of non-interest in the video decoding device.

[0184] (Supplementary Note 22) The video coding method of Supplementary Note 16 or Supplementary Note 21, wherein information capable of identifying the detected region of interest is included in the auxiliary information bitstream.

[0185] (Supplementary Note 23) The video encoding method of Supplementary Note 16 or Supplementary Note 21, further comprising: replacing pixel values ​​of a region of non-interest in the input video frame with predetermined pixel values; and including the predetermined pixel values ​​in the side information bitstream.

[0186] (Supplementary Note 24) The video encoding method of Supplementary Note 16 or Supplementary Note 21, further comprising: removing the region of non-interest in the input video frame; and including pixel values ​​set for the removed region of non-interest in the side information bitstream.

[0187] (Supplementary Note 25) The video coding method of Supplementary Note 16 or any one of Supplementary Notes 21 to 24, wherein the video bitstream and the auxiliary information bitstream are multiplexed.

[0188] (Supplementary Note 26) The video decoding method of Supplementary Note 17, wherein the information for controlling the replacement process includes information indicating whether the video decoding device should perform the replacement process or not and pixel values ​​that replace pixel values ​​of the region of non-interest, and the replacement process is performed when the information indicating whether the replacement process or not indicates that the replacement process should be performed.

[0189] (Supplementary Note 27) The video decoding method of Supplementary Note 17, wherein the information for controlling the setting process includes pixel values ​​to be set in the deleted region of non-interest, and the deleted region of non-interest is padded with pixel values ​​included in the information for controlling the setting process.

[0190] (Supplementary Note 28) The video decoding method of Supplementary Note 17, further comprising identifying a region of predetermined pixel values ​​as the region of non-interest.

[0191] (Supplementary Note 29) The video decoding method of Supplementary Note 17 or Supplementary Note 26, wherein the region of non-interest is identified based on pixel values ​​of the region of non-interest included in the side information bitstream.

[0192] (Supplementary Note 30) The video decoding method of Supplementary Note 17 or any one of Supplementary Notes 26 to 29, wherein the input bitstream is demultiplexed to obtain the video bitstream and the side information bitstream.

[0193] (Supplementary Note 31) A video encoding program that causes a computer to execute the video encoding method described in any one of Supplementary Notes 16, 21-25.

[0194] (Supplementary Note 32) A video decoding program that causes a computer to execute the video decoding method described in any one of Supplementary Notes 17, 26-30.

[0195] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0196] This application claims priority based on Japanese Patent Application No. 2023-172565, filed on October 4, 2023, the disclosure of which is incorporated herein in its entirety.

[0197] REFERENCE SIGNS LIST 10 Video encoding device 11 Region of interest detection means 12 Video encoding means 13 Decoder control means 20 Video decoding device 22 Video decoding means 23 Region of non-interest identification means 24 Region of non-interest processing control means 101, 102, 103, 104 Video encoder 201, 202, 203, 204 Video decoder 701 Processor 702 Program memory 703, 704 Storage medium 1001, 1001A Region of interest detection unit 1002, 1002B Region of non-interest replacement processing unit 1003 Encoder 1004 Multiplexer 1005, 1005A, 1005B Decoder processing control unit 1006 Encoder processing control unit 1010 RTG processing unit 2001 Region of non-interest replacement processing unit 2002, 2002A, 2002B Region of non-interest replacement processing control unit 2003, 2003A, 2003B Region of non-interest identification unit 2004 Decoder 2005 Demultiplexer 2010 RTG processing unit 8231 ROI information acquisition unit 8232 Partial region restoration unit 8233 Parameter acquisition unit 8234 Conversion unit

Claims

1. A video encoding device comprising: a region of interest detection means for detecting a region of interest from an input video frame; a video encoding means for generating a video bitstream; and a decoder control means for generating an auxiliary information bitstream including information for controlling a process of setting pixel values ​​of regions of non-interest in the video decoding device.

2. The video encoding device of claim 1, wherein the information for controlling the setting process includes information indicating whether the video decoding device should execute the setting process and pixel values ​​to be set in the region of non-interest by the video decoding device.

3. A video encoding device according to claim 1 or 2, wherein said decoder control means includes information capable of identifying the detected region of interest in said auxiliary information bitstream.

4. A video encoding device as claimed in claim 1 or claim 2, further comprising a region of non-interest replacement means for replacing pixel values ​​of the region of non-interest in an input video frame with a predetermined pixel value, wherein the decoder control means includes the predetermined pixel value in the auxiliary information bitstream.

5. A video encoding device as described in claim 1 or claim 2, further comprising a deletion means for deleting the non-interested region in the input video frame, wherein the decoder control means includes pixel values ​​set in the deleted non-interested region in the auxiliary information bitstream.

6. The video encoding device according to claim 1, further comprising a multiplexing means for multiplexing said video bit stream and said auxiliary information bit stream.

7. The video encoding device according to claim 6, further comprising a transmitting means for transmitting said video bitstream and said auxiliary information bitstream.

8. The video encoding device according to claim 7, wherein said transmitting means transmits the bit stream multiplexed by said multiplexing means.

9. A video decoding device comprising: a video decoding means for decoding a video bitstream to obtain a decoded video frame; a region of non-interest identifying means for identifying a region of non-interest in the decoded video frame; and a region of non-interest processing control means for decoding an auxiliary information bitstream including information for controlling a setting process of pixel values ​​of the region of non-interest to obtain information for controlling the setting process.

10. A video decoding device as described in claim 9, wherein the information for controlling the setting process includes information indicating whether the video decoding device should perform a replacement process to replace pixel values ​​of the region of non-interest with a predetermined pixel value and a pixel value with which to replace the pixel values ​​of the region of non-interest, and further comprising a replacement process execution means for executing the replacement process, wherein the replacement process execution means executes the replacement process when the information indicating whether the replacement process should be executed indicates that the replacement process should be executed.

11. A video decoding device as described in claim 9, wherein the information for controlling the setting process includes pixel values ​​to be set in the deleted region of non-interest, and the device further comprises a padding means for padding the deleted region of non-interest with pixel values ​​included in the information for controlling the setting process.

12. A video decoding device according to claim 9 or 10, wherein said non-interest region identifying means identifies a region having a predetermined pixel value as said non-interest region.

13. A video decoding device according to claim 9 or 10, wherein said non-interest identifying means identifies said non-interest region based on information capable of identifying said region of interest contained in said auxiliary information bitstream.

14. A video decoding device according to claim 9 or 10, wherein said region of non-interest identifying means identifies said region of non-interest based on pixel values ​​of said region of non-interest contained in said auxiliary information bitstream.

15. The video decoding device according to claim 9, further comprising demultiplexing means for demultiplexing an input bitstream to obtain the video bitstream and the auxiliary information bitstream.

16. A video encoding method comprising: detecting a region of interest from an input video frame; generating a video bitstream; and generating an auxiliary information bitstream including information for controlling a process of setting pixel values ​​of regions of non-interest in a video decoding device.

17. A video decoding method, comprising: decoding a video bitstream to obtain a decoded video frame; identifying a region of non-interest in the decoded video frame; and decoding an auxiliary information bitstream including information for controlling a process of setting pixel values ​​of the region of non-interest to obtain information for controlling the setting process.

18. A computer-readable recording medium storing a video encoding program for causing a computer to execute the following processes: detecting a region of interest from an input video frame; generating a video bitstream; and generating an auxiliary information bitstream including information for controlling the process of setting pixel values ​​of regions of non-interest in a video decoding device.

19. A computer-readable recording medium storing a video decoding program for causing a computer to execute the following processes: decoding a video bitstream to obtain a decoded video frame; identifying an area of ​​non-interest in the decoded video frame; and decoding an auxiliary information bitstream including information for controlling a process of setting pixel values ​​of the area of ​​non-interest to obtain information for controlling the setting process.

20. A storage medium for storing a bitstream generated by a video encoding device comprising a region of interest detection means for detecting a region of interest from an input video frame, a video encoding means for generating a video bitstream, and a decoder control means for generating an auxiliary information bitstream including information for controlling the process of setting pixel values ​​of regions of non-interest in the video decoding device.

Images