Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

A video encoding device 10 according to the present invention includes: an interest area detection means 11 for detecting an interest area from input video frames; a non-interest area replacement means 12 for replacing a pixel value of a non-interest area in the input video frames with a predetermined pixel value; a block division means 13 for dividing, into one or more blocks, the frames in which the pixel value of the non-interest area has been replaced; a code string generation means 14 for encoding a processing target block to generate a code string; a local decoding means 15 for decoding the code string; and a restoration means 16 for restoring the pixel value of the non-interest area on the basis of information capable of specifying the interest area. The block division means 13 includes a processing target block selection means 131 for selecting a processing target block on the basis of information capable of specifying the interest area.

Description

Video encoding device, video decoding device, video encoding method, and video decoding method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to transmit or record video efficiently, a video encoding device is used to generate a coded representation (hereinafter referred to as a bitstream) of input video, and a video decoding device is used to decode the bitstream to generate decoded video.

[0003] Video coding standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), and H.266 / VVC (Versatile Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. Another recent video coding technology is the technology described in Non-Patent Document 1.

[0004] In these video encoding methods, lossy processing is generally performed in the video encoding device. The use of lossy processing reduces the amount of data in the generated bitstream. However, the quality of the video decoded by the video decoding device (decoded video) deteriorates compared to the quality of the video input to the video encoding device (input video). In other words, there is a trade-off between the amount of data in the bitstream and the degree of deterioration in the quality of the decoded video.

[0005] In addition to the video coding methods mentioned above, various methods have been developed for human viewing. Generally, the performance of these methods is evaluated in terms of the degree to which the quality of the decoded video degrades compared to the quality of the input video, when the decoded video is decoded from bitstreams of similar data amounts generated according to the processing of each method.

[0006] In recent years, video is increasingly used not only for human viewing but also for machine processing, such as object detection, which detects people and other objects in a video.

[0007] Development of video coding methods that are intended for processing by machines is underway. For example, ISO / IEC / SC29 is working on standardizing a video coding method for machines called Video for Coding Machines (VCM), which is described in Non-Patent Document 2.

[0008] The performance of a video coding method for machines is generally evaluated by the performance of the processing performed by the machine when using video decoded from a bitstream with a similar data volume. Therefore, even if the quality of the decoded video is significantly degraded compared to the quality of the input video, the performance of the coding method is considered good as long as the degree of degradation in the performance of the processing performed by the machine when using the decoded video is small compared to the performance when using the input video. In other words, the difference in quality between the input video and the decoded video is not considered an issue. Therefore, video coding devices that perform video coding for machines may apply pre-processing that has not been used in methods intended for human viewing in order to improve the performance of the coding method.

[0009] As a pre-processing for improving the performance of the encoding method, for example, there is a process of replacing pixel values ​​in regions other than the region of interest of the input video, i.e., regions of non-interest, with arbitrary pixel values. Non-Patent Document 3 discloses an example of pre-processing for machine-oriented video encoding, including such a process.

[0010] In addition, in a method that can refer to the H.274 / VSEI standard, it is possible to transmit the coordinates of a region of interest detected in a video frame as auxiliary information called annotated_regions, which is defined in the H.274 / VSEI standard.

[0011] A bitstream is generated by applying an arbitrary encoding method to the pre-processed video frames. The encoding method may be a general video encoding method without machine intervention, such as H.265 / HEVC or H.266 / VVC, or may be a method using deep learning or a method combining multiple methods.

[0012] In video coding methods such as H.266 / VVC, a video frame is divided into blocks, and a prediction signal is generated for each block to be processed. A prediction error signal is then generated from the difference between the block to be processed and a reference block formed from the prediction signal. After frequency transformation and quantization are applied to the prediction error signal, the quantized transform coefficients are entropy coded.

[0013] In the process of generating a prediction signal, a region within the same frame as the frame to be processed or a region of a frame at a different time is referenced, and a prediction signal is generated based on the pixel values ​​of the referenced block. That is, a coding method using a general video coding system codes the difference between the pixel values ​​of a reference block formed from the prediction signal and the pixel values ​​of the block to be processed. If this difference is small, the amount of code in the bitstream will be small.

[0014] In a method using deep learning, for example, a bitstream is generated by inputting video frames into a convolutional neural network (CNN) and entropy-encoding the output values. Non-Patent Document 4 discloses an example of such a method.

[0015] "Algorithm description of Enhanced Compression Model 9 (ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023"CfP response report for Video Coding for Machines", ISO / IEC JTC 1 / SC 29 / WG 2 N248, ISO / IEC JTC 1 / SC 29 / WG 2, October 2022"Optimization of encoders and receiving systems for machine analysis of coded video content (draft 2)", JVET-AD2030, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023N. Le, H. Zhang, F. Cricri, R. Ghaznavi-Youvalari and E. Rahtu, "Image Coding For Machines: An End-To-End Learned Approach", in proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1590-1594, 2021

[0016] When a substitution process is applied to pixel values ​​of a region of non-interest, the amount of code in the bitstream is reduced compared to the encoding result when the substitution process is not applied. However, when the substitution process is not taken into consideration in the video encoding method, a bitstream containing redundant code sequences may be transmitted. Note that the substitution process is a process of replacing pixel values ​​of a region of non-interest with predetermined pixel values.

[0017] The redundant code string is, for example, a code string of pixel values ​​of a region of non-interest and information necessary for decoding the pixel values. If the pixel values ​​that replace the region of non-interest are shared between the encoder and the decoder, the region can be restored by the decoder if the coordinates of the region of non-interest are known. Therefore, transmitting pixel values ​​corresponding to the region of non-interest and information necessary for their restoration can be redundant. Furthermore, even if the region of non-interest is coded, coding degradation may result in the restored value being different from the pixel value replaced during coding when decoded. If a prediction signal is generated from pixel values ​​that have suffered such coding degradation, the prediction accuracy may decrease, increasing the value of the prediction error signal to be coded, which may result in a loss of coding amount.

[0018] Furthermore, when the current block is a region of interest and predictive coding is performed using a reference block that includes both the region of interest and a region of non-interest, the generated code sequence may include an extra code sequence, because even if the difference between the pixel values ​​of the region of interest in the reference block and the pixel values ​​of the corresponding region in the current block is small, the difference between the pixel values ​​of the corresponding region of non-interest in the reference block and the current block may be large, resulting in an increase in the amount of code in the generated bitstream.

[0019] The present invention aims to provide a video encoding device, a video decoding device, a video encoding method, and a video decoding method that can suppress an increase in the amount of transmission of code sequences related to regions of non-interest in input video frames.

[0020] A video encoding device based on the present disclosure includes a region of interest detection means for detecting a region of interest from an input video frame, a region of non-interest replacement means for replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values, a block division means for dividing the frame after the pixel values ​​of the regions of non-interest have been replaced into one or more blocks, a code string generation means for encoding the block to be processed to generate a code string, a local decoding means for decoding the code string, and a restoration means for restoring the pixel values ​​of the regions of non-interest based on information that can identify the region of interest, and the block division means includes a block to be processed selection means for selecting the block to be processed based on the information that can identify the region of interest.

[0021] A video decoding device based on the present disclosure includes a region of interest identification means for acquiring information capable of identifying a region of interest from a bitstream in which information capable of identifying a region of interest in a video frame is added to a code sequence, a video decoding means for decoding the code sequence to acquire pixel values ​​of the region of interest, and a restoration means for restoring pixel values ​​of regions of non-interest based on the information capable of identifying the region of interest.

[0022] A video encoding method according to the present disclosure replaces pixel values ​​of regions of non-interest in an input video frame with predetermined pixel values, divides the frame in which the pixel values ​​of the regions of non-interest have been replaced into one or more blocks, and when dividing the blocks, selects a block to be processed based on information that can identify the region of interest, encodes the block to be processed to generate a code string, decodes the code string, and restores the pixel values ​​of the regions of non-interest based on the information that can identify the region of interest.

[0023] A video decoding method based on the present disclosure obtains information capable of identifying a region of interest from a bitstream in which information capable of identifying a region of interest in a video frame is added to a code string, decodes the code string to obtain pixel values ​​of the region of interest, and restores pixel values ​​of regions of non-interest based on the information capable of identifying the region of interest.

[0024] A video encoding program based on the present disclosure causes a computer to perform the following processes: detecting a region of interest from an input video frame; replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values; dividing the frame after the pixel values ​​of the regions of non-interest have been replaced into one or more blocks; selecting a block to be processed based on information that can identify the region of interest when dividing the blocks; encoding the block to be processed to generate a code string; decoding the code string; and restoring the pixel values ​​of the regions of non-interest based on the information that can identify the region of interest.

[0025] A video decoding program based on the present disclosure causes a computer to perform the following processes: obtaining information that can identify a region of interest from a bit stream in which information that can identify a region of interest in a video frame is added to a code sequence; decoding the code sequence to obtain pixel values ​​of the region of interest; and restoring pixel values ​​of regions of non-interest based on the information that can identify the region of interest.

[0026] According to the present invention, it is possible to suppress an increase in the amount of transmission of code strings related to regions of non-interest in a video frame.

[0027] 1 is a block diagram showing an embodiment of a video encoder and a video decoder of the present disclosure. FIG. 2 is a flowchart showing the operation of the video encoder of the first embodiment. FIG. 3 is a block diagram showing an embodiment of a video encoder and a video decoder of the present disclosure. FIG. 4 is a flowchart showing the operation of the video encoder of the second embodiment. FIG. 5 is an explanatory diagram for explaining a correction process. FIG. 6 is a flowchart showing a correction process of a reference block in the video decoder of the second embodiment. FIG. 7 is a block diagram showing an example configuration of an information processing system. FIG. 8 is a block diagram showing main parts of a video encoder. FIG. 9 is a block diagram showing main parts of a video decoder.

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0029] Embodiment 1. Figure 1 is a block diagram illustrating an embodiment of a video encoder and a video decoder.

[0030] The video encoder 101 shown in Fig. 1 is an encoder having a function of performing pre-processing. In this embodiment and the following embodiments, the pre-processing is a process of replacing pixel values ​​of regions of non-interest in a video frame. The video encoder 101 includes a region-of-interest detection unit 1001, a region-of-non-interest replacement processing unit 1002, a block division unit 1003, a code string generator 1004, a bitstream generator 1005, a local decoder 1006, a filter application unit 1007, a frame buffer 1008, and a correction processing unit 1009. The block division unit 1003 includes a block selection unit 1003A that selects a block to be processed.

[0031] The video decoder 201 is a decoder having a function of executing a predetermined correction process when a video frame includes a region of non-interest, and includes a code string decoder 2001, a bit stream decoder 2002, a filter application unit 2003, a frame buffer 2004, a correction processing unit 2006, and a region of interest identification unit 2007.

[0032] The video encoder 101 and the video decoder 201 may use any video encoding method as long as it performs predictive encoding by referring to previously encoded frames. However, a common video encoding method is used by the video encoder and the video decoder. For example, a predictive encoding method such as H.266 / VVC may be used. Furthermore, a method using deep learning may also be used.

[0033] 1 simply indicates the direction of signal (data) flow, but does not exclude bidirectionality. This also applies to other block diagrams.

[0034] [Description on the Encoding Side] In the video encoder 101, the region of interest detection unit 1001 extracts frame features from an input frame (input video frame). The region of interest detection unit 1001 detects a region to be determined as a region of interest based on the extracted features. The region of interest detection unit 1001 supplies information capable of identifying the region of interest to the region of non-interest replacement processing unit 1002 along with the input frame. The region of interest detection unit 1001 also supplies the information capable of identifying the region of interest to the block division unit 1003. In this embodiment, the region of interest detection unit 1001 uses a set of coordinate information of the region of interest (a group of coordinates) as information capable of identifying the region of interest. The coordinate information of the region of interest may be the coordinates of two pixels (top left and bottom right), or may be the coordinates of four pixels (top left, bottom right, top right, and bottom left). Note that the region of interest detection unit 1001 may output information capable of identifying a region of non-interest (e.g., coordinates of the region of non-interest) instead of the coordinates of the region of interest. Hereinafter, the set of coordinate information supplied to the region of interest replacement processor 1002 will be simply referred to as coordinate information. Furthermore, the region of interest detector 1001 supplies the coordinates of the region of interest to the block divider 1003, the bitstream generator 1005, and the frame buffer 1008. Note that detecting a region to be determined as a region of interest essentially means detecting a region of non-interest.

[0035] The number of regions of interest detected by the region of interest detection unit 1001 may be plural or zero.

[0036] One method for detecting a region of interest is to use a neural network trained to detect a specific object. In this case, the neural network receives, for example, pixel values ​​of an input video frame and outputs the coordinates of a region determined to contain an object. The output coordinates are designated as the coordinates of the region of interest.

[0037] The region of interest for each frame may be calculated from only the processing result of the frame, or may be calculated based on the processing results of multiple frames. In the case where the region of interest is calculated based on the processing results of multiple frames, the region of interest detection unit 1001 buffers the frames until a predetermined number of frames have been input to the video encoder.

[0038] The region of non-interest replacement processing unit 1002 replaces pixel values ​​of regions other than the region of interest in the frame to be processed, i.e., the region of non-interest, with predetermined pixel values ​​based on the frame (the input frame to the video encoder 101) supplied from the region of interest detection unit 1001 and information about the region of interest (e.g., coordinate information). In this embodiment, the predetermined pixel values ​​are assumed to be values ​​agreed upon in advance with the video decoder.

[0039] The block division unit 1003 divides the frame supplied from the region of no interest replacement processing unit 1002 into one or more rectangular regions (blocks) having a size usable in the video encoding method to obtain blocks to be processed. That is, the video encoder 101 may treat one frame as one block or as a set of multiple blocks.

[0040] In the block division unit 1003, the block selection unit 1003A selects a target block for generating a code string, which is uniquely identified based on the coordinate information of the region of interest. First, the block selection unit 1003A scans the coordinates of all regions of interest detected by the region of interest detection unit 1001 to identify the regions of interest located at the leftmost, topmost, rightmost, and bottommost positions within the video frame, and defines a rectangular region that includes all of them. Any block that is even partially included in this rectangular region is supplied to the code string generator 1004 as the target block.

[0041] The bitstream generator 1004 performs processing according to the video encoding method to be used for each target block obtained from the block selector 1003A, and generates a bitstream for the target block. The bitstream generator 1004 references pixel values ​​of the corrected frame supplied from the frame buffer 1008 as necessary. The bitstream generator 1004 supplies the generated bitstream, together with coordinate information of the region of interest, to the bitstream generator 1005 and the local decoder 1006.

[0042] The bitstream generator 1005 performs entropy coding on the code strings supplied from the codestring generator 1004. The bitstream generator 1005 then concatenates the code strings of each block to be coded and generates a bitstream by adding coordinate information of all regions of interest and other information required for decoding, such as frame size.

[0043] The local decoder 1006 can restore the region of interest of the locally decoded frame on a block-by-block basis by decoding the code string of the block to be processed that includes the region of interest, which is supplied from the code string generator 1004 .

[0044] The filter application unit 1007 is a block having a function of applying a filter to a decoded frame according to the video encoding method used. When applying a filter, the filter application unit 1007 applies filter processing to a rectangular area made up of decoded blocks supplied from the local decoding unit 1006. The filter application unit 1007 supplies the rectangular area made up of decoded blocks after the filter has been applied to the correction processing unit 1009. If the video encoding method used does not have a function of applying a filter to a decoded frame, or if a filter is not to be applied, the filter application unit 1007 supplies the rectangular area made up of decoded blocks to the correction processing unit 1009 without applying filter processing.

[0045] Based on the coordinates of the region of interest, the correction processing unit 1009 corrects the region of non-interest in a rectangular region made up of decoded blocks supplied from the filter application unit 1007. The correction processing unit 1009 identifies pixels in the region of non-interest in the decoded block, replaces the pixel values ​​with pixel values ​​obtained by replacing the pixel values ​​of the region of non-interest by the region of non-interest replacement processing unit 1002, and obtains a corrected decoded block. This makes it possible to improve prediction accuracy in predictive coding.

[0046] Because the bitstream generator 1004 does not encode some blocks that include only the region of non-interest, the local decoder 1006 cannot restore the region of non-interest in the target block. Therefore, the correction processor 1009 first restores pixel values ​​of blocks that were not encoded, i.e., pixel values ​​of the region of non-interest (pixel values ​​with which the region of non-interest replacement processor 1002 has replaced pixel values ​​of the region of non-interest), based on the width and height of the input video frame, block division information, coordinates of the region of interest, and pixel values ​​with which the region of non-interest replacement processor 1002 has replaced pixel values ​​of the region of non-interest. The restored pixel values ​​are pixel values ​​after replacement by the region of interest detection unit 1001. The correction processor 1009 combines a block restored from a block whose encoding was skipped with the corrected decoded block to create a single frame. This makes it possible to avoid transmitting redundant information.

[0047] The correction processing unit 1009 applies the above two processes to the frame supplied from the filter application unit 1007, and the filter application unit 1007 stores the frame obtained by synthesizing the restored region of interest and region of non-interest as a corrected frame in the frame buffer 1008.

[0048] By the above-described process, even if the bitstream generator 1004 does not encode the region of non-interest due to the exclusion of blocks by the block division unit 1003, a frame in which pixel values ​​of the region of non-interest are restored is stored in the frame buffer 1008. Therefore, the bitstream generator 1004 can use the encoded frame stored in the frame buffer 1008 as, for example, a reference frame. Furthermore, by using the coordinates of the region of interest, it is possible to uniquely select the block to be processed and identify the region of non-interest. If the bitstream includes the coordinates of the region of interest, similar processing can be performed in a decoder that decodes the bitstream.

[0049] Next, the operation of the video encoder 101 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the operation of the video encoder 101.

[0050] First, the region of interest detection unit 1001 detects a region to be a region of interest from an input frame of video (step S100). The region of interest detection unit 1001 outputs information that can identify the region of interest, such as a group of coordinates of the detected region of interest, to the region of non-interest replacement processing unit 1002.

[0051] The region of non-interest replacement processor 1002 performs replacement processing (step S101). That is, the region of non-interest replacement processor 1002 identifies a region of non-interest based on information capable of identifying a region of interest from the region of interest detection unit 1001, and replaces pixel values ​​of the region corresponding to the region of non-interest with predetermined pixel values. The region of non-interest replacement processor 1002 supplies the pixel values ​​of the frame after the replacement processing to the block division unit 1003.

[0052] After the block division unit 1003 divides the frame supplied from the non-interest region replacement processing unit 1002 into one or more blocks, the block selection unit 1003A selects a block to be processed based on the coordinates of the region of interest (step S102).

[0053] The code string generator 1004 selects one unprocessed block to be processed in the frame to be processed (step S103).

[0054] The code string generator 1004 encodes the target block (step S104). The local decoder 1006 decodes the code string of the target block supplied from the code string generator 1004 (step S105). If the filter application unit 1007 applies a filter to the decoded block, the filter application unit 1007 performs the filter processing on a rectangular area formed from the blocks supplied from the local decoder 1006 (step S106). Thereafter, the correction processing unit 1009 applies correction processing to the block supplied from the filter application unit 1007 based on the coordinates of the region of interest (step S107).

[0055] After the processes of steps S103 to S107 have been performed for all target blocks input to the bitstream generator 1004 (step S108), the correction processor 1009 performs a correction process to restore pixel values ​​of regions corresponding to blocks whose coding has been skipped, i.e., regions of no interest (step S109).The bitstream generator 1005 performs a process to generate a bitstream (step S110).

[0056] [Description on the Decoding Side] In the video decoder 201, the bitstream decoder 2002 decodes a code string corresponding to each block from the bitstream and supplies the decoded code string to the codestring decoder 2001. The bitstream decoder 2002 also obtains coordinate information and block division information of the region of interest from the bitstream and supplies them to a region of interest identification unit 2007.

[0057] The region of interest identifying unit 2007 identifies a region of non-interest based on the coordinate information of the region of interest supplied from the bitstream decoder 2002 , and supplies information indicating the region of non-interest to the correction processing unit 2006 .

[0058] The code string decoder 2001 can restore a rectangular area including the region of interest in a decoded frame on a block-by-block basis by decoding the code string of the block to be processed that includes the region of interest. The code string decoder 2001 supplies information about the restored rectangular area to the filter application unit 2003.

[0059] The filter application unit 2003 is a block having a function of applying a filter to a decoded frame according to the video encoding method used. When applying a filter, the filter application unit 2003 applies filter processing to a rectangular area supplied from the codestream decoder 2001. The filter application unit 2003 supplies the rectangular area after filtering to the correction processing unit 2006. When the video encoding method used does not have a loop filter function or when a filter is not to be applied, the filter application unit 2003 supplies the rectangular area supplied from the codestream decoder 2001 to the correction processing unit 2004 without applying filter processing.

[0060] The correction processing unit 2006 first corrects a region of non-interest in a rectangular region composed of decoded blocks, based on the coordinates of the region of interest and the pixel values ​​into which the video encoder 101 has replaced pixel values ​​of the region of non-interest, for the rectangular region supplied from the filter application unit 2003. The correction processing unit 2006 identifies pixels in the region of non-interest in the decoded block and replaces the pixel values ​​with pixel values ​​into which the video encoder 101 has replaced pixel values ​​of the region of non-interest, thereby obtaining a corrected decoded block. This improves prediction accuracy in predictive coding. Furthermore, the correction processing unit 2006 restores pixel values ​​of blocks not coded by the video encoder 101, i.e., the region of non-interest, on a block-by-block basis, based on the block division information, the coordinates of the region of interest, the width and height of the video frame coded by the video encoder 101, and the pixel values ​​into which the video encoder 101 has replaced pixel values ​​of the region of non-interest. The correction processing unit 2006 supplies the corrected frame to the frame buffer 2004. The frame output by the correction processing unit 2006 is the frame output from the video decoder 201.

[0061] Next, the operation of the video decoder 201 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the operation of the video decoder 201.

[0062] First, the bitstream decoder 2002 entropy-decodes the input bitstream to obtain the code string of each block and coordinate information of the region of interest (step S200). The bitstream decoder 2002 supplies the code string to the codestring decoder 2001. The bitstream decoder 2002 supplies the coordinates of the region of interest to the codestring decoder 2001 and the region of interest identifying unit 2007.

[0063] The code string decoder 2001 can restore a rectangular area made up of blocks including a region of interest on a block-by-block basis from the code string supplied from the bit stream decoder 2002 (step S201).

[0064] If it is determined that a filter is to be applied, the filter application unit 2003 applies filter processing to the decoded rectangular area supplied from the code string decoder 2001 (step S202). The filter application unit 2003 supplies the rectangular area after filtering to the correction processing unit 2006. If it is determined that a filter is not to be applied, the filter application unit 2003 supplies the rectangular area supplied from the code string decoder 2001 to the correction processing unit 2006 without applying filter processing.

[0065] Based on the supplied coordinate information, the region of interest identification unit 2007 identifies a block including a region of non-interest that was excluded from the target block by the encoder 101, and supplies information indicating that the block includes a region of non-interest to the correction processing unit 2006 (step S203). When the correction processing unit 2006 receives the rectangular region and information indicating that the block includes a region of non-interest from the filter application unit 2003, the correction processing unit 2006 performs a correction process (step S204). Specifically, the correction processing unit 2006 first corrects the pixel values ​​of the region of non-interest in the rectangular region supplied from the filter application unit 2003 by a predetermined pixel value based on the coordinates of the region of interest supplied from the bitstream decoding unit 2001. Then, based on the coordinates of the region of interest supplied from the bitstream decoding unit 2001, the correction processing unit 2006 restores the block whose encoding was skipped and combines it with the corrected rectangular region to generate an output frame (step S205), and stores the generated block including the region of non-interest in the frame buffer 2004.

[0066] In this embodiment, the predetermined pixel value is the pixel value after replacement by the region of interest detection unit 1001.

[0067] By the above-described process, even if the bitstream generator 1004 in the video encoder 101 does not encode the region of non-interest, the frame buffer 2004 stores frames in which pixel values ​​of the region of non-interest are restored. Therefore, the bitstream decoder 2001 can use the decoded frames stored in the frame buffer 2004 as, for example, reference frames.

[0068] In this embodiment, the bitstream generator 1004 in the video encoder 101 can skip entropy coding of the target blocks in the regions of non-interest. This prevents an increase in the amount of codestreams transmitted related to the regions of non-interest. Furthermore, by always maintaining the same fixed values ​​as before encoding for the pixels in the regions of non-interest in the frames output from the video encoder 101, the accuracy of predictive coding of frames that reference the output frames can be improved. Furthermore, since the video decoder 201 receives coordinate information for the regions of interest from the video encoder 101, it can recognize the coordinate information for the regions of non-interest and restore the regions of non-interest from the coordinate information for the regions of non-interest.

[0069] Embodiment 2. Figure 4 is a block diagram illustrating an embodiment of a video encoder and a video decoder.

[0070] In the first embodiment, the correction process is performed on the processing target frame based on the coordinate information of the region of interest.

[0071] For example, a portion of an encoded block (reference block) referenced by a target block may be a region of non-interest, and the pixel values ​​of the region of interest in the reference block may be closer to the pixel values ​​of the target block than the pixel values ​​into which the video encoder replaces the pixel values ​​of the region of non-interest. In this case, replacing the pixel values ​​of the reference block's region of non-interest with the pixel values ​​of the region of interest reduces the difference between the reference block and the target block. As a result, the target block is encoded with a smaller amount of code. In the second embodiment, correction processing is also applied to the encoded frame based on the coordinate information of the region of interest.

[0072] The video encoder 102 shown in FIG. 4 includes a region of interest detection unit 1001, a region of non-interest replacement processing unit 1002, a block division unit 1003, a code string generator 1004A, a bit stream generator 1005, a local decoder 1006, a filter application unit 1007, a frame buffer 1008, and a correction processing unit 1009A.

[0073] The video decoder 202 includes a code string decoder 2001A, a bit stream decoder 2002, a filter application unit 2003, a frame buffer 2004, a correction processing unit 2006A, and a region of interest identification unit 2007A.

[0074] [Description on the Encoding Side] The configurations and functions of the region of interest replacement processor 1002, block divider 1003, bitstream generator 1005, local decoder 1006, filter applicator 1007, and frame buffer 1008 are the same as those in the first embodiment. However, in this embodiment, the region of interest detector 1001 supplies a set of coordinate information of the region of interest to the frame buffer 1008. In the frame buffer 1008, the coordinate information of the region of interest is stored in association with the frame supplied from the filter applicator 1007.

[0075] The bitstream generator 1004A has the following functions in addition to the functions of the bitstream generator 1004 in the first embodiment. That is, when the bitstream generator 1004A encodes a target block by referencing a certain region of an encoded frame (e.g., a reference block), it supplies the correction processing unit 1009A with the proportion of the region of interest included in the target block (the proportion of the target block that the region of interest occupies) and information specifying the encoded frame to be referenced and the reference block. The bitstream generator 1004A then performs predictive encoding based on the reference block output by the correction processing unit 1009A in accordance with the information supplied to the correction processing unit 1009A.

[0076] The correction processing unit 1009A performs a correction process on the reference block. In this embodiment, the correction processing unit 1009A requests the frame buffer 1008 to supply the block to be used as the reference block and coordinate information on the region of interest based on information supplied from the bitstream generator 1004A. If the proportion of the region of interest included in the block to be processed is smaller than the proportion of the region of non-interest, the correction processing unit 1009A replaces the pixel values ​​of the region of interest in the reference block with pixel values ​​obtained by replacing the region of non-interest. If the proportion of the region of interest is equal to or greater than the proportion of the region of non-interest, the correction processing unit 1009A replaces the pixel values ​​of the region of non-interest in the reference block with pixel values ​​of the closest region of interest in the reference block. The correction processing unit 1009A supplies the reference block after the replacement process (correction process) to the bitstream generator 1004A.

[0077] Note that comparing the ratio of the region of interest to the ratio of the region of non-interest in the reference block is equivalent to comparing the area of ​​the region of interest to the area of ​​the region of non-interest in the reference block.

[0078] Next, the operation of the video encoder 102 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the operation of the video encoder 102.

[0079] The operation of the video encoder 102 is basically the same as the operation of the video encoder 101 of the first embodiment.

[0080] However, in this embodiment, the code string generator 1004A calculates the ratio of the region of interest in the block to be processed (size of the region of interest / size of the block to be processed) (step S107A). The code string generator 1004A supplies the ratio of the region of interest included in the block to be processed and information specifying the reference frame and reference block to the correction processing unit 1009A. The ratio of the region of non-interest is obtained by calculating (1 - ratio of the region of interest).

[0081] The correction processing unit 1009A performs correction processing on the reference block (step S107B). Then, the correction processing unit 1009A supplies the corrected reference block to the bitstream generator 1004A. Note that if correction processing is not performed, the correction processing unit 1009A supplies the reference block read from the frame buffer 1008 to the bitstream generator 1004A as is.

[0082] FIG. 6 is an explanatory diagram for explaining the correction process. In each of FIGS. 6(A) to 6(C), one square corresponds to one pixel. The numerical value in the square indicates the pixel value. In the example shown in FIG. 6, the pixel value is any value between 0 and 255. "127" is the pixel value for gray.

[0083] Fig. 6(A) shows an example of pixel values ​​of the reference block before the correction process. Fig. 6(B) shows an example of pixel values ​​after correction of the reference block shown in Fig. 6(A) when the ratio of the region of interest in the processing target block is equal to or greater than the ratio of the region of non-interest (when the proportion of the region of interest in the processing target block is equal to or greater than the proportion occupied by the region of non-interest). Fig. 6(C) shows an example of pixel values ​​after correction of the reference block shown in Fig. 6(A) when the proportion of the region of interest in the processing target block is smaller than the proportion of the region of non-interest (when the proportion occupied by the region of interest in the processing target block is less than the proportion occupied by the region of non-interest).

[0084] In the example shown in Figure 6, if the ratio of the region of interest in the processing target block is equal to or greater than the ratio of the region of non-interest, the correction processing unit 1009A replaces the pixel values ​​of the region of non-interest in the reference block with the pixel values ​​of the pixel in the region of interest located closest to it. However, if there are multiple pixels at the same distance, the pixel closest in the horizontal direction is given priority. In the example shown in Figure 6, the pixel value of the region of non-interest is "127".

[0085] If the ratio of the region of interest in the target block is smaller than the ratio of the region of non-interest, the correction processing unit 1009A treats the reference block as a region of non-interest, i.e., replaces pixel values ​​of the region of interest with pixel values ​​of the region of non-interest.

[0086] In other words, when the target block contains a mixture of pixels from the region of interest and pixels from the region of non-interest, the correction processing unit 1009A performs a correction process to bring the pixel values ​​of the coded block (i.e., the reference block) referenced by the target block closer to those of the target block as a whole, as a whole reference block.

[0087] In the example shown in Figure 6, if the pixels of the region of non-interest predominate in the block to be processed, the correction processing unit 1009A sets all pixel values ​​of the reference block to the pixel values ​​of the region of non-interest (see Figure 6C). If the pixels of the region of interest predominate in the block to be processed, the correction processing unit 1009A sets all pixel values ​​of the reference block to the pixel values ​​of the region of interest (see Figure 6B). In this case, the correction processing unit 1009A uses the values ​​of pixels of the region of interest that are near the pixels of the region of non-interest as pixel values ​​to replace the pixel values ​​of the region of non-interest.

[0088] By performing such a correction process, when a target block contains pixels from regions of interest and pixels from regions of non-interest, the pixel values ​​of the target block are corrected so that the difference between the pixel values ​​of the target block and the pixel values ​​of the corresponding reference block is reduced, thereby preventing an increase in the amount of code in the generated bitstream.

[0089] In step S107A, the bitstream generator 1004A performs encoding processing on the target block. That is, the bitstream generator 1004A performs predictive encoding processing based on the reference block output by the correction processing unit 1009A in accordance with the information supplied to the correction processing unit 1009A. As described above, the information supplied from the bitstream generator 1004A to the correction processing unit 1009A is information that identifies the proportion of the region of interest included in the target block and the coded frame (reference frame) and reference block to be referenced.

[0090] The other processes are the same as those of the video encoder 101 in the first embodiment.

[0091] [Description on the Decoding Side] In the video decoder 202, the bitstream decoder 2002, the filter application unit 2003, and the frame buffer 2004 have the same configurations and functions as those in the first embodiment. However, in this embodiment, the region of interest identification unit 2007A supplies a set of coordinate information of the region of interest to the frame buffer 2004. In the frame buffer 2004, the coordinate information of the region of interest is stored in association with the frame supplied from the filter application unit 2003.

[0092] The code string decoder 2001A has the following functions in addition to the functions of the code string decoder 2001 in the first embodiment. That is, when the code string decoder 2001A decodes a block to be processed by referencing a certain area of ​​a decoded frame, it supplies the correction processing unit 2006A with information specifying the proportion of the region of interest included in the block to be processed and the decoded frame to be referenced and the reference block. The code string generator 1004A then performs predictive decoding based on the reference block output by the correction processing unit 1009A in accordance with the information supplied to the correction processing unit 2006A.

[0093] The correction processing unit 2006A performs the correction processing (correction processing on the processing target frame) in the first embodiment, and also performs correction processing on the reference block. When performing correction processing on the reference block, the correction processing unit 2006A requests the frame buffer 2004 to supply the block to be used as the reference block and coordinate information of the region of interest based on information supplied from the codestream decoder 2001A. If the proportion of the region of interest included in the processing target block is smaller than the proportion of the region of non-interest, the correction processing unit 2006A replaces the pixel values ​​of the region of interest in the reference block with pixel values ​​obtained by replacing the pixel values ​​of the region of non-interest by the video encoder 102. If the proportion of the region of interest is equal to or greater than the proportion of the region of non-interest, the correction processing unit 2006A replaces the pixel values ​​of the region of non-interest in the reference block with pixel values ​​of the closest region of interest in the reference block. The correction processing unit 2006A supplies the reference block subjected to the replacement processing (correction processing) to the codestream decoder 2001A.

[0094] Next, the operation of the video decoder 202 will be described. The video decoder 202 operates in the same manner as the video decoder 201 in the first embodiment (see FIG. 3 ). However, as described above, the region of interest identifying unit 2007A, the correction processing unit 2006A, and the codestream decoder 2001A also perform processes not performed in the first embodiment (processes related to the correction processing of the reference block), in addition to the processes performed by the region of interest identifying unit 2007, the correction processing unit 2006, and the codestream decoder 2001 in the first embodiment.

[0095] In this embodiment, the video decoder 202 further performs a reference block correction process. Fig. 7 is a flowchart showing the reference block correction process. The reference block correction process is executed by the codestream decoder 2001A during the decoding process.

[0096] In the reference block correction process, the codestream decoder 2001A calculates the ratio of the region of interest in the current block. The codestream decoder 2001A supplies the ratio of the region of interest included in the current block and information specifying the reference frame and reference block to the correction processor 2006A. The correction processor 2006A receives the ratio of the region of interest and the information specifying the reference frame and reference block (steps S210 and S211).

[0097] The correction processing unit 2006A requests the frame buffer 2004 to output coordinate information of the reference block and the region of interest based on the information input from the code string decoder 2001A. The correction processing unit 2006A receives the coordinate information of the reference block and the region of interest output from the frame buffer 2004 in response to the request (step S212).

[0098] The correction processing unit 2006A compares the ratio of the region of interest with the ratio of the region of non-interest (step S213).

[0099] If the ratio of the region of interest included in the target block is equal to or greater than the ratio of the region of non-interest, the correction processor 2006A replaces the pixel values ​​of the region of non-interest in the reference block with the pixel values ​​of the closest region of interest in the reference block (step S214: see FIG. 6B). If the ratio of the region of interest is smaller than the ratio of the region of non-interest, the correction processor 2006A replaces the pixel values ​​of the region of interest in the reference block with the pixel values ​​of the region of non-interest (step S215: see FIG. 6C). The correction processor 2006A supplies the reference block after the replacement process to the codestream decoder 2001A.

[0100] The code string decoder 2001A uses the reference block supplied from the correction processing unit 2006A when executing predictive decoding processing with reference to the reference block.

[0101] In this embodiment, the video encoder 102 and the video decoder 202 perform correction processing on the reference block, i.e., processing to convert the pixel values ​​of the reference block, so that the block to be processed is coded with a smaller amount of code, thereby reducing the amount of code to be transmitted.

[0102] [Variation 1] In the second embodiment, the video encoder 102 and the video decoder 202 perform both a correction process on the frame to be processed and a correction process on the reference block, but the video encoder 102 and the video decoder 202 may perform only a correction process on the reference block.

[0103] [Variation 2] In the second embodiment, the size relationship between the size of the region of interest and the size of the region of non-interest is used as a criterion for determining whether to replace pixel values ​​of the region of interest or the region of non-interest in the reference block. However, this determination method is merely an example, and other determination criteria may be used. As another determination method, for example, it is possible to determine whether the size of the region of interest is larger than (2 / 3) the size of the region of non-interest. Furthermore, in the second embodiment, when pixel values ​​of the region of interest are replaced, a replacement method such as that illustrated in FIG. 6(C) is used, and when pixel values ​​of the region of non-interest are replaced, a replacement method such as that illustrated in FIG. 6(B) is used. However, pixel values ​​after replacement may be pixel values ​​other than those illustrated in FIGS. 6(B) and 6(C).

[0104] [Variation 3] A flag indicating whether or not to apply the correction process to the reference block may be introduced into the second embodiment. Such a flag is transmitted from the video encoder 102 to the video decoder 202. If the video encoder 102 determines that it is more efficient not to apply the correction process, it sets the value of the flag to a value that indicates that the correction process to the reference block is not applied. By performing such control, it is expected that the coding efficiency will be further improved.

[0105] [Variation 4] In each of the first and second embodiments, the video encoder may include, in the bitstream, coordinate information of a rectangular area formed by the current block and information on the size (width and height) of the rectangular area. Based on this information, the video decoder may identify blocks that were not encoded from the coordinate and size information of the rectangular area formed by the decoded blocks and restore their pixel values. The coordinates and size may be expressed as the number and number of blocks obtained by the block division unit 1003 in raster scan order.

[0106] [Variation 5] In each of the first and second embodiments, the video encoder may divide the regions of interest into multiple groups, encode the rectangular regions for each group, and concatenate the generated code sequences to generate a bitstream. In this case, the video encoder may include in the bitstream information indicating which region of interest belongs to each group and information associating the code sequences of each rectangular region with the region of interest group. The video decoder identifies which region of interest group the code sequence belongs to based on the information associating the code sequences of the rectangular regions with the region of interest group. The video decoder then restores each rectangular region based on the information about the region of interest belonging to the corresponding group and performs correction processing. Note that the coordinate information of each rectangular region may also be transmitted in the above Variation 4.

[0107] Each of the above embodiments can be configured by hardware, but can also be realized by a computer program.

[0108] The information processing system shown in Fig. 8 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing a bitstream. The storage medium 703 and the storage medium 704 may be separate storage media or may be storage areas formed by the same storage medium. A magnetic storage medium such as a hard disk can be used as the storage medium.

[0109] In the information processing system, a program memory 702 stores a program (video encoding program or video decoding program) for implementing the functions of each block shown in each of the above embodiments.

[0110] The processor 701 then executes processing in accordance with the program stored in the program memory 702, thereby realizing the functions of the video encoders 101 and 102 and the video decoders 201 and 202 shown in each embodiment.

[0111] For example, the processor 701 executes processing in accordance with a video encoding program for realizing the functions of each block (except for the frame buffer 1008) in the video encoders 101 and 102 shown in Figures 1 and 4, thereby realizing the functions of the video encoders 101 and 102.

[0112] Furthermore, for example, the processor 701 executes processing in accordance with a video decoding program for realizing the functions of each block (excluding the frame buffer 2004) in the video decoders 201 and 202 shown in Figures 1 and 4, thereby realizing the functions of the video decoders 201 and 202.

[0113] At least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transitory computer-readable medium. The program is supplied to the transitory computer-readable medium, for example, via a wired or wireless communication channel, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0114] 9 is a block diagram showing the main components of a video encoding device. The video encoding device 10 shown in FIG. 9 (implemented by video encoders 101 and 102 in the embodiment) includes region of interest detection means 11 (implemented by a region of interest detection unit 1001 in the embodiment) for detecting a region of interest from an input video frame, region of non-interest replacement means 12 (implemented by a region of non-interest replacement processing unit 1002 in the embodiment) for replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values, block division means 13 (implemented by a block division unit 1003 in the embodiment) for dividing the frame in which the pixel values ​​of the regions of non-interest have been replaced into one or more blocks, and a processing target block division unit 1004. The image processing system 1000 includes a code string generating means 14 (realized by the code string generator 1004 in the embodiment) that encodes the blocks to generate a code string, a local decoding means 15 (realized by the local decoder 1006 in the embodiment) that decodes the code string, and a restoration means 16 (realized by the correction processing unit 1009 in the embodiment) that restores pixel values ​​of regions of non-interest based on information that can identify the region of interest, and the block dividing means 13 includes a target block selecting means 131 (realized by the block dividing unit 1003 in the embodiment) that selects a target block to be processed based on the information that can identify the region of interest.

[0115] The video encoding device 10 may be configured to include a region of interest detection means for detecting a region of interest from an input video frame, a code string generation means for encoding the input video frame to generate a code string, a calculation means (realized by the code string generator 1004 in the embodiment) for calculating the proportion of the region of interest in the block to be processed, and a pixel value conversion means (realized by the correction processing unit 1009A in the embodiment) for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes a region of interest and a region of non-interest.

[0116] Fig. 10 is a block diagram showing the main components of a video decoding device. The video decoding device 20 shown in Fig. 10 (implemented by video decoders 201 and 202 in the embodiment) includes region-of-interest identifying means 21 (implemented by a region-of-interest identifying unit 2007 in the embodiment) that acquires information for identifying a region of interest from a bitstream in which information for identifying a region of interest in a video frame is added to a codestream, video decoding means 22 (implemented by a codestream decoder 2001 in the embodiment) that decodes the codestream to acquire pixel values ​​of the region of interest, and restoration means 23 (implemented by a correction processing unit 2006 in the embodiment) that restores pixel values ​​of regions of non-interest based on the information for identifying the region of interest.

[0117] The video decoding device 20 may be configured to include a region of interest identification means for acquiring information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest has been added to the code string; a video decoding means for decoding the code string to acquire pixel values ​​of the region of interest; a calculation means (realized by the code string decoder 2001A in the embodiment) for calculating the proportion of the region of interest in the block to be processed; and a pixel value conversion means (realized by the correction processing unit 2006A in the embodiment) for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes a region of interest and a region of non-interest.

[0118] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0119] (Supplementary Note 1) A video encoding device comprising: region of interest detection means for detecting a region of interest from an input video frame; region of non-interest replacement means for replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values; block division means for dividing the frame after the pixel values ​​of the regions of non-interest have been replaced into one or more blocks; code sequence generation means for encoding a block to be processed to generate a code sequence; local decoding means for decoding the code sequence; and restoration means for restoring the pixel values ​​of the regions of non-interest based on information that can identify the region of interest, wherein the block division means includes target block selection means for selecting the block to be processed based on the information that can identify the region of interest.

[0120] (Supplementary Note 2) The video encoding device of Supplementary Note 1, wherein the restoration means generates an encoded frame by combining the region of interest restored by the local decoding means and the region of non-interest restored by the restoration means.

[0121] (Supplementary Note 3) A video encoding device according to Supplementary Note 1 or Supplementary Note 2, comprising: a calculation means for calculating a proportion of a target block occupied by the region of interest; and a pixel value conversion means for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes the region of interest and the region of non-interest.

[0122] (Supplementary Note 4) The video encoding device of Supplementary Note 3, wherein the pixel value conversion means replaces pixel values ​​of the region of non-interest in the reference block with pixel values ​​of pixels of the region of interest that are closest to the pixels of the reference block when the proportion of the region of interest in the block to be processed that is equal to or greater than the proportion of the region of non-interest, and replaces pixel values ​​of the region of interest with pixel values ​​of the region of non-interest when the proportion of the block to be processed that is represented by the region of interest that is less than the proportion of the region of non-interest.

[0123] (Supplementary Note 5) A video encoding device comprising: region of interest detection means for detecting a region of interest from an input video frame; code sequence generation means for encoding the input video frame to generate a code sequence; calculation means for calculating a proportion of a block to be processed that is occupied by the region of interest; and pixel value conversion means for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes the region of interest and regions of non-interest.

[0124] (Supplementary Note 6) A video decoding device comprising: a region of interest identification means for acquiring information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest has been added to a code string; a video decoding means for decoding the code string to acquire pixel values ​​of the region of interest; and a restoration means for restoring pixel values ​​of regions of non-interest based on the information capable of identifying the region of interest.

[0125] (Supplementary Note 7) The video decoding device according to Supplementary Note 6, wherein the region of interest acquired by the video decoding means and the region of non-interest restored by the restoration means are combined to generate an output frame.

[0126] (Supplementary Note 8) The video decoding device of Supplementary Note 6 or Supplementary Note 7, comprising: a calculation means for calculating a proportion of the region of interest in a target block; and a pixel value conversion means for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes the region of interest and the region of non-interest.

[0127] (Supplementary Note 9) The video decoding device of Supplementary Note 8, wherein the pixel value conversion means replaces pixel values ​​of the region of non-interest in the reference block with pixel values ​​of pixels of the region of interest that are closest to the pixels of the reference block when the proportion of the region of interest in the block to be processed is equal to or greater than the proportion of the region of non-interest; and replaces pixel values ​​of the region of interest with pixel values ​​of the region of non-interest when the proportion of the block to be processed represented by the region of interest is less than the proportion of the region of non-interest.

[0128] (Supplementary Note 10) A video decoding device comprising: a region of interest identification means for acquiring information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest is added to a code sequence; a video decoding means for decoding the code sequence to acquire pixel values ​​of the region of interest; a calculation means for calculating a proportion of a block to be processed that is occupied by the region of interest; and a pixel value conversion means for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes the region of interest and a region of non-interest.

[0129] (Supplementary Note 11) A video encoding method comprising: detecting a region of interest from an input video frame; replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values; dividing the frame after the pixel values ​​of the regions of non-interest have been replaced into one or more blocks; selecting a block to be processed based on information that can identify the region of interest when dividing the blocks; encoding the block to be processed to generate a code string; decoding the code string; and restoring the pixel values ​​of the regions of non-interest based on the information that can identify the region of interest.

[0130] (Supplementary Note 12) The video coding method of Supplementary Note 11, further comprising: pixel value conversion means for calculating a proportion of the target block occupied by the region of interest; and, when a reference block includes the region of interest and the region of non-interest, converting pixel values ​​of the reference block according to the proportion.

[0131] (Supplementary Note 13) The video coding method of Supplementary Note 12, wherein, if the proportion of the target block represented by the region of interest is equal to or greater than the proportion of the region of non-interest, pixel values ​​of the region of non-interest in the reference block are replaced with pixel values ​​of pixels of the region of interest that are closest to the pixels of the reference block; and, if the proportion of the target block represented by the region of interest is less than the proportion of the region of non-interest, pixel values ​​of the region of interest are replaced with pixel values ​​of the region of non-interest.

[0132] (Supplementary Note 14) A video encoding method comprising: detecting a region of interest from an input video frame; encoding the input video frame to generate a code string; calculating a proportion of a block to be processed that is occupied by the region of interest; and, if a reference block includes the region of interest and a region of non-interest, converting pixel values ​​of the reference block in accordance with the proportion.

[0133] (Supplementary Note 15) A video decoding method comprising: obtaining information capable of identifying a region of interest in a video frame from a bitstream in which information capable of identifying the region of interest is added to the codestream; decoding the codestream to obtain pixel values ​​of the region of interest; and restoring pixel values ​​of regions of non-interest based on the information capable of identifying the region of interest.

[0134] (Supplementary Note 16) The video decoding method of Supplementary Note 15, further comprising: calculating a proportion of the target block occupied by the region of interest; and, if the reference block includes both the region of interest and the region of non-interest, converting pixel values ​​of the reference block according to the proportion.

[0135] (Supplementary Note 17) The video decoding method of Supplementary Note 16, wherein, if the proportion of the target block represented by the region of interest is equal to or greater than the proportion occupied by the region of non-interest, pixel values ​​of the region of non-interest in the reference block are replaced with pixel values ​​of pixels of the region of interest that are closest to the pixels of the reference block; and, if the proportion of the target block represented by the region of interest is less than the proportion occupied by the region of non-interest, pixel values ​​of the region of interest are replaced with pixel values ​​of the region of non-interest.

[0136] (Supplementary Note 18) A video decoding method comprising: obtaining information capable of identifying a region of interest in a video frame from a bitstream in which information capable of identifying the region of interest is added to a code sequence; decoding the code sequence to obtain pixel values ​​of the region of interest; calculating a proportion of a block to be processed that is occupied by the region of interest; and, if a reference block includes the region of interest and a region of non-interest, converting pixel values ​​of the reference block in accordance with the proportion.

[0137] (Supplementary Note 19) A video encoding program that causes a computer to execute the following processes: detecting a region of interest from an input video frame; replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values; dividing the frame after the pixel values ​​of the regions of non-interest have been replaced into one or more blocks; selecting a block to be processed based on information that can identify the region of interest when dividing the blocks; encoding the block to be processed to generate a code string; decoding the code string; and restoring the pixel values ​​of the regions of non-interest based on the information that can identify the region of interest.

[0138] (Supplementary Note 20) A video decoding program for causing a computer to execute the following processes: obtaining information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest has been added to the code string; decoding the code string to obtain pixel values ​​of the region of interest; and restoring pixel values ​​of regions of non-interest based on the information capable of identifying the region of interest.

[0139] (Supplementary Note 21) The video encoding device of Supplementary Note 5, wherein the pixel value conversion means replaces pixel values ​​of the region of non-interest in the reference block with pixel values ​​of pixels of the region of interest that are closest to the pixels of the reference block when the proportion of the region of interest in the block to be processed is equal to or greater than the proportion of the region of non-interest; and replaces pixel values ​​of the region of interest with pixel values ​​of the region of non-interest when the proportion of the block to be processed represented by the region of interest is less than the proportion of the region of non-interest.

[0140] (Supplementary Note 22) The video decoding device of Supplementary Note 10, wherein the pixel value conversion means replaces pixel values ​​of the region of non-interest in the reference block with pixel values ​​of pixels of the region of interest that are closest to the pixels of the reference block when the proportion of the region of interest in the block to be processed is equal to or greater than the proportion of the region of non-interest; and replaces pixel values ​​of the region of interest with pixel values ​​of the region of non-interest when the proportion of the block to be processed represented by the region of interest is less than the proportion of the region of non-interest.

[0141] (Supplementary Note 23) The video coding method of Supplementary Note 14, wherein, if the proportion of the target block represented by the region of interest is equal to or greater than the proportion occupied by the region of non-interest, pixel values ​​of the region of non-interest in the reference block are replaced with pixel values ​​of pixels of the region of interest that are closest to the pixels of the reference block; and, if the proportion of the target block represented by the region of interest is less than the proportion occupied by the region of non-interest, pixel values ​​of the region of interest are replaced with pixel values ​​of the region of non-interest.

[0142] (Supplementary Note 24) The video decoding method of Supplementary Note 18, wherein, if the proportion of the target block represented by the region of interest is equal to or greater than the proportion occupied by the region of non-interest, pixel values ​​of the region of non-interest in the reference block are replaced with pixel values ​​of pixels of the region of interest that are closest to the pixels of the reference block; and, if the proportion of the target block represented by the region of interest is less than the proportion occupied by the region of non-interest, pixel values ​​of the region of interest are replaced with pixel values ​​of the region of non-interest.

[0143] (Supplementary Note 25) A video encoding program that causes a computer to execute any one of the video encoding methods of Supplementary Note 11 to Supplementary Note 14.

[0144] (Supplementary Note 26) A video encoding program that causes a computer to execute any one of the video decoding methods of Supplementary Note 15 to Supplementary Note 18.

[0145] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0146] This application claims priority based on Japanese Patent Application No. 2023-173571, filed on October 5, 2023, the disclosure of which is incorporated herein in its entirety.

[0147] 101, 102 Video encoder 201, 202 Video decoder 701 Processor 702 Program memory 703, 704 Storage medium 1001 Region of interest detection unit 1002 Region of non-interest replacement processing unit 1003 Block division unit 1004, 1004A Code string generator 1005 Bit stream generator 1006 Local decoder 1007 Filter application unit 1008 Frame buffer 1009, 1009A Correction processing unit 2001, 2001A Code string decoder 2002 Bit stream decoder 2003 Filter application unit 2004 Frame buffer 2005 Correction processing unit 2006, 2006A Correction processing unit 2007, 2007A Region of interest identification unit

Claims

1. A video encoding device comprising: region of interest detection means for detecting a region of interest from an input video frame; region of non-interest replacement means for replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values; block division means for dividing the frame after the pixel values ​​of the regions of non-interest have been replaced into one or more blocks; and code string generation means for encoding a target block to generate a code string, wherein the block division means includes target block selection means for selecting the target block based on information capable of identifying the region of interest, and the code string generation means includes local decoding means for decoding the code string, and restoration means for restoring pixel values ​​of the regions of non-interest based on the information capable of identifying the region of interest.

2. A video encoding device according to claim 1, wherein the restoration means generates an encoded frame by combining the region of interest restored by the local decoding means and the region of non-interest restored by the restoration means.

3. A video encoding device as described in claim 1 or claim 2, comprising: a calculation means for calculating the proportion of the region of interest in the block to be processed; and a pixel value conversion means for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes the region of interest and the region of non-interest.

4. The video encoding device of claim 3, wherein the pixel value conversion means replaces pixel values ​​of the region of non-interest in the reference block with pixel values ​​of pixels of the region of interest that are located closest to the pixels of the reference block when the proportion of the block represented by the region of interest is equal to or greater than the proportion of the region of non-interest, and replaces pixel values ​​of the region of interest with pixel values ​​of the region of non-interest when the proportion of the block represented by the region of interest is less than the proportion of the block represented by the region of non-interest.

5. A video encoding device comprising: a region of interest detection means for detecting a region of interest from an input video frame; a code string generation means for encoding the input video frame to generate a code string; a calculation means for calculating a proportion of a block to be processed that is occupied by the region of interest; and a pixel value conversion means for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes the region of interest and regions of non-interest.

6. A video decoding device comprising: a region of interest identification means for acquiring information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest has been added to a code sequence; a video decoding means for decoding the code sequence to acquire pixel values ​​of the region of interest; and a restoration means for restoring pixel values ​​of regions of non-interest based on the information capable of identifying the region of interest.

7. The video decoding device according to claim 6, wherein the region of interest acquired by the video decoding means and the region of non-interest restored by the restoration means are combined to generate an output frame.

8. A video decoding device as described in claim 6 or claim 7, comprising: a calculation means for calculating a proportion of a block to be processed that is occupied by the region of interest; and a pixel value conversion means for converting pixel values ​​of the reference block in accordance with the proportion when the reference block includes the region of interest and the region of non-interest.

9. A video decoding device as described in claim 8, wherein the pixel value conversion means replaces pixel values ​​of the region of non-interest in the reference block with pixel values ​​of pixels of the region of interest that are located closest to the pixels of the reference block when the proportion of the region of interest in the block to be processed that is equal to or greater than the proportion of the region of non-interest, and replaces pixel values ​​of the region of interest with pixel values ​​of the region of non-interest when the proportion of the block to be processed that is represented by the region of interest that is less than the proportion of the region of non-interest.

10. A video decoding device comprising: a region of interest identification means for acquiring information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest is added to a code sequence; a video decoding means for decoding the code sequence to acquire pixel values ​​of the region of interest; a calculation means for calculating the proportion of the region of interest in a block to be processed; and a pixel value conversion means for converting pixel values ​​of the reference block in accordance with the proportion when a reference block includes the region of interest and regions of non-interest.

11. A video encoding method comprising: detecting a region of interest from an input video frame; replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values; dividing the frame after the pixel values ​​of the regions of non-interest have been replaced into one or more blocks; when dividing the blocks, selecting a block to be processed based on information capable of identifying the region of interest; encoding the block to be processed to generate a code string; decoding the code string; and restoring the pixel values ​​of the regions of non-interest based on the information capable of identifying the region of interest.

12. A video coding method as described in claim 11, further comprising a pixel value conversion means for calculating a proportion of the block to be processed that is occupied by the region of interest, and converting pixel values ​​of the reference block in accordance with said proportion when the reference block includes both the region of interest and the region of non-interest.

13. A video encoding method as described in claim 12, wherein, if the proportion of the block to be processed represented by the region of interest is equal to or greater than the proportion of the region of non-interest, pixel values ​​of the region of non-interest in the reference block are replaced with pixel values ​​of pixels of the region of interest that are located closest to the pixels of the reference block, and, if the proportion of the block to be processed represented by the region of interest is less than the proportion of the block to be processed represented by the region of non-interest, pixel values ​​of the region of interest are replaced with pixel values ​​of the region of non-interest.

14. A video encoding method comprising: detecting a region of interest from an input video frame; encoding the input video frame to generate a code string; calculating a proportion of a block to be processed that is occupied by the region of interest; and, if a reference block includes both the region of interest and a region of non-interest, converting pixel values ​​of the reference block in accordance with the proportion.

15. A video decoding method comprising: obtaining information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest has been added to the code string; decoding the code string to obtain pixel values ​​of the region of interest; and restoring pixel values ​​of regions of non-interest based on the information capable of identifying the region of interest.

16. The video decoding method according to claim 15, further comprising: calculating a proportion of the block to be processed that is occupied by the region of interest; and, if a reference block includes both the region of interest and the region of non-interest, converting pixel values ​​of the reference block in accordance with the proportion.

17. A video decoding method as described in claim 16, further comprising the steps of: replacing pixel values ​​of the region of interest in the reference block with pixel values ​​of pixels of the region of interest that are closest to pixels of the reference block when the proportion of the region of interest in the block to be processed is equal to or greater than the proportion of the region of non-interest; and replacing pixel values ​​of the region of interest in the reference block with pixel values ​​of the region of non-interest when the proportion of the region of interest in the block to be processed is less than the proportion of the region of non-interest.

18. A video decoding method comprising: obtaining information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest has been added to the code string; decoding the code string to obtain pixel values ​​of the region of interest; calculating a proportion of a block to be processed that is occupied by the region of interest; and, if a reference block includes the region of interest and regions of non-interest, converting pixel values ​​of the reference block in accordance with the proportion.

19. A video encoding program causing a computer to execute the following processes: detecting a region of interest from an input video frame; replacing pixel values ​​of regions of non-interest in the input video frame with predetermined pixel values; dividing the frame after the pixel values ​​of the regions of non-interest have been replaced into one or more blocks; when dividing the blocks, selecting a block to be processed based on information capable of identifying the region of interest; encoding the block to be processed to generate a code string; decoding the code string; and restoring the pixel values ​​of the regions of non-interest based on the information capable of identifying the region of interest.

20. A video decoding program for causing a computer to execute the following processes: obtaining information capable of identifying a region of interest in a video frame from a bit stream in which information capable of identifying the region of interest has been added to a code sequence; decoding the code sequence to obtain pixel values ​​of the region of interest; and restoring pixel values ​​of regions of non-interest based on the information capable of identifying the region of interest.

Images