Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

In order to ensure the quality of output video on the video decoding side, this video encoding device includes a control means for controlling the order of processing a plurality of gaze regions such that the amount of information of the gaze regions in the video is reduced.

Description

Video encoding device, video decoding device, video encoding method, and video decoding method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to transmit or record video efficiently, a video encoding device is used to generate a coded representation (hereinafter referred to as a bitstream) of input video, and a video decoding device is used to decode the bitstream to generate decoded video.

[0003] [Video Coding Based on Predictive Coding in Coding Units] Video coding standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), and H.266 / VVC (Versatile Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. Another recent video coding technology is the technology described in Non-Patent Document 1.

[0004] In these video coding methods, video data is coded and decoded while being managed in a hierarchical structure. The hierarchical structure is made up of, for example, pictures that make up the video data, slices (or tiles) obtained by dividing pictures, coding tree units (CTUs) obtained by dividing slices, and coding units (CUs) obtained by dividing coding tree units.

[0005] The input image of the target CU is usually encoded earlier than the target CU and is predictively coded based on a predicted image generated based on the decoded image that has been decoded. That is, a prediction error image obtained by subtracting the predicted image from the input image is coded and decoded. Predictive coding includes intra-picture prediction (intra-prediction) that uses a decoded image included in a picture with the same display time as the target CU, and inter-picture prediction (inter-prediction) that uses a decoded image included in a picture with a different display time from the target CU.

[0006] The prediction error image is encoded based on frequency transform, quantization, and entropy coding. The prediction error image is decoded based on entropy decoding, inverse quantization, and inverse frequency transform. The frequency transform value of the quantized prediction error image is called a quantized value.

[0007] [Video Coding Based on Neural Networks] Non-Patent Document 2 describes a new video coding technique that combines an auto-encoder, which is a type of neural network, quantization, and entropy coding.

[0008] An autoencoder compresses input data into a low-dimensional feature vector that contains only important features. The autoencoder then generates reconstructed data by reconstructing the low-dimensional feature vector back to its original dimensions. Figure 1 is an explanatory diagram showing the autoencoder algorithm. In Figure 1, the circular parts are called nodes and the arrows are called edges. The process of reducing the data into a low-dimensional feature vector (the first half) is called encoding. The process of generating reconstructed data (the second half) is called decoding.

[0009] The autoencoder is trained to minimize the reconstruction error (the difference between the input data and the reconstructed data). To obtain meaningful features, the autoencoder is designed to impose constraints on the encoding structure and to add regularization terms to the network's loss function.

[0010] "Algorithm description of Enhanced Compression Model 9(ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023J. Ball'e, V. Laparra, and EP Simoncelli, "End-to-end Optimized Image Compression", published as a conference paper at ICLR 2017S. Ren et al., "Faster R-CNN: Towards real-time object detection with region proposal networks", IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016

[0011] In applications where video coding based on predictive coding in the coding unit, video coding based on a neural network, or video coding based on a combination thereof (hereinafter referred to as internal video coding) is optimized for a machine processing task, it is necessary to ensure the quality of the output video from the video decoding side (e.g., a video decoding device). The machine processing task is, for example, video coding for machines (VCM) used for machine recognition.

[0012] For example, on the video encoding side (e.g., a video encoding device), input video for internal video encoding is classified into a region of interest (ROI) and a non-region of interest for the machine processing task as an optimization for the machine processing task. Furthermore, by reducing the region of interest by a factor corresponding to its importance, only the video information necessary for the machine processing task can be efficiently represented. Hereinafter, the classification process and the reduction process are collectively referred to as preprocessing.

[0013] In the optimization for the above machine processing task, when there is overlap between multiple attention areas and the reduction ratios are different, there is a problem that the output video on the video decoding side may not be identical to the original video. This is because the area reduced on the video encoding side is enlarged to its original size on the video decoding side, but the video decoding side does not have information on which attention area (specifically, the reduction ratio) should be applied to the overlapping area. In addition, for example, if the video decoding side uses the attention area that has been reduced to a smaller size to restore the overlapping area, there is a problem that the video quality is lower than when the non-reduced attention area is used.

[0014] In summary, the video decoder cannot reproduce the same video as the original video, and the quality of the output video from the video decoder is reduced. In other words, there is a problem in that the quality of the output video from the video decoder cannot be guaranteed.

[0015] An object of the present invention is to provide a video encoding device, a video decoding device, a video encoding method, and a video decoding method that can ensure the quality of the output video on the video decoding side.

[0016] A video encoding device according to the present disclosure is a video encoding device that reduces the amount of information about a fixation area in a video, and includes a control means for controlling the processing order of a plurality of fixation areas.

[0017] A video decoding device according to the present disclosure is a video decoding device that reduces the amount of information about a focus area in a video, and includes a control means for controlling the processing order of a plurality of focus areas.

[0018] A video coding method according to the present disclosure is a video coding method that reduces the amount of information about a fixation region in a video, and controls the processing order of a plurality of fixation regions.

[0019] A video decoding method according to the present disclosure is a video decoding method that reduces the amount of information about a focus region in a video, and controls the processing order of a plurality of focus regions.

[0020] A video encoding program according to the present disclosure is a video encoding program that reduces the amount of information about a fixation area in a video, and causes a computer to control the processing order of a plurality of fixation areas.

[0021] A video decoding program according to the present disclosure is a video decoding program that reduces the amount of information about a gaze region in a video, and causes a computer to control the processing order of a plurality of gaze regions.

[0022] According to the present invention, the quality of the output video on the video decoding side can be guaranteed.

[0023] FIG. 1 is an explanatory diagram showing an algorithm of an autoencoder. FIG. 2 is a block diagram showing a video encoder and a video decoder that encode and decode each picture that constitutes video data. FIG. 3 is an explanatory diagram of pre-processing and post-processing. FIG. 4 is an explanatory diagram of pre-processing and post-processing. FIG. 5 is an explanatory diagram of pre-processing and post-processing. FIG. 6 is an explanatory diagram showing an example of a data structure of auxiliary information. FIG. 7 is a flowchart showing an example operation of a video encoder. FIG. 8 is a flowchart showing an example operation of a video decoder. FIG. 9 is a block diagram showing an example configuration of an information processing system. FIG. 10 is a block diagram showing main parts of a video encoding device. FIG. 11 is a block diagram showing main parts of a video decoding device.

[0024] Hereinafter, an embodiment will be described with reference to the drawings.

[0025] Embodiment 1 Fig. 2 is a block diagram showing a video encoder 100 and a video decoder 200 that encode and decode each picture that constitutes video data.

[0026] The video encoder 100 includes a pre-processing unit 101 , an internal video encoder 102 , a multiplexer 103 , and a control unit 104 .

[0027] The video decoder 200 includes a post-processing unit 201 , an internal video decoder 202 , a demultiplexer 203 , and a control unit 204 .

[0028] 2 simply indicates the direction of signal (data) flow, but does not exclude bidirectionality. This also applies to other block diagrams.

[0029] [Description on the Encoding Side] In the video encoder 100, the preprocessing unit 101 identifies a region of interest in an input picture. The preprocessing unit 101, for example, detects an object in the input picture and sets a rectangular region including the object as the region of interest. Furthermore, the preprocessing unit 101 reduces the region of interest.

[0030] The internal video encoder 102 encodes the image output from the pre-processing unit 101 to create a video bitstream.

[0031] The control unit 104 generates an auxiliary information bitstream containing the coordinates of the top left and bottom right of each gaze area and a magnification as auxiliary information for post-processing on the video decoding side.

[0032] The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream to generate a bitstream.

[0033] 3A to 3D, a description will be given of the processing by the pre-processing unit 101. A case where the input picture illustrated in FIG.

[0034] The pre-processing unit 101 can obtain a rectangular region of the machine processing task as illustrated in Fig. 3B. As a method for obtaining the rectangular region of the machine processing task, the pre-processing unit 101 may use, for example, the object detection method described in Non-Patent Document 3. Figs. 3B to 3D illustrate four attention regions.

[0035] In the examples shown in Figures 3B to 3D, a magnification of 0.5 indicates that the region is reduced by 0.5 times in both the horizontal and vertical directions. The same applies to magnifications of 0.75 and 1.00. Note that these magnification values ​​are merely examples. A smaller reduction ratio means a lower degree of reduction. For example, a magnification of 0.75 is a smaller reduction ratio than a magnification of 0.5.

[0036] 3B and 3D also illustrate overlapping regions. As mentioned above, there are problems with overlapping regions, and solutions to these problems will be described later.

[0037] The larger the magnification value, the more important the region. In other words, the preprocessing unit 101 reduces the region of interest by increasing the magnification of the region of interest that is assumed to be important and decreasing the magnification of the region of interest that is not important. The reduction is achieved, for example, by thinning out the samples (downsampling).

[0038] The preprocessing unit 101 may change the non-attention area to, for example, gray in order to reduce the amount of information in the area other than the attention area (non-attention area). Note that gray is an example. In addition, in Figures 3C and 3D, the gray area is indicated by diagonal lines.

[0039] Next, an example of a method for solving the problem regarding the overlapping area by the control unit 104 will be described.

[0040] 3B illustrates an example of an overlapping area that spans a fixation area with a magnification of 0.50 and a fixation area with a magnification of 0.75. The control unit 104 controls so that the magnification used in such an overlapping area can be uniquely determined in post-processing on the video decoding side.

[0041] 4 is an explanatory diagram showing an example of the data structure of auxiliary information. As described above, the control unit 104 generates an auxiliary information bit stream including the coordinates of the upper left and lower right of each gaze area and the magnification as auxiliary information. For example, the control unit 104 controls the sending of auxiliary information for a gaze area with a lower priority first so that the image is overwritten last with the sample values ​​of the gaze area that should be prioritized.

[0042] 4 shows the data structure of auxiliary information corresponding to one gaze area. Referring to the example shown in FIG. 3B etc., the control unit 104 generates an auxiliary information bitstream including four pieces of auxiliary information. In the auxiliary information bitstream, the control unit 104 sets auxiliary information related to a gaze area with a high priority before auxiliary information related to a gaze area with a low priority.

[0043] For example, if it is desired to treat the overlapping area as a fixation area with a magnification of 0.75, the control unit 104 first sets auxiliary information regarding a fixation area with a magnification of 0.5 as auxiliary information for a fixation area with a low priority in the auxiliary information bit stream, and then embeds auxiliary information regarding a fixation area with a magnification of 0.75 as auxiliary information for a fixation area with a high priority.

[0044] Next, a description will be given of the operation of the video encoder 100. FIG.

[0045] The pre-processing unit 101 identifies a region of interest in an input picture (step S101), and then reduces the region of interest (step S102).

[0046] The internal video encoder 102 encodes the image output from the pre-processing unit 101 to create a video bitstream (step S103).

[0047] The control unit 104 embeds the auxiliary information of each gaze area in the auxiliary information bit stream (step S104). As described above, the control unit 104 determines the setting order (embedding order) of the auxiliary information according to the priority of the gaze area, that is, according to the reduction ratio of the gaze area. Note that the processing of step S103 and the processing of step S104 can be performed in any order.

[0048] The embedding order of the auxiliary information essentially indicates the processing order of the attention area. Specifically, the control unit 104 controls the processing order of the attention area in the video decoder 200 by controlling the embedding order of the auxiliary information in the auxiliary information bitstream. This is because the video decoder 200 performs the decoding process of the attention area in accordance with the embedding order of the auxiliary information.

[0049] The multiplexer 103 multiplexes the video bitstream supplied from the internal video encoder 102 and the auxiliary information bitstream supplied from the control unit 104 to generate and output a multiplexed bitstream (step S105).

[0050] [Description on the Decoding Side] In the video decoder 200, the demultiplexer 203 demultiplexes the bitstream to obtain a video bitstream and a side information bitstream. The side information bitstream includes a plurality of pieces of side information, such as those illustrated in FIG. 4, for a plurality of attention regions. As described above, the embedding order of the plurality of pieces of side information essentially indicates the processing order of the attention regions. This is because the video decoder 200 performs post-processing on the attention regions according to the embedding order of the side information.

[0051] The internal video decoder 202 decodes the video bitstream supplied from the demultiplexer 203 to obtain decoded video of the internal video encoding.

[0052] The control unit 204 decodes the auxiliary information bitstream to obtain the auxiliary information, and uses the auxiliary information as control information for post-processing. As described above, the control information includes the coordinates of the upper left and lower right corners of the gaze area and the magnification.

[0053] The post-processing unit 201 applies post-processing based on control information (corresponding to auxiliary information) supplied from the control unit 204 to the decoded video of the internal video encoding to obtain a final decoded video. That is, the post-processing unit 201 enlarges the gaze area at the position indicated by the coordinate information based on a magnification factor. The enlargement is achieved, for example, by sample interpolation (upsampling).

[0054] By the processing of the post-processing unit 201, the gaze area in the original image is restored as shown in FIG. 3D.

[0055] Next, a description will be given of the operation of the video decoder 200. FIG.

[0056] The demultiplexer 203 demultiplexes the bitstream to obtain a video bitstream and a side information bitstream (step S201).

[0057] The internal video decoder 202 decodes the video bitstream supplied from the demultiplexer 203 to obtain decoded video of the internal video encoding (step S202).

[0058] The control unit 204 decodes the auxiliary information bitstream supplied from the demultiplexer 203 to obtain control information (corresponding to auxiliary information) for post-processing (step S203). Note that the order of the processing in step S202 and the processing in step S203 is not important.

[0059] The post-processing unit 201 applies post-processing based on the control information supplied from the control unit 204 to the decoded video of the internal video encoding supplied from the internal video decoder 202, thereby obtaining the final decoded video (step S204).

[0060] As described above, in this embodiment, the video encoder 100 reduces the amount of information to be transmitted by reducing the region of interest. Furthermore, in this embodiment, the video encoder 100 controls the processing order of the region of interest, for example, by controlling the setting order of auxiliary information related to the region of interest in the auxiliary information bitstream, allowing the video decoder 200 to uniquely identify the magnification of the overlapping region. Specifically, on the video decoding side, the magnification of the image of the overlapping region in the region of interest becomes the same as the magnification of the region of interest with the larger magnification value. In other words, the video decoder 200 can faithfully reproduce the region of interest in the original image.

[0061] [Variation 1] In the above-described embodiment, the control unit 104 in the video encoder 100 uses the setting order of the auxiliary information in the auxiliary information bitstream to perform control so that the magnification of an overlapping area spanning attention areas with different magnifications can be uniquely identified in post-processing on the video decoding side. However, the video encoder 100 may also apply a larger magnification among the magnifications of multiple attention areas as the magnification of the overlapping area so that the image quality of the overlapping area is high.

[0062] In this case, the control unit 104 does not explicitly change the auxiliary bitstream, and the control unit 204 in the video decoder 200 performs control so that, for an overlapping area spanning attention areas with different magnifications, the larger magnification is applied as the magnification of the overlapping area. Note that not explicitly changing the auxiliary bitstream means that the control unit 104 does not perform control such as setting auxiliary information of a attention area with a lower priority first in the auxiliary bitstream.

[0063] According to this modification, the overlapping area restored on the video decoding side has higher image quality.

[0064] The above-described embodiment can be configured by hardware, but can also be realized by a computer program.

[0065] The information processing system shown in Fig. 7 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing a bitstream. The storage medium 703 and the storage medium 704 may be separate storage media or may be storage areas formed by the same storage medium. A magnetic storage medium such as a hard disk can be used as the storage medium. The information processing system may also include multiple processors 701.

[0066] In the information processing system, a program memory 702 stores a program (video encoding program or video decoding program) for realizing the functions of each block shown in the above embodiment.

[0067] The processor 701 executes processing in accordance with the program stored in the program memory 702, thereby realizing the functions of the video encoder 100 or the video decoder 200 described in the above embodiments.

[0068] For example, the processor 701 executes processing in accordance with a video encoding program for implementing the functions of each block in the video encoder 100 shown in FIG. 2, thereby implementing the functions of the video encoder 100.

[0069] Also, for example, the processor 701 executes processing in accordance with a video decoding program for realizing the functions of each block in the video decoder 200 shown in FIG. 2, thereby realizing the functions of the video decoder.

[0070] At least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transitory computer-readable medium. The program is supplied to the transitory computer-readable medium, for example, via a wired or wireless communication channel, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0071] 8 is a block diagram showing the main components of a video encoding device 10 (implemented by a video encoder 100 in the embodiment) including a control unit 11 (implemented by a control unit 104 in the embodiment) that controls the processing order of a plurality of attention regions.

[0072] 9 is a block diagram showing the main components of a video decoding device 20 (implemented by a video decoder 200 in the embodiment) including a control unit 21 (implemented by a control unit 204 in the embodiment) that controls the processing order of a plurality of attention regions.

[0073] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0074] (Supplementary Note 1) A video encoding device for reducing an amount of information of a gaze area in a video, the video encoding device comprising: a control means for controlling a processing order of the plurality of gaze areas.

[0075] (Supplementary Note 2) The video encoding device according to Supplementary Note 1, wherein the control means controls the processing order in accordance with a reduction ratio of the attention area.

[0076] (Supplementary Note 3) The video encoding device according to Supplementary Note 2, wherein the control means controls the attention area with a smaller reduction rate so that the attention area with a larger reduction rate is processed later.

[0077] (Supplementary Note 4) A video decoding device for reducing an amount of information of a gaze area in a video, the video decoding device comprising: a control means for controlling a processing order of the plurality of gaze areas.

[0078] (Supplementary Note 5) The video decoding device according to Supplementary Note 4, wherein the control means controls the processing order according to a reduction ratio of the gaze area.

[0079] (Supplementary Note 6) The video decoding device according to Supplementary Note 5, wherein the control means controls so that the attention area with a smaller reduction rate is processed after the attention area with a larger reduction rate.

[0080] (Supplementary Note 7) A video coding method for reducing an amount of information of a fixation area in a video, the video coding method comprising: controlling a processing order of a plurality of the fixation areas.

[0081] (Supplementary Note 8) A video decoding method for reducing an amount of information of a focus area in a video, the video decoding method comprising: controlling a processing order of a plurality of the focus areas.

[0082] (Supplementary Note 9) A video encoding program for reducing the amount of information in a gaze area in a video, the video encoding program causing a computer to control the processing order of a plurality of the gaze areas.

[0083] (Supplementary Note 10) A video decoding program for reducing the amount of information of a gaze area in a video, the video decoding program causing a computer to control the processing order of a plurality of the gaze areas.

[0084] (Supplementary Note 11) A recording medium having recorded thereon a video encoding program for reducing the amount of information of a gaze area in a video, the video encoding program causing a computer to control the processing order of a plurality of the gaze areas.

[0085] (Supplementary Note 12) A recording medium having recorded thereon a video decoding program for reducing the amount of information of a gaze area in a video, the video decoding program causing a computer to control the order of processing a plurality of the gaze areas.

[0086] (Supplementary Note 13) A recording medium for recording a bitstream generated by a video coding method for reducing the amount of information of a gaze area in a video, the video coding method controlling the processing order of a plurality of the gaze areas.

[0087] (Supplementary Note 14) A recording medium according to Supplementary Note 13, which records a bitstream generated by a video coding method that controls the processing order according to the reduction ratio of the gaze area.

[0088] (Appendix 15) A recording medium according to Appendix 14, which records a bitstream generated by a video encoding method that controls the processing of the gaze area with a smaller reduction ratio after the gaze area with a larger reduction ratio.

[0089] Some or all of the configurations described in Supplementary Notes 2 and 3, which are dependent on Supplementary Note 1, may be made dependent on Supplementary Notes 7, 9, and 11 in the same dependency relationship as Supplementary Notes 2 and 3. Furthermore, some or all of the configurations described in Supplementary Notes 5 and 6, which are dependent on Supplementary Note 4, may be made dependent on Supplementary Notes 8, 10, and 12 in the same dependency relationship as Supplementary Notes 5 and 6. Furthermore, without being limited to Supplementary Notes 1, 4, and 7 to 12, some or all of the configurations described as the above Supplements may be made dependent on various hardware, software, various recording means for recording software, or systems, provided that they do not deviate from the above-described embodiments.

[0090] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0091] This application claims priority based on Japanese Patent Application No. 2024-061470, filed April 5, 2024, the disclosure of which is incorporated herein in its entirety.

[0092] REFERENCE SIGNS LIST 10 Video encoding device 11 Control means 20 Video decoding device 21 Control means 100 Video encoder 101 Pre-processing unit 102 Internal video encoder 103 Multiplexer 104 Control unit 200 Video decoder 201 Post-processing unit 202 Internal video decoder 203 Demultiplexer 204 Control unit 701 Processor 702 Program memory 703, 704 Storage medium

Claims

1. A video encoding device that reduces the amount of information in a gaze area in a video, the video encoding device comprising a control means that controls the processing order of a plurality of the gaze areas.

2. The video encoding device according to claim 1, wherein said control means controls the processing order according to the reduction ratio of said fixation area.

3. The video encoding device according to claim 2, wherein said control means controls so that the region of interest with a smaller reduction rate is processed after the region of interest with a larger reduction rate.

4. A video decoding device that reduces the amount of information of a gaze area in a video, the video decoding device comprising: a control means that controls the processing order of a plurality of the gaze areas.

5. The video decoding device according to claim 4, wherein the control means controls the processing order according to the reduction ratio of the fixation area.

6. The video decoding device according to claim 5, wherein said control means controls so that the gaze area with a smaller reduction rate is processed after the gaze area with a larger reduction rate.

7. A video coding method for reducing the amount of information of a gaze area in a video, the video coding method comprising controlling the order of processing a plurality of the gaze areas.

8. A video decoding method for reducing the amount of information of a gaze area in a video, the video decoding method comprising: controlling the order in which a plurality of the gaze areas are processed.

9. A video encoding program for reducing the amount of information in a gaze area in a video, the video encoding program causing a computer to control the processing order of a plurality of the gaze areas.

10. A video decoding program for reducing the amount of information in a gaze area in a video, the video decoding program causing a computer to control the processing order of multiple gaze areas.

11. A recording medium for recording a bitstream generated by a video encoding device that reduces the amount of information in a gaze area in a video, the video encoding device having a control means for controlling the processing order of a plurality of the gaze areas.

Images