Method and apparatus for video encoding / decoding for optimized post-processing

Abstract

The present invention provides a method and an apparatus for video encoding / decoding for optimized post-processing, which can be used in video encoding and decoding areas. A video decoding method according to an embodiment of the present invention includes the steps of: receiving a bitstream including an encoded image and supplemental enhancement information of the encoded image; extracting the encoded image and the supplemental enhancement information from the bitstream; obtaining a decoded image by decoding the encoded image; and performing post-processing on the decoded image on the basis of the supplemental enhancement information, wherein the supplemental enhancement information includes information on at least one of a post-processing method and a post-processing purpose for the encoded image.

Description

Video encoding / decoding method and device for optimized post-processing

[0001] The present invention relates to the field of encoding and decoding of digital video, and relates to a method for encoding and decoding digital video, a method for recording such data, and components, devices, and systems for realizing such a method.

[0002] The present invention may be in the same technical field as at least one of the digital video compression technology standards known by the names of standards such as MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG, or may be in the technical field for improving the inherent efficiency of the standards, or may be in the technical field for improving or replacing the standards.

[0003] Digital video encoding and decoding are widely used in various digital video applications. For example, digital television broadcasting, video transmission over communication networks, video calls / video conversations / video chats, recording and providing video content using optical media including video compact discs (VCDs), digital versatile discs (DVDs), and Blu-Rays, all processes for producing, editing, collecting, and distributing video content, and devices such as video recording devices and camcorders for various purposes, including personal, commercial, industrial, and security purposes, all depend on video encoding and decoding technologies.

[0004] Accordingly, implementations that may be referred to as digital video encoders and decoders may form part of a wide range of devices related to the generation, recording, and provision of digital video, including digital televisions, digital broadcasting systems, wireless broadcasting systems, computers in the form of notebooks / desktops / tablets, e-book readers, digital cameras, digital recording devices, digital multimedia playback devices, video game devices / terminals / consoles, mobile phones (including smartphones) with multimedia playback capabilities, equipment for video conferencing, and other devices.

[0005] The above digital video encoders and decoders can be implemented by a digital video compression standard that is widely used and understood by those skilled in the art. The digital video compression standard may include at least one of compression standards known by a standard name such as MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG.

[0006] Video encoders and decoders can be implemented to more efficiently encode or decode digital video information while complying with the above standards, or by improving or modifying the above standards. Attempts to modify the above standards can also lead to the development of new standards. A well-known example is the so-called enhanced compression model (ECM), an attempt to improve and replace the existing H.266 / VVC standard, currently being developed by the Joint Video Experts Team (JVET), a joint international standardization group of ISO, IEC, and ITU-T.

[0007] Digital video is known to require a significant amount of information to describe its content in its uncompressed state. Therefore, recording or transmitting this information in its original form can be inefficient. Therefore, digital video is compressed using various methods prior to recording or transmission. These compression methods include lossy and lossless encoding. Lossy encoding sacrifices some image quality to achieve high compression performance, while lossless encoding sacrifices some compression performance to prevent image quality degradation. Regardless of the encoding method, there is a need to implement a technology that achieves a high compression ratio while minimizing image quality degradation to meet the growing demand for high-quality digital video within the constraints of limited memory storage capacity and communication transmission bandwidth.

[0008] As described above, the encoding process for compression requires various operations, such as spatial segmentation of digital video, segmentation and / or processing in color channels, removal of spatial redundancy, removal of temporal redundancy, tracking of motion vectors within the video, encoding of differential images, quantization, coefficient scan, run-length coding, entropy coding, and loop filtering. These encoding operations generally consume computing resources and take a certain amount of time to complete. Similarly, the decoding operation for the encoding operation also requires certain computing resources and a certain amount of time. The main goal of video encoding and decoding technology is to ensure that the above resource consumption and time consumption do not interfere with the production, recording, distribution, and viewing of digital videos.

[0009] Accordingly, the present invention provides a new technology that can contribute to at least one of the following: improvement of encoding efficiency, improvement of decoding efficiency, improvement of video quality, reduction of computational amount, reduction of software size, reduction of hardware size, and improvement of other performances related to encoding and decoding in the technical tasks in the field of video encoding and decoding.

[0010] According to an embodiment of the present invention for solving the above-described technical problem, a video decoding method includes the steps of receiving a bitstream including an encoded image and supplemental enhancement information about the encoded image, extracting the encoded image and the supplemental enhancement information from the bitstream, decoding the encoded image to obtain a decoded image, and performing post-processing on the decoded image based on the supplemental enhancement information, wherein the supplemental enhancement information may include information about at least one of a post-processing method and a post-processing purpose for the encoded image.

[0011] The encoded image and the auxiliary enhancement information may be characterized in that they are combined by a high-level syntax within the bit string.

[0012] The above auxiliary improvement information may include information related to at least one of general visual quality improvement, chroma upsampling, resolution resampling, picture rate upsampling, bit depth upsampling, and colorization.

[0013] The above post-processing is performed by a neural network post-processing filter, and the auxiliary improvement information may include information describing at least one neural network included in the neural network post-processing filter.

[0014] The above post-processing may be sequentially executed by a plurality of neural network-based post-processing filters, and the auxiliary improvement information may be characterized by including descriptive information about at least one of the plurality of neural networks.

[0015] The encoded image may be an image processed from a first media format into an image by a preprocessing technique, the postprocessing includes a process of restoring the decoded image to the first media format, and the auxiliary improvement information may be characterized in that it includes information related to at least one postprocessing method related to restoring the decoded image to the first media format.

[0016] The method may be such that the first media format is a multi-view image, and the auxiliary enhancement information may include at least one of information for generating a three-dimensional view from the encoded image, information for merging images for multiple viewpoints, and information for imparting a three-dimensional effect.

[0017] The above method may be characterized in that the first medium format is a holographic image, and the encoded image includes at least one of real information, imaginary information, amplitude information, and phase information expressing the holographic image.

[0018] The method may be characterized in that the first media format is neural network information, and the encoded image includes information visualizing at least one of the parameters of the neural network constituting the neural network information and the feature values ​​of the neural network.

[0019] The above auxiliary improvement information may be characterized in that it includes at least one bit string information generated by the encoder of the first media format in the same format.

[0020] A method in which the first media format is a media format included in at least one standard technical specification among MIV (MPEG Immersive Video), INVR (Implicit Neural Visual Representation), NNC (Neural Network Compression), VCM (Video Coding for Machines), and PCC (Point Cloud Coding).

[0021] The above post-processing purpose may include one of a first purpose for viewing the decoded image or a second purpose for a vision task based on the decoded image, and the auxiliary improvement information may be characterized in that it includes instruction information indicating one of the first purpose and the second purpose.

[0022] The method may further include, when the instruction information indicates the second purpose, information indicating at least one of object recognition, object detection, object segmentation, information retrieval, and captioning by the computer vision.

[0023] The above auxiliary improvement information may be characterized by including information indicating different post-processing methods for at least two segmentation units included in the encoded image.

[0024] A video encoding method according to an embodiment of the present invention for solving the above-described technical problem may include a step of determining at least one of a post-processing method and a post-processing purpose for an input image, a step of generating auxiliary enhancement information for the input image based on at least one of the method and the purpose, a step of encoding the input image to obtain an encoded image, and a step of generating a bit string including the encoded image and the auxiliary enhancement information.

[0025] The above method may further include a step of performing preprocessing on the input image based on the above method and purpose.

[0026] According to an embodiment of the present invention for solving the above-described technical problem, a video decoding device includes a receiving unit that receives a bitstream including an encoded image and supplementary enhancement information about the encoded image, a high-level syntax parsing unit that extracts the encoded image and the supplementary enhancement information from the bitstream, a decoding unit that decodes the encoded image to obtain a decoded image, and a postprocessing unit that performs postprocessing on the decoded image based on the supplementary enhancement information, wherein the supplementary enhancement information may include information about at least one of a postprocessing method and a postprocessing purpose for the encoded image.

[0027] The above post-processing unit is configured based on a neural network post-processing filter, and the auxiliary improvement information may include information describing at least one neural network included in the neural network post-processing filter.

[0028] The encoded image may be an image processed from a first media format into an image using a preprocessing technique, the post-processing unit may be configured to restore the decoded image to the first media format, and the auxiliary improvement information may be characterized by including information related to at least one post-processing method related to restoring the decoded image to the first media format.

[0029] The above post-processing purpose may include one of a first purpose for viewing the decoded image or a second purpose for a vision task based on the decoded image, and the auxiliary improvement information may be characterized in that it includes instruction information indicating one of the first purpose and the second purpose.

[0030] According to the present invention, in the technical task of the video encoding and decoding field, it is possible to obtain advantageous effects in at least one of improvement in encoding efficiency, improvement in decoding efficiency, improvement in video quality, reduction in computational amount, reduction in software size, reduction in hardware size, and improvement in other performances related to encoding and decoding.

[0031] Figure 1 is a conceptual diagram of a video communication system according to one embodiment of the present invention;

[0032] FIG. 2 is a conceptual diagram of the arrangement of an encoder and decoder in a real-time video streaming environment according to one embodiment of the present invention.

[0033] Figure 3 is a functional unit conceptual diagram of a video decoder according to one embodiment of the present invention;

[0034] Figure 4 is a functional unit conceptual diagram of a video encoder according to one embodiment of the present invention;

[0035] Figure 5 is a conceptual diagram of a frame type according to one embodiment of the present invention;

[0036] Figure 6 is a conceptual diagram showing the structure of a video encoder using VVC.

[0037] Figure 7 is a conceptual diagram of preprocessing and postprocessing of a video encoding / decoding process according to one embodiment of the present invention.

[0038] FIG. 8 is an exemplary diagram showing a video-mediated application information processing method according to one embodiment of the present invention;

[0039] Figure 9 is an exemplary diagram showing an example of using the MIV standard according to one embodiment of the present invention;

[0040] Figure 10 is an exemplary diagram showing the use of neural network information according to one embodiment of the present invention;

[0041] FIG. 11 is an exemplary diagram of a method for optimizing a post-processing filter and a post-processing method according to one embodiment of the present invention.

[0042] FIG. 12 is an exemplary diagram of another method for optimizing a post-processing filter and a post-processing method according to one embodiment of the present invention;

[0043] FIG. 13 is an exemplary diagram showing an example of application of input information for multiple post-processors according to one embodiment of the present invention; and

[0044] Figure 14 is an exemplary diagram showing an example of application of different post-processing methods according to one embodiment of the present invention.

[0045] The video can be used by dividing it into a video for viewing and a video for non-vision mission. Referring to FIG. 14, after decoding the video, the decoder (1430) extracts information included in the SEI accompanying the video bit stream (1420), and / or information of the decoding process derived from the video bit stream (1420), such as quantization parameter information, motion information, mode information, or filtering information within the loop, and according to the analysis (1440) of the post-processing information depending on whether the purpose of use of the video is viewing by a human or execution of a vision mission by a computer device, the decoder can be configured to supply necessary information to a corresponding appropriate post-processor (1445, 1446) and operate the corresponding post-processor. Preferably, the distinction for the purpose of use can be made by information derived from the process of preprocessing performed in the encoder (1410) that encodes the image (1420) to be decoded by the decoder (1430) and / or in the preprocessor (1405).

[0046]

[0047] Although the present invention has been described with reference to drawings and embodiments, as already mentioned above, it does not mean that the scope of protection of the present invention is limited to the drawings or embodiments presented above, and it will be understood that a person skilled in the relevant technical field can modify and change the present invention in various ways within a scope that does not depart from the spirit and scope of the present invention described in the claims of the present invention patent.

Claims

1. In the video decryption method, A step of receiving a bitstream including an encoded image and supplemental enhancement information regarding the encoded image; A step of extracting the encoded image and the auxiliary enhancement information from the bit string; A step of decoding the encoded image to obtain a decoded image; and A step of performing post-processing on the decoded image based on the above auxiliary improvement information; including, A method characterized in that the auxiliary improvement information includes information on at least one of a post-processing method and a post-processing purpose for the encoded image.

2. In paragraph 1, A method, characterized in that the encoded image and the auxiliary enhancement information are combined by a high-level syntax within the bit string.

3. In paragraph 1, The above auxiliary improvement information is, A method comprising information related to at least one of general visual quality improvement, chroma upsampling, resolution resampling, picture rate upsampling, bit depth upsampling, and colorization.

4. In paragraph 1, The above post-processing is performed by a neural network post-filter, A method wherein the auxiliary improvement information includes information describing at least one neural network included in the neural network-based post-processing filter.

5. In paragraph 4, The above post-processing is sequentially executed by multiple neural network-based post-processing filters, A method, characterized in that the auxiliary improvement information includes descriptive information about at least one of the plurality of neural networks.

6. In paragraph 1, The above encoded image is an image processed from a first media format into an image using a preprocessing technique, The above post-processing includes a process of restoring the decrypted image to the first media format, A method characterized in that the auxiliary improvement information includes information related to at least one post-processing method related to restoring the decrypted image into the first media format.

7. In paragraph 6, The above first media format is a multi-view video, A method wherein the auxiliary improvement information includes at least one of information for generating a three-dimensional viewpoint from the encoded image, information for merging images for multiple viewpoints, and information for imparting a three-dimensional effect.

8. In paragraph 6, The above first media format is a holographic image, A method characterized in that the encoded image includes at least one of real information, imaginary information, amplitude information, and phase information representing the holographic image.

9. In paragraph 6, The above first media format is neural network information, A method characterized in that the encoded image includes information visualizing at least one of the parameters of the neural network and the feature values ​​of the neural network that constitute the neural network information.

10. In paragraph 6, A method characterized in that the auxiliary improvement information includes at least one bit string information generated by the encoder of the first media format in the same format.

11. In paragraph 10, A method in which the first media format is a media format included in at least one standard technical specification among MIV (MPEG Immersive Video), INVR (Implicit Neural Visual Representation), NNC (Neural Network Compression), VCM (Video Coding for Machines), and PCC (Point Cloud Coding).

12. In paragraph 1, The above post-processing purpose includes one of the first purpose for viewing the decoded image or the second purpose for a vision task based on the decoded image, A method characterized in that the auxiliary improvement information includes instruction information indicating one of the first purpose and the second purpose.

13. In paragraph 12, If the above instruction information indicates the second purpose, A method wherein the auxiliary improvement information further includes information indicating at least one of object recognition, object detection, object segmentation, information retrieval, and captioning by the computer vision.

14. In paragraph 1, The above auxiliary improvement information is, A method characterized in that it includes information indicating different post-processing methods for at least two segmentation units included in the encoded image.

15. In a video encoding method, A step of determining at least one of a post-processing method and a post-processing purpose for an input image; A step of generating auxiliary improvement information for the input image based on at least one of the above methods and purposes; A step of encoding the input image to obtain an encoded image; and A method comprising: generating a bit string including the encoded image and the auxiliary enhancement information.

16. In paragraph 15, A method further comprising a step of performing preprocessing on the input image based on the above method and purpose.

17. In a video decryption device, A receiving unit that receives a bitstream including an encoded image and supplemental enhancement information regarding the encoded image; A high-level syntax parsing unit that extracts the encoded image and the auxiliary enhancement information from the bit string; A decoding unit that decodes the encoded image to obtain a decoded image; and A post-processing unit that performs post-processing on the decoded image based on the above auxiliary improvement information; A device characterized in that the auxiliary improvement information includes information on at least one of a post-processing method and a post-processing purpose for the encoded image.

18. In paragraph 17, The above post-processing unit is configured based on a neural network post-filter, A device wherein the auxiliary improvement information includes information describing at least one neural network included in the neural network-based post-processing filter.

19. In paragraph 17, The above encoded image is an image processed from a first media format into an image using a preprocessing technique, The above post-processing unit is configured to restore the decrypted image into the first media format, A device characterized in that the auxiliary improvement information includes information related to at least one post-processing method related to restoring the decrypted image into the first media format.

20. In paragraph 17, The above post-processing purpose includes one of the first purpose for viewing the decoded image or the second purpose for a vision task based on the decoded image, A method characterized in that the auxiliary improvement information includes instruction information indicating one of the first purpose and the second purpose.

Images