Video processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video processing method and device, electronic equipment, a storage medium and a program product. The video processing method comprises the following steps: acquiring coding information of a coded video; on the basis of the coding information, a post-processing strategy executed for a decoded video is determined, post-processing strategy information is obtained, and the decoded video is a video obtained after the coding video is decoded by the decoding end; adding the post-processing policy information to supplemental enhancement information (SEI) of the encoded video; and sending the SEI to a decoding end to indicate the decoding end to perform post-processing on the decoded video according to the post-processing strategy information in the SEI. According to the method, the post-processing decision is made at the coding end with more sufficient information and computing power, so that the post-processing performance can be improved, and the video quality is enhanced. By adding the post-processing strategy information to the supplementary enhancement information as the selectable information, reliable transmission of the post-processing strategy information can be ensured, and decoding of the existing code stream according to the standard is not affected.

Description

Technical Field

[0001] The present disclosure relates to video coding and compression. More specifically, the present disclosure relates to a video processing method, device, electronic device, storage medium and program product. Background Art

[0002] Post-processing in the video coding link is to enhance the decoded video, and its main function is to improve the quality of the video and reduce the distortion and error caused by compression. The post-processing module is outside the video coding loop, that is, it is not a module specified by the video coding standard. It is independent of video coding and can be freely customized.

[0003] Existing post-processing is usually performed entirely by the decoding end, and the same post-processing is performed on all frames in the video, resulting in limited performance improvement due to the diversity of video content. Summary of the invention

[0004] Embodiments of the present disclosure provide a video processing method, apparatus, electronic device, storage medium, and program product to solve at least one of the above problems.

[0005] According to one aspect of the present disclosure, a video processing method is provided, which is applied to an encoding end, and the video processing method includes: obtaining encoding information of an encoded video; based on the encoding information, determining a post-processing strategy to be performed on a decoded video to obtain post-processing strategy information, wherein the decoded video is a video obtained after a decoding end decodes the encoded video; adding the post-processing strategy information to supplemental enhancement information (SEI) of the encoded video; and sending the SEI to a decoding end to instruct the decoding end to post-process the decoded video according to the post-processing strategy information in the SEI.

[0006] Optionally, the post-processing strategy information includes information of a post-processing strategy for a post-processing unit, wherein the post-processing unit includes at least one of an image group, a frame, and a region, and the region includes at least one of a strip, a tile, and a block.

[0007] Optionally, adding the post-processing strategy information to the supplemental enhancement information (SEI) of the encoded video includes: adding the post-processing strategy information to the SEI of the level corresponding to the post-processing unit or a higher level; or for multiple consecutive post-processing units using the same post-processing strategy, adding the post-processing strategy information to the SEI corresponding to the starting post-processing unit and the SEI corresponding to the ending post-processing unit among the multiple post-processing units.

[0008] Optionally, the post-processing strategy includes at least one of a skip strategy and a video quality processing strategy, wherein the skip strategy indicates whether to skip post-processing, and the video quality processing strategy indicates which post-processing solution to use.

[0009] Optionally, the post-processing scheme includes at least one of the following algorithms: a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a denoising algorithm, and a color enhancement algorithm, and each algorithm includes models of different strengths and / or models of different structures.

[0010] Optionally, in the case that the post-processing strategy includes the skip strategy, determining the post-processing strategy to be executed for the decoded video based on the encoding information includes: based on the encoding information, determining a post-processing unit that meets a preset skip condition as a skip post-processing unit, wherein the preset skip condition includes a condition indicating that the human eye is insensitive to the corresponding image.

[0011] Optionally, determining the post-processing strategy to be executed on the decoded video based on the encoding information also includes: determining other post-processing units that are similar to the post-processing unit that satisfies the preset skip condition as skip post-processing units; and / or if there are multiple skip post-processing units belonging to the same higher-level post-processing unit, and the proportion of the multiple skip post-processing units in the higher-level post-processing units exceeds a second proportion threshold, then determining the higher-level post-processing unit as a skip post-processing unit.

[0012] Optionally, determining the post-processing strategy to be executed on the decoded video based on the encoding information also includes: performing quality detection processing on the post-processing units that do not meet the preset skipping conditions based on the encoding information, and determining the post-processing units that pass the quality detection as skipped post-processing units.

[0013] Optionally, in the case where the post-processing strategy includes the video quality processing strategy, the post-processing strategy to be executed on the decoded video is determined based on the encoding information, including: in the case where it is determined based on the encoding information that the post-processing unit has a preset quality problem, using a preset post-processing scheme corresponding to the preset quality problem as the post-processing scheme of the post-processing unit; and / or post-processing the post-processing unit according to multiple different post-processing schemes, respectively, and selecting the post-processing scheme with the best post-processing quality as the post-processing scheme of the post-processing unit.

[0014] According to another aspect of the present disclosure, a video processing method is provided, which is applied to a decoding end, and the video processing method includes: receiving supplemental enhancement information (SEI) sent by an encoding end, wherein the SEI includes post-processing strategy information, and the post-processing strategy information represents a post-processing strategy executed after decoding the encoded video, and the SEI is obtained by processing the video processing method applied to the encoding end as described above; after decoding the encoded video, post-processing the decoded video based on the post-processing strategy information.

[0015] Optionally, the post-processing strategy information includes information of a post-processing strategy for a post-processing unit, wherein the post-processing unit includes at least one of an image group, a frame, and a region, and the region includes at least one of a strip, a tile, and a block.

[0016] Optionally, the video processing method also includes: obtaining the post-processing strategy information from the SEI of the level corresponding to the post-processing unit or a higher level; or, for multiple consecutive post-processing units using the same post-processing strategy, obtaining the post-processing strategy information of the multiple post-processing units from the SEI corresponding to the starting post-processing unit and the SEI corresponding to the ending post-processing unit among the multiple post-processing units.

[0017] Optionally, the post-processing strategy includes at least one of a skip strategy and a video quality processing strategy, wherein the skip strategy indicates whether to skip post-processing, and the video quality processing strategy indicates which post-processing solution to use.

[0018] Optionally, the post-processing scheme includes at least one of the following algorithms: a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a denoising algorithm, and a color enhancement algorithm, and each algorithm includes models of different strengths and / or models of different structures.

[0019] According to another aspect of the present disclosure, there is provided a video processing device, which is applied to an encoding end, and the video processing device includes: an acquisition unit, configured to acquire encoding information of an encoded video; a determination unit, configured to determine a post-processing strategy to be performed on a decoded video based on the encoding information, and obtain post-processing strategy information, wherein the decoded video is a video obtained after the decoding end decodes the encoded video; an adding unit, configured to add the post-processing strategy information to the supplemental enhancement information (SEI) of the encoded video; and a sending unit, configured to send the SEI to the decoding end, so as to instruct the decoding end to post-process the decoded video according to the post-processing strategy information in the SEI.

[0020] Optionally, the post-processing strategy information includes information of a post-processing strategy for a post-processing unit, wherein the post-processing unit includes at least one of an image group, a frame, and a region, and the region includes at least one of a strip, a tile, and a block.

[0021] Optionally, the adding unit is also configured to: add the post-processing strategy information to the SEI of the corresponding level or higher level of the post-processing unit; or for multiple consecutive post-processing units using the same post-processing strategy, add the post-processing strategy information to the SEI corresponding to the starting post-processing unit and the SEI corresponding to the ending post-processing unit among the multiple post-processing units.

[0022] Optionally, the post-processing strategy includes at least one of a skip strategy and a video quality processing strategy, wherein the skip strategy indicates whether to skip post-processing, and the video quality processing strategy indicates which post-processing solution to use.

[0023] Optionally, the post-processing scheme includes at least one of the following algorithms: a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a denoising algorithm, and a color enhancement algorithm, and each algorithm includes models of different strengths and / or models of different structures.

[0024] Optionally, in the case that the post-processing strategy includes the skip strategy, the determination unit is further configured to: based on the encoding information, determine a post-processing unit that satisfies a preset skip condition as a skip post-processing unit, wherein the preset skip condition includes a condition indicating that the human eye is insensitive to the corresponding image.

[0025] Optionally, the determination unit is further configured to: determine other post-processing units that are similar to the post-processing unit that meets the preset skip condition as skipped post-processing units; and / or if there are multiple skipped post-processing units belonging to the same higher-level post-processing unit, and the proportion of the multiple skipped post-processing units in the higher-level post-processing units exceeds a second proportion threshold, then determine the higher-level post-processing unit as a skipped post-processing unit.

[0026] Optionally, the determination unit is further configured to: based on the coding information, perform quality detection processing on the post-processing units that do not meet the preset skipping condition, and determine the post-processing units that pass the quality detection as skipped post-processing units.

[0027] Optionally, in the case where the post-processing strategy includes the video quality processing strategy, the determination unit is further configured to: when it is determined based on the encoding information that the post-processing unit has a preset quality problem, use a preset post-processing scheme corresponding to the preset quality problem as the post-processing scheme of the post-processing unit; and / or post-process the post-processing unit according to multiple different post-processing schemes, respectively, and select the post-processing scheme with the best post-processing quality as the post-processing scheme of the post-processing unit.

[0028] According to another aspect of the present disclosure, a video processing device is provided, which is applied to a decoding end, and the video processing device includes: a receiving unit, configured to receive supplemental enhancement information (SEI) sent by an encoding end, wherein the SEI includes post-processing strategy information, and the post-processing strategy information represents a post-processing strategy executed after decoding the encoded video, and the SEI is obtained by processing the video processing method applied to the encoding end as described above; a processing unit, configured to post-process the decoded video based on the post-processing strategy information after decoding the encoded video.

[0029] Optionally, the post-processing strategy information includes information of a post-processing strategy for a post-processing unit, wherein the post-processing unit includes at least one of an image group, a frame, and a region, and the region includes at least one of a strip, a tile, and a block.

[0030] Optionally, the video processing device also includes a reading unit configured to: obtain the post-processing strategy information from the SEI of the level corresponding to the post-processing unit or a higher level; or for multiple consecutive post-processing units using the same post-processing strategy, obtain the post-processing strategy information of the multiple post-processing units from the SEI corresponding to the starting post-processing unit and the SEI corresponding to the ending post-processing unit among the multiple post-processing units.

[0031] Optionally, the post-processing strategy includes at least one of a skip strategy and a video quality processing strategy, wherein the skip strategy indicates whether to skip post-processing, and the video quality processing strategy indicates which post-processing solution to use.

[0032] Optionally, the post-processing scheme includes at least one of the following algorithms: a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a denoising algorithm, and a color enhancement algorithm, and each algorithm includes models of different strengths and / or models of different structures.

[0033] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the video processing method as described above.

[0034] According to another aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the video processing method as described above.

[0035] According to another aspect of the present disclosure, a computer program product is provided, comprising computer instructions, and when the computer instructions are executed by at least one processor, the video processing method as described above is implemented.

[0036] According to another aspect of the present disclosure, a computer program product is provided, having instructions for storing a bit stream, wherein the bit stream includes: supplemental enhancement information (SEI) of a coded video generated according to the video processing method applied to the encoding end as described above; or supplemental enhancement information received according to the video processing method applied to the decoding end as described above.

[0037] According to another aspect of the present disclosure, a method for generating a bitstream is provided, comprising: generating supplemental enhancement information (SEI) of an encoded video according to the video processing method applied to an encoding end as described above.

[0038] According to another aspect of the present disclosure, a method for processing a bit stream is provided, comprising: receiving a bit stream; and decoding and post-processing the bit stream according to the video processing method applied to a decoding end as described above.

[0039] According to the video processing method, device, electronic device, storage medium and program product of the exemplary embodiments of the present disclosure, by making post-processing decisions at the encoding end with more information and computing power, and then transmitting the post-processing strategy information to the decoding end for post-processing, the post-processing performance can be improved and the video quality can be enhanced. By adding the post-processing strategy information to the supplementary enhancement information as optional information, the reliable transmission of the post-processing strategy information can be guaranteed without affecting the decoding of the existing code stream according to the standard.

[0040] It is to be understood that both the foregoing general description and the following detailed description are examples only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate examples according to the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0042] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

[0043] Figure 2is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0044] Figure 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0045] Figure 4 It is a schematic diagram showing the overall framework flow of a video link of the related technology.

[0046] Figure 5 is a schematic diagram showing an exemplary overall framework flow of a video link according to some embodiments of the present disclosure.

[0047] Figure 6 is a flowchart illustrating an exemplary video processing method applied to an encoding end according to some embodiments of the present disclosure.

[0048] Figure 7 is a flowchart illustrating an exemplary video processing method applied to a decoding end according to some embodiments of the present disclosure.

[0049] Figure 8 is a block diagram showing an exemplary video processing device applied to an encoding end according to some embodiments of the present disclosure.

[0050] Fig. 9 is a block diagram showing an exemplary video processing device applied to a decoding end according to some embodiments of the present disclosure.

[0051] Fig.10 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0052] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the drawings are used to distinguish objects, but not to describe any specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure.

[0054] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0055] In some implementations, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14.

[0056] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28.

[0057] like Figure 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as, or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video.

[0058] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0059] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0060] The video encoder 20 and the video decoder 30 can operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or an extension of such a standard. It should be understood that the present disclosure is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data according to any of these current standards or future standards.

[0061] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0062] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present disclosure. The video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. Intra-frame prediction coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0063] like Figure 2 As shown in , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 such as a deblocking filter can be located between the adder 62 and the DPB 64 to filter the block boundary to remove the block effect from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can also be used to filter the output of the adder 62. In some examples, the loop filter may be omitted, and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0064] Video data memory 40 may store video data to be encoded by components of video encoder 20. Figure 1 Video source 18 is shown obtaining video data in video data memory 40. DPB 64 is a buffer that stores reference video data (eg, reference frames or pictures) for use by video encoder 20 in encoding the video data (eg, in intra or inter prediction coding modes).

[0065] like Figure 2 As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree (QT) structure. It should be noted that the terms "block" or "video block" used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0066] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0067] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0068] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame associated with a current block being encoded within the current frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vector determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine a block vector.

[0069] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0070] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0071] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform).

[0072] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0073] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0074] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0075] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0076] Figure 3 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present disclosure. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 2 The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0077] In some examples, embodiments of the present disclosure may be dispersed in one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present disclosure alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0078] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. Video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk).

[0079] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0080] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from a previously decoded block of the current frame.

[0081] When the video frame is encoded as an inter-frame prediction coding (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vector and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0082] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0083] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded.

[0084] Motion compensation unit 82 may also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video blocks to calculate interpolated values ​​for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 based on the received syntax elements and use these interpolation filters to produce the prediction blocks.

[0085] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0086] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video blocks in a given frame are then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (e.g., Figure 1 on a display device 34).

[0087] Figure 4 FIG. 1 is a schematic diagram showing the overall framework flow of a video link in the related art. Figure 4 As shown, in the existing video link, the cloud end (cloud) usually uses an encoder to encode the video frame, and the code stream usually complies with the video coding standard (such as H.264, H.265, etc.), and then the encoded code stream is transmitted to the decoding end (end) to be decoded by a decoder. Because the decoded video frame is distorted, the decoded video frame is usually post-processed and enhanced (the post-processing enhancement is outside the encoding and decoding loop), and finally the post-processing enhanced video frame is displayed and played.

[0088] Existing post-processing methods usually directly perform the same enhancement on all decoded video frames. Due to the limited computing power of the decoding end (such as the mobile phone end), the existing direct post-processing methods have two problems: First, some video frames are suitable for enhancement, such as some static scenes, which can significantly improve the quality of decoded video frames. However, some frames are not suitable for enhancement, such as some motion blurred scenes. Enhancement does not bring subjective visual quality gain, but increases the amount of post-processing calculations, affecting the mobile phone playback experience. Second, due to the diversity of video frames, different video frames or even different areas of the same frame may be suitable for different enhancement methods. Using the same post-processing enhancement method for all frames may limit the post-processing performance.

[0089] Based on this analysis, we are inspired to consider the characteristics of different video frames and use different post-processing strategies for different video frames or even different areas within the frame, including whether to do post-processing and what method to choose for post-processing, to improve the post-processing performance. To realize this idea, one way is to analyze the video frames on the decoding end and analyze different strategies according to the frame content, but due to the limited computing power of the decoding end, analyzing the video frames will bring huge complexity and affect the playback. Another way is to parse the information in the bitstream (such as quantization parameter QP, motion vector MV, etc.) during post-processing to decide what strategy to use for the frame, but there are also problems: 1) If all information is not stored during decoding, some information of some areas may be in the middle of the bitstream, and the entire bitstream needs to be parsed to obtain the required information; 2) If all information is stored during decoding, it will consume a lot of storage space; 3) The existing information in the bitstream may not be helpful for the post-processing strategy.

[0090] Based on the above analysis, the present disclosure proposes a method for end-cloud collaborative guidance post-processing. Figure 5 FIG. 2 is a schematic diagram showing an exemplary overall framework flow of a video link according to some embodiments of the present disclosure. Figure 5As shown, because the encoding end itself has a large amount of information about the frame content (such as MV, rate-distortion cost, degree of distortion, etc.), and the encoding end has sufficient computing power, it can be analyzed at the encoding end (cloud) to obtain the post-processing strategy for each frame and specific area, including whether to do post-processing and what method to choose for post-processing, and then the post-processing strategy information is transmitted to the decoding end for post-processing. However, if the post-processing strategy information is directly written into the existing bitstream in the form of a new syntax element, the bitstream will not meet the standard and cannot be widely disseminated. Therefore, the present disclosure also proposes to use supplemental enhancement information (SEI, Supplemental Enhancement Information) to transmit the post-processing strategy information to the decoding end, because SEI is supplementary optional information and is not in the video coding standard specification. Even without SEI information, the correct video can be decoded. Using SEI to transmit post-processing strategy information can greatly improve flexibility, which can not only ensure the reliable transmission of post-processing strategy information, but also will not affect the existing bitstream decoding according to the standard.

[0091] Figure 6 1 is a flowchart showing an exemplary video processing method applied to an encoding end according to some embodiments of the present disclosure. The video processing method can be implemented in the source device 12 (specifically, for example, a cloud server) described above.

[0092] Reference Figure 6 In step S601, the encoding information of the encoded video is obtained. The encoding information is the information generated in the process of encoding the video at the encoding end, including but not limited to QP, MV, rate distortion cost, distortion degree, etc., so as to provide rich data support for post-processing decisions.

[0093] In step S602, based on the coding information, a post-processing strategy to be executed on the decoded video is determined to obtain post-processing strategy information. The decoded video is a video obtained after the decoding end decodes the coded video.

[0094] Optionally, the post-processing strategy information includes information about the post-processing strategy for the post-processing unit. In other words, the post-processing strategy information includes two aspects of information, one is the post-processing unit targeted, and the other is the post-processing strategy executed for the post-processing unit. Among them, the post-processing unit includes at least one of a group of pictures (GOP), a frame, and a region, and the region includes at least one of a strip, a tile, and a block. By determining the post-processing strategy for the post-processing units of different coding levels (i.e., the picture group level, the frame level, and the region level), it is possible to implement differentiated post-processing for different post-processing units, thereby determining a corresponding post-processing strategy at a certain coding level according to the diversity of the video content, thereby improving the flexibility and adaptability of the post-processing and helping to improve the post-processing performance. It should be understood that a region is a part of a frame, which may be at least one of a strip, a tile, and a block. As an example, a block may further include at least one of CTU, CU, PU, ​​TU, CTB, CB, PB, and TB. It should also be understood that the post-processing strategy of the post-processing unit at the region level will be applied to the corresponding entire region; the post-processing strategy of the post-processing unit at the frame level will be applied to the corresponding entire frame; and the post-processing strategy of the post-processing unit at the image group level will be applied to all frames in the corresponding image group.

[0095] The content and determination process of the post-processing strategy will be introduced in detail later, which will not be expanded here.

[0096] In step S603, the post-processing strategy information is added to the supplemental enhancement information (SEI) of the encoded video. SEI is usually used at the frame level and the picture group level. As an example, the post-processing strategy information can be added to the frame level SEI. Of course, the post-processing strategy information can also be added to the picture group level and the region level SEI, which is not limited in the present disclosure.

[0097] Optionally, taking the aforementioned post-processing strategy information including information on post-processing strategies for post-processing units as an example, each post-processing unit has its own corresponding level, some post-processing units have relatively high levels, and some post-processing units have relatively low levels, for example, the level of the image group is higher than the level of the frame, and the level of the frame is higher than the level of the region. Step S603 may include: adding the post-processing strategy information to the SEI of the level corresponding to the post-processing unit or a higher level, that is, the information on the post-processing strategy for a post-processing unit can be added to the SEI of the level where the post-processing unit is located (that is, the information on the post-processing strategy for a post-processing unit is added to the SEI corresponding to the targeted post-processing unit, for example, the information on the post-processing strategy for a frame is added to the SEI corresponding to the frame, and at this time, the location where the information is added indicates the post-processing unit to which the information is targeted, and there is no need to specify it specifically), and it can also be added to the SEI of the level where the post-processing unit is located. In the SEI of a higher level of the post-processing unit (for example, adding the information of the post-processing strategy for a post-processing unit to the SEI corresponding to the high-level post-processing unit to which the targeted post-processing unit belongs, for example, adding the information of the post-processing strategy for a frame to the SEI corresponding to the image group to which the frame belongs, and for example, adding the information of the post-processing strategy for a region to the SEI corresponding to the frame to which the region belongs. At this time, since the high-level post-processing unit contains multiple low-level post-processing units, it is also necessary to indicate which specific post-processing unit the information is targeted for), to achieve flexible addition of post-processing strategies. It should be understood that at this time, the same SEI may include the information of the post-processing strategy of the post-processing unit at the level where it is located, and may also include the information of the post-processing strategy of the sub-units of the post-processing unit at a lower level than itself. For the latter, the low-level sub-unit targeted by the post-processing strategy needs to be indicated in the SEI. Alternatively, step S603 may also include: for a plurality of consecutive post-processing units using the same post-processing strategy, adding the post-processing strategy information to the SEI corresponding to the starting post-processing unit (i.e., the first post-processing unit) and the SEI corresponding to the ending post-processing unit (i.e., the last post-processing unit) among the plurality of post-processing units. In the case where a plurality of consecutive post-processing units (these post-processing units have the same level) use the same post-processing strategy, by considering the continuity of the time domain, the post-processing strategy information is added to the SEI of the starting post-processing unit and the ending post-processing unit, which can effectively reduce the amount of information added.

[0098] As an example, you can use only the former adding method, that is, add the post-processing strategy information to the SEI corresponding to a single post-processing unit, or use only the latter adding method, that is, for multiple consecutive post-processing units, consider the continuity in the time domain and add the post-processing strategy information only to the SEI corresponding to the starting post-processing unit and the ending post-processing unit. You can also use both adding methods at the same time.

[0099] For the latter adding method, if only this adding method is used, then by default, when the post-processing strategy information appears for an odd number of times (e.g., the 1st or 3rd time) in the entire video sequence, the corresponding post-processing strategy is executed starting from the corresponding post-processing unit, and when the post-processing strategy information appears for an even number of times (e.g., the 2nd or 4th time), the corresponding post-processing unit stops executing the post-processing strategy, that is, the post-processing strategy is no longer executed starting from the next post-processing unit after the even number of times the post-processing strategy information appears. If both adding methods are used at the same time, special marks, such as a start mark and an end mark, can be added to the latter adding method to indicate that multiple consecutive post-processing units use the same post-processing strategy, thereby distinguishing them from the information of the post-processing strategy for a single post-processing unit. In particular, for the case where multiple consecutive post-processing units are all frames in an image group, the image group's own mark can also be used to distinguish its start frame and end frame without adding a special mark in SEI, and the present disclosure does not limit this.

[0100] As an example, step S603 may include: adding information on post-processing strategies for regions and frames to the SEI of corresponding frames. At this time, the information on post-processing strategies for regions needs to indicate the region, for example, a CTU index may be used, or region coordinates may be used directly, and the present disclosure does not limit this; adding information on post-processing strategies for multiple consecutive frames to the SEI of the starting frame (i.e., the first frame) and the ending frame (i.e., the last frame) in these multiple frames. At this time, these multiple consecutive frames may be frames in the same image group, or may not be frames in the same image group, and the present disclosure does not limit this. By using the frame-level SEI as the object of adding post-processing strategy information, the consistency of information storage can be guaranteed, which facilitates the decoding end to identify the post-processing strategy information.

[0101] As an example, step S603 may include: adding information on post-processing strategies for multiple consecutive regions to the SEI of the starting region (i.e., the first region) and the ending region (i.e., the last region) among the multiple regions, thereby realizing the addition of post-processing strategy information for multiple consecutive regions.

[0102] In step S604, the SEI is sent to the decoding end to instruct the decoding end to post-process the decoded video according to the post-processing strategy information in the SEI.

[0103] Next, the content and determination process of the post-processing strategy are introduced.

[0104] In some embodiments, the post-processing strategy optionally includes at least one of a skip strategy and a video quality processing strategy. The skip strategy indicates whether to skip post-processing, and can skip post-processing of a specified post-processing unit, thereby reasonably reducing the application scope of post-processing, reducing the complexity of post-processing performed by the decoding end, improving playback fluency, achieving acceleration, and improving the performance benefits of post-processing. The video quality processing strategy indicates which post-processing scheme to use, and can specify the most appropriate post-processing scheme according to the content characteristics, and fully improve the post-processing effect.

[0105] It should be understood that for the post-processing unit that does not skip post-processing, a unified post-processing scheme can be used with reference to relevant technologies, or a specified post-processing scheme can be used according to the video quality processing strategy, or a part can use the specified post-processing scheme according to the video quality processing strategy and the other part can use a unified post-processing scheme. Of course, other reasonable methods can also be adopted to determine how to perform post-processing, and the present disclosure does not limit this.

[0106] In the case where the post-processing strategy includes a skip strategy, optionally, step S602 includes: based on the coding information, determining the post-processing unit that meets the preset skip condition as a skip post-processing unit, and obtaining information on the post-processing strategy for the skip post-processing unit. By configuring the preset skip condition to identify the skip post-processing unit suitable for skip post-processing, the range of post-processing units suitable for skip post-processing can be efficiently determined to achieve acceleration and efficiency improvement. It should be understood that the information on the post-processing strategy for the skip post-processing unit is information indicating that no post-processing is performed on the skip post-processing unit.

[0107] Specifically, since the human eye is not sensitive to motion blurred images and smooth images, the benefits of post-processing enhancement of such post-processing units are limited. Based on this, the preset skipping condition includes a condition indicating that the human eye is not sensitive to the corresponding image, which may specifically include at least one of the following: motion blur condition, smooth image condition, and can skip post-processing for motion blurred and smooth post-processing units to achieve acceleration and efficiency improvement. Of course, corresponding preset skipping conditions can also be configured for other post-processing units suitable for skipping, and the present disclosure does not limit this.

[0108] The motion blur condition includes at least one of the following: the cost of the intra prediction mode calculated in the adaptive quantization algorithm is less than the first cost threshold, the cost of the inter prediction mode calculated in the adaptive quantization algorithm is greater than the second cost threshold, the proportion of the selected intra prediction mode in the adaptive quantization algorithm is higher than the first proportion threshold, the proportion of the selected inter prediction mode in the adaptive quantization algorithm is lower than the second proportion threshold, the statistical value of the motion vector obtained in the pre-analysis stage is higher than the statistical threshold, and the quantization parameter increment is less than the increment threshold. When there is motion blur in a certain area, the prediction result obtained by motion estimation ME and motion compensation MC in the reference frame of the area is usually inaccurate, so when calculating the cost of the inter prediction mode (Inter) and the intra prediction mode (Intra) in the temporal and spatial domain adaptive quantization algorithms (CuTree and AQ), the cost of Intra will be lower and the cost of Inter will be higher, so that the proportion of Intra selected (IntraMbs) increases and the proportion of Inter selected (InterMbs) decreases. Based on this, at least one of the Intra cost, Inter cost, the proportion of Intra selected, and the proportion of Inter selected can be used to realize the judgment of motion blur. At the same time, since the importance of the motion blur area is relatively low, a larger quantization parameter QP is required (the QP value is negatively correlated with the importance of the corresponding area), that is, the quantization parameter increment QPoffset analyzed by CuTree and AQ is larger, so the judgment can be achieved in combination with the quantization parameter increment. In addition, the information in the pre-analysis module of the encoding end can also be used for detection. Intuitively, the motion state is directly related to the characteristics (mean and variance Var, etc.) of the motion vector MV(x, y) of the encoding end, so the judgment of motion blur can also be achieved in combination with the statistical characteristics of the motion vector. As an example, when the number of conditions selected for use is two or more, the motion blur condition can be considered to be met when at least one of the conditions is met, or the motion blur condition can be considered to be met when all conditions are met. The present disclosure does not limit this.

[0109] The smooth image condition includes at least one of the following: the gradient value of the pixel value is less than the gradient threshold, the variance of the pixel value is less than the variance threshold, and the proportion of the high-frequency component of the pixel value is less than the first proportion threshold. By using at least one of the above conditions, a smooth image with small changes can be detected, and reliable detection of the smooth image can be achieved. Similar to the motion blur condition, when the number of conditions selected for use in the smooth image condition is two or more, the smooth image condition can also be considered to be satisfied when at least one of the conditions is met, or the smooth image condition can be considered to be satisfied when all the conditions are met, and the present disclosure does not limit this.

[0110] In addition to the judgment of the preset skip condition, optionally, step S602 also includes: determining other post-processing units similar to the post-processing unit that meets the preset skip condition as skip post-processing units; and / or if there are multiple skip post-processing units belonging to the same higher-level post-processing unit, and the proportion of multiple skip post-processing units in the higher-level post-processing unit exceeds the second proportion threshold, then the higher-level post-processing unit is determined as a skip post-processing unit. By configuring the above two extension schemes, the recognition scope of the skip post-processing unit can be expanded on the basis of the preset skip condition, which helps to further achieve acceleration and efficiency. Specifically, by finding other post-processing units similar to the already determined skip post-processing unit, the similarity between the post-processing units can be used to expand the skip post-processing unit. As an example, because the encoding end in video encoding has MV information to describe the motion between two frames, if the previous frame can be skipped, the overall MV of the next frame is almost 0 (that is, the two frames before and after are almost not moving, and a sufficiently small threshold can be configured in the specific implementation, and the MV is considered to be almost 0 when the MV is less than the threshold), then the next frame can also be skipped. The same is true for regions. If some regions in the previous frame are skippable, and some regions in the next frame happen to refer to the skippable regions in the previous frame, then those regions in the next frame are also considered to be skippable. By identifying the entire high-level post-processing unit as a skipped post-processing unit for multiple bottom-level post-processing units belonging to the same higher-level post-processing unit, when the proportion of bottom-level post-processing units identified as skipped post-processing units reaches a second proportion threshold, the cross-level expansion of skipped post-processing units can be achieved. As an example, when calculating the proportion, the quantity proportion or the area proportion can be calculated, and the present disclosure does not limit this. As an example, if the area of ​​the skipped region in a frame accounts for a proportion of the entire frame area that exceeds the second proportion threshold, the entire frame is set as a skipped frame, and 1 is used in the SEI of the frame to indicate that a skip strategy is executed for the frame; if the number of skipped frames in an image group accounts for a proportion of the total number of frames in the entire image group that exceeds the second proportion threshold (the specific values ​​of the second proportion threshold used in the two places may be equal or unequal), all frames in the image group are set as skipped frames, and 1 and a start mark are used in the SEI of the start frame of the image group to indicate that post-processing is skipped starting from this frame, and 1 and an end mark are used in the SEI of the end frame of the image group to indicate that post-processing is skipped for this frame, but post-processing is no longer skipped for frames after this frame.

[0111] Optionally, step S602 also includes: based on the encoding information, performing quality detection processing on the post-processing units that do not meet the preset skipping conditions, and determining the post-processing units that pass the quality detection as skipped post-processing units. By further adding quality detection processing on the basis of the preset skipping conditions, post-processing can also be skipped for post-processing units with better quality, thereby further achieving acceleration and efficiency improvement. As an example, the quality detection processing can be achieved by calculating the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) of the reconstructed post-processing unit and the original post-processing unit (for example, the reconstructed frame and the original frame, and the reconstructed area and the original area). As an example, for the post-processing units with poor quality that fail the quality detection, a unified post-processing scheme in the relevant technology can be used, or when the post-processing strategy of the present disclosure includes a video quality processing strategy, a specified post-processing scheme can be used according to the video quality processing strategy, and the present disclosure does not limit this.

[0112] In the case where the post-processing strategy includes a video quality processing strategy, in some embodiments, optionally, step S602 includes: in the case where it is determined based on the coding information that the post-processing unit has a preset quality problem, the preset post-processing scheme corresponding to the preset quality problem is used as the post-processing scheme of the post-processing unit, and the information of the post-processing strategy for the post-processing unit is obtained. By purposefully finding the preset quality problem existing in the post-processing unit and then determining the corresponding preset post-processing scheme, the efficiency of determining the post-processing scheme can be improved. As an example, the preset quality problem can be a specific problem type, such as blur, the presence of more block effects, weak illumination or overexposure, large noise, and color shift, and the corresponding preset post-processing schemes can be a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a de-noising algorithm, and a color enhancement algorithm. The preset problem can also be the severity of a specific problem type, so that models of different strengths or models of different structures can be used as the corresponding preset post-processing schemes based on the processing model algorithm corresponding to the specific problem type. These different algorithms and models can optimize video quality from different angles and to different degrees. Taking the blur problem as an example, since blurry images usually have fewer significant edges, an edge detection algorithm (such as Sobel or Laplacian) can be used to measure the clarity of the post-processing unit, and further set a clarity threshold to divide different clarity levels. For example, three clarity levels are divided by setting two clarity thresholds. For the post-processing unit with the highest clarity, it is considered that there is no blur problem and there is no need to use a deblurring model. For the post-processing unit with a middle clarity, a simple deblurring model is used. For the post-processing unit with the lowest clarity, a stronger deblurring model is used.

[0113] In other embodiments, optionally, step S602 includes: post-processing the post-processing unit according to a plurality of different post-processing schemes, selecting the post-processing scheme with the best post-processing quality as the post-processing scheme of the post-processing unit, and obtaining information on the post-processing strategy for the post-processing unit. By trying a plurality of different post-processing schemes, it is possible to make full use of the sufficient information and computing power of the encoding end to find the most suitable post-processing scheme for the post-processing unit, which helps to improve the post-processing effect. As an example, similar to some of the aforementioned embodiments, the post-processing schemes tried here may be different types of processing model algorithms, or different strengths or models of the same type of processing model algorithms, or may include different types of processing model algorithms, and at least one of the processing model algorithms further includes different strengths or models. As an example, for different post-processing units, each post-processing unit may be tried multiple times to determine a suitable post-processing scheme for each post-processing unit.

[0114] As an example, for the same decoded video, the two embodiments of determining the post-processing scheme above may be used selectively or simultaneously. For the case of using them simultaneously, the first embodiment (i.e., the above-mentioned “some embodiments”) may be used for a part of the post-processing units, and the second embodiment (i.e., the above-mentioned “some other embodiments”) may be used for another part of the post-processing units. Alternatively, the first embodiment may be used preferentially for all post-processing units, and the second embodiment may be used when it is determined that the post-processing units do not have preset quality problems. The present disclosure does not impose any restrictions on this.

[0115] Optionally, before specifically determining the post-processing scheme of the post-processing unit, step S602 may also include: based on the coding information, performing quality detection processing on multiple post-processing units, and determining the post-processing units that fail the quality detection as post-processing units that use the video quality processing strategy. By determining the post-processing units with poor quality and using the video quality processing strategy to determine the applicable post-processing scheme, the post-processing scheme can be determined specifically for the low-quality post-processing units with strong post-processing requirements, thereby increasing the efficiency of improving the post-processing effect. It should be understood that at this time, there is no need to first determine whether the post-processing unit meets the preset skip condition.

[0116] Next, a video processing method applied to an encoding end according to the present disclosure is introduced in conjunction with a specific embodiment.

[0117] In this specific embodiment, a method for end-cloud collaborative guidance of post-processing is proposed, and the overall process includes: the encoding end (cloud side) uses the encoder to analyze the post-processing strategy information required for the decoding end post-processing, and then attaches the post-processing strategy information to the video bitstream in SEI mode to ensure that the bitstream still meets the standard, the decoding end (end side) performs normal decoding to obtain the reconstructed frame, and then the post-processing module parses the SEI information in the bitstream to guide the post-processing of the decoded reconstructed frame.

[0118] According to the functions required for post-processing, the end-cloud collaborative post-processing strategies can be divided into two categories: post-processing acceleration (corresponding to the skip strategy) and quality improvement (corresponding to the video quality processing strategy). The following will introduce the strategy details according to the function.

[0119] 1. End-cloud collaborative post-processing acceleration.

[0120] Due to the limited computing power of the decoding end, if all regions of all frames are post-processed, the decoding end may be too complex and affect the smoothness of playback. By using the cloud to analyze which frames and regions have limited performance benefits or are not suitable for post-processing, and then transmitting information about the skip strategy for these frames and regions to the decoding end, the post-processing module of the decoding end can be told to skip the post-processing process of these frames and regions, thereby achieving acceleration.

[0121] An intuitive strategy is that post - processing enhancement is usually not required for fast - moving regions because the human eye is insensitive to motion blur and the benefits brought by post - processing enhancement are limited. Therefore, as long as it is detected at the encoding end which frames or which regions have motion blur, these frames and regions are marked (i.e., recorded as skipping the post - processing unit) and transmitted to the decoding end through SEI, and the post - processing module can directly skip these regions. For motion blur, the information in the pre - analysis module at the encoding end can be used for detection. Intuitively, motion is directly related to the characteristics (mean and variance Var, etc.) of the motion vector MV(x, y) at the encoding end. At the same time, when there is motion blur, the prediction results obtained from motion estimation ME and motion compensation MC in these regions from the reference frame are inaccurate. Therefore, when calculating the costs of Inter and Intra in the temporal and spatial adaptive quantization algorithms (CuTree and AQ), the proportion of selecting Inter will decrease, and the proportion of selecting Intra (IntraMbs) will increase. At the same time, the importance of these regions is relatively low, that is, the QPoffset analyzed by CuTree and AQ is relatively small. Based on these analyses, the above - mentioned characteristics can be used to detect which frames and which regions have motion blur and set thresholds. It can be represented by the following pseudo - code: Motion = IntraMbs>thd_intra && (Var_MVX + Var_MVY)>thd_mv && QPoffset<thd_qp. Among them, thd_intra is the first proportion threshold introduced above, Var_MVX and Var_MVY respectively represent the variance of the motion vector in the x - direction and the variance of the motion vector in the y - direction, thd_mv is the statistical threshold introduced above, and thd_qp is the increment threshold introduced above. This pseudo - code means that when a certain frame or a certain region simultaneously meets the three conditions that the proportion of selecting the intra - frame prediction mode Intra in the adaptive quantization algorithm is higher than the first proportion threshold thd_intra, the sum of the variance of the motion vector in the x - direction Var_MVX and the variance of the motion vector in the y - direction Var_MVX is greater than the statistical threshold thd_mv, and the quantization parameter increment QPoffset is less than the increment threshold thd_qp, it is determined to skip the post - processing of this frame or this region. Thus, the post - processing units at the frame - level and region - level (such as CTU - level) can be calculated respectively to determine which frames and which regions are motion - blurred and can skip the post - processing.

[0122] In order to ensure the stability of the algorithm, it can also be set that when the motion blur area in the frame exceeds the threshold of the proportion of the entire frame area (the second proportion threshold introduced above), the entire frame is set as a motion blur frame; and when the number of motion blur frames in an image group exceeds the threshold of the proportion of the number of frames in the entire image group (the second proportion threshold introduced above, and it can be different from the value of the threshold of the proportion of the entire frame area), all frames in the entire image group are set as motion blur frames. At the same time, considering that in the case of motion blur in the previous frame, the next frame may not move and still be motion blurred, but the above detection algorithm will fail, so the correlation between consecutive frames can also be considered. When the MV of the next frame is almost 0, the next frame is also determined as a motion blur frame, which can achieve more comprehensive motion blur detection. Similarly, for motion blur areas, if some areas in the previous frame are motion blurred, and some areas in the next frame happen to refer to the motion blur area of ​​the previous frame, it is considered that the area of ​​the next frame is also motion blurred.

[0123] After detecting which frames and regions can skip post-processing, the corresponding post-processing strategy information is transmitted using SEI. One transmission method is to write SEI at the frame level, which can be information about frame-level skipping or region-level skipping, where frame-level skipping can be represented by 0 or 1, and region-level skipping can be represented by CTU index or directly by region coordinates. When the decoder performs post-processing analysis, if it is found that the frame has SEI indicating skipping post-processing, then the corresponding post-processing is skipped. Another method is to consider the continuity of the time domain, and the corresponding SEI information can be written into the start frame and the end frame of the skipped post-processing. When the decoder performs post-processing analysis, if it is found that the frame has SEI for skipping post-processing, then the subsequent frames maintain the SEI state of the frame until a frame reads a new SEI state and then switches.

[0124] 2. Improve quality through end-cloud collaborative post-processing.

[0125] Since different video contents may be suitable for different post-processing schemes during post-processing, the encoder can specify the most appropriate post-processing scheme according to the content characteristics. Different algorithms can be specified for different types of content characteristics, and different intensities or models can be used based on the same algorithm. For the former, for example, if the reconstructed frame is blurry, a deblurring algorithm can be used for post-processing; if there are many block effects in the reconstructed frame, a deblocking algorithm can be used for post-processing; if the reconstructed frame has weak lighting or is overexposed, a lighting compensation algorithm can be used for post-processing; if the reconstructed frame has a lot of noise, a de-noising algorithm can be used; if the reconstructed frame has color shift, a color enhancement algorithm can be used, etc. For the latter, for example, some frames are suitable for a simple deblurring model, while some frames are suitable for a stronger deblurring model.

[0126] The encoder can analyze the video content (for example, it can calculate indicators such as PSNR and SSIM for the encoded reconstructed frames and the original frames to check whether the quality of the reconstructed frames is good and determine whether to use quality enhancement for certain frames or areas), or use different post-processing schemes for the reconstructed frames to check the effects (for example, use PSNR to indicate the post-processing effect, or detect the noise of the reconstructed frames after processing. Small noise indicates good post-processing effect). The encoder selects the best scheme based on the post-processing quality and transmits it to the decoder through SEI. The post-processing module of the decoder parses SEI for each image group, each frame, or each area to determine the corresponding post-processing scheme and maximize the post-processing effect.

[0127] This specific embodiment transmits the post-processing strategy information to the post-processing module on the terminal side through SEI through cloud-side analysis. When the bitstream standard is met, frames or areas with smaller post-processing benefits are skipped and accelerated to reduce complexity and improve the terminal-side playback experience; or different post-processing algorithms are selected for different frames or areas to maximize the subjective effect of post-processing and improve efficiency.

[0128] In addition, as an extension, in the end-cloud collaborative acceleration, this specific embodiment only analyzes the motion blur situation and can skip post-processing enhancement. There are other situations, such as completely smooth images (sky, night, white walls, etc.), which can also skip post-processing. For example, the mean and variance of the pixel values ​​in the area are calculated. If the variance is very small, it can be judged as a smooth image.

[0129] Only a few of the mentioned features are used in the motion blur detection method. Other features can also be used, such as the mean of MV, the cost of Inter, etc. The mentioned features can be met at the same time to be judged as motion blur, or the "or" method can be used to judge it as motion blur when any one of them is met.

[0130] In summary, this specific embodiment proposes a method for end-cloud collaborative guidance of post-processing, which combines the post-processing algorithm originally independent at the decoding end with the encoding end, and the encoding end guides the post-processing of the decoding end by transmitting information. Specifically, the encoding end (cloud) performs analysis to obtain the post-processing strategy for each frame and specific area, and then transmits the information to the decoding end (end) through SEI for post-processing, which does not affect the standard bitstream and improves the performance of post-processing.

[0131] Figure 7 1 is a flowchart showing an exemplary video processing method applied to a decoding end according to some embodiments of the present disclosure. The video processing method can be implemented in the target device 14 described above.

[0132] Reference Figure 7In step S701, supplemental enhancement information (SEI) sent by the encoder is received. The SEI includes post-processing strategy information, which indicates a post-processing strategy executed after decoding the encoded video. The SEI is obtained by the video processing method applied to the encoder as described above. Accordingly, the encoder here is the encoder that processes the SEI.

[0133] In step S702, after decoding the encoded video, post-processing is performed on the decoded video based on the post-processing strategy information.

[0134] Corresponding to the video processing method applied to the encoding end as described above, in the exemplary video processing method applied to the decoding end according to some embodiments of the present disclosure, optionally, the post-processing strategy information includes information about the post-processing strategy for the post-processing unit. In other words, the post-processing strategy information includes two aspects of information, one is the post-processing unit targeted, and the other is the post-processing strategy executed by the post-processing unit. Among them, the post-processing unit includes at least one of a group of images, a frame, and a region, and the region includes at least one of a strip, a tile, and a block. By receiving information about the post-processing strategies used by the post-processing units at different coding levels (i.e., the group of images level, the frame level, and the region level), different post-processing can be achieved for different post-processing units, which improves the flexibility and adaptability of post-processing and helps to improve the post-processing performance. It should be understood that a region is a part of a frame, which can be at least one of a strip, a tile, and a block. As an example, a block may further include at least one of CTU, CU, PU, ​​TU, CTB, CB, PB, and TB. It should also be understood that the post-processing strategy of the post-processing unit at the region level will be applied to the corresponding entire region; the post-processing strategy of the post-processing unit at the frame level will be applied to the corresponding entire frame; and the post-processing strategy of the post-processing unit at the image group level will be applied to all frames in the corresponding image group.

[0135] Taking the aforementioned post-processing strategy information including the information on the post-processing strategy for the post-processing unit as an example, each post-processing unit has its own corresponding level, some post-processing units have relatively high levels, and some post-processing units have relatively low levels. For example, the level of the image group is higher than the level of the frame, and the level of the frame is higher than the level of the region.

[0136] Accordingly, in some embodiments, optionally, the exemplary video processing method applied to the decoding end according to some embodiments of the present disclosure further includes: obtaining post-processing strategy information from the SEI of the corresponding level or higher level of the post-processing unit, which can realize targeted information reading for post-processing units of different levels, and ensure the flexibility of information reading. It should be understood that at this time, from the perspective of the SEI corresponding to a post-processing unit, what is read therefrom is the information of the post-processing strategy for the post-processing unit and / or the sub-units of the post-processing unit.

[0137] In other embodiments, optionally, the exemplary video processing method applied to the decoding end according to some embodiments of the present disclosure further includes: for a plurality of consecutive post-processing units using the same post-processing strategy, the post-processing strategy information of the plurality of post-processing units is obtained from the SEI corresponding to the starting post-processing unit and the SEI corresponding to the ending post-processing unit in the plurality of post-processing units, and the acquisition of the post-processing strategies of the plurality of post-processing units can be realized by using two information readings, thereby improving the efficiency of information acquisition. It should be understood that at this time, after the post-processing strategy information is read from the SEI corresponding to the starting post-processing unit, the starting post-processing unit can be post-processed according to the post-processing strategy recorded in the information, and the same post-processing strategy is also used for the post-processing units after the starting post-processing unit, until the post-processing strategy information is read from the SEI corresponding to the ending post-processing unit, the post-processing strategy of the post-processing unit after the ending post-processing unit is switched, that is, the previous post-processing strategy is no longer used.

[0138] It should be understood that the embodiments of the present disclosure may support at least one of the above two information reading methods. As long as the information in the SEI conforms to the information reading method supported by the embodiments of the present disclosure, the information can be read.

[0139] Optionally, the post-processing strategy includes at least one of a skip strategy and a video quality processing strategy, wherein the skip strategy indicates whether to skip post-processing, and can skip post-processing of a specified post-processing unit, thereby reasonably reducing the application scope of post-processing, reducing the complexity of executing post-processing, improving playback fluency, achieving acceleration, and improving the performance benefits of post-processing. The video quality processing strategy indicates which post-processing solution to use, and can specify the most appropriate post-processing solution according to content characteristics, and fully improve the post-processing effect.

[0140] Optionally, the post-processing scheme includes at least one of the following algorithms: a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a denoising algorithm, and a color enhancement algorithm, and each algorithm includes models of different intensities and / or models of different structures, thereby optimizing video quality from different angles and to different degrees.

[0141] It should be understood that for the post-processing unit that does not skip post-processing, a unified post-processing scheme can be used with reference to relevant technologies, or a specified post-processing scheme can be used according to the video quality processing strategy, or a part can use the specified post-processing scheme according to the video quality processing strategy and the other part can use a unified post-processing scheme. Of course, other reasonable methods can also be adopted to determine how to perform post-processing, and the present disclosure does not limit this.

[0142] Figure 8 is a block diagram showing an exemplary video processing device applied to an encoding end according to some embodiments of the present disclosure.

[0143] Reference Figure 8 The video processing device 800 includes an acquiring unit 801, a determining unit 802, an adding unit 803, and a sending unit 804.

[0144] The acquisition unit 801 may acquire encoding information of the encoded video.

[0145] The determination unit 802 may determine a post-processing strategy to be executed on a decoded video based on the encoding information to obtain post-processing strategy information, wherein the decoded video is a video obtained after a decoding end decodes the encoded video.

[0146] The adding unit 803 may add the post-processing strategy information to the supplemental enhancement information (SEI) of the encoded video.

[0147] The sending unit 804 may send the SEI to the decoding end to instruct the decoding end to post-process the decoded video according to the post-processing strategy information in the SEI.

[0148] Optionally, the post-processing strategy information includes information of a post-processing strategy for a post-processing unit, wherein the post-processing unit includes at least one of a group of images, a frame, and a region, and the region includes at least one of a stripe, a tile, and a block.

[0149] Optionally, the adding unit 803 may also: add the post-processing strategy information to the SEI of the corresponding level or higher level of the post-processing unit; or for multiple consecutive post-processing units using the same post-processing strategy, add the post-processing strategy information to the SEI corresponding to the starting post-processing unit and the SEI corresponding to the ending post-processing unit among the multiple post-processing units.

[0150] Optionally, the post-processing strategy includes at least one of a skip strategy and a video quality processing strategy, wherein the skip strategy indicates whether to skip post-processing, and the video quality processing strategy indicates which post-processing solution to use.

[0151] Optionally, the post-processing scheme includes at least one of the following algorithms: a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a denoising algorithm, and a color enhancement algorithm, and each algorithm includes models of different strengths and / or models of different structures.

[0152] Optionally, when the post-processing strategy includes a skip strategy, the determination unit 802 may also: based on the encoding information, determine a post-processing unit that meets a preset skip condition as a skip post-processing unit, wherein the preset skip condition includes a condition indicating that the human eye is insensitive to the corresponding image.

[0153] Optionally, the determination unit 802 may also: determine other post-processing units that are similar to the post-processing units that meet the preset skipping conditions as skipped post-processing units; and / or if there are multiple skipped post-processing units belonging to the same higher-level post-processing unit, and the proportion of the multiple skipped post-processing units in the higher-level post-processing units exceeds a second proportion threshold, then determine the higher-level post-processing unit as a skipped post-processing unit.

[0154] Optionally, the determining unit 802 may further: perform quality detection processing on the post-processing units that do not meet the preset skipping condition based on the coding information, and determine the post-processing units that pass the quality detection as skipped post-processing units.

[0155] Optionally, when the post-processing strategy includes a video quality processing strategy, the determination unit 802 may also: when it is determined based on the encoding information that the post-processing unit has a preset quality problem, use a preset post-processing scheme corresponding to the preset quality problem as the post-processing scheme of the post-processing unit; and / or post-process the post-processing unit according to multiple different post-processing schemes, and select the post-processing scheme with the best post-processing quality as the post-processing scheme of the post-processing unit.

[0156] Fig. 9 is a block diagram showing an exemplary video processing device applied to a decoding end according to some embodiments of the present disclosure.

[0157] Reference Fig. 9 The video processing device 900 includes a receiving unit 901 and a processing unit 902.

[0158] The receiving unit 901 can receive supplemental enhancement information (SEI) sent by the encoding end, wherein the SEI includes post-processing strategy information, which represents the post-processing strategy executed after decoding the encoded video. The SEI is obtained by the video processing method applied to the encoding end as described above.

[0159] The processing unit 902 may post-process the decoded video based on the post-processing strategy information after decoding the encoded video, wherein the SEI is obtained by the video processing method applied to the encoding end as described above.

[0160] Optionally, the post-processing strategy information includes information of a post-processing strategy for a post-processing unit, wherein the post-processing unit includes at least one of a group of images, a frame, and a region, and the region includes at least one of a stripe, a tile, and a block.

[0161] Optionally, the video processing device 900 also includes a reading unit (not shown in the figure), which can obtain post-processing strategy information from the SEI of the corresponding level or higher level of the post-processing unit; or for multiple consecutive post-processing units using the same post-processing strategy, obtain the post-processing strategy information of multiple post-processing units from the SEI corresponding to the starting post-processing unit and the SEI corresponding to the ending post-processing unit among the multiple post-processing units.

[0162] Optionally, the post-processing strategy includes at least one of a skip strategy and a video quality processing strategy, wherein the skip strategy indicates whether to skip post-processing, and the video quality processing strategy indicates which post-processing solution to use.

[0163] Optionally, the post-processing scheme includes at least one of the following algorithms: a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a denoising algorithm, and a color enhancement algorithm, and each algorithm includes models of different strengths and / or models of different structures.

[0164] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0165] Fig.10 A computing environment 1010 is shown coupled to a user interface 1050. The computing environment 1010 may be part of a data processing server. The computing environment 1010 includes a processor 1020, a memory 1030, and an input / output (I / O) interface 1040.

[0166] The processor 1020 generally controls the overall operation of the computing environment 1010, such as operations associated with display, data acquisition, data communication, and image processing. The processor 1020 may include one or more processors for executing instructions to perform all or some of the steps in the above method. In addition, the processor 1020 may include one or more modules that facilitate the interaction between the processor 1020 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0167] The memory 1030 is configured to store various types of data to support the operation of the computing environment 1010. The memory 1030 may include predetermined software 1032. Examples of such data include instructions for any application or method operating on the computing environment 1010, video data sets, image data, etc. The memory 1030 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0168] The I / O interface 1040 provides an interface between the processor 1020 and peripheral interface modules (e.g., a keyboard, a click wheel, buttons, etc.). The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 1040 may be coupled to an encoder and a decoder.

[0169] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 1030 and / or storing a bitstream generated or received by the above-mentioned video processing method is also provided, wherein the plurality of programs can be executed by a processor 1020 in a computing environment 1010 to perform the above-mentioned method. In one example, the plurality of programs can be executed by a processor 1020 in a computing environment 1010 to (for example, from Figure 2 The video encoder 20 in the computing environment 1010 receives a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, SEI, etc.), and can also be executed by the processor 1020 in the computing environment 1010 to perform the above-mentioned video processing method applied to the decoding end according to the received bitstream or data stream. In another example, the multiple programs can be executed by the processor 1020 in the computing environment 1010 to perform the above-mentioned video processing method applied to the encoding end to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, SEI, etc.) into a bitstream or data stream, and can also be executed by the processor 1020 in the computing environment 1010 to (e.g., to Figure 3 Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in the example uses the video processing method applied to the encoding end as described above to generate a video signal for the decoder (e.g., Figure 3A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, SEI, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium can be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0170] In an embodiment, a bit stream generated by the above-mentioned video processing method applied to the encoding end or a bit stream to be decoded and post-processed by the above-mentioned video processing method applied to the decoding end is provided. In an embodiment, a bit stream including coded video information generated by the above-mentioned video processing method applied to the encoding end or a bit stream including coded video information to be decoded and post-processed by the above-mentioned video processing method applied to the decoding end is provided.

[0171] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 1020); and a non-transitory computer-readable storage medium or memory 1030 in which multiple programs that can be executed by the one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0172] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream is also provided, wherein the bitstream includes coded video information generated by the above-mentioned video processing method applied to the encoding end or coded video information to be decoded and post-processed by the above-mentioned video processing method applied to the decoding end. In an embodiment, a computer program product including, for example, a plurality of programs in a memory 1030 is also provided, and the plurality of programs can be executed by a processor 1020 in a computing environment 1010 for performing the above-mentioned method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0173] In an embodiment, the computing environment 1010 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0174] In an embodiment, a method for storing a bit stream is also provided, comprising: storing the bit stream on a digital storage medium, wherein the bit stream includes encoded video information generated by the above-mentioned video processing method applied to the encoding end or encoded video information that will be decoded and post-processed by the above-mentioned video processing method applied to the decoding end.

[0175] In an embodiment, a method for transmitting a bit stream generated by the above encoder is also provided. In an embodiment, a method for receiving a bit stream to be decoded by the above decoder is also provided.

[0176] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0177] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0178] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A video processing method, characterized in that: Applied to the encoding end, the video processing method includes: Get the encoding information of the encoded video; Based on the encoding information, determining a post-processing strategy to be executed on a decoded video, and obtaining post-processing strategy information, wherein the decoded video is a video obtained after a decoding end decodes the encoded video; Adding the post-processing strategy information to supplemental enhancement information (SEI) of the encoded video; The SEI is sent to a decoding end to instruct the decoding end to post-process the decoded video according to the post-processing strategy information in the SEI.

2. The video processing method according to claim 1, characterized in that: The post-processing strategy information includes information of a post-processing strategy for a post-processing unit, wherein the post-processing unit includes at least one of a group of images, a frame, and a region, and the region includes at least one of a stripe, a tile, and a block.

3. The video processing method according to claim 2, characterized in that: The adding the post-processing strategy information to the supplemental enhancement information (SEI) of the coded video comprises: Adding the post-processing strategy information to the SEI of the corresponding level or higher level of the post-processing unit; or For a plurality of consecutive post-processing units using the same post-processing strategy, the post-processing strategy information is added to the SEI corresponding to the starting post-processing unit and the SEI corresponding to the ending post-processing unit among the plurality of post-processing units.

4. The video processing method according to claim 1, characterized in that: The post-processing strategy includes at least one of a skip strategy and a video quality processing strategy, wherein the skip strategy indicates whether to skip post-processing, and the video quality processing strategy indicates which post-processing scheme to use, and the post-processing scheme includes at least one of the following algorithms: a deblurring algorithm, a deblocking algorithm, an illumination compensation algorithm, a denoising algorithm, and a color enhancement algorithm, and each algorithm includes models of different strengths and / or models of different structures.

5. The video processing method according to claim 4, characterized in that: In the case where the post-processing strategy includes the skip strategy, determining the post-processing strategy to be executed on the decoded video based on the encoding information includes: Based on the encoding information, a post-processing unit that meets a preset skipping condition is determined as a skipped post-processing unit, wherein the preset skipping condition includes a condition indicating that human eyes are insensitive to the corresponding image.

6. The video processing method according to claim 5, characterized in that: The step of determining a post-processing strategy to be executed on the decoded video based on the encoding information further includes: Determine other post-processing units similar to the post-processing unit satisfying the preset skip condition as skipped post-processing units; and / or If there are multiple skipped post-processing units belonging to the same higher-level post-processing unit, and the proportion of the multiple skipped post-processing units in the higher-level post-processing unit exceeds a second proportion threshold, the higher-level post-processing unit is determined as a skipped post-processing unit.

7. The video processing method according to claim 5, characterized in that: The step of determining a post-processing strategy to be executed on the decoded video based on the encoding information further includes: Based on the coding information, quality detection processing is performed on the post-processing units that do not meet the preset skipping condition, and the post-processing units that pass the quality detection are determined as skipped post-processing units.

8. The video processing method according to claim 4, characterized in that: In the case where the post-processing strategy includes the video quality processing strategy, determining the post-processing strategy to be executed on the decoded video based on the encoding information includes: In a case where it is determined based on the coding information that the post-processing unit has a preset quality problem, using a preset post-processing solution corresponding to the preset quality problem as a post-processing solution of the post-processing unit; and / or The post-processing unit is post-processed according to a plurality of different post-processing schemes, and a post-processing scheme with the best post-processing quality is selected as the post-processing scheme of the post-processing unit.

9. A video processing method, characterized in that: Applied to the decoding end, the video processing method includes: Receiving supplemental enhancement information (SEI) sent by an encoder, wherein the SEI includes post-processing strategy information, the post-processing strategy information represents a post-processing strategy executed after decoding the encoded video, and the SEI is obtained by processing the video processing method applied to the encoder according to any one of claims 1 to 8; After decoding the encoded video, post-processing the decoded video is performed based on the post-processing strategy information.

10. A video processing device, characterized in that: Applied to the encoding end, the video processing device: An acquisition unit, configured to acquire encoding information of the encoded video; A determination unit is configured to determine a post-processing strategy to be executed on a decoded video based on the encoding information, and obtain post-processing strategy information, wherein the decoded video is a video obtained after a decoding end decodes the encoded video; an adding unit, configured to add the post-processing strategy information to supplemental enhancement information (SEI) of the encoded video; The sending unit is configured to send the SEI to the decoding end to instruct the decoding end to post-process the decoded video according to the post-processing strategy information in the SEI.

11. A video processing device, characterized in that: Applied to a decoding end, the video processing device comprises: A receiving unit, configured to receive supplemental enhancement information (SEI) sent by an encoder, wherein the SEI includes post-processing strategy information, the post-processing strategy information represents a post-processing strategy executed after decoding the encoded video, and the SEI is obtained by processing the video processing method applied to the encoder according to any one of claims 1 to 8; The processing unit is configured to post-process the decoded video based on the post-processing strategy information after decoding the encoded video.

12. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by at least one processor, the at least one processor is prompted to execute the video processing method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the video processing method according to any one of claims 1 to 9.

14. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the video processing method according to any one of claims 1 to 9 is implemented.

15. A computer program product having instructions for storing a bit stream, characterized in that The bitstream includes: Supplemental Enhancement Information (SEI) of a coded video generated according to the video processing method according to any one of claims 1 to 8; or The supplemental enhancement information received according to the video processing method of claim 9.

Images