Method and device for video processing and medium

Abstract

The embodiment of the invention provides a video processing scheme. A method for video processing is presented. The method comprises: during a transition between a video unit of a video and a bitstream of the video, applying signal processing to the video unit based at least in part on a window-based attention module; and performing a conversion based on the processed video unit.

Description

Technical Field

[0002] Embodiments of the present disclosure relate generally to video processing techniques, and more particularly, to learned image compression. Background Art

[0003] Nowadays, digital video capabilities are being applied to all aspects of people's lives. Various types of video compression technologies have been proposed for video encoding / decoding, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard. However, it is generally expected to further improve the encoding and decoding efficiency of video encoding and decoding technologies. Summary of the invention

[0004] Embodiments of the present disclosure provide a solution for video processing.

[0005] In a first aspect, a method for video processing is provided. The method comprises: applying signal processing to the video unit at least in part based on a window-based attention module during conversion between a video unit of the video and a bitstream of the video; and performing conversion based on the processed video unit. The method according to the first aspect of the present disclosure can achieve more compact signal representation and more refined recovery.

[0006] In a second aspect, another method for video processing is proposed. The method includes: applying a data augmentation process for training a window-based attention module during conversion between a video unit of a video and a bitstream of the video; performing signal processing on the video unit based on the trained window-based attention module; and performing conversion based on the processed video unit. The method according to the second aspect of the present disclosure can achieve a better window-based attention module.

[0007] In a third aspect, a device for video processing is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect or the second aspect of the present disclosure.

[0008] In a fourth aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, and the instructions enable a processor to execute the method according to the first aspect or the second aspect of the present disclosure.

[0009] In a fifth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method includes: applying signal processing to a video unit of the video based at least in part on a window-based attention module; and generating a bitstream of the video based on the processed video unit.

[0010] In a sixth aspect, a method for storing a bitstream of a video is provided. The method includes: applying signal processing to video units of the video based at least in part on a window-based attention module; generating a bitstream of the video based on the processed video units; and storing the bitstream in a non-transitory computer-readable recording medium.

[0011] In a seventh aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method includes: applying a data augmentation process for training a window-based attention module; performing signal processing on a video unit of the video based on the trained window-based attention module; and generating a bitstream of the video based on the processed video unit.

[0012] In an eighth aspect, a method for storing a bitstream of a video is provided. The method includes: applying a data augmentation process for training a window-based attention module; performing signal processing on a video unit of the video based on the trained window-based attention module; generating a bitstream of the video based on the processed video unit; and storing the bitstream in a non-transitory computer-readable recording medium.

[0013] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings.In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0015] Figure 1 shows a block diagram illustrating an example video encoding and decoding system according to some embodiments of the present disclosure;

[0016] Figure 2 shows a block diagram illustrating a first example video encoder according to some embodiments of the present disclosure;

[0017] Figure 3shows a block diagram illustrating an example video decoder according to some embodiments of the present disclosure;

[0018] Figure 4 A framework according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A coding block based on a structured shift window transformer (swin-transformer) according to an embodiment of the present disclosure is shown;

[0020] Figure 6 A flowchart of a method for video processing according to an embodiment of the present disclosure is shown;

[0021] Figure 7 A flowchart showing a method for video processing according to an embodiment of the present disclosure; and

[0022] Figure 8 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0023] Throughout the drawings, the same or similar reference numbers generally refer to the same or similar elements. DETAILED DESCRIPTION

[0024] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are only for illustrative purposes, and contribute to those skilled in the art to understand and implement the present disclosure, without implying any limitation to the scope of the present disclosure. The disclosure described herein can be realized in various ways different from the modes described below.

[0025] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0026] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, it is considered that it is within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0027] It should be understood that although the terms "first" and "second" etc. can be used to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0028] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "including", "comprising", and / or "having" when used herein indicate the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0029] Example Environment

[0030] Figure 1 1 is a block diagram illustrating an example video encoding and decoding system 100 that may utilize the techniques of the present disclosure. As shown, the video encoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0031] The video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface for receiving video data from a video content provider, a computer graphics system for generating video data, and / or a combination thereof.

[0032] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a sequence of bits that form a coded representation of the video data. The bitstream may include coded pictures and associated data. The coded pictures are coded representations of pictures. The associated data may include sequence parameter sets, picture parameter sets, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded video data may be directly transmitted to the destination device 120 via the I / O interface 116 through the network 130A. The coded video data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0033] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, and the destination device 120 may be configured to be connected to an external display device interface.

[0034] The video encoder 114 and the video decoder 124 may operate according to a video compression standard such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or future standards.

[0035] Figure 2 is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is shown in FIG.

[0036] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , the video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of the video encoder 200. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0037] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding and decoding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.

[0038] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0039] Furthermore, although some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be integrated, for the purpose of explanation, these components are described in detail below. Figure 2 are shown separately in the example.

[0040] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0041] The mode selection unit 203 may select one of a plurality of codec modes (intra-frame coding or inter-frame coding), for example, based on the error result, and provide the generated intra-frame codec block or inter-frame codec block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the codec block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution for the motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.

[0042] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0043] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.

[0044] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0045] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.

[0046] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of a neighboring video block.

[0047] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.

[0048] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0049] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0050] The intra prediction unit 206 may perform intra prediction on the current video block. When performing intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0051] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.

[0052] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0053] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0054] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0055] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0056] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0057] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the data is received, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0058] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is shown.

[0059] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0060] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.

[0061] The entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy encoded video data, and the motion compensation unit 302 can determine motion information from the entropy decoded video data, the motion information including motion vectors, motion vector resolution, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures of neighboring PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B stripes, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks that are adjacent in space or time.

[0062] The motion compensation unit 302 may generate motion compensated blocks, possibly performing interpolation based on interpolation filters.Identifiers for the interpolation filters used with sub-pixel precision may be included in the syntax elements.

[0063] The motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filters used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filters used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filters to generate a prediction block.

[0064] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.

[0065] The intra prediction unit 303 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 304 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[0066] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also generates the decoded video for presentation on a display device.

[0067] Some example embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, rather than limiting the embodiments disclosed in the section to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video codec technologies. In addition, although some embodiments describe the video encoding steps in detail, it should be understood that the corresponding decoding steps of the unencoding will be implemented by the decoder. In addition, the term video processing includes video encoding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates.

[0068] 1. Brief Overview

[0069] The present disclosure relates to image / video processing technology, in particular, to algorithm design for image compression. These ideas can be applied to any image / video codec system or part of the encoding and decoding process, either alone or in various combinations.

[0070] 2. Introduction

[0071] Image compression plays a more prominent role in handling the ever-increasing amount of image data. A series of efforts have been devoted to improving the compression efficiency in literature and industry. Traditional lossy image compression standards including JPEG2000, BPG and VVC have been developed in the past few decades based on a block-based hybrid codec framework. Traditional codec schemes include advanced prediction modules, 2D transforms, scalar quantization, arithmetic entropy coding and decoding, and loop filters, so that redundant information in images can be effectively eliminated.

[0072] With the rise of deep learning, learning-based strategies have permeated image compression. Compared with traditional codec methods, variational autoencoders (VAEs) have been reported to achieve amazing compression performance, demonstrating the compression potential of end-to-end optimized codec schemes. Typically, end-to-end compression frameworks involve convolution operations and generalized divisive normalization (GDN), so that the visual signal is linearly and nonlinearly transformed into a hidden code and synthesized back into the reconstructed visual signal. Using entropy coding and decoding, the hidden code is compressed into the bitstream. The network can be optimized in a global end-to-end manner with the goal of minimizing rate and distortion costs. In the literature, a series of efforts have been made to enhance the rate-distortion performance of end-to-end image compression. Advanced probability estimation and entropy coding and decoding schemes such as hyper-prior models, joint autoregressive models, and entropy estimation with Gaussian mixture models have been proposed. Visual transformer networks have gradually come into people's field of vision and have been successfully applied to visual tasks such as segmentation and restoration.

[0073] 3. Question

[0074] The main problem of convolution-based schemes lies in the weakness of fine local analysis, which makes local details degraded with a purely convolution-based end-to-end compression framework. Therefore, it is necessary to study new methods for enhancing compression performance.

[0075] 4. Detailed solution

[0076] To further improve the rate and distortion performance, we propose a transformer-based end-to-end image compression framework. The proposed method is constructed with interleaved transformer layers and convolutional layers, so that it can take advantage of both the convolution operation for information distillation and the shifted window transformer for local analysis and non-local perception, resulting in more compact signal representation and refined restoration. In the following description, the term "image compression" may refer to various signal processing methods that compress or process the current input. The input image / video includes but is not limited to screen content and natural content. To address this problem, one or more of the following methods are disclosed:

[0077] 1. Introducing Transformer Networks for Image and Video Restoration.

[0078] a) In one example, the Transformer Network can be applied to image / video compression, including traditional compression and learning-based compression schemes.

[0079] b) In one example, the transformer network can be applied to image / video super-resolution.

[0080] c) In one example, the transformer network can be applied to loop filtering in image / video compression.

[0081] d) In one example, a transformer network may be applied for pre-processing and / or post-processing.

[0082] 2. Introduce the combination of transformer network and convolutional network.

[0083] a) In one example, the convolutional layers in a convolutional network can be replaced by transformer layers.

[0084] i. In one example, convolutional layers in a convolution-based end-to-end image / video compression network can be replaced by Transformer layers.

[0085] ii. In one example, the convolutional layers in a convolutional-based image / video super-resolution network can be replaced by Transformer layers.

[0086] b) In one example, some modules in the convolutional network can be replaced by the transformer network.

[0087] i. In one example, the encoder in a convolution-based end-to-end image / video compression network can be replaced by a Transformer network.

[0088] ii. In one example, the decoder in a convolution-based end-to-end image / video compression network can be replaced by a Transformer network.

[0089] iii. The residual blocks in the image / video super-resolution network can be replaced by the Transformer network.

[0090] 3. Introduce the transformer network into the end-to-end compression framework.

[0091] a) In one example, the entire framework adopts a transformer network to replace the convolutional based backbone.

[0092] b) In one example, the transformer layers can be a subset of the end-to-end compression framework in collaboration with the convolutional layers.

[0093] 4. A transformer network is involved in the encoding process.

[0094] a) In one example, the Transformer network is applied directly to visual images.

[0095] i. In one example, when extracting window-based attention, the transformer divides the input image into p×p patches, and the output size of the transformer is the same as the input size.

[0096] 1) In one example, p is equal to 2 n .

[0097] 2) In one example, p is equal to 8.

[0098] ii. In one example, when extracting window-based attention, the transformer splits the input image into p×p small patches, and h is smaller than h of the input.

[0099] 1) In one example, p is equal to 2 n .

[0100] 2) In one example, p is equal to 8.

[0101] 3) In one example, the output feature size is half the input size.

[0102] 4) In one example, the output feature size is 1 / 4 of the input size.

[0103] b) In one example, a transformer is applied to a feature. The number of channels of the input feature is denoted as N.

[0104] i. In one example, when extracting window-based attention, the transformer splits the input features into p×p small blocks, and the spatial size of the transformer's output is the same as that of the input.

[0105] 1) In one example, p is equal to 2 n .

[0106] 2) In one example, p is equal to 8.

[0107] 3) In one example, the number of channels of the output features is the same as N.

[0108] 4) In one example, the number of channels of the output features is less than N.

[0109] 5) In one example, the number of channels of the output features is greater than N.

[0110] ii. In one example, when extracting window-based attention, the transformer splits the input features into p×p small blocks, and the size of the transformer's output is smaller than that of the input.

[0111] 1) In one example, p is equal to 2 n .

[0112] 2) In one example, p is equal to 8.

[0113] 3) In one example, the output feature size is half the input size.

[0114] 4) In one example, the output feature size is 1 / 4 of the input size.

[0115] 5) In one example, the number of channels of the output features is the same as N.

[0116] 6) In one example, the number of channels of the output features is less than N.

[0117] 7) In one example, the number of channels of the output features is greater than N.

[0118] 5. The transformer layer is involved in the decoding process.

[0119] a) In one example, the transformer network is applied directly to the latent feature map.

[0120] i. In one example, when extracting window-based attention, the transformer splits the latent feature map into p×p small blocks, and the output size of the transformer is the same as the input size.

[0121] 1) In one example, p is equal to 2 n .

[0122] 2) In one example, p is equal to 8.

[0123] i. In one example, when extracting window-based attention, the transformer splits the latent feature map into p×p small blocks, and the output size of the transformer is larger than the input size.

[0124] 1) In one example, p is equal to 2 n .

[0125] 2) In one example, p is equal to 8.

[0126] 3) In one example, the output feature size is twice the input size.

[0127] 4) In one example, the output feature size is four times the input size.

[0128] b) In one example, a transformer is applied to a feature. The number of channels of the input feature is denoted as N.

[0129] i. In one example, when extracting window-based attention, the transformer splits the input features into p×p small blocks, and the spatial size of the transformer's output is the same as that of the input.

[0130] 1) In one example, p is equal to 2 n .

[0131] 2) In one example, p is equal to 8.

[0132] 3) In one example, the number of channels of the output features is the same as N.

[0133] 4) In one example, the number of channels of the output features is less than N.

[0134] 5) In one example, the number of channels of the output features is greater than N.

[0135] ii. In one example, when extracting window-based attention, the transformer splits the input features into p×p small blocks, and the output size of the transformer is smaller than the input size.

[0136] 1) In one example, p is equal to 2 n .

[0137] 2) In one example, p is equal to 8.

[0138] 3) In one example, the output feature size is half the input size.

[0139] 4) In one example, the output feature size is 1 / 4 of the input size.

[0140] 5) In one example, the number of channels of the output features is the same as N.

[0141] 6) In one example, the number of channels of the output features is less than N.

[0142] 7) In one example, the number of channels of the output features is greater than N.

[0143] 6. Data augmentation strategies are proposed to enhance the learning process of compression or restoration.

[0144] a) In one example, only natural scene images / videos are used to train the network.

[0145] b) In one example, only game scene images / videos are used to train the network.

[0146] c) In one example, natural scene and game scene images / videos are mixed into a training set containing x% game scene images / videos and (1-x%) natural scene images / videos.

[0147] d) In one example, the network is first trained using natural scene images / videos and then fine-tuned using game scene images / videos.

[0148] e) Alternatively, for example, the network is first trained using game scene images / videos and then fine-tuned using natural scene images / videos.

[0149] 5. Examples

[0150] An example of an end-to-end image compression process is shown below. A shift window transformer based encoding block and a shift window transformer based decoding block involve a compression network. The input can be a color picture / frame / video with three channels (e.g., RGB, YUV) or a signal channel picture / frame / video.

[0151] Example #1

[0152] In this embodiment, if Figure 4 As shown, the encoding analysis branch consists of sequentially connected shifted window transformer-based encoding blocks. The GDN layer is connected as an activation function. Similarly, the decoding synthesis branch is stacked with a shifted window transformer-based decoding block and an IGDN layer. The joint autoregressive entropy encoding and decoding module and the context modeling module are used for more accurate probability estimation. The shifted window transformer-based decoding block shares a similar architecture with the encoding block, except that the last convolutional layer is replaced by a deconvolutional layer with stride 2, which expands the spatial scale of the reconstructed feature map.

[0153] Assume that the input of the i-th shifted window transformer based coding block is x with size W×H×N i , the feature forward projection is first applied to x i , transforming the input signal into a higher dimensional feature space, but with a more compact representation f i .f i The size is W×H×E, where E should be less than N. Subsequently, a shifted window transformer is applied to make finer local predictions. Then, convolutional layers are operated with residual skip connections. Feature back-projection is performed, which projects the features back to the input x. i The same size. Finally, a convolutional layer with a stride of 2 is involved, which reduces the spatial scale of the feature map. The output is x i+1 Used as input to the next stage.

[0154] Details of the shift window transformer layer are given in Figure 5 More specifically, a shifted window transformer is constructed during multi-head self-attention extraction, where the shifted window self-attention is blurred to achieve cross-window decorrelation in the second stage. Given a feature f with size W×H×E i , block embedding is performed, which divides the input features into a set of non-overlapping p×p blocks. In this way, f iis embedded as a patch-based feature. Here, p×p is the patch size corresponding to the window size. Self-attention is calculated separately within each window. Specifically, layer normalization and multi-head self-attention are performed to extract local attention. At the same time, multi-layer perception is used after layer normalization. Next, the window is shifted horizontally and vertically by p / 2, and the local attention within the shifted window is calculated. Residual skip connection is used. Finally, the window-based patches are aggregated to obtain the result sf i .

[0155] As used herein, the term "video unit" or "video block" may be a sequence, picture, slice, tile, block, sub-picture, codec tree unit (CTU) / codec tree block (CTB), CTU / CTB row, one or more codec units (CU) / codec blocks (CB), one or more CTU / CTB, one or more virtual pipeline data units (VPDU), sub-region within a picture / slice / tile / block. The term "image compression" may refer to various signal processing methods that compress or process the current input. Input images / videos include, but are not limited to, screen content and natural content.

[0156] The term “window-based attention module” used herein is a model in natural language processing and may also be referred to as a “transformer.” The terms “window-based attention module” and “transformer” may be used interchangeably hereinafter.

[0157] Figure 6 A flow chart of a method 600 for video processing according to an embodiment of the present disclosure is shown. The method 600 is implemented during conversion between a video unit of a video and a bitstream of the video.

[0158] At block 610, for conversion between a video unit of a video and a bitstream of the video, signal processing is applied to the video unit based at least in part on a window-based attention module. The window-based attention module can be applied at different stages during the conversion. For example, the window-based attention module can be located at different locations in the encoder and / or decoder.

[0159] At block 620, a conversion is performed based on the processed video unit. In some embodiments, the conversion may include encoding the video unit into a bitstream. Alternatively or additionally, the conversion may include decoding the video unit from the bitstream. The method 600 may combine interleaved transformer layers and convolutional layers so that it can take advantage of both the convolution operation for information distillation and the shifted window transformer for local analysis and non-local perception, thereby resulting in a more compact signal representation and refined recovery.

[0160] In some embodiments, the signal processing includes restoration of the video unit. In some embodiments, a window-based attention module is applied to compression of the video unit. Compression can include non-learning-based compression and learning-based compression. In one example, the transformer network can be applied to image / video compression, including traditional compression and learning-based compression schemes.

[0161] In some embodiments, the window-based attention module is applied to super-resolution of the video unit. In some embodiments, the window-based attention module is applied to loop filtering in compression of the video unit. Alternatively, the window-based attention module is applied to at least one of pre-processing or post-processing of the video unit.

[0162] In some embodiments, applying the signal processing includes applying the signal processing to the video unit based on a combination of a window-based attention module and a convolutional network. For example, the window-based attention module and the convolutional network can be combined.

[0163] In some embodiments, convolutional layers in a convolutional network are replaced by layers of a window-based attention module. In some embodiments, the convolutional network includes a convolutional-based compression network, and the convolutional layers of the convolutional-based compression network are replaced by layers of a window-based attention module. In some embodiments, the convolutional network includes a convolutional-based super-resolution network, and the convolutional layers of the convolutional-based super-resolution network are replaced by layers of a window-based attention module.

[0164] In some embodiments, a portion of the modules in the convolutional network are replaced by a window-based attention module. In some embodiments, the convolutional network includes a convolutional-based compression network, and the encoder in the convolutional-based compression network is replaced by a window-based attention module. In some other embodiments, the convolutional network includes a convolutional-based compression network, and the decoder in the convolutional-based compression network is replaced by a window-based attention module. In other embodiments, the convolutional network includes a super-resolution network, and the residual block in the super-resolution network is replaced by a window-based attention module.

[0165] In some embodiments, the window-based attention module is applied in the compression framework. In some embodiments, the compression framework uses the window-based attention module instead of the convolution-based network. In some other embodiments, the layers of the window-based attention module are a subset of the compression framework that cooperates with the convolutional layers.

[0166] In some embodiments, the signal processing is an encoding process and the window-based attention module is applied during the encoding process. In some embodiments, the window-based attention module is applied directly to the visual image of the video unit.

[0167] In some embodiments, the window-based attention module splits the input image into p×p small blocks when extracting the window-based attention, and the size of the output feature of the window-based attention module is the same as the size of the input image. In this case, p can be an integer. In some embodiments, p is equal to 2 n , and n can be an integer. Alternatively, p is equal to 8.

[0168] In some embodiments, the window-based attention module divides the input image into p×p small blocks when extracting the window-based attention, and the size of the output feature of the window-based attention module is smaller than the size of the input image, where p is an integer. In some embodiments, p is equal to 2 n , and wherein n is an integer. In some embodiments, p is equal to 8. In some embodiments, the size of the output image is half the size of the input. In some embodiments, the size of the output image is one quarter the size of the input.

[0169] In some embodiments, the window-based attention module is applied to the features of the video unit. In some embodiments, the window-based attention module divides the input features into p×p small blocks when extracting the window-based attention, and the spatial size of the output features of the window-based attention module is the same as the spatial size of the input features. In this case, p can be an integer. In some embodiments, p is equal to 2 n , and wherein n is an integer. Alternatively, p is equal to 8.

[0170] In some embodiments, the window-based attention module divides the input feature into p×p small blocks when extracting the window-based attention, and the size of the output feature of the window-based attention module is smaller than the size of the input feature. In this case, p can be an integer. In some embodiments, p is equal to 2 n , and wherein n is an integer. Alternatively, p is equal to 8. In some embodiments, the size of the output feature is half the size of the input. In some embodiments, the size of the output feature is one quarter the size of the input.

[0171] In some embodiments, the number of channels of the output features is the same as the number of channels of the input features. In some other embodiments, the number of channels of the output features is less than the number of channels of the input features. In some embodiments, the number of channels of the output features is greater than the number of channels of the input features.

[0172] In some embodiments, the signal processing is a decoding process, and the window-based attention module is applied in the decoding process. In some embodiments, the window-based attention module is directly applied to the latent feature map of the video unit.

[0173] In some embodiments, the window-based attention module divides the latent feature map into p×p small blocks when extracting the window-based attention, and the size of the output feature of the window-based attention module is the same as the size of the input latent feature of the window-based attention module. In this case, p can be an integer. In some embodiments, p is equal to 2 n , and wherein n is an integer. In some embodiments, p is equal to 8.

[0174] In some embodiments, the window-based attention module divides the latent feature map into p×p small blocks when extracting the window-based attention, and the size of the output feature of the window-based attention module is larger than the size of the input latent feature of the window-based attention module. In this case, p can be an integer. In some embodiments, p is equal to 2 n , and wherein n is an integer. In some embodiments, p is equal to 8. In some embodiments, the size of the output is twice the size of the input. In some embodiments, the size of the output is four times the size of the input.

[0175] In some embodiments, the window-based attention module can be applied to the features of the video unit. In some embodiments, the window-based attention module divides the input features into p×p small blocks when extracting the window-based attention, and the spatial size of the output features of the window-based attention module is the same as the spatial size of the input features. In this case, p can be an integer. In some embodiments, p is equal to 2 n , and wherein n is an integer. In some embodiments, p is equal to 8.

[0176] In some embodiments, the window-based attention module divides the input feature into p×p small blocks when extracting the window-based attention, and the size of the output feature of the window-based attention module is smaller than the size of the input feature. In this case, p can be an integer. In some embodiments, p is equal to 2 n , and wherein n is an integer. In some embodiments, p is equal to 8. In some embodiments, the size of the output feature is half the size of the input. In some embodiments, the size of the output feature is one quarter the size of the input.

[0177] In some embodiments, the number of channels of the output features is the same as the number of channels of the input features. Alternatively, the number of channels of the output features is less than the number of channels of the input features. In some other embodiments, the number of channels of the output features is greater than the number of channels of the input features.

[0178] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method includes: applying signal processing to a video unit of the video based at least in part on a window-based attention module; and generating a bitstream of the video based on the processed video unit.

[0179] According to some other embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: applying signal processing to video units of the video based at least in part on a window-based attention module; generating a bitstream of the video based on the processed video units; and storing the bitstream in a non-transitory computer-readable recording medium.

[0180] Figure 7 A flow chart of a method 700 for video processing according to an embodiment of the present disclosure is shown. The method 700 is implemented during the conversion between a video unit of a video and a bitstream of the video. In some embodiments, the method 700 can be implemented in combination with the method 600. Alternatively, the method 700 can be implemented independently.

[0181] At block 710, a data augmentation process for training a window-based attention module is applied during the conversion between a video unit of a video and a bitstream of the video. For example, a data augmentation strategy can be used to enhance the learning process of compression or restoration.

[0182] At block 720, signal processing is performed on the video unit based on the trained window-based attention module. For example, the signal processing may be a compression process or a restoration process.

[0183] At block 730, a conversion is performed based on the processed video unit. In some embodiments, the conversion may include encoding the video unit into a bitstream. Alternatively or additionally, the conversion may include decoding the video unit from the bitstream. The method 700 enables an appropriate training window-based attention module.

[0184] In some embodiments, only natural scene video units are used to train the window-based attention module. In some embodiments, only game scene video units are used to train the window-based attention module.

[0185] In some embodiments, the natural scene video unit and the game scene video unit are mixed as a training set for the window-based attention module. In some embodiments, the window-based attention module is first trained using the natural scene video unit and then fine-tuned using the game scene video unit. In one example, the natural scene and game scene images / videos are mixed into a training set containing x% game scene images / videos and (1-x%) natural scene images / videos. In some embodiments, the window-based attention module is first trained using the game scene video unit and then fine-tuned using the natural scene video unit.

[0186] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method includes: applying a data enhancement process for training a window-based attention module; performing signal processing on a video unit of the video based on the trained window-based attention module; and generating a bitstream of the video based on the processed video unit.

[0187] According to some other embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: applying a data enhancement process for training a window-based attention module; performing signal processing on a video unit of the video based on the trained window-based attention module; generating a bitstream of the video based on the processed video unit; and storing the bitstream in a non-transitory computer-readable recording medium.

[0188] Embodiments of the present disclosure may be described according to the following clauses, and the features of the following clauses may be combined in any reasonable way.

[0189] Item 1. A method of video processing, comprising: applying signal processing to the video unit during conversion between a video unit of a video and a bitstream of the video, at least in part based on a window-based attention module; and performing the conversion based on the processed video unit.

[0190] Clause 2. The method of clause 1, wherein the signal processing comprises restoration of the video unit.

[0191] Clause 3. A method according to clause 1, wherein the window-based attention module is applied to compression of the video unit, and wherein the compression includes non-learning-based compression and learning-based compression.

[0192] Clause 4. The method of clause 1, wherein the window-based attention module is applied for super-resolution of the video unit.

[0193] Clause 5. The method of clause 1, wherein the window-based attention module is applied to in-loop filtering of compression of the video unit.

[0194] Clause 6. The method of clause 1, wherein the window-based attention module is applied to at least one of pre-processing or post-processing of the video unit.

[0195] Clause 7. A method according to clause 1, wherein applying the signal processing comprises: applying the signal processing to the video unit based on a combination of the window-based attention module and a convolutional network.

[0196] Clause 8. A method according to clause 7, wherein the convolutional layers in the convolutional network are replaced by layers of the window-based attention module.

[0197] Clause 9. A method according to clause 8, wherein the convolutional network includes a convolution-based compression network, and the convolutional layers of the convolution-based compression network are replaced by the layers of the window-based attention module.

[0198] Clause 10. A method according to clause 8, wherein the convolutional network includes a convolutional-based super-resolution network, and the convolutional layers of the convolutional-based super-resolution network are replaced by the layers of the window-based attention module.

[0199] Clause 11. A method according to Clause 7, wherein a portion of the modules in the convolutional network are replaced by the window-based attention module.

[0200] Clause 12. A method according to clause 11, wherein the convolutional network includes a convolution-based compression network, and the encoder in the convolution-based compression network is replaced by the window-based attention module.

[0201] Clause 13. A method according to clause 11, wherein the convolutional network includes a convolution-based compression network, and the decoder in the convolution-based compression network is replaced by the window-based attention module.

[0202] Clause 14. A method according to clause 11, wherein the convolutional network includes a super-resolution network, and the residual blocks in the super-resolution network are replaced by the window-based attention module.

[0203] Clause 15. The method of clause 1, wherein the window-based attention module is applied in a compression framework.

[0204] Clause 16. The method of clause 15, wherein the compression framework uses the window-based attention module instead of a convolution-based network.

[0205] Clause 17. A method according to clause 15, wherein the layers of the window-based attention module are a subset of the compression framework in collaboration with convolutional layers.

[0206] Clause 18. A method according to clause 1, wherein the signal processing is an encoding process and the window-based attention module is applied in the encoding process.

[0207] Clause 19. The method of clause 18, wherein the window-based attention module is applied directly to the visual image of the video unit.

[0208] Clause 20. A method according to Clause 19, wherein the window-based attention module divides the input image into p×p small blocks when extracting window-based attention, and the size of the output feature of the window-based attention module is the same as the size of the input image, where p is an integer.

[0209] Clause 21. The method according to clause 20, wherein p is equal to 2 n , and wherein n is an integer, or wherein p is equal to 8.

[0210] Clause 22. A method according to Clause 19, wherein the window-based attention module divides the input image into p×p small blocks when extracting window-based attention, and the size of the output feature of the window-based attention module is smaller than the size of the input image, where p is an integer.

[0211] Clause 23. The method according to clause 22, wherein p is equal to 2 n , and wherein n is an integer, or wherein p is equal to 8, or wherein the size of the output image is half the input size, or wherein the size of the output image is one quarter the input size.

[0212] Clause 24. The method of clause 18, wherein the window-based attention module is applied to features of the video units.

[0213] Clause 25. A method according to Clause 24, wherein the window-based attention module divides the input features into p×p small blocks when extracting window-based attention, and the spatial domain size of the output features of the window-based attention module is the same as the spatial domain size of the input features, where p is an integer.

[0214] Clause 26. The method according to clause 25, wherein p is equal to 2 n , and wherein n is an integer, or wherein p is equal to 8.

[0215] Clause 27. A method according to Clause 24, wherein the window-based attention module divides the input features into p×p small blocks when extracting window-based attention, and the size of the output features of the window-based attention module is smaller than the size of the input features, where p is an integer.

[0216] Clause 28. The method according to clause 27, wherein p is equal to 2 n , and wherein n is an integer, or wherein p is equal to 8, or wherein the size of the output feature is half the size of the input, or wherein the size of the output feature is one quarter the size of the input.

[0217] Clause 29. A method according to clause 26 or 28, wherein the number of channels of the output feature is the same as the number of channels of the input feature, wherein the number of channels of the output feature is less than the number of channels of the input feature, or wherein the number of channels of the output feature is greater than the number of channels of the input feature.

[0218] Clause 30. The method of clause 1, wherein the signal processing is a decoding process and the window-based attention module is applied in the decoding process.

[0219] Clause 31. A method according to clause 30, wherein the window-based attention module is directly applied to the latent feature map of the video unit.

[0220] Clause 32. A method according to clause 31, wherein the window-based attention module divides the latent feature map into p×p small blocks when extracting window-based attention, and the size of the output feature of the window-based attention module is the same as the size of the input latent feature of the window-based attention module, where p is an integer.

[0221] Clause 33. The method according to clause 32, wherein p is equal to 2 n , and wherein n is an integer, or wherein p is equal to 8.

[0222] Clause 34. A method according to Clause 31, wherein the window-based attention module divides the latent feature map into p×p small blocks when extracting window-based attention, and the size of the output feature of the window-based attention module is larger than the size of the input latent feature of the window-based attention module, where p is an integer.

[0223] Clause 35. The method according to clause 34, wherein p is equal to 2 n, and wherein n is an integer, or wherein p is equal to 8, or wherein the size of the output is twice the size of the input, or wherein the size of the output is four times the size of the input.

[0224] Clause 36. The method of clause 30, wherein the window-based attention module is applied to features of the video unit.

[0225] Clause 37. A method according to Clause 36, wherein the window-based attention module divides the input features into p×p small blocks when extracting window-based attention, and the spatial domain size of the output features of the window-based attention module is the same as the spatial domain size of the input features, where p is an integer.

[0226] Clause 38. The method according to clause 37, wherein p is equal to 2 n , and wherein n is an integer, or wherein p is equal to 8.

[0227] Clause 39. A method according to Clause 36, wherein the window-based attention module divides the input features into p×p small blocks when extracting window-based attention, and the size of the output features of the window-based attention module is smaller than the size of the input features, where p is an integer.

[0228] Clause 40. The method according to clause 39, wherein p is equal to 2 n , and wherein n is an integer, or wherein p is equal to 8, or wherein the size of the output feature is half the size of the input, or wherein the size of the output feature is one quarter the size of the input.

[0229] Clause 41. A method according to clause 38 or 40, wherein the number of channels of the output feature is the same as the number of channels of the input feature, wherein the number of channels of the output feature is less than the number of channels of the input feature, or wherein the number of channels of the output feature is greater than the number of channels of the input feature.

[0230] Item 42. A method of video processing, comprising: applying a data augmentation process for training a window-based attention module during conversion between a video unit of a video and a bitstream of the video; performing signal processing on the video unit based on the trained window-based attention module; and performing the conversion based on the processed video unit.

[0231] Clause 43. A method according to clause 42, wherein only natural scene video units are used to train the window-based attention module, or wherein only game scene video units are used to train the window-based attention module.

[0232] Clause 44. A method according to Clause 42, wherein natural scene video units and game scene video units are mixed as a training set for the window-based attention module.

[0233] Clause 45. A method according to clause 42, wherein the window-based attention module is first trained using natural scene video units and then fine-tuned using game scene video units.

[0234] Clause 46. A method according to clause 42, wherein the window-based attention module is first trained using game scene video units and then fine-tuned using natural scene video units.

[0235] Clause 47. The method of any one of clauses 1 to 46, wherein the converting comprises encoding the video unit into the bitstream.

[0236] Clause 48. The method of any one of clauses 1 to 46, wherein the converting comprises decoding the video unit from the bitstream.

[0237] Clause 49. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of clauses 1 to 48.

[0238] Clause 50. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any one of clauses 1 to 48.

[0239] Item 51. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: applying signal processing to video units of the video based at least in part on a window-based attention module; and generating a bitstream of the video based on the processed video units.

[0240] Item 52. A method for storing a bitstream of a video, comprising: applying signal processing to video units of the video based at least in part on a window-based attention module; generating a bitstream of the video based on the processed video units; and storing the bitstream in a non-transitory computer-readable recording medium.

[0241] Item 53. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: applying a data enhancement process for training a window-based attention module; performing signal processing on video units of the video based on the trained window-based attention module; and generating a bitstream of the video based on the processed video units.

[0242] Item 54. A method for storing a bitstream of a video, comprising: applying a data augmentation process for training a window-based attention module; performing signal processing on video units of the video based on the trained window-based attention module; generating a bitstream of the video based on the processed video units; and storing the bitstream in a non-transitory computer-readable recording medium.

[0243] Example Device

[0244] Figure 8 A block diagram of a computing device 800 in which various embodiments of the present disclosure may be implemented is shown. The computing device 800 may be implemented as a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300), or may be included in a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300).

[0245] It should be understood that Figure 8 The computing device 800 shown in FIG. 8 is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of the present disclosure in any way.

[0246] like Figure 8 As shown, computing device 800 includes a general computing device 800. Computing device 800 may include at least one or more processors or processing units 810, memory 820, storage unit 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860.

[0247] In some embodiments, the computing device 800 can be implemented as any user terminal or server terminal with computing power. The server side can be a server, a large computing device, etc. provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / video camera, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, including the accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 800 can support any type of interface to the user (such as a "wearable" circuit, etc.).

[0248] Processing unit 810 may be a physical processor or a virtual processor and may implement various processes based on programs stored in memory 820. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 800. Processing unit 810 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0249] The computing device 800 typically includes various computer storage media. Such media can be any media accessible to the computing device 800, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory), or any combination thereof. The storage unit 830 can be any removable or non-removable medium, and can include a machine-readable medium, such as a memory, a flash drive, a disk, or another other medium that can be used to store information and / or data and can be accessed in the computing device 800.

[0250] The computing device 800 may also include additional removable / non-removable volatile / non-volatile storage media. Figure 8 Although not shown in the figure, a magnetic disk drive for reading from and / or writing to a removable and non-volatile magnetic disk and an optical disk drive for reading from and / or writing to a removable, non-volatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more data medium interfaces.

[0251] The communication unit 840 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 800 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the computing device 800 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0252] The input device 850 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. The output device 860 may be one or more of various output devices, such as a display, a speaker, a printer, etc. With the aid of the communication unit 840, the computing device 800 may also communicate with one or more external devices (not shown), such as storage devices and display devices, and the computing device 800 may also communicate with one or more devices that enable a user to interact with the computing device 800, or any device that enables the computing device 800 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0253] In some embodiments, instead of being integrated in a single device, some or all components of the computing device 800 may also be arranged in a cloud computing architecture. In a cloud computing architecture, components may be provided remotely and work together. In some embodiments, cloud computing provides computing, software, data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (e.g., the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a remote server. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote data center. Cloud computing infrastructure may provide services through a shared data center, although they appear as a single access point for users. Therefore, the cloud computing architecture may be used to provide the components and functions described herein from a service provider at a remote location. Alternatively, they may be provided by a conventional server, or installed directly or otherwise on a client device.

[0254] In an embodiment of the present disclosure, the computing device 800 may be used to implement video encoding / decoding. The memory 820 may include one or more video encoding / decoding modules 825 having one or more program instructions. These modules may be accessed and executed by the processing unit 810 to perform the functions of the various embodiments described herein.

[0255] In an example embodiment performing video encoding, input device 850 may receive video data as input to be encoded 870. The video data may be processed, for example, by video codec module 825 to produce an encoded bitstream. The encoded bitstream may be provided as output 880 via output device 860.

[0256] In an example embodiment performing video decoding, the input device 850 may receive an encoded bitstream as input 870. The encoded bitstream may be processed, for example, by the video codec module 825 to produce decoded video data. The decoded video data may be provided as output 880 via the output device 860.

[0257] Although the present disclosure has been specifically shown and described with reference to its preferred embodiments, it will be appreciated by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the above description of the embodiments of the present application is not intended to be limiting.

Claims

1. A video processing method, comprising: applying signal processing to the video unit based at least in part on a window-based attention module during conversion between a video unit of a video and a bitstream of the video; as well as The conversion is performed based on the processed video unit.

2. The method of claim 1, wherein the signal processing comprises restoration of the video unit.

3. The method of claim 1, wherein the window-based attention module is applied to the compression of the video unit, and The compression includes non-learning-based compression and learning-based compression.

4. The method of claim 1, wherein the window-based attention module is applied for super-resolution of the video unit.

5. The method of claim 1, wherein the window-based attention module is applied to in-loop filtering in compression of the video unit.

6. The method of claim 1, wherein the window-based attention module is applied to at least one of pre-processing or post-processing of the video unit.

7. The method of claim 1 , wherein applying the signal processing comprises: The signal processing is applied to the video unit based on a combination of the window-based attention module and a convolutional network.

8. The method of claim 7, wherein the convolutional layers in the convolutional network are replaced by layers of the window-based attention module.

9. The method of claim 8, wherein the convolutional network comprises a convolution-based compression network, and the convolutional layers of the convolution-based compression network are replaced by the layers of the window-based attention module.

10. The method of claim 8, wherein the convolutional network comprises a convolutional-based super-resolution network, and convolutional layers of the convolutional-based super-resolution network are replaced by the layers of the window-based attention module.

11. The method according to claim 7, wherein a portion of the modules in the convolutional network are replaced by the window-based attention module.

12. The method of claim 11, wherein the convolutional network comprises a convolution-based compression network, and the encoder in the convolution-based compression network is replaced by the window-based attention module.

13. The method of claim 11, wherein the convolutional network comprises a convolution-based compression network, and a decoder in the convolution-based compression network is replaced by the window-based attention module.

14. The method of claim 11, wherein the convolutional network comprises a super-resolution network, and the residual block in the super-resolution network is replaced by the window-based attention module.

15. The method of claim 1, wherein the window-based attention module is applied in a compression framework.

16. The method of claim 15, wherein the compression framework uses the window-based attention module instead of a convolutional-based network.

17. The method of claim 15, wherein the layers of the window-based attention module are a subset of the compression framework in collaboration with convolutional layers.

18. The method of claim 1, wherein the signal processing is an encoding process and the window-based attention module is applied in the encoding process.

19. The method of claim 18, wherein the window-based attention module is applied directly to the visual image of the video unit.

20. The method of claim 19, wherein the window-based attention module divides the input image into p×p small blocks when extracting the window-based attention, and the size of the output feature of the window-based attention module is the same as the size of the input image, where p is an integer.

21. The method of claim 20, wherein p is equal to 2 n , and where n is an integer, or Where p is equal to 8.

22. The method of claim 19, wherein the window-based attention module divides the input image into p×p small blocks when extracting the window-based attention, and the size of the output feature of the window-based attention module is smaller than the size of the input image, where p is an integer.

23. The method of claim 22, wherein p is equal to 2 n , and where n is an integer, or where p is equal to 8, or wherein the size of the output image is half the size of the input, or Wherein the size of the output image is one quarter of the input size.

24. The method of claim 18, wherein the window-based attention module is applied to features of the video units.

25. The method according to claim 24, wherein the window-based attention module divides the input features into p×p small blocks when extracting window-based attention, and the spatial domain size of the output features of the window-based attention module is the same as the spatial domain size of the input features, where p is an integer.

26. The method of claim 25, wherein p is equal to 2 n , and where n is an integer, or Where p is equal to 8.

27. The method of claim 24, wherein the window-based attention module divides the input features into p×p small blocks when extracting the window-based attention, and the size of the output features of the window-based attention module is smaller than the size of the input features, where p is an integer.

28. The method of claim 27, wherein p is equal to 2 n , and where n is an integer, or where p is equal to 8, or wherein the size of the output features is half the size of the input, or Wherein the size of the output features is one quarter of the input size.

29. The method according to claim 26 or 28, wherein the number of channels of the output features is the same as the number of channels of the input features, wherein the number of channels of the output feature is less than the number of channels of the input feature, or The number of channels of the output features is greater than the number of channels of the input features.

30. The method of claim 1, wherein the signal processing is a decoding process and the window-based attention module is applied in the decoding process.

31. The method of claim 30, wherein the window-based attention module is applied directly to the latent feature map of the video unit.

32. A method according to claim 31, wherein the window-based attention module divides the latent feature map into p×p small blocks when extracting window-based attention, and the size of the output feature of the window-based attention module is the same as the size of the input latent feature of the window-based attention module, where p is an integer.

33. The method of claim 32, wherein p is equal to 2 n , and where n is an integer, or Where p is equal to 8.

34. A method according to claim 31, wherein the window-based attention module divides the latent feature map into p×p small blocks when extracting window-based attention, and the size of the output feature of the window-based attention module is larger than the size of the input latent feature of the window-based attention module, where p is an integer.

35. The method of claim 34, wherein p is equal to 2 n , where n is an integer, or where p is equal to 8, or wherein the size of the output is twice the size of the input, or wherein the size of the output is four times the size of the input.

36. The method of claim 30, wherein the window-based attention module is applied to features of the video units.

37. A method according to claim 36, wherein the window-based attention module divides the input features into p×p small blocks when extracting window-based attention, and the spatial domain size of the output features of the window-based attention module is the same as the spatial domain size of the input features, where p is an integer.

38. The method of claim 37, wherein p is equal to 2 n , and where n is an integer, or Where p is equal to 8.

39. A method according to claim 36, wherein the window-based attention module divides the input features into p×p small blocks when extracting window-based attention, and the size of the output feature of the window-based attention module is smaller than the size of the input feature, where p is an integer.

40. The method of claim 39, wherein p is equal to 2 n , where n is an integer, or where p is equal to 8, or wherein the size of the output features is half the size of the input, or Wherein the size of the output features is one quarter of the input size.

41. The method according to claim 38 or 40, wherein the number of channels of the output features is the same as the number of channels of the input features, wherein the number of channels of the output feature is less than the number of channels of the input feature, or, The number of channels of the output features is greater than the number of channels of the input features.

42. A method for video processing, comprising: applying a data augmentation process for training a window-based attention module during conversion between video units of a video and a bitstream of said video; performing signal processing on the video unit based on the trained window-based attention module; as well as The conversion is performed based on the processed video unit.

43. The method of claim 42, wherein only natural scene video units are used to train the window-based attention module, or Only the game scene video unit is used to train the window-based attention module.

44. The method of claim 42, wherein natural scene video units and game scene video units are mixed as a training set for the window-based attention module.

45. A method according to claim 42, wherein the window-based attention module is first trained using natural scene video units and then fine-tuned using game scene video units.

46. ​​A method according to claim 42, wherein the window-based attention module is first trained using game scene video units and then fine-tuned using natural scene video units.

47. The method of any one of claims 1 to 46, wherein the converting comprises encoding the video unit into the bitstream.

48. The method of any one of claims 1 to 46, wherein the converting comprises decoding the video unit from the bitstream.

49. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 48.

50. A non-transitory computer readable storage medium storing instructions, the instructions causing a processor to execute the method according to any one of claims 1 to 48.

51. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: applying signal processing to video units of the video based at least in part on a window-based attention module; as well as A bitstream of the video is generated based on the processed video unit.

52. A method for storing a bitstream of a video, comprising: applying signal processing to video units of the video based at least in part on a window-based attention module; generating a bitstream of the video based on the processed video unit; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

53. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: Apply the data augmentation process for training the window-based attention module; performing signal processing on video units of the video based on the trained window-based attention module; as well as A bitstream of the video is generated based on the processed video unit.

54. A method for storing a bitstream of a video, comprising: Apply the data augmentation process for training the window-based attention module; performing signal processing on video units of the video based on the trained window-based attention module; generating a bitstream of the video based on the processed video unit; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images