End-to-end video compression method and apparatus using block partition information as additional information

Abstract

Disclosed in the present embodiment is an end-to-end video compression method and apparatus using block partition information as additional information. In this embodiment, an image decoding device decodes block partition information of a current image, and inputs the decoded block partition information into a deep learning-based extraction model in order to generate block partition potential features. An image decoding device decodes a code of a current image that has been compressed, and then inputs the decoded code into a converter-based video decoder in order to generate a reconstructed image of the current image. During generation of the reconstructed image, the image decoding device fuses the block partition potential features with the intermediate potential features in a video decoder.

Description

Technical Field

[0001] The present invention relates to an end-to-end video compression method and device using block partition information as additional information. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] Since video data has a large amount of data compared to audio data or still image data, video data requires a large amount of hardware resources (including memory) to store or transmit the uncompressed video data.

[0004] Accordingly, an encoder is typically used to compress and store or transmit video data. A decoder receives the compressed video data, decompresses the received compressed video data, and plays the decompressed video data. Video compression technologies include H.264 / Advanced Video Codec (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Codec (VVC), which improves codec efficiency by approximately 30% or more compared to HEVC.

[0005] However, as image size, resolution, and frame rate continue to increase, the amount of data to be encoded will also increase. Accordingly, new compression technologies are needed that provide higher encoding and decoding efficiency and improved image enhancement effects than existing compression technologies.

[0006] In order to improve the prediction performance in video compression, end-to-end video compression based on deep learning is utilized. The structure for end-to-end video compression adopts the form of the basic Joint Photographic Experts Group (JPEG) compression algorithm involving transformation → quantization → entropy coding and decoding → inverse transformation. In order to apply deep learning to the algorithm, the existing transformation method, namely the Discrete Cosine Transform (DCT), is replaced by a neural network. The quantization process of converting a continuous signal into a discrete signal is an operation that is nowhere differentiable and therefore cannot withstand backpropagation for deep learning. Therefore, during training, quantization is approximated as additive noise. The inverse transformation process is not a complete inverse function of the transformation process implemented by the neural network. However, since the neural network corresponding to the inverse transformation process is trained by calculating the final loss as the mean square error (MSE), the inverse transformation process can be approximated as the inverse function of the transformation process. On the other hand, in order to improve the efficiency of video coding and decoding and enhance video quality, a method for effectively utilizing additional information in the above-mentioned end-to-end video compression process is needed. Summary of the Invention

[0007] Technical issues

[0008] The present invention is directed to providing an end-to-end video compression method and apparatus, which are used to generate block partition structure information and block partition characteristics as additional information in end-to-end video compression, and then utilize the generated additional information in video encoding / decoding.

[0009] Technical Solution

[0010] At least one aspect of the present invention provides a method for reconstructing a current video by a video decoding device. The method includes decoding block partition information of the current video from a bitstream. The method also includes generating block partition latent features by inputting the block partition information into a deep learning-based extraction model. The method also includes decoding a compressed code that is a compression of the current video from the bitstream. The method also includes inputting the decoded code into a transformer-based video decoder to generate a reconstructed video of the current video. When generating the reconstructed video, the method also includes fusing the block partition latent features into intermediate latent features in the video decoder.

[0011] Another aspect of the present invention provides a method for encoding a current video using a video encoding device. The method includes generating block partition information for the current video. The method also includes generating block partition latent features by inputting the block partition information into a deep learning-based extraction model. The method also includes generating a compressed code as a compression of the current video by inputting the current video into a transformer-based video encoder. The method also includes encoding the block partition information. During the generation of the compressed code, the method also includes fusing the block partition latent features into intermediate latent features in the video encoder.

[0012] Another aspect of the present invention provides a computer-readable recording medium storing a bitstream generated by a video encoding method. The video encoding method includes generating block partition information of a current video. The video encoding method also includes generating block partition latent features by inputting the block partition information into a deep learning-based extraction model. The video encoding method also includes generating a compressed code as a compression of the current video by inputting the current video into a transformer-based video encoder. The video encoding method also includes encoding the block partition information. When generating the compressed code, the video encoding method also includes fusing the block partition latent features into intermediate latent features in the video encoder.

[0013] Beneficial effects

[0014] As described above, the present invention provides an end-to-end video compression method and apparatus that generates block partition structure information and block partition characteristics as additional information during end-to-end video compression, and then utilizes the generated additional information for video encoding / decoding. Consequently, the end-to-end video compression method and apparatus improve video encoding and decoding efficiency and enhance video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of a video encoding device that can implement the technology of the present invention.

[0016] Figure 2 A method of partitioning a block using a quadtree plus binary tree ternary tree (QTBTTT) structure is shown.

[0017] Figure 3a and Figure 3b A plurality of intra prediction modes including a wide-angle intra prediction mode are shown.

[0018] Figure 4 Shows the neighboring blocks of the current block.

[0019] Figure 5 is a block diagram of a video decoding device that can implement the technology of the present invention.

[0020] Figure 6 is a schematic diagram illustrating the operation of a convolutional layer.

[0021] Figure 7 is a schematic diagram showing a single image super resolution (SISR) network.

[0022] Figure 8 is a schematic diagram showing a residual block used in SISR.

[0023] Figure 9 is a schematic diagram illustrating a fixed coefficient in-loop filter based on a convolutional neural network or CNN.

[0024] Figure 10a and Figure 10b is a schematic diagram showing the self-attention of a vector.

[0025] Figure 11a and Figure 11b is a schematic diagram showing the self-attention of a matrix.

[0026] Figure 12 is a schematic diagram showing multi-head attention.

[0027] Figure 13 is a schematic diagram showing the structure of a converter.

[0028] Figure 14 is a schematic diagram showing the input preprocessing of the visual transformer.

[0029] Figure 15 is a schematic diagram illustrating a deep learning-based end-to-end video compression model according to at least one embodiment of the present invention.

[0030] Figure 16 is a schematic diagram illustrating an analysis transform and a synthesis transform according to at least one embodiment of the present invention.

[0031] Figure 17 is a schematic diagram illustrating an end-to-end video compression apparatus including a super-prior model according to at least one embodiment of the present invention.

[0032] Figure 18 2 is a schematic diagram illustrating an end-to-end video compression apparatus including a super-prior model according to another embodiment of the present invention.

[0033] Figure 19 is a schematic diagram illustrating an end-to-end video compression apparatus utilizing block partition information according to at least one embodiment of the present invention.

[0034] Figure 20 is a schematic diagram illustrating block partition information according to at least one embodiment of the present invention.

[0035] Figure 21 is a schematic diagram illustrating block partition information according to another embodiment of the present invention.

[0036] Figure 22 is a schematic diagram illustrating the generation of potential features according to at least one embodiment of the present invention.

[0037] Figure 23 is a schematic diagram illustrating a fusion of potential features according to at least one embodiment of the present invention.

[0038] Figure 24 is a flowchart of a method for encoding a current video by a video encoding apparatus according to at least one embodiment of the present invention.

[0039] Figure 25 is a flowchart of a method for reconstructing a current video by a video decoding device according to at least one embodiment of the present invention. DETAILED DESCRIPTION

[0040] Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying illustrative drawings. In the following description, the same reference numerals represent the same elements, even though the elements are shown in different drawings. In addition, in the following description of some embodiments, when it is believed that the detailed description of related known components and functions obscures the subject matter of the present invention, the detailed description of the related known components and functions may be omitted for the sake of clarity and brevity.

[0041] Figure 1 FIG. 1 is a block diagram of a video encoding device that can implement the technology of the present invention. Figure 1 , a video encoding device and components of the device are described.

[0042] The encoding device may include: an image splitter 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filtering unit 180 and a memory 190.

[0043] Each component of the encoding device can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the function of the software corresponding to each component.

[0044] A video consists of one or more sequences including multiple images. Each image is divided into multiple regions, and encoding is performed on each region. For example, an image is divided into one or more tiles or / and slices. Here, one or more tiles can be defined as a tile group. Each tile or / and slice is divided into one or more coding tree units (CTUs). In addition, each CTU is divided into one or more coding units (CUs) through a tree structure. The information applied to each coding unit (CU) is encoded as the syntax of the CU, and the information commonly applied to the CUs included in a CTU is encoded as the syntax of the CTU. In addition, the information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and the information applied to all blocks constituting one or more images is encoded as a picture parameter set (PPS) or a picture header. In addition, information commonly referenced by multiple images is encoded as a sequence parameter set (SPS). In addition, information commonly referenced by one or more SPSs is encoded as a video parameter set (VPS). In addition, information commonly applied to one tile or tile group may also be encoded as syntax of the tile or tile group header.The syntax included in the SPS, PPS, slice header, tile or tile group header may be referred to as a high-level syntax.

[0045] The image splitter 110 determines the size of a codec tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as a syntax of an SPS or PPS and transmitted to a video decoding apparatus.

[0046] The image splitter 110 splits each image constituting a video into a plurality of codec tree units (CTUs) of a predetermined size, and then recursively splits the CTUs using a tree structure. Leaf nodes in the tree structure become codec units (CUs), which are basic units of coding.

[0047] The tree structure can be a quadtree (QT), in which a higher node (or parent node) is split into four lower nodes (or child nodes) of the same size. The tree structure can also be a binary tree (BT), in which a higher node is split into two lower nodes. The tree structure can also be a ternary tree (TT), in which a higher node is split into three lower nodes at a ratio of 1:2:1. The tree structure can also be a structure in which two or more structures of the QT structure, the BT structure, and the TT structure are mixed. For example, a quadtree plus binary tree (QTBT) structure can be used, or a quadtree plus binary tree ternary tree (QTBTTT) structure can be used. Here, a binary tree ternary tree (BTTT) is added to the tree structure to be called a multiple-type tree (MTT).

[0048] Figure 2 It is a schematic diagram for describing a method of dividing a block by using a QTBTTT structure.

[0049] like Figure 2 As shown, the CTU can first be split into a QT structure. The quadtree splitting can be recursive until the size of the split block reaches the minimum block size (MinQTSize) of the leaf node allowed in QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder 155 and notified to the video decoding device with a signal. When the leaf node of QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, the leaf node can be further split into at least one of a BT structure or a TT structure. There can be multiple splitting directions in the BT structure and / or the TT structure. For example, there can be two directions, namely, the direction of splitting the blocks of the corresponding node horizontally and the direction of splitting the blocks of the corresponding node vertically. As shown in FIG. Figure 2 As shown, when MTT splitting starts, the entropy encoder 155 encodes a second flag (mtt_split_flag) indicating whether the node is split, and a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (binary or trifurcated) when the node is split, and notifies the video decoding device of the same with a signal.

[0050] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes in the lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may also be encoded. When the value of the CU split flag (split_cu_flag) indicates that each node is not split, the block of the corresponding node becomes a leaf node in the partition tree structure and becomes a CU, which is the basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that each node is split, the video encoding device first starts encoding the first flag in the above-mentioned scheme.

[0051] When QTBT is used as another example of a tree structure, there may be two types, namely, a type in which the block of the corresponding node is horizontally split into two blocks of the same size (i.e., symmetrical horizontal splitting) and a type in which the block of the corresponding node is vertically split into two blocks of the same size (i.e., symmetrical vertical splitting). The entropy encoder 155 encodes a split flag (split_flag) indicating whether each node of the BT structure is split into blocks of the lower layer and split type information indicating the split type, and transmits them to the video decoding device. On the other hand, there may also be a type in which the block of the corresponding node is split into two blocks that are asymmetric to each other. The asymmetric form may include a form in which the block of the corresponding node is split into two rectangular blocks with a size ratio of 1:3, or may also include a form in which the block of the corresponding node is split in a diagonal direction.

[0052] The CU can have various sizes depending on the QTBT or QTBTTT partitioned from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." When QTBTTT partitioning is adopted, the shape of the current block can also be a rectangular shape in addition to a square shape.

[0053] The predictor 120 predicts the current block to generate a predicted block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.

[0054] Typically, each current block in an image can be predictively encoded. Typically, prediction of the current block can be performed using intra-frame prediction techniques (which utilize data from the image that includes the current block) or inter-frame prediction techniques (which utilize data from an image that was encoded before the image that includes the current block). Inter-frame prediction includes both unidirectional prediction and bidirectional prediction.

[0055] The intra-frame predictor 122 predicts pixels in the current block by using pixels (reference pixels) located adjacent to the current block in the current image including the current block. There are multiple intra-frame prediction modes according to the prediction direction. For example, Figure 3aAs shown, the plurality of intra prediction modes may include two non-directional modes including a planar mode and a DC mode, and may include 65 directional modes. Neighboring pixels to be used and algorithm equations are defined differently according to each prediction mode.

[0056] In order to perform efficient directional prediction for a current block with a rectangular shape, we can additionally use Figure 3b The directional modes are shown in the figure with dotted arrows (#67 to #80, intra-frame prediction modes #-1 to #-14). The directional modes can be called "wide angle intra-frame prediction modes". Figure 3b , the arrows indicate the corresponding reference samples used for prediction, rather than indicating the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra prediction mode is a mode in which prediction is performed in a direction opposite to the specific direction mode without additional bit transmission. In this case, in the wide-angle intra prediction mode, some wide-angle intra prediction modes available for the current block can be determined by the ratio of the width to the height of the current block having a rectangular shape. For example, when the current block has a rectangular shape with a height less than the width, a wide-angle intra prediction mode (intra prediction modes #67 to #80) with an angle less than 45 degrees is available. When the current block has a rectangular shape with a width greater than the height, a wide-angle intra prediction mode with an angle greater than -135 degrees is available.

[0057] The intra-frame predictor 122 can determine the intra-frame prediction to be used for encoding the current block. In some examples, the intra-frame predictor 122 can encode the current block by utilizing multiple intra-frame prediction modes, and can also select an appropriate intra-frame prediction mode to use from a test mode. For example, the intra-frame predictor 122 can calculate the rate-distortion value by utilizing a rate-distortion analysis of multiple test intra-frame prediction modes, and can also select the intra-frame prediction mode with the best rate-distortion characteristics from the test mode.

[0058] The intra-frame predictor 122 selects an intra-frame prediction mode from a plurality of intra-frame prediction modes and predicts the current block by using adjacent pixels (reference pixels) and an algorithm equation determined according to the selected intra-frame prediction mode. The entropy encoder 155 encodes information about the selected intra-frame prediction mode and transmits it to the video decoding device.

[0059] The inter-frame predictor 124 generates a prediction block for the current block by utilizing motion compensation processing. The inter-frame predictor 124 searches for a block that is most similar to the current block in a reference image that has been encoded and decoded earlier than the current image, and generates a prediction block for the current block by utilizing the searched block. In addition, a motion vector (MV) is generated, which corresponds to the displacement between the current block in the current image and the prediction block in the reference image. Typically, motion estimation is performed on the luminance (luma) component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. The motion information including the information of the reference image and the information about the motion vector used to predict the current block is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0060] The inter-frame predictor 124 may also interpolate a reference image or reference block to increase prediction accuracy. In other words, subsamples are interpolated between two consecutive integer samples by applying filter coefficients to a plurality of consecutive integer samples. When searching for a block most similar to the current block in the interpolated reference image, fractional precision rather than integer sample precision may be used for the motion vector. The precision or resolution of the motion vector may be set differently for each target region to be encoded, such as a unit such as a slice, tile, CTU, or CU. When such adaptive motion vector resolution (AMVR) is applied, information regarding the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information regarding the motion vector resolution to be applied to each CU is signaled. The information regarding the motion vector resolution may be information indicating the precision of the motion vector difference, as described below.

[0061] On the other hand, the inter-frame predictor 124 can perform inter-frame prediction using bidirectional prediction. Bidirectional prediction uses two reference images and two motion vectors representing the block positions most similar to the current block in each reference image. The inter-frame predictor 124 selects a first reference image and a second reference image from reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for blocks most similar to the current block in the respective reference images to generate first and second reference blocks. Furthermore, a prediction block for the current block is generated by averaging or weighted averaging the first and second reference blocks. Motion information, including information about the two reference images used to predict the current block and information about the two motion vectors, is transmitted to the entropy encoder 155. Reference image list 0 may consist of images preceding the current image in display order among the pre-reconstructed images, and reference image list 1 may consist of images following the current image in display order among the pre-reconstructed images. However, while not particularly limited to this, pre-reconstructed images following the current image in display order may also be included in reference image list 0. Conversely, a pre-reconstructed image preceding the current image may also be additionally included in the reference image list 1 .

[0062] To minimize the amount of bits consumed for encoding motion information, various methods may be used.

[0063] For example, when the reference image and motion vector of the current block are the same as those of a neighboring block, information identifying the neighboring block is encoded to transmit the motion information of the current block to the video decoding device. This method is called merge mode.

[0064] In the merge mode, the inter predictor 124 selects a predetermined number of merge candidate blocks (hereinafter, referred to as “merge candidates”) from neighboring blocks of the current block.

[0065] As the adjacent blocks for deriving the merge candidate, all or some of the left block A0, the lower left block A1, the upper block B0, the upper right block B1 and the upper left block B2 adjacent to the current block in the current image may be used, such as Figure 4 As shown. In addition, in addition to the current image where the current block is located, blocks located in the reference image (which may be the same as or different from the reference image used to predict the current block) can also be used as merge candidates. For example, the co-located block of the current block in the reference image or a block adjacent to the co-located block can be additionally used as a merge candidate. If the number of merge candidates selected by the above method is less than the preset number, a zero vector is added to the merge candidates.

[0066] The inter-frame predictor 124 configures a merge list including a predetermined number of merge candidates using neighboring blocks. From the merge candidates included in the merge list, a merge candidate to be used as motion information for the current block is selected, and merge index information is generated to identify the selected candidate. The generated merge index information is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0067] Merge skip mode is a special case of merge mode. After quantization, when all transform coefficients used for entropy coding are close to zero, only the neighboring block selection information is transmitted, without the residual signal. By utilizing merge skip mode, relatively high coding efficiency can be achieved for images with minimal motion, still images, and images with on-screen content.

[0068] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0069] Another method for encoding motion information is the advanced motion vector prediction (AMVP) mode.

[0070] In the AMVP mode, the inter-frame predictor 124 derives a motion vector prediction candidate for the motion vector of the current block by using the neighboring blocks of the current block. As the neighboring blocks for deriving the motion vector prediction candidate, the neighboring blocks may be used. Figure 4 All or some of the left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2 adjacent to the current block in the current image shown. Furthermore, in addition to the current image in which the current block is located, blocks within a reference image (which may be the same as or different from the reference image used to predict the current block) may also be used as adjacent blocks for deriving motion vector prediction candidates. For example, a co-located block of the current block within the reference image or a block adjacent to the co-located block may be used. If the number of motion vector candidates selected by the above method is less than a preset number, a zero vector is added to the motion vector candidates.

[0071] The inter-frame predictor 124 derives motion vector prediction candidates by using the motion vectors of the neighboring blocks, and determines the motion vector prediction of the motion vector of the current block by using the motion vector prediction candidates. In addition, the motion vector difference is calculated by subtracting the motion vector prediction from the motion vector of the current block.

[0072] Motion vector predictions can be obtained by applying a predefined function (e.g., median and mean calculations) to motion vector prediction candidates. In this case, the video decoding device also knows the predefined function. Furthermore, since the neighboring blocks used to derive motion vector prediction candidates are blocks that have already been encoded and decoded, the video decoding device may also already know the motion vectors of the neighboring blocks. Therefore, the video encoding device does not need to encode information for identifying motion vector prediction candidates. Accordingly, in this case, information about the motion vector difference and information about the reference image used to predict the current block are encoded.

[0073] Alternatively, the motion vector prediction may be determined by selecting any one of the motion vector prediction candidates. In this case, information identifying the selected motion vector prediction candidate is additionally encoded together with information about the motion vector difference used to predict the current block and information about the reference image.

[0074] The subtractor 130 generates a residual block by subtracting a prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block.

[0075] The transformer 140 transforms the residual signal in the residual block having pixel values in the spatial domain into transform coefficients in the frequency domain. The transformer 140 may transform the residual signal in the residual block by using the entire size of the residual block as a transform unit, or may divide the residual block into multiple sub-blocks and perform the transform using the sub-blocks as transform units. Alternatively, the residual block is divided into two sub-blocks, namely a transform region and a non-transform region, to transform the residual signal using only the transform region sub-block as a transform unit. Here, the transform region sub-block may be one of two rectangular blocks with a size ratio of 1:1 based on the horizontal axis (or vertical axis). In this case, the entropy encoder 155 encodes a flag (cu_sbt_flag) indicating that only the sub-block is transformed, as well as direction (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag), and signals these to the video decoding device. In addition, the size of the transform region subblock may have a size ratio of 1:3 based on the horizontal axis (or vertical axis). In this case, the entropy encoder 155 additionally encodes a flag (cu_sbt_quad_flag) for dividing the corresponding partition and signals it to the video decoding apparatus.

[0076] On the other hand, the transformer 140 can perform transformations of the residual block separately in the horizontal direction and the vertical direction. For this transformation, various types of transformation functions or transformation matrices can be used. For example, paired transformation functions for horizontal transformation and vertical transformation can be defined as a multiple transform set (MTS). The transformer 140 can select a transformation function pair with the highest transformation efficiency in the MTS and can transform the residual block in each of the horizontal and vertical directions. The information (mts_idx) about the transformation function pair in the MTS is encoded by the entropy encoder 155 and notified to the video decoding device with a signal.

[0077] The quantizer 145 quantizes the transform coefficients output from the transformer 140 using a quantization parameter and outputs the quantized transform coefficients to the entropy encoder 155. The quantizer 145 can also immediately quantize the relevant residual block without transforming any block or frame. The quantizer 145 can also apply different quantization coefficients (scaling values) depending on the position of the transform coefficient in the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions can be encoded and signaled to the video decoding device.

[0078] The rearrangement unit 150 may perform rearrangement of coefficient values on the quantized residual value.

[0079] The rearrangement unit 150 can change the 2D coefficient array into a 1D coefficient sequence by using coefficient scanning. For example, the rearrangement unit 150 can use a zigzag scan or a diagonal scan to scan the DC coefficient to the coefficient of the high-frequency region to output a 1D coefficient sequence. Depending on the size of the transform unit and the intra-frame prediction mode, the zigzag scan can also be replaced by a vertical scan that scans the 2D coefficient array in the column direction and a horizontal scan that scans the 2D block type coefficients in the row direction. In other words, depending on the size of the transform unit and the intra-frame prediction mode, the scanning method to be used can be determined from zigzag scanning, diagonal scanning, vertical scanning, and horizontal scanning.

[0080] The entropy encoder 155 encodes the sequence of 1D quantized transform coefficients output from the rearrangement unit 150 by using various encoding schemes including Context-based Adaptive Binary Arithmetic Code (CABAC), Exponential Golomb, etc. to generate a bitstream.

[0081] In addition, the entropy encoder 155 encodes information related to block partitioning (e.g., CTU size, CTU partition flag, QT partition flag, MTT partition type, and MTT partition direction, etc.) so that the video decoding device can partition the block in the same manner as the video encoding device. In addition, the entropy encoder 155 encodes information about the prediction type indicating whether the current block is encoded by intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (merge index in the case of merge mode, and information about reference image index and motion vector difference in the case of AMVP mode) according to the prediction type. In addition, the entropy encoder 155 encodes information related to quantization (i.e., information about quantization parameters and information about quantization matrices).

[0082] The inverse quantizer 160 inversely quantizes the quantized transform coefficient output from the quantizer 145 to generate a transform coefficient. The inverse transformer 165 transforms the transform coefficient output from the inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct a residual block.

[0083] The adder 170 reconstructs the current block by adding the reconstructed residual block and the prediction block generated by the predictor 120. When performing intra prediction on the next block, the pixels in the reconstructed current block are used as reference pixels.

[0084] The loop filtering unit 180 performs filtering on the reconstructed pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filtering unit 180, as an in-loop filter, may include all or some of the deblocking filter 182, the sample adaptive offset (SAO) filter 184, and the adaptive loop filter (ALF) 186.

[0085] The deblocking filter 182 filters the boundaries between reconstructed blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter 184 and ALF 186 perform additional filtering on the deblocked video. The SAO filter 184 and ALF 186 are filters used to compensate for the difference between reconstructed and original pixels caused by lossy coding. The SAO filter 184 applies an offset per CTU to enhance subjective image quality and coding efficiency. On the other hand, the ALF 186 performs block-based filtering and applies different filters to compensate for distortion based on the boundaries of the corresponding blocks and the degree of change. Information about the filter coefficients to be used for the ALF can be encoded and signaled to the video decoding device.

[0086] The reconstructed blocks filtered by the deblocking filter 182, the SAO filter 184, and the ALF 186 are stored in the memory 190. When all blocks in one image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks within an image to be subsequently encoded.

[0087] The video encoding apparatus may store a bit stream of encoded video data in a non-volatile storage medium or transmit the bit stream to a video decoding apparatus through a communication network.

[0088] Figure 5 is a functional block diagram of a video decoding device that can implement the technology of the present invention. Figure 5 , describes a video decoding device and components of the device.

[0089] The video decoding apparatus may include an entropy decoder 510 , a rearrangement unit 515 , an inverse quantizer 520 , an inverse transformer 530 , a predictor 540 , an adder 550 , a loop filtering unit 560 , and a memory 570 .

[0090] Similar to Figure 1 Each component of the video encoding device and the video decoding device can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the function of the software corresponding to each component.

[0091] The entropy decoder 510 extracts information related to block partitioning by decoding a bitstream generated by a video encoding apparatus to determine a current block to be decoded, and extracts prediction information required to reconstruct the current block and information about a residual signal.

[0092] The entropy decoder 510 extracts information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS) to determine the size of the CTU and partitions the image into CTUs of the determined size. Furthermore, the CTU is determined as the highest level (i.e., the root node) of the tree structure, and partition information of the CTU is extracted to partition the CTU using the tree structure.

[0093] For example, when a CTU is segmented using a QTBTTT structure, a first flag (QT_split_flag) related to QT segmentation is first extracted to segment each node into four nodes in the lower layer. Furthermore, a second flag (mtt_split_flag) related to MTT segmentation, a segmentation direction (vertical / horizontal), and / or a segmentation type (binary / trifurcated) are extracted with respect to nodes corresponding to QT leaf nodes to segment the corresponding leaf nodes into an MTT structure. As a result, each node below the QT leaf node is recursively segmented into a BT or TT structure.

[0094] As another example, when a CTU is split using the QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU is split is extracted. When the corresponding block is split, a first flag (QT_split_flag) may also be extracted. During the splitting process, for each node, zero or more recursive MTT splits may occur after zero or more recursive QT splits. For example, for a CTU, MTT splits may occur immediately, or, conversely, multiple QT splits may occur.

[0095] As another example, when a CTU is split using a QTBT structure, a first flag (QT_split_flag) related to the splitting of the QT is extracted to split each node into four nodes in the lower layer. In addition, a split flag (split_flag) indicating whether a node corresponding to a leaf node of the QT is further split into a BT and split direction information are extracted.

[0096] On the other hand, when the entropy decoder 510 determines the current block to be decoded by partitioning using a tree structure, the entropy decoder 510 extracts information about the prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts syntax elements of intra-prediction information (intra-prediction mode) for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts information of syntax elements representing inter-prediction information, that is, a motion vector and a reference image referenced by the motion vector.

[0097] In addition, the entropy decoder 510 extracts quantization-related information and extracts information on a quantized transform coefficient of the current block as information on a residual signal.

[0098] The rearrangement unit 515 may change the sequence of 1D quantized transform coefficients entropy-decoded by the entropy decoder 510 into a 2D coefficient array (ie, block) again in the reverse order of the coefficient scanning order performed by the video encoding device.

[0099] The inverse quantizer 520 inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients by using a quantization parameter. The inverse quantizer 520 can also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in 2D. The inverse quantizer 520 can perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from the video encoding device to the 2D array of quantized transform coefficients.

[0100] The inverse transformer 530 reconstructs a residual signal by inversely transforming the inversely quantized transform coefficient from the frequency domain to the spatial domain to generate a residual block of the current block.

[0101] In addition, when the inverse transformer 530 inverse-transforms a partial region (sub-block) of a transform block, the inverse transformer 530 extracts a flag (cu_sbt_flag) indicating that only the sub-block of the transform block is transformed, direction (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block. The inverse transformer 530 also inverse-transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the uninverse-transformed region with a value of "0" as the residual signal to generate the final residual block of the current block.

[0102] In addition, when MTS is applied, the inverse transformer 530 determines the transformation function or transformation matrix to be applied in each of the horizontal and vertical directions by using the MTS information (mts_idx) signaled from the video encoding device. The inverse transformer 530 also performs inverse transformation on the transformation coefficients in the transformation block in the horizontal and vertical directions by using the determined transformation function.

[0103] The predictor 540 may include an intra predictor 542 and an inter predictor 544. The intra predictor 542 is activated when the prediction type of the current block is intra prediction, and the inter predictor 544 is activated when the prediction type of the current block is inter prediction.

[0104] The intra predictor 542 determines an intra prediction mode of a current block from among a plurality of intra prediction modes according to syntax elements of the intra prediction mode extracted from the entropy decoder 510. The intra predictor 542 also predicts the current block by using neighboring reference pixels of the current block according to the intra prediction mode.

[0105] The inter predictor 544 determines a motion vector of a current block and a reference image to which the motion vector refers by using a syntax element of the inter prediction mode extracted from the entropy decoder 510 .

[0106] The adder 550 reconstructs the current block by adding the residual block output from the inverse transformer 530 to the prediction block output from the inter predictor 544 or the intra predictor 542. When intra-predicting a subsequent block to be decoded, pixels within the reconstructed current block are used as reference pixels.

[0107] The loop filter unit 560, which serves as an in-loop filter, may include a deblocking filter 562, an SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundaries between reconstructed blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter 564 and the ALF 566 perform additional filtering on the reconstructed blocks after deblocking filtering to compensate for differences between reconstructed and original pixels caused by lossy encoding. The filter coefficients of the ALF are determined by utilizing information about the filter coefficients decoded from the bitstream.

[0108] The reconstructed blocks filtered by the deblocking filter 562, the SAO filter 564, and the ALF 566 are stored in the memory 570. When all blocks in one image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks within an image to be subsequently encoded.

[0109] In some embodiments, the present invention relates to encoding and decoding video images as described above. More specifically, the present invention provides an end-to-end video compression method and apparatus for generating block partition structure information and block partition features as additional information in end-to-end video compression, and then utilizing the generated additional information for video encoding / decoding.

[0110] The following embodiments may be performed by a video encoding device or a video decoding device.

[0111] The video encoding device may generate signaling information associated with this embodiment in terms of optimizing rate distortion when encoding the current block. The video encoding device may encode the signaling information using the entropy encoder 155 and transmit the encoded signaling information to the video decoding device. The video decoding device may decode the signaling information related to decoding of the current block from the bitstream using the entropy decoder 510.

[0112] In the following description, the term "target block" may be used interchangeably with a current block or a coding unit (CU), or may refer to some areas of a coding unit.

[0113] Furthermore, a flag value of true indicates a case where the flag is set to 1. Furthermore, a flag value of false indicates a case where the flag is set to 0.

[0114] I. Convolutional Neural Network (CNN)

[0115] A CNN is a neural network composed of multiple convolutional and pooling layers and is considered the most suitable deep learning technique for image processing. Convolutional layers use multiple kernels, or filters, to extract feature maps, also known as "features." The kernel coefficients that make up the filters are parameters determined during the learning process.

[0116] In the convolutional layers of a CNN, the front layers close to the input extract feature maps that respond to simple, low-level image features (e.g., lines, points, or surfaces). In the convolutional layers of a CNN, the back layers close to the output extract feature maps that respond to higher-level features (e.g., textures and object parts).

[0117] Figure 6 is a schematic diagram illustrating the operation of a convolutional layer according to at least one embodiment of the present invention.

[0118] The convolutional layer generates feature maps from the input image using convolution operations. Figure 6 The example depicts a kernel (or filter) with a kernel size of 3×3. The kernel size is also called the filter size. The kernel has kernel parameters or filter parameters, also called weights. Figure 6 The kernel shown has a total of nine kernel parameters. The kernel parameters can be initially set to random values, and their values can be updated based on training.

[0119] The convolutional layer performs a convolution operation by utilizing a patch of the input image that is equal to the kernel size. In this case, the patch that is equal to the kernel size in the input image is called a window.

[0120] When filtering the input image in raster scan order, the amount by which the window is moved is called the stride. Figure 6 In the example, the stride is 1. If the stride is set to 2, a convolution operation is performed, shifting the window by 2 samples, resulting in a feature map with horizontal and vertical dimensions that are half the horizontal and vertical dimensions of the input image.

[0121] As mentioned above, a convolutional layer can include multiple filters. The number of filters, or kernels, is called the number of channels. In other words, the number of channels is equal to the number of filters. The number of filters also determines the dimensionality of the feature map.

[0122] Padding is a method of expanding the input data by filling it with a certain value before performing a convolution operation. Padding is often used to control the spatial size of the output data. The value used for padding can be determined by hyperparameters, but zero padding is commonly used. Without padding, the spatial size of the output data decreases with each pass through the convolutional layer, which can lead to loss of boundary information. Therefore, padding is used to avoid this problem. Padding can be used to balance the spatial size of the output data and the input data of the convolutional layer.

[0123] The deconvolution layer performs the opposite operation of the convolution layer. The deconvolution layer generates the desired data image as output from the feature map as input.

[0124] The pooling layer performs pooling, which is the process of subsampling the feature maps generated by the convolutional layer. The pooling layer uses a 2×2 window to select samples so that the output is half the width and half the height of the input. In other words, the pooling layer is used to reduce the size of the input image or input feature map by compressing the 2×2 area into a single sample.

[0125] The opposite concept of pooling layer is defined as unpooling layer. In contrast to pooling layer, unpooling layer is used to increase the dimensionality and is usually used after deconvolution layer.

[0126] The convolutional encoder-decoder architecture is a network structure consisting of pairs of convolutional and deconvolutional layers. The convolutional encoder consists of convolutional and pooling layers to output a feature map (or feature vector) based on the input image. The final output vector of the convolutional encoder is also called the latent vector. The convolutional decoder consists of deconvolutional and depooling layers to generate the output image from the feature map or latent vector.

[0127] The input and output of the convolutional encoder-decoder can vary depending on the application and the purpose of the network. For example, the input and output can be optical flow maps, saliency maps, image frames, etc.

[0128] Figure 7 is a schematic diagram showing a SISR network.

[0129] An example application of CNN is Single Image Super Resolution (SISR). A SISR network generates a high-resolution output image from a low-resolution input image. A SISR network can include multiple convolutional layers, such as Figure 7As shown in Figure 1, each convolutional layer includes an activation function, such as a rectified linear unit (ReLU). The parameters of the SISR network can be trained so that the resulting super-resolution (SR) image is close to the ground truth (GT).

[0130] SR methods using CNN can improve SR performance by increasing the depth (e.g., by increasing the number of convolutional layers). To overcome the overfitting problem in learning that may occur with increasing depth, SISR networks can utilize residual blocks that can perform skip connections and residual learning. Figure 8 As shown, in addition to the input feature x l In addition to the path where the convolution operation is applied, the residual block also includes a skip path. In addition, when generating the output x l+1 The residual block can choose the path to apply convolution operation or skip the path based on learning efficiency. Figure 8 In the example of , the residual block includes a batch normalization (BN) layer.

[0131] In one example, the Enhanced Deep Residual Networks for SISR (EDSR) enhances the performance of the network by continuously splicing residual blocks to increase the depth. Another example is the accurate image super-resolution using a very deep convolutional network (VDSR), which is a CNN model based on the Visual Geometry Group Network (VGG network) that uses residual learning, that is, adding a residual frame to the final output. VDSR adds a residual signal at the very end of the network, thereby adding the residual signal to the input signal.

[0132] As another example, CNN can be used as an in-loop filter in a video encoding device or a video decoding device. In this case, the in-loop filter based on deep learning can be used at any position within the existing loop filter unit 180 or 560 consisting of a deblocking filter, a SAO filter, and an ALF.

[0133] The deep learning-based in-loop filter may be a fixed coefficient in-loop filter. The deep learning-based fixed coefficient in-loop filter utilizes common CNN kernel parameters stored on both sides of the video encoding device and the video decoding device.

[0134] Figure 9 is a schematic diagram showing a fixed coefficient in-loop filter based on CNN.

[0135] The input block (or input frame) is passed through the normalized QP map and passed to the subsequent stage. The normalized QP map is used to reduce the inference error when mixing quantization noise of different intensities during learning and inference. The kernel parameters that constitute the dense residual unit (DRU) and convolutional layer can be stored and used identically in the video encoding device and the video decoding device. Figure 9 In the example of , each DRU may include all or part of a convolutional layer, a ReLU layer, and a depth-wise separable convolutional (DSC) layer.

[0136] Deep learning based fixed-coefficient in-loop filters need to provide general performance over a wide range of video frames, so they have the disadvantage of increasing the number of layers deep into the CNN and correspondingly extending the computation time.

[0137] II. Converter

[0138] Transformers are models proposed for performing sequence-to-sequence tasks in natural language processing (such as machine translation).

[0139] Existing sequence-to-sequence models include encoder-decoder structures that utilize recurrent neural networks (RNNs) or long short-term memory (LSTMs) in response to input words to sequentially pop out output words. The encoder represents the input sequence as a vector representation, and the decoder generates an output sequence based on the vector representation.

[0140] However, RNN-based models suffer from the limited length of input vectors. In addition, because some input sequence information is lost during encoding, RNN models are limited in processing complex sentences. To compensate for this, context vectors are generated based on the attention mechanism, which associates the output of the decoder at a specific time with the output of the encoder. However, RNN models do not overcome the above limitations.

[0141] The Transformer outperforms existing models in translation performance even when it maintains the existing encoder-decoder structure by only utilizing attention, and it does not utilize RNNs or LSTMs.

[0142] Before introducing the overall structure of the Transformer, we first describe the self-attention utilized in the Transformer.

[0143] Figure 10a and Figure 10b is a schematic diagram showing the self-attention of a vector.

[0144] Self-attention is performed by an attention module (or “attention layer”) within the Transformer. The following description assumes that the Transformer processes the sentence “I am a student” sequentially, with the attention module currently processing the input word “student”.

[0145] First, if Figure 10a As shown in Figure 1, the attention module multiplies the word vector corresponding to each word initially input to the transformer by a weight matrix to generate a query vector (Q vector), a key vector (K vector), and a value vector (V vector). Figure 10a In the example, the weight matrix W Q 、W K and W V are the trained parameters. In addition, for the input word “student”, Q student , K student and V student Represent the Q vector, K vector and V vector respectively.

[0146] Then, for a given Q vector, the attention module calculates an attention score using all K vectors. The attention module then uses the relevant similarity (i.e., attention score) as a weight to reflect the relevant similarity on the V vector mapped to the corresponding K vector. Finally, the attention module can perform a weighted summation of all V vectors reflecting similarity to calculate the attention value for the input word.

[0147] On the other hand, the attention score can be calculated by scaled dot product attention, which utilizes the inner product of two vectors, as Figure 10b As shown in Figure 1, the scaled dot product attention can be expressed as Equation 1.

[0148] [Equation 1]

[0149] Score function

[0150] Here, d k are predefined constants.

[0151] The attention score function represents the degree of correlation between the q vector and the k vector. In Equation 1, the score function uses the inner product, but it can also use any similarity between the two vectors, such as cosine similarity, correlation, etc.

[0152] The above operation can be applied in parallel to all "I", "am" and "a" except "student", as in Figure 11a and Figure 11b As shown. Figure 11a and Figure 11b In the example of , for the input sentence “I am a student”, Q, K, and V represent the Q matrix, K matrix, and V matrix respectively. The final attention value matrix can be calculated as shown in Equation 2.

[0153] [Equation 2]

[0154] attention

[0155] Here, B is a matrix of predefined positional encoding values. Figure 11b In the example, B is omitted.

[0156] Figure 12 is a schematic diagram showing multi-head attention.

[0157] Performing parallel attention is more effective for generalization than performing single attention. This is due to the use of different weights to collect information from different perspectives on each word vector. In the Transformer, parallel attention is represented by multi-head attention.

[0158] To apply multi-head attention, the weight matrix W Q 、W K and W V It can be trained by the number of attention heads (num_heads). Figure 12 As shown, the weight matrix W Q 0...num_head-1 、W K 0...num_head-1 and W V 0...num_head-1 to perform parallel attention. Then, the attention outputs from the corresponding heads can be concatenated, multiplied by the weight matrix, and the final attention can be generated.

[0159] Figure 13 is a schematic diagram showing the structure of a converter.

[0160] The transformer includes N encoders and N decoders.

[0161] The encoder consists of N layers. One encoder layer consists of two sublayers: self-attention and feedforward neural network. Here, self-attention refers to the case where Q, K, and V are equal. In addition, the encoder's self-attention layer performs multi-head attention as described above.

[0162] On the other hand, the input sentence of the encoder is positionally encoded. The transformer may not receive input words sequentially. Therefore, in order to represent the position information of the word, the position information is added in the token order by utilizing the sine function. Figure 13 In the examples, “input embedding” refers to the input sentence represented in the form of an embedding.

[0163] Like the encoder, the decoder consists of N layers. One decoder layer consists of three sub-layers. Like the encoder, the sentence matrix after positional encoding is input to the decoder. Figure 13 In the example above, “output embedding” refers to the decoder’s input sentence, represented as an embedding.

[0164] As with sequence-to-sequence, the first sublayer of the decoder is trained based on teacher forcing, where the GT is the next input to the decoder, so the decoder is trained to predict words at each time point when it receives the matrix of the target sentence. In the first sublayer, the matrix based on the input sentence is masked to avoid referencing words at future time points, thereby referencing the word at each time point and its predecessor. Essentially, the first sublayer of the decoder is structurally identical to the first sublayer of the encoder because it is self-attentional and performs multi-head attention. The behavior of the first sublayer of the decoder is called look-ahead attention.

[0165] The second sublayer of the decoder uses the output of the first self-attention layer as the Q matrix, and uses the K and V matrices received from the last layer of the encoder to calculate the attention score matrix. The second sublayer of the decoder is called the encoder-decoder attention. Meanwhile, the third sublayer is a feedforward neural network.

[0166] In the field of vision, CNN structures are often used. Based on the performance of transformers in natural language processing, standard transformers can be modified to a minimum, and then the modified transformer can be applied directly to the image. To do this, the image can be divided into patches, which can then be processed in the same way as tokens in NLP. That is, a linear embedding sequence of these patches can be fed into the transformer. Transformers used in vision applications are called vision transformers.

[0167] Figure 14 is a schematic diagram showing the input preprocessing of the visual transformer.

[0168] The standard transformer receives as input a sequence of one-dimensional token embeddings. In order to feed the one-dimensional sequence into the visual transformer, the two-dimensional image needs to be preprocessed. For this purpose, Figure 14 As shown, the two-dimensional input image x∈R W×H×C is tuned into a series of flat patches, x p ∈R N×P×P×C When a two-dimensional input image is partitioned, the size of each patch is P × P. N is the number of patches, and C is the dimension of color information.

[0169] Alternatively, instead of utilizing image patches, feature maps from a CNN can be used as the input sequence.

[0170] The following embodiments are described with respect to a video encoding apparatus, but they are equally applicable to a video decoding apparatus.

[0171] III. End-to-end video compression technology based on deep learning

[0172] In order to improve the prediction performance in video compression, end-to-end video compression based on deep learning is utilized. The structure for end-to-end video compression adopts the form of a basic Joint Photographic Experts Group (JPEG) compression algorithm including transform → quantization → entropy coding and decoding → inverse transform. In order to apply deep learning to the algorithm, the existing transform method, namely Discrete Cosine Transform (DCT), is replaced by a neural network. The quantization process of converting a continuous signal into a discrete signal is a non-differentiable operation, so it cannot be back-propagated by deep learning. Therefore, during training, quantization is approximated as additive noise. The inverse transform process is not a complete inverse function of the transform process implemented by the neural network. However, since the neural network corresponding to the inverse transform process is trained by calculating the final loss as the mean square error (MSE), the inverse transform process can be approximated as the inverse function of the transform process.

[0173] Figure 15 is a schematic diagram illustrating an end-to-end video compression model based on deep learning according to at least one embodiment of the present invention.

[0174] like Figure 15 As shown, the end-to-end video compression model includes an analysis transformation unit g a 1510, quantization unit Q1520 and synthesis transformation unit g s 1530. The end-to-end video compression model is an autoencoder that generates a smaller dimensional code y from an input image x. Analysis of the transform unit g a 1510 encodes the input image x to generate code y. Q quantizes y to generate a bit stream R. Synthesizes the transform unit g s 1530 generates a reconstructed image x_hat from the code y_hat. y_hat is decoded from the bitstream. A perceptual transformation unit, g p 1540 is a fixed transformation for generating a representation w or w_hat in the perceptual space from the input image x or the reconstructed image x_hat. The distortion D between w and w_hat is used to train g a and g s .exist Figure 15In the example of , the input image x and the reconstructed image x_hat reside in the data space, and the codes y and y_hat reside in the code space. According to the embodiment, g p May be omitted during training. If g p is omitted, the distortion D can be calculated based on the difference between the image x and x_hat.

[0175] Hereinafter, in the end-to-end video compression model, the analysis transform unit 1510 and the video encoder are used interchangeably. In addition, the synthesis transform unit 1530 and the video decoder are used interchangeably.

[0176] Figure 16 is a schematic diagram illustrating an analysis transform and a synthesis transform according to at least one embodiment of the present invention.

[0177] exist Figure 16 In the example of FIG, the structures of the analysis transformation unit 1510 and the synthesis transformation unit 1530 are shown. The analysis transformation unit 1510 includes a convolution layer, a pooling layer for performing downsampling, and a structure in which the activation function (i.e., Generalized Divisive Normalization (GDN)) is repeated three times. The synthesis transformation unit 1530 includes a convolution layer, a reverse pooling layer for performing upsampling, and a structure in which the activation function (i.e., Inverse GDN (IGDN)) is repeated three times. Figure 16 The structures of the analysis transform unit 1510 and the synthesis transform unit 1530 are shown for compression of 16×16 blocks. Compared with JPEG processing of 8×8 blocks, Figure 16 The structure shown performs a greater energy compression.

[0178] The loss function for end-to-end video compression is defined as shown in Equation 3.

[0179] [Equation 3]

[0180] L(g a ,g s ,P q )=-E[log2P q ]+λE[d(w,w_hat)]

[0181] In Equation 3, the first term on the right-hand side represents the length of the bitstream required for image compression, and the second term represents the distortion between the original image w and the reconstructed image w_hat used for training. E represents the average calculation on the training dataset, and λ is a constant that scales the ratio between the two terms. The end-to-end video compression model is trained according to Equation 3 to reconstruct a high-quality image based on the minimum bitstream. In the second term on the right-hand side of Equation 3, w and w_hat in the perceptual space are used as the original image and the reconstructed image. However, if g p is omitted, the second term of Equation 3 can be calculated based on the difference between x and x_hat.

[0182] During training, quantization is replaced by adding uniform noise in the range -1 / 2 and +1 / 2 to the code y values. This is appropriate for specific values that exist in the interval (m-1 / 2, m+1 / 2) of the random integer m, where the rounding operation of quantization is the process of mapping the specific value to the random integer m.

[0183] The super-prior model provides additional information to the above end-to-end video compression model. In general, since the latent tensor y_hat contains spatial dependencies, spatially adjacent elements tend to change with scaling. Therefore, by dividing the large values of y_hat by the scaled value of y_hat, making y_hat have small values, the super-prior model allocates fewer bits to the code y_hat.

[0184] Figure 17 is a schematic diagram illustrating an end-to-end video compression apparatus including a super-prior model according to at least one embodiment of the present invention.

[0185] like Figure 17 As shown, the end-to-end video compression device includes an end-to-end video compression model and a super prior model. As mentioned above, the end-to-end video compression model includes an analysis transformation unit g a 1510, quantization unit Q 1520 and synthesis transformation unit g s 1530. The super-prior model includes a priori encoder 1710, quantization unit Q 1720, and a priori decoder 1730. The super-prior model approximates y_hat as a Gaussian model with mean 0 and standard deviation σ, and compresses the standard deviation σ as additional information z. When decoding, the super-prior model simultaneously decodes y_hat and the standard deviation σ of y_hat.

[0186] exist Figure 17 In the example, the analysis transformation unit g a 1510 includes the structure of convolutional layer, pooling layer and GDN repetition as described above. s 1530 includes a convolutional layer, an anti-pooling layer, and an IGDN repeated structure. The prior encoder h of the super prior modela 1710 includes a structure of repeated convolutional layers, pooling layers, and activation functions (i.e., rectified linear units (ReLU)). Priori decoder h s 1730 includes a structure in which a convolution layer, an unpooling layer, and an activation function (ReLU) are repeated. In addition, AE represents an arithmetic encoder, and AD represents an arithmetic decoder.

[0187] CNN-based end-to-end compression techniques significantly improve the rate-distortion trade-off compared to traditional codecs. However, due to the large number of parameters characteristic of the CNN structure, CNN-based end-to-end compression techniques still suffer from the problem of slow decoding time. Therefore, by replacing the CNN-based end-to-end video compression model and the CNN-based super-prior model with a visual transformer-based structure (e.g., Swin transformer), improved compression efficiency can be achieved while utilizing fewer parameters and shorter decoding time.

[0188] Figure 18 FIG. 4 is a schematic diagram illustrating an end-to-end video compression apparatus including a super-prior model according to another embodiment of the present invention.

[0189] Figure 18 An exemplary transformer-based implementation of an end-to-end video compression model and a super-prior model is shown in FIG. a 1510 includes a structure of repeated patch merging blocks and Swin transformer (SwinT) blocks. The patch merging block contains, in order: a spatial to depth layer (spatial downsampling + channel upsampling layer), a normalization layer, and a linear layer. The SwinT block performs local self-attention within each window where the feature maps do not overlap and maintains the size of the feature maps. Consecutive SwinT blocks that generate the same feature map size can be window-shifted relative to the previous block to better capture information based on the adjacent windows of the previous block. The provided synthetic transformation unit g s 1530 includes a structure with a patch segmentation block and a SwinT block repeated. The synthesis transform is symmetrical to the analysis transform and operates in the opposite order of the analysis transform. The patch segmentation block includes, in order: a linear layer, a normalization layer, and a depth-to-spatial layer (spatial upsampling + channel downsampling layer).

[0190] The prior encoder h of the hyper-prior model a The structure of 1710 is similar to the analysis transformation unit g a 1510, and its a priori decoder h s The structure of 1730 is similar to the synthesis transformation unit g s1530. A channel-wise autoregressive model (ChARM) 1810 is provided, and uses the output of the prior decoder 1730 and the output of the AD attached to the end-to-end video compression model to generate the mean and standard deviation of y_hat as additional information. ChARM 1810 provides the standard deviation to the AE and AD attached to the end-to-end video compression model. ChARM 1810 subtracts the mean from the code y and adds the mean to the output of the AD. In addition, a factorized model 1820 generates a learned factorized prior as additional information and provides it to the AE and AD attached to the hyper-prior model.

[0191] IV. Embodiments of the Invention

[0192] Figure 19 is a schematic diagram illustrating an end-to-end video compression apparatus utilizing block partition information according to at least one embodiment of the present invention.

[0193] The end-to-end video compression apparatus according to the present invention obtains block partition information as additional information, and then applies the block partition information to the end-to-end video compression model. Figure 19 As shown, in addition to the end-to-end video compression model, the end-to-end video compression apparatus further includes a block partitioner 1910 and a latent feature generator 1920. When dividing a video image into one or more regions, the end-to-end video compression apparatus can utilize information about the local region of the video as block partition information to compress the video.

[0194] The block partitioner 1910 generates block partition information indicating a block partition structure.

[0195] In one example, to generate block partition information, the block partitioner 1910 utilizes a block-based partition structure used by a conventional video codec. For example, the quadtree block partition structure of the High Efficiency Video Codec (HEVC) or the multitree block partition structure of the Versatile Video Codec (VVC) can be used as block partition information. The block partitioner 1910 extracts information about the position, size, and shape of the codec unit provided by the video codec during intra-frame compression of the video, and uses the extracted information as block partition information. Alternatively, in order to reduce the time required for block partitioning, a deep learning-based neural network is used, which outputs a block partition structure from the input video. For example, the block partitioner 1910 can utilize a deep learning-based partitioning model (e.g., Deep Quad-Tree plus Multi-type Tree (Deep QTMT)) to generate block partition information from the input image.

[0196] As another example, the block partitioner 1910 may represent the block partition information by using domain regions for random positions and sizes of the image, such as Figure 20 As shown in . A domain represents a group of pixels in an image that have similar characteristics in terms of color, brightness, object, background, etc. Figure 20 In the example, regions of different colors / patterns / shapes represent different domain regions.

[0197] As yet another example, the block partitioner 1910 may represent the block partition information by utilizing a quantization parameter (QP) for each region of the image. Figure 21 In an example image of , the block partition information is represented by a single quantization parameter. Figure 21 In another example image of FIG, block partition information is represented by more than one quantization parameter. To represent the block partition information using the quantization parameter, the block partitioner 1910 represents each pixel value as a corresponding quantization parameter value in a map having the same dimension as the image. Alternatively, the block partitioner 1910 may normalize the quantization parameter to a value ranging between 0 and 1 by setting the minimum value of the quantization parameter to 0 and the maximum value to 1, and then the block partitioner 1910 may represent each pixel value by the normalized value.

[0198] In addition, the block partition information may be represented using one of the following methods.

[0199] In one example, the block partitioner 1910 may use boundaries (edges) of blocks and domains to represent block partition information.

[0200] In another example, the block partitioner 1910 represents the interior of the block and domain as the average value of the luma component within the block.

[0201] As yet another example, the block partitioner 1910 represents the interior of the block and the domain as quantization parameters.

[0202] Figure 22 is a schematic diagram illustrating the generation of potential features according to at least one embodiment of the present invention.

[0203] To be used as additional information in end-to-end video compression, a latent feature generator 1920 generates latent features from the block partition information. The latent features may be generated according to one of the following methods.

[0204] exist Figure 22 In one example shown above, the latent feature generator 1920 generates latent features from block partition information by utilizing a transformer-based extraction model similar to the neural network used in the end-to-end video compression model. In this case, no training parameters are shared between the neural network used in the end-to-end video compression model and the neural network used to generate the latent features.

[0205] As in Figure 22 As another example shown on the lower side of , when the end-to-end video compression model is implemented as a CNN-based neural network, the latent feature generator 1920 generates latent features from block partition information by utilizing a CNN-based extraction model.

[0206] Both types of latent feature generation neural networks start from the block partition image X∈R W×H×1 Generate latent tensor embedded with block partition information In the following, the latent tensor embedded with the block partition information is denoted as the block partition latent feature. Here, C P represents the number of channels in the block partition latent feature. p The dimension of is adjusted to have the same intermediate latent features as those generated by the end-to-end video compression model After the same dimension, the block partition latent features f p Fusion with intermediate latent features of video compression models, such as Figure 22 Here, C C Represents the number of channels of intermediate latent features generated by the end-to-end video compression model.

[0207] At this time, the extracted block partition latent features are fused at the video encoder and the video decoder of the end-to-end video compression model. As described above, the video encoder that performs the analysis transform includes a patch merging layer and a transformer block layer that are alternated at a preset number of times, and the video decoder that performs the synthesis transform includes a patch segmentation layer and a transformer block layer that are alternated at a preset number of times. For example, the video encoder generates intermediate latent features by applying a preset number of spatial downsampling and channel upsampling, and the video decoder generates intermediate latent features by applying a preset number of spatial upsampling and channel downsampling. Here, for fusion, the intermediate latent features within the video encoder or the video decoder With potential features of block partitioning Same dimensions.

[0208] Figure 23 is a schematic diagram illustrating a fusion of potential features according to at least one embodiment of the present invention.

[0209] The end-to-end video compression model fuses two latent features, a block partition latent feature and an intermediate latent feature, according to one of the following methods.

[0210] The end-to-end video compression model is based on element-wise addition to fuse two latent features. After fusion, the dimension of the fused latent feature remains at

[0211] The end-to-end video compression model fuses two latent features by using a splicing operation. After fusion, the dimension of the fused latent feature is This is twice the dimension of the channels before fusion. Therefore, after the concatenation operation, a channel downsampling layer is added inside the video encoder to adjust the dimension of the fused latent features.

[0212] like Figure 23 As shown in , the fusion of two latent features can be equally applied to video encoders and video decoders. In this case, considering the complexity of the video decoder, the block partition latent features and the intermediate latent features may not be fused at the decoder side. Figure 23 In the example of , a single transformer block is used as an extraction model to generate block partition latent features. The patch embedding layer in the video encoder embeds the input video to fit the transformer block layer. The de-embedding layer in the video decoder de-embeds the output of the transformer block layer to fit the shape of the reconstructed video. In addition, in Figure 23 In an example, the end-to-end video compression apparatus includes a hyper-prior model to estimate the mean and standard deviation of the reconstructed code y_hat.

[0213] When the input video is the original video, Figure 1 The video encoding apparatus shown in FIG. 1 can be implemented as a video encoding apparatus including Figure 23 The components shown in are the video encoder, the prior encoder of the super prior model, the extraction model for generating block partition latent features, the quantizer Q and the AE. During the training process, Figure 1 The video encoding apparatus shown in further includes a video decoder, a priori decoder of a super prior model, and AD. The video encoding apparatus may further include a block partitioner 1910 that generates block partition potential information. The video encoding apparatus uses AE to encode the code of the input video, the output of the priori encoder, and the block partition information to generate a bitstream, and then transmits the generated bitstream to the video decoding apparatus.

[0214] Figure 5 The video decoding apparatus shown can be implemented as including a video decoder, a priori decoder in a super priori model, an extraction model for generating block partition latent features, and AD. The video decoding apparatus decodes the code of the input video, the output of the priori encoder, and block partition information from a bitstream.

[0215] On the other hand, a training module (not shown) can train an end-to-end video compression model, a super-prior model, and an extraction model for generating block partition latent features, for example, according to Equation 3. The video encoding device can be implemented with an installed training module or in combination with an external training module. The parameters of the trained neural network can be pre-shared between the video encoding device and the video decoding device.

[0216] Reference below Figure 24 and Figure 25 , used to describe an end-to-end video compression method that utilizes block partition information as additional information. In the following description, the hyper-prior model is not included.

[0217] Figure 24 is a flowchart of a method for encoding a current video by a video encoding apparatus according to at least one embodiment of the present invention.

[0218] The video encoding apparatus generates block partition information of a current video ( S2400 ).

[0219] After obtaining the current video, the video encoding device generates block partition information for the current video. For example, the video encoding device extracts information about the position, size, and shape of the codec unit provided by the video codec during intra-frame compression of the video, and uses the extracted information as the block partition information. Alternatively, the video encoding device can generate the block partition information by inputting the current video into a deep learning-based partition model (e.g., deep QTMT).

[0220] As another example, the video encoding device may derive a domain area constituting the current video and use the derived domain area as the block partition information. Alternatively, the video encoding device may derive a quantization parameter for each pixel constituting the current video and use the quantization parameter of the pixel as the block partition information.

[0221] The video encoding device inputs the block partition information into a deep learning-based extraction model to generate block partition potential features (S2402). Here, depending on the type of video encoder, the extraction model can be implemented as a transformer-based neural network or a CNN-based neural network.

[0222] The video encoding apparatus inputs the current video to a transformer-based video encoder to generate a compressed code that is a compression of the current video ( S2404 ).

[0223] As another example, a CNN-based neural network can be used as a video encoder.

[0224] In the process of generating the compressed code, the video encoding apparatus fuses the block partition latent features into the intermediate latent features in the video encoder. In this case, the block partition latent features and the intermediate latent features have the same dimension.

[0225] The video encoding apparatus encodes the block partition information (S2406). For example, the video encoding apparatus may apply arithmetic coding to the block partition information to generate a bit stream. The generated bit stream is transmitted to the video decoding apparatus.

[0226] The video encoding apparatus quantizes the compressed code (S2408).

[0227] The video encoding apparatus generates a bitstream from the quantized code (S2410). For example, the video encoding apparatus may apply arithmetic coding to the quantized code to generate a bitstream. The generated bitstream is transmitted to the video decoding apparatus.

[0228] The following steps can be further performed to train the extraction model, video encoder, and video decoder.

[0229] The video encoding apparatus decodes the quantized code from the bitstream ( S2412 ).

[0230] The video encoding apparatus inputs the quantized code to a transformer-based video decoder to generate a reconstructed video of the current video ( S2414 ).

[0231] As another example, a CNN-based neural network can be used as a video decoder.

[0232] In the process of generating the reconstructed video, the video encoding apparatus fuses the block partition latent features into the intermediate latent features in the video decoder. In this case, the block partition latent features and the intermediate latent features have the same dimension.

[0233] The video encoding apparatus uses a loss function to pre-train the extraction model, video encoder, and video decoder end-to-end based on the difference between the current video for training and the reconstructed video. In this case, the video encoding apparatus may use a loss function according to Equation 3.

[0234] Figure 25 is a flowchart of a method for reconstructing a current video by a video decoding device according to one embodiment of the present invention.

[0235] The video decoding apparatus decodes block partition information of a current video from a bitstream ( S2500 ).

[0236] The video decoding device inputs the block partition information into a deep learning-based extraction model to generate block partition potential features (S2502). Here, depending on the type of video encoder, the extraction model can be implemented as a transformer-based neural network or a CNN-based neural network.

[0237] The video decoding apparatus decodes the compressed code generated by compressing the current video from the bit stream (S2504).

[0238] The video decoding apparatus inputs the decoded code to a transformer-based video decoder to generate a reconstructed video of the current video ( S2506 ).

[0239] As another example, a CNN-based neural network can be used as a video decoder.

[0240] In the process of generating the reconstructed video, the video decoding apparatus fuses the block partition latent features into the intermediate latent features in the video decoder. In this case, the block partition latent features and the intermediate latent features have the same dimension.

[0241] Although the steps in the various flowcharts are described as being performed sequentially, these steps merely illustrate the technical concepts of some embodiments of the present invention. Therefore, a person skilled in the art to which the present invention pertains can implement the steps by changing the order in which they are described in the various figures or by performing two or more steps in parallel. Therefore, the steps in the various flowcharts are not limited to the chronological order shown.

[0242] It should be understood that the above description presents illustrative embodiments that can be implemented in various other ways. The functions described in some embodiments can be implemented by hardware, software, firmware, and / or a combination thereof. It should also be understood that the functional components described in the present invention are labeled as "units" to highlight their ability to be implemented independently.

[0243] On the other hand, the various methods or functions described in some embodiments can be implemented as instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. The non-volatile recording medium can include, for example, various types of recording devices that store data in a form readable by a computer system. For example, the non-volatile recording medium can include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical disk drives, magnetic hard drives, and solid-state drives (SSDs), etc.

[0244] Although exemplary embodiments of the present invention have been described for illustrative purposes, it will be understood by those skilled in the art that various modifications, additions, and substitutions may be made without departing from the spirit and scope of the present invention. Therefore, embodiments of the present invention have been described for brevity and clarity. The scope of the technical ideas of the embodiments of the present invention is not limited by the examples. Accordingly, it will be understood by those skilled in the art that the scope of the present invention should not be limited by the embodiments clearly described above, but by the claims and their equivalents.

[0245] Reference numerals

[0246] 1510: Analysis and transformation unit

[0247] 1520: Quantizer

[0248] 1530: Synthesis Transformation Unit

[0249] 1910: Block Partitioner

[0250] 1920: Latent feature generator.

[0251] CROSS-REFERENCE TO RELATED APPLICATIONS

[0252] This application claims priority to and the benefit of Korean Patent Application No. 10-2022-0172790, filed on December 12, 2022, and Korean Patent Application No. 10-2023-0172041, filed on December 1, 2023, the entire contents of each of which are incorporated herein by reference.

Claims

1. A method for reconstructing a current video using a video decoding device, the method comprising: Decoding block partition information of the current video from the bitstream; The block partition latent features are generated by inputting the block partition information into a deep learning based extraction model; Decoding a compressed code as a compression of the current video from a bitstream; as well as Generate a reconstructed video of the current video by inputting the decoded code into a transformer-based video decoder, Among them, generating the reconstructed video includes: Fusion of block partition latent features into intermediate latent features in video decoder.

2. The method according to claim 1, wherein The block partition information includes: The position, size, or shape of the codec unit generated by intra prediction of the current video.

3. The method according to claim 1, wherein The block partition information is transmitted after being generated by the video encoding apparatus by inputting an original video of a current video into a deep learning-based partition model.

4. The method according to claim 1, wherein The block partition information indicates a domain area constituting a current video.

5. The method according to claim 1, wherein The block partition information represents quantization parameters of pixels constituting the current video.

6. The method according to claim 1, wherein Potential features of generated block partitions include: Utilizes transformer-based neural networks for extraction models.

7. The method according to claim 1, wherein Video decoders include: Alternate patch segmentation layers and transformer block layers a preset number of times.

8. The method according to claim 1, wherein The potential features of fusion block partitions include: The block partition latent features are fused with the intermediate latent features by utilizing element-wise addition.

9. The method according to claim 1, wherein The potential features of fusion block partitions include: The block partition latent features and the intermediate latent features are fused by utilizing a splicing operation, and downsampling is utilized to reduce the dimensionality increase of the fused latent features caused by the splicing operation.

10. A method for encoding a current video using a video encoding device, the method comprising: Generate block partition information of the current video; The block partition latent features are generated by inputting the block partition information into a deep learning based extraction model; generating a compressed code as a compression of the current video by inputting the current video into a transformer-based video encoder; as well as Encode block partition information, The generated compressed code includes: Fusion of block partition latent features into intermediate latent features in video encoders.

11. The method according to claim 10, wherein: Generate block partition information includes: The position, size or shape of the codec unit generated by intra-frame prediction of the current video is generated instead of the block partition information.

12. The method according to claim 10, wherein: Generate block partition information includes: The current video is input into the deep learning based partitioning model to generate block partitioning information.

13. The method according to claim 10, wherein: Video decoders include: Alternate patch merging layers and transformer block layers a preset number of times.

14. The method according to claim 10, further comprising: Quantized compressed code; as well as Generate a bitstream from the quantized code.

15. The method according to claim 14, further comprising: Decoding quantized codes from a bitstream; as well as Generate a reconstructed video of the current video by inputting the quantized code into a transformer-based video decoder, Among them, generating the reconstructed video includes: Fusion of block partition latent features into intermediate latent features in video decoder.

16. The method according to claim 15, further comprising: Based on the difference between the current video used for training and the reconstructed video, a loss function is used to pre-train the extraction model, video encoder, and video decoder end-to-end.

17. A computer-readable recording medium storing a bitstream generated by a video encoding method, the video encoding method comprising: Generate block partition information of the current video; The block partition latent features are generated by inputting the block partition information into a deep learning based extraction model; generating a compressed code as a compression of the current video by inputting the current video into a transformer-based video encoder; as well as Encode block partition information, The generated compressed code includes: Fusion of block partition latent features into intermediate latent features in video encoders.

Images