Method and apparatus for video coding and decoding using meta-information

Abstract

The embodiment discloses a method and a device for video coding and decoding by utilizing meta information. In this embodiment, an image decoding device decodes compression parameters for a current block, and then generates meta-information by combining one or more of the compression parameters. Here, the meta-information includes each original compression parameter, or includes a meta-information vector generated from each compression parameter. The image decoding apparatus inputs meta-information into a deep learning-based extraction model to generate a meta-information feature vector, and then performs a decoding process on a current block using the meta-information feature vector.

Description

Technical Field

[0001] The invention relates to a video encoding and decoding method and device using meta information. Background Art

[0002] The statements in this section merely provide background information related to the present invention and may not necessarily constitute prior art.

[0003] Since video data has a large amount of data compared to audio data or still image data, video data requires a large amount of hardware resources (including memory) to store or transmit the video data without being compressed.

[0004] Accordingly, the encoder is generally used to compress and store or transmit video data. The decoder receives the compressed video data, decompresses the received compressed video data, and plays the decompressed video data. Video compression technologies include H.264 / Advanced Video Codec (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Codec (VVC), which improves the coding efficiency of HEVC by about 30% or more.

[0005] However, as image size, resolution and frame rate gradually increase, the amount of data to be encoded will also increase. Accordingly, new compression technologies are needed that provide higher encoding and decoding efficiency and improved image enhancement effects than existing compression technologies.

[0006] Meta information is defined as a combination of one or more parameters utilized in the video encoding and decoding process, such as block structure, intra prediction, inter prediction, quantization parameter, and motion vector. Feature vectors derived from the meta information can then be used for video encoding / decoding. Therefore, in order to improve video coding efficiency and video quality, it is necessary to consider methods for effectively deriving feature vectors from meta information and methods for effectively utilizing feature vectors. Summary of the invention

[0007] Technical issues

[0008] The present invention seeks to provide a video coding method and apparatus for extracting feature vectors from meta information by utilizing a deep learning-based extraction model. The video coding method and apparatus utilize the extracted feature vectors for video coding / decoding.

[0009] Technical Solution

[0010] At least one aspect of the present invention provides a method for reconstructing a current block performed by a video decoding device. The method includes decoding a compression parameter for the current block from a bitstream. The method also includes generating meta information by combining one or more of the compression parameters. Here, the meta information includes each compression parameter without modification, or includes a meta information vector generated from each compression parameter. The method also includes generating a feature vector of the meta information by inputting the meta information into an extraction model based on deep learning. The method also includes performing a decoding process on the current block by utilizing the feature vector.

[0011] Another aspect of the present invention provides a method for encoding a current block performed by a video encoding device. The method includes: obtaining a compression parameter for the current block. The method also includes generating meta information by combining one or more of the compression parameters. Here, the meta information includes each compression parameter without modification, or includes a meta information vector generated from each compression parameter. The method also includes generating a feature vector of the meta information by inputting the meta information into a deep learning-based extraction model. The method also includes performing an encoding process on the current block by using the feature vector.

[0012] Another aspect of the present invention provides a computer-readable recording medium storing a bitstream generated by a video encoding method. The video encoding method includes obtaining a compression parameter for a current block. The video encoding method also includes generating meta information by combining one or more of the compression parameters. Here, the meta information includes each compression parameter without modification, or includes a meta information vector generated from each compression parameter. The video encoding method also includes generating a feature vector of the meta information by inputting the meta information into an extraction model based on deep learning. The video encoding method also includes performing encoding processing on the current block by using the feature vector.

[0013] Beneficial Effects

[0014] As described above, the present invention provides a video coding method and apparatus that extracts feature vectors from meta information by using a deep learning-based extraction model and uses the extracted feature vectors for video coding / decoding. Therefore, the video coding method and apparatus improve video coding efficiency and enhance video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of a video encoding device that can implement the technology of the present invention.

[0016] Figure 2 A method of partitioning blocks using a quadtree plus binary tree ternary tree (QTBTTT) structure is shown.

[0017] Figure 3a and Figure 3bMultiple intra prediction modes are shown including a wide-angle intra prediction mode.

[0018] Figure 4 Shows the neighboring blocks of the current block.

[0019] Figure 5 is a block diagram of a video decoding device that can implement the technology of the present invention.

[0020] Figure 6 Matrix-weighted intra prediction is shown.

[0021] Figure 7 A feed-forward network as a deep learning based neural network is shown.

[0022] Figure 8 A video decoding apparatus using meta information according to an embodiment of the present invention is shown.

[0023] Fig. 9 The utilization of meta information according to one embodiment of the present invention is shown.

[0024] Fig.10 The use of meta information according to another embodiment of the present invention is shown.

[0025] Fig.11 A method of encoding a current block performed by a video encoding apparatus according to an embodiment of the present invention is illustrated.

[0026] Fig.12 A method of decoding a current block performed by a video decoding apparatus according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] Hereinafter, some embodiments of the present invention will be described in detail with reference to the attached illustrative drawings. In the following description, the same reference numerals represent the same elements, although the elements are shown in different drawings. In addition, in the following description of some embodiments, when it is considered that the detailed description of related known components and functions obscures the subject matter of the present invention, the detailed description of related known components and functions may be omitted for the sake of clarity and brevity.

[0028] Figure 1 is a block diagram of a video encoding device that can implement the technology of the present invention. Figure 1 A video encoding device and components of the device are described in detail.

[0029] The encoding device may include: an image divider 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filtering unit 180 and a memory 190.

[0030] Each component of the encoding device can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the function of the software corresponding to each component.

[0031] A video consists of one or more sequences including multiple images. Each image is divided into multiple regions, and encoding is performed on each region. For example, an image is divided into one or more tiles or / and slices. Here, one or more tiles can be defined as a tile group. Each tile or / and slice is divided into one or more coding tree units (coding tree unit, CTU). In addition, each CTU is divided into one or more coding units (coding unit, CU) through a tree structure. The information applied to each coding unit (CU) is encoded as the syntax of the CU, and the information commonly applied to the CU included in a CTU is encoded as the syntax of the CTU. In addition, the information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and the information applied to all blocks constituting one or more images is encoded as a picture parameter set (Picture Parameter Set, PPS) or a picture header. In addition, the information commonly referenced by multiple images is encoded as a sequence parameter set (Sequence Parameter Set, SPS). In addition, the information commonly referenced by one or more SPSs is encoded as a video parameter set (Video Parameter Set, VPS). In addition, information commonly applied to one tile or tile group may also be encoded as syntax of the tile or tile group header.The syntax included in the SPS, PPS, slice header, tile or tile group header may be referred to as a high-level syntax.

[0032] The image divider 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as a syntax of an SPS or a PPS and transmitted to a video decoding device.

[0033] The image divider 110 divides each image constituting a video into a plurality of coding tree units (CTUs) having a predetermined size, and then recursively divides the CTUs by using a tree structure. A leaf node in the tree structure becomes a coding unit (CU), which is a basic unit of coding.

[0034] The tree structure can be a quadtree (quadtree, QT), in which a higher node (or parent node) is split into four lower nodes (or child nodes) of the same size. The tree structure can also be a binary tree (binarytree, BT), in which a higher node is split into two lower nodes. The tree structure can also be a ternary tree (ternarytree, TT), in which a higher node is split into three lower nodes at a ratio of 1:2:1. The tree structure can also be a structure in which two or more structures in the QT structure, BT structure, and TT structure are mixed. For example, a quadtree plus binary tree (quadtree plus binarytree, QTBT) structure can be utilized, or a quadtree plus binary tree ternary tree (quadtree plus binarytreeternarytree, QTBTTT) structure can be utilized. Here, a binary tree ternary tree (BTTT) is added to the tree structure to be called a multiple-type tree (multiple-type tree, MTT).

[0035] Figure 2 is a schematic diagram for describing a method of partitioning a block by using a QTBTTT structure.

[0036] like Figure 2 As shown, the CTU can be first split into a QT structure. The quadtree splitting can be recursive until the size of the split block reaches the minimum block size (MinQTSize) of the leaf node allowed in QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder 155 and notified to the video decoding device with a signal. When the leaf node of QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, the leaf node can be further split into at least one of a BT structure or a TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions, namely, a direction for horizontally splitting blocks of the corresponding node and a direction for vertically splitting blocks of the corresponding node. As Figure 2 As shown, when MTT splitting starts, the entropy encoder 155 encodes a second flag (mtt_split_flag) indicating whether the node is split, and when the node is split, a flag additionally indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (binary or trifurcated) and notifies the video decoding device of the same by signal.

[0037] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of the lower layer, the CU split flag (split_cu_flag) indicating whether the node is split may also be encoded. When the value of the CU split flag (split_cu_flag) indicates that each node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU, which is the basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that each node is split, the video encoding device first starts encoding the first flag in the above scheme.

[0038] When QTBT is used as another example of a tree structure, there may be two types, namely, a type in which the block of the corresponding node is horizontally split into two blocks of the same size (i.e., symmetrical horizontal splitting) and a type in which the block of the corresponding node is vertically split into two blocks of the same size (i.e., symmetrical vertical splitting). The entropy encoder 155 encodes a split flag (split_flag) indicating whether each node of the BT structure is split into blocks of the lower layer and split type information indicating the split type, and transmits them to the video decoding device. On the other hand, there may be another type in which the block of the corresponding node is split into two blocks that are asymmetric to each other. The asymmetric form may include a form in which the block of the corresponding node is split into two rectangular blocks with a size ratio of 1:3, or may also include a form in which the block of the corresponding node is split in a diagonal direction.

[0039] The CU may have various sizes according to the QTBT or QTBTTT partitioned from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block". When QTBTTT partitioning is adopted, the shape of the current block may also be a rectangular shape in addition to a square shape.

[0040] The predictor 120 predicts the current block to generate a predicted block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.

[0041] Typically, each of the current blocks in an image may be predictively encoded. Typically, the prediction of the current block may be performed by utilizing an intra-prediction technique (which utilizes data from an image including the current block) or an inter-prediction technique (which utilizes data from an image encoded prior to the image including the current block). Inter-prediction includes both unidirectional prediction and bidirectional prediction.

[0042] The intra-frame predictor 122 predicts pixels in the current block by using pixels (reference pixels) located adjacent to the current block in the current image including the current block. There are multiple intra-frame prediction modes according to the prediction direction. For example, Figure 3aAs shown, the plurality of intra prediction modes may include two non-directional modes including a planar mode and a DC mode, and may include 65 directional modes. Neighboring pixels and algorithm equations to be used are defined differently according to each prediction mode.

[0043] In order to perform efficient directional prediction for a current block with a rectangular shape, we can additionally utilize Figure 3b The directional modes are shown in the dashed arrows (#67 to #80, intra-frame prediction modes #-1 to #-14). The directional modes can be called "wide angle intra-frame prediction modes". Figure 3b In the figure, the arrows indicate the corresponding reference samples used for prediction, rather than representing the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra prediction mode is a mode in which prediction is performed in a direction opposite to the specific direction mode without additional bit transmission. In this case, in the wide-angle intra prediction mode, some wide-angle intra prediction modes available for the current block can be determined by the ratio of the width to the height of the current block having a rectangular shape. For example, when the current block has a rectangular shape with a height less than the width, a wide-angle intra prediction mode (intra prediction modes #67 to #80) with an angle less than 45 degrees is available. When the current block has a rectangular shape with a width greater than the height, a wide-angle intra prediction mode with an angle greater than -135 degrees is available.

[0044] The intra-frame predictor 122 may determine an intra-frame prediction to be used for encoding the current block. In some examples, the intra-frame predictor 122 may encode the current block by utilizing multiple intra-frame prediction modes, and may also select an appropriate intra-frame prediction mode to be utilized from a test mode. For example, the intra-frame predictor 122 may calculate a rate-distortion value by utilizing a rate-distortion analysis of multiple tested intra-frame prediction modes, and may also select an intra-frame prediction mode having the best rate-distortion characteristic in the test mode.

[0045] The intra-frame predictor 122 selects an intra-frame prediction mode among a plurality of intra-frame prediction modes and predicts the current block by using adjacent pixels (reference pixels) and an algorithm equation determined according to the selected intra-frame prediction mode. Information about the selected intra-frame prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0046] The inter-frame predictor 124 generates a prediction block for the current block by using a motion compensation process. The inter-frame predictor 124 searches for a block that is most similar to the current block in a reference image that has been encoded and decoded earlier than the current image, and generates a prediction block for the current block by using the searched block. In addition, a motion vector (MV) is generated, which corresponds to the displacement between the current block in the current image and the prediction block in the reference image. Typically, motion estimation is performed on the luminance (luma) component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. The motion information including information of the reference image and information about the motion vector used to predict the current block is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0047] The inter-frame predictor 124 may also perform interpolation of a reference image or a reference block to increase the accuracy of the prediction. In other words, a subsample is interpolated between two consecutive integer samples by applying a filter coefficient to a plurality of consecutive integer samples including two integer samples. When performing a process of searching for a block most similar to the current block on the interpolated reference image, a decimal unit precision may be represented for the motion vector instead of an integer sample unit precision. For each target region to be encoded, for example, a unit such as a slice, a tile, a CTU, a CU, etc., the precision or resolution of the motion vector may be set differently. When such an adaptive motion vector resolution (AMVR) is applied, information about the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information about the motion vector resolution applied to each CU is signaled. The information about the motion vector resolution may be information representing the precision of the motion vector difference to be described below.

[0048] On the other hand, the inter-frame predictor 124 may perform inter-frame prediction by utilizing bidirectional prediction. In the case of bidirectional prediction, two reference images and two motion vectors representing the block positions most similar to the current block in each reference image are utilized. The inter-frame predictor 124 selects a first reference image and a second reference image from a reference image list 0 (RefPicList0) and a reference image list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for a block most similar to the current block in the corresponding reference image to generate a first reference block and a second reference block. In addition, a prediction block of the current block is generated by averaging or weighted averaging the first reference block and the second reference block. In addition, motion information including information about two reference images used to predict the current block and including information about two motion vectors is transmitted to the entropy encoder 155. Here, the reference image list 0 may be composed of images that are before the current image in the display order in the pre-reconstructed image, and the reference image list 1 may be composed of images that are after the current image in the display order in the pre-reconstructed image. However, although not particularly limited thereto, the pre-reconstructed image that is after the current image in the display order may be additionally included in the reference image list 0. Conversely, a pre-reconstructed image preceding the current image may also be additionally included in the reference image list 1 .

[0049] In order to minimize the amount of bits consumed for encoding motion information, various methods may be utilized.

[0050] For example, when the reference image and motion vector of the current block are the same as those of the neighboring block, information capable of identifying the neighboring block is encoded to transmit the motion information of the current block to the video decoding device. This method is called merge mode.

[0051] In the merge mode, the inter predictor 124 selects a predetermined number of merge candidate blocks (hereinafter, referred to as 'merge candidates') from neighboring blocks of the current block.

[0052] As the neighboring blocks for deriving the merge candidate, all or some of the left block A0, the lower left block A1, the upper block B0, the upper right block B1 and the upper left block B2 adjacent to the current block in the current image may be used, such as Figure 4 As shown. In addition, in addition to the current image where the current block is located, a block located in a reference image (which may be the same as or different from the reference image used to predict the current block) may also be used as a merge candidate. For example, a co-located block of the current block in the reference image or a block adjacent to the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the above method is less than the preset number, a zero vector is added to the merge candidate.

[0053] The inter-frame predictor 124 configures a merge list including a predetermined number of merge candidates by using adjacent blocks. A merge candidate to be used as motion information of the current block is selected from among the merge candidates included in the merge list, and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0054] Merge skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy coding are close to zero, only the neighbor block selection information is transmitted without the residual signal. By using merge skip mode, relatively high coding efficiency can be achieved for images with slight motion, still images, screen content images, etc.

[0055] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0056] Another method for encoding motion information is the advanced motion vector prediction (AMVP) mode.

[0057] In the AMVP mode, the inter-frame predictor 124 derives a motion vector prediction candidate for a motion vector of a current block by using neighboring blocks of the current block. As neighboring blocks for deriving motion vector prediction candidates, Figure 4 All or some of the left block A0, lower left block A1, upper block B0, upper right block B1 and upper left block B2 adjacent to the current block in the current image shown. In addition, in addition to the current image where the current block is located, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) may also be used as adjacent blocks for deriving motion vector prediction candidates. For example, the same block of the current block in the reference image or a block adjacent to the same block may be used. If the number of motion vector candidates selected by the above method is less than the preset number, a zero vector is added to the motion vector candidates.

[0058] The inter-frame predictor 124 derives a motion vector prediction candidate by using the motion vectors of the neighboring blocks, and determines a motion vector prediction of the motion vector of the current block by using the motion vector prediction candidate. In addition, a motion vector difference is calculated by subtracting the motion vector prediction from the motion vector of the current block.

[0059] The motion vector prediction can be obtained by applying a predefined function (e.g., median and mean calculations, etc.) to the motion vector prediction candidate. In this case, the video decoding device also knows the predefined function. In addition, since the neighboring blocks used to derive the motion vector prediction candidate are blocks that have completed encoding and decoding, the video decoding device may also already know the motion vectors of the neighboring blocks. Therefore, the video encoding device does not need to encode the information for identifying the motion vector prediction candidate. Accordingly, in this case, information about the motion vector difference and information about the reference image used to predict the current block are encoded.

[0060] On the other hand, the motion vector prediction can also be determined by selecting any one of the motion vector prediction candidates. In this case, information for identifying the selected motion vector prediction candidate is additionally encoded together with information about the motion vector difference used to predict the current block and information about the reference image.

[0061] The subtractor 130 generates a residual block by subtracting a prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block.

[0062] The transformer 140 transforms the residual signal in the residual block having the pixel value of the spatial domain into the transform coefficient of the frequency domain. The transformer 140 may transform the residual signal in the residual block by using the entire size of the residual block as the transform unit, or may also divide the residual block into a plurality of sub-blocks, and may perform the transform by using the sub-block as the transform unit. Alternatively, the residual block is divided into two sub-blocks, namely, a transform region and a non-transform region, so as to transform the residual signal by using only the transform region sub-block as the transform unit. Here, the transform region sub-block may be one of two rectangular blocks having a size ratio of 1:1 based on the horizontal axis (or the vertical axis). In this case, the entropy encoder 155 encodes a flag (cu_sbt_flag) indicating that only the sub-block is transformed, and direction (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag), and notifies the video decoding device with a signal. In addition, the size of the transform region subblock may have a size ratio of 1:3 based on the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_quad_flag) for dividing the corresponding partition is additionally encoded by the entropy encoder 155 and signaled to the video decoding apparatus.

[0063] On the other hand, the transformer 140 may perform transformation of the residual block separately in the horizontal direction and the vertical direction. For the transformation, various types of transformation functions or transformation matrices may be used. For example, paired transformation functions for horizontal transformation and vertical transformation may be defined as a multiple transform set (MTS). The transformer 140 may select a transformation function pair with the highest transformation efficiency in the MTS, and may transform the residual block in each of the horizontal and vertical directions. Information (mts_idx) about the transformation function pair in the MTS is encoded by the entropy encoder 155 and notified to the video decoding device with a signal.

[0064] The quantizer 145 quantizes the transform coefficients output from the transformer 140 using the quantization parameters and outputs the quantized transform coefficients to the entropy encoder 155. The quantizer 145 may also quantize the relevant residual block immediately without transforming any block or frame. The quantizer 145 may also apply different quantization coefficients (scaling values) according to the position of the transform coefficients in the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to the video decoding device.

[0065] The rearrangement unit 150 may perform rearrangement of coefficient values ​​on the quantized residual value.

[0066] The rearrangement unit 150 may change the 2D coefficient array into a 1D coefficient sequence by utilizing coefficient scanning. For example, the rearrangement unit 150 may utilize a zig-zag scan or a diagonal scan to scan the DC coefficient to the coefficient of the high frequency region to output a 1D coefficient sequence. Depending on the size of the transform unit and the intra-frame prediction mode, the zig-zag scan may also be replaced by a vertical scan that scans the 2D coefficient array in the column direction and a horizontal scan that scans the 2D block type coefficient in the row direction. In other words, depending on the size of the transform unit and the intra-frame prediction mode, the scanning method to be utilized may be determined among zig-zag scanning, diagonal scanning, vertical scanning, and horizontal scanning.

[0067] The entropy encoder 155 encodes the sequence of 1D quantized transform coefficients output from the rearrangement unit 150 by using various encoding schemes including Context-based Adaptive Binary Arithmetic Code (CABAC), Exponential Golomb, etc. to generate a bitstream.

[0068] In addition, the entropy encoder 155 encodes information related to block segmentation (e.g., CTU size, CTU segmentation flag, QT segmentation flag, MTT segmentation type, and MTT segmentation direction, etc.) so that the video decoding device can segment the block equivalent to the video encoding device. In addition, the entropy encoder 155 encodes information about the prediction type indicating whether the current block is encoded by intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about intra-frame prediction mode) or inter-frame prediction information (merge index in the case of merge mode, and information about reference image index and motion vector difference in the case of AMVP mode) according to the prediction type. In addition, the entropy encoder 155 encodes information related to quantization (i.e., information about quantization parameters and information about quantization matrices).

[0069] The inverse quantizer 160 inversely quantizes the quantized transform coefficient output from the quantizer 145 to generate a transform coefficient. The inverse transformer 165 transforms the transform coefficient output from the inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct a residual block.

[0070] The adder 170 adds the reconstructed residual block and the prediction block generated by the predictor 120 to reconstruct the current block. When intra-prediction is performed on the next block, the pixels in the reconstructed current block are used as reference pixels.

[0071] The loop filtering unit 180 performs filtering on the reconstructed pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transform / quantization. The loop filtering unit 180 as an in-loop filter may include all or some of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[0072] The deblocking filter 182 filters the boundaries between the reconstructed blocks to remove blocking artifacts that occur due to block unit encoding / decoding, and the SAO filter 184 and the ALF 186 perform additional filtering on the deblocking filtered video. The SAO filter 184 and the ALF 186 are filters for compensating for the difference between the reconstructed pixels and the original pixels that occur due to lossy coding. The SAO filter 184 applies an offset in units of CTU to enhance subjective image quality and encoding efficiency. On the other hand, the ALF 186 performs block unit filtering and applies different filters to compensate for distortion by dividing the boundaries of the corresponding blocks and the degree of the amount of change. Information about the filter coefficients to be used for the ALF can be encoded and notified to the video decoding device with a signal.

[0073] The reconstructed blocks filtered by the deblocking filter 182, the SAO filter 184, and the ALF 186 are stored in the memory 190. When all blocks in one image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks within an image to be subsequently encoded.

[0074] The video encoding device may store a bit stream of encoded video data in a nonvolatile storage medium or transmit the bit stream to a video decoding device through a communication network.

[0075] Figure 5 is a functional block diagram of a video decoding device that can implement the technology of the present invention. Figure 5 , describes a video decoding device and components of the device.

[0076] The video decoding apparatus may include an entropy decoder 510 , a rearrangement unit 515 , an inverse quantizer 520 , an inverse transformer 530 , a predictor 540 , an adder 550 , a loop filtering unit 560 , and a memory 570 .

[0077] Similar to Figure 1 Each component of the video encoding device and the video decoding device can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the function of the software corresponding to each component.

[0078] The entropy decoder 510 extracts information related to block partitioning by decoding a bitstream generated by a video encoding apparatus to determine a current block to be decoded, and extracts prediction information required to reconstruct the current block and information about a residual signal.

[0079] The entropy decoder 510 determines the size of the CTU by extracting information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), and divides the image into CTUs of the determined size. In addition, the CTU is determined as the highest level (i.e., the root node) of the tree structure, and the partition information of the CTU can be extracted to partition the CTU by using the tree structure.

[0080] For example, when CTU is segmented by using the QTBTTT structure, the first flag (QT_split_flag) related to the segmentation of QT is first extracted to segment each node into four nodes of the lower layer. In addition, the second flag (mtt_split_flag) related to the segmentation of MTT, the segmentation direction (vertical / horizontal) and / or the segmentation type (binary / tripartite) are extracted with respect to the node corresponding to the leaf node of QT to segment the corresponding leaf node into the MTT structure. As a result, each node below the leaf node of QT is recursively segmented into a BT or TT structure.

[0081] As another example, when a CTU is split by using a QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU is split is extracted. When the corresponding block is split, a first flag (QT_split_flag) may also be extracted. During the splitting process, for each node, 0 or more recursive MTT splits may occur after 0 or more recursive QT splits. For example, for a CTU, MTT splits may occur immediately, or, conversely, multiple QT splits may occur only.

[0082] As another example, when CTU is segmented by using the QTBT structure, the first flag (QT_split_flag) related to the segmentation of the QT is extracted to segment each node into four nodes of the lower layer. In addition, the segmentation flag (split_flag) indicating whether the node corresponding to the leaf node of the QT is further segmented into BT and the segmentation direction information are extracted.

[0083] On the other hand, when the entropy decoder 510 determines the current block to be decoded by using the partitioning of the tree structure, the entropy decoder 510 extracts information about the prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts syntax elements of intra-prediction information (intra-prediction mode) for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts information representing syntax elements of inter-prediction information, that is, a motion vector and a reference image to which the motion vector refers.

[0084] In addition, the entropy decoder 510 extracts quantization-related information and extracts information on a quantized transform coefficient of the current block as information on a residual signal.

[0085] The rearrangement unit 515 may change the sequence of 1D quantized transform coefficients entropy-decoded by the entropy decoder 510 into a 2D coefficient array (ie, block) again in the reverse order of the coefficient scanning order performed by the video encoding device.

[0086] The inverse quantizer 520 inversely quantizes the quantized transform coefficients, and inversely quantizes the quantized transform coefficients by using quantization parameters. The inverse quantizer 520 may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in 2D. The inverse quantizer 520 may perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from a video encoding device to a 2D array of quantized transform coefficients.

[0087] The inverse transformer 530 reconstructs a residual signal by inversely transforming the inversely quantized transform coefficient from the frequency domain to the spatial domain to generate a residual block of the current block.

[0088] In addition, when the inverse transformer 530 inversely transforms a partial region (sub-block) of the transform block, the inverse transformer 530 extracts a flag (cu_sbt_flag) that only the sub-block of the transform block is transformed, direction (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block. The inverse transformer 530 also inversely transforms the transform coefficient of the corresponding sub-block from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the region that is not inversely transformed with a value of "0" as the residual signal to generate a final residual block of the current block.

[0089] In addition, when MTS is applied, the inverse transformer 530 determines the transform index or transform matrix to be applied in each of the horizontal and vertical directions by using the MTS information (mts_idx) signaled from the video encoding device. The inverse transformer 530 also performs inverse transform on the transform coefficients in the transform block in the horizontal and vertical directions by using the determined transform function.

[0090] The predictor 540 may include an intra predictor 542 and an inter predictor 544. The intra predictor 542 is activated when the prediction type of the current block is intra prediction, and the inter predictor 544 is activated when the prediction type of the current block is inter prediction.

[0091] The intra predictor 542 determines an intra prediction mode of a current block among a plurality of intra prediction modes according to syntax elements of the intra prediction mode extracted from the entropy decoder 510. The intra predictor 542 also predicts the current block by using neighboring reference pixels of the current block according to the intra prediction mode.

[0092] The inter predictor 544 determines a motion vector of a current block and a reference image to which the motion vector refers by using a syntax element of the inter prediction mode extracted from the entropy decoder 510 .

[0093] The adder 550 reconstructs the current block by adding the residual block output from the inverse transformer 530 and the prediction block output from the inter predictor 544 or the intra predictor 542. When intra-predicting a subsequent block to be decoded, pixels within the reconstructed current block are used as reference pixels.

[0094] The loop filtering unit 560 as an in-loop filter may include a deblocking filter 562, an SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundary between the reconstructed blocks in order to remove block artifacts that occur due to block unit decoding. The SAO filter 564 and the ALF 566 perform additional filtering on the reconstructed blocks after deblocking filtering in order to compensate for the difference between the reconstructed pixels and the original pixels that occurs due to lossy encoding. The filter coefficients of the ALF are determined by using information about the filter coefficients decoded from the bitstream.

[0095] The reconstructed blocks filtered by the deblocking filter 562, the SAO filter 564, and the ALF 566 are stored in the memory 570. When all blocks in one image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks within an image to be subsequently encoded.

[0096] In some embodiments, the present invention relates to encoding and decoding video images as described above. More specifically, the present invention provides a video encoding and decoding method and apparatus that extracts feature vectors from meta-information using a deep learning-based extraction model and uses the extracted feature vectors for video encoding / decoding.

[0097] The following embodiments may be performed by various components in a video encoding device. The following embodiments may also be performed by various components in a video decoding device.

[0098] The video encoding device may generate signaling information associated with the present embodiment from the perspective of optimizing rate distortion when encoding the current block. The video encoding device may encode the signaling information using the entropy encoder 155 and send the encoded signaling information to the video decoding device. The video decoding device may decode the signaling information associated with the decoding of the current block from the bitstream using the entropy decoder 510.

[0099] In the following description, the term 'target block' may be used interchangeably with a current block or a coding unit (CU), or may refer to some regions of a coding unit.

[0100] In addition, a flag value of true indicates a case where the flag is set to 1. In addition, a flag value of false indicates a case where the flag is set to 0.

[0101] I. Matrix weighted Intra Prediction (MIP)

[0102] Figure 6 Matrix-weighted intra prediction is shown.

[0103] When intra prediction is performed on a current block having a width of W and a height of H, a prediction factor may be generated based on a predefined matrix operation using neighboring pixels of the current block and encoding information of the current block, such as Figure 6 This rule-based prediction method is called Matrix Weighted Intra Prediction (MIP).

[0104] MIP generates all or part of the intra prediction factors using predefined matrix operations. If a partial prediction factor is generated, MIP can additionally perform interpolation using the partial prediction factor for upsampling or upscaling to generate a final intra prediction sample having the same size as the current block.

[0105] On the other hand, the MIP may selectively select some pixels that are spatially adjacent to the current block and may use some pixels as neighboring pixels of the current block. In another embodiment, the MIP may utilize values ​​derived from operations such as subsampling and reduction for matrix operations.

[0106] Figure 6 An example of generating a partial predictor of a current block using a value derived from an operation and a matrix having a size smaller than that of the current block is shown. Figure 6 An example of an implementation of MIP is described in FIG.

[0107] In the first step, a certain number of samples are generated from the boundary samples of the current block using an averaging operation. For example, the upper boundary pixel bdry is generated using a predefined rule according to the block size. top and left border pixel bdry left Generate reduced border pixels bdry top red and bdry left red Here, the predefined rule may be a combination of averaging and downsampling. In addition, by combining the bdry reduced according to the predefined rule top red and bdry leftred To generate the reduced boundary vector bdry red According to the size and mode of the current block, the reduced boundary vector bdry can be generated as shown in Equation 1 red .

[0108] [Equation 1]

[0109]

[0110] Here, mode represents the MIP mode to be described later. According to equation 1, when W = H = 4 and mode is less than 18, it is connected by connecting bdry top red and bdry left red To generate bdry red On the other hand, when the mode is 18 or higher, by connecting bdry in sequence left red and bdry top red Generate bdry in the order of red On the other hand, when W=H=4, bdry red The size of bdry is 4; in all other cases, red The size is 8.

[0111] In the second step, Figure 6 As shown, for the reduced boundary vector bdry red Apply a predefined matrix operation to generate a reduced prediction factor pred for a specific portion of the current block red Here, pred red is of width W red and height H red The width W can be determined according to the size of the current block. red and height H red , as shown in Equation 2.

[0112] [Equation 2]

[0113]

[0114] On the other hand, the reduced prediction factor pred can be calculated according to Equation 3 using the matrix and the offset vector red .

[0115] [Equation 3]

[0116] pred red =A k ·bdry red +bk

[0117] Here, A k is a predefined matrix with W red ·H red The sum of the number of rows is equal to the dimension bdry red Therefore, when W=H=4, A k has 4 columns; otherwise, it has 8 columns. On the other hand, the offset vector b k is a predefined vector with dimension W red ·H red .. A k and b k The subscript k is an index indicating one of the predefined matrices and vectors.

[0118] A k and b k Can be selected from Set S 0 , S 1 and S 2 At this time, as shown in Equation 4, an index idx(W,H) indicating a specific set is extracted based on the block size.

[0119] [Equation 4]

[0120]

[0121] Set S 0 Includes 18 matrices Each matrix has 16 rows and 4 columns and 18 offset vectors Each matrix is ​​of size 16 and is used for 4×4 blocks. Set S 1 Includes 10 matrices Each matrix has 16 rows and 8 columns and 10 offset vectors Each matrix is ​​of size 16 and is used for 4×4, 8×4, and 8×8 blocks. In addition, the set S 2 Includes 6 matrices Each matrix has 64 rows and 8 columns and 6 offset vectors Each matrix is ​​of size 64 and is used for blocks of various sizes.

[0122] For each index idx(W,H) based on the block size, the number of MIP modes is 35, 19, and 11, respectively. On the other hand, in order to reduce memory consumption, a pair of MIP modes can use the same matrix and offset vector. Therefore, in A of Equation 3 k and b k In this paper, k can be expressed by Equation 5.

[0123] [Equation 5]

[0124]

[0125] In the third step, linear interpolation is applied to the reduced predictor pred red , to generate prediction samples for the remaining positions of the current block. Linear interpolation is first applied in the horizontal direction and then in the vertical direction, regardless of the size and shape of the block.

[0126] For example, in order to indicate whether the MIP is activated, the video encoding device may encode a matrix-based prediction flag, and then the encoded flag may be sent to the video decoding device. When the MIP mode is applied, the most probable mode (MPM) flag is signaled to indicate whether the prediction mode is one of the MPM modes. In MIP, 3-MPM is applied so that the MPM mode is encoded with a truncated binary code and the non-MPM mode is encoded with a fixed length code (FLC). At this time, if there are blocks (referred to as "regular blocks") to which the MIP mode is not applied around the block to which the MIP mode is applied, the MPM derivation of the block to which the MIP mode is applied may be performed using a mapping table. In other words, the mapping table is used to derive a MIP mode having similar characteristics from an adjacent block to which a regular mode is applied. The MIP mode derived in this way is used to derive the MPM of the block to which the MIP mode is applied.

[0127] In addition, if there are adjacent blocks to which MIP is applied around the conventional block, the mapping table can be used to perform MPM derivation for the conventional block. In other words, the mapping table is used to derive a conventional pattern with similar characteristics from the MIP pattern of the adjacent block to which MIP is applied. The conventional pattern derived in this way is used to derive the MPM of the conventional block. Similarly, even if the luma block at the same position used in the derivation of the chroma DM applies MIP, the mapping table can be used to derive the conventional pattern, and the derived conventional pattern can be used for the derivation of the chroma DM.

[0128] The mapping between the normal mode and the MIP mode using the mapping table can be expressed as shown in Equation 6.

[0129] [Equation 6]

[0130] predmode ALWIP =map_angular_to_alwip idx [predmode Angular ]

[0131] predmode Angular =map_alwip_to_angular idx(PU)[predmode ALWIP ]

[0132] Here, affine linear weighted intra prediction (ALWIP) is another representative of MIP. In Equation 6, the former equation maps the normal mode to the MIP mode, and the latter equation maps the MIP mode to the normal mode.

[0133] A portion of the mapping table that maps the normal mode to the MIP mode for each index idx(W,H) may be expressed as shown in Table 1 based on the block size.

[0134] [Table 1]

[0135]

[0136] In Table 1, MipSizeID is the same value as the index idx(W,H). [xNbX][yNbX] represents the position of the adjacent block, and IntraPreModeY[xNbX][yNbX] represents the normal mode at the corresponding position. According to Table 1, the normal intra prediction mode ( Figure 3a 0 to 66) in can be mapped to the corresponding MIP mode.

[0137] A portion of the mapping table that maps the MIP mode to the normal mode for each index idx(W,H) may be expressed as shown in Table 2 based on the block size.

[0138] [Table 2]

[0139]

[0140] In Table 2, MipSizeID is the same value as the index idx(W,H). [xNbX][yNbX] represents the position of the neighboring block, and IntraPreModeY[xNbX][yNbX] represents the normal mode at the corresponding position. According to Table 2, the MIP mode corresponding to each index idx(W,H) can be mapped to the normal intra prediction mode ( Figure 3a 0 to 66 in the range.

[0141] In another example, in order to indicate whether the MIP is activated, the video encoding device may encode a matrix-based prediction flag, and then the encoded flag may be sent to the video decoding device. At this time, the MIP may be performed separately from the conventional intra-frame prediction mode using the MPM. In addition, the video encoding device may encode the MIP mode to indicate one of the predefined matrices and one of the predefined vectors, and then the encoded MIP mode may be sent to the video decoding device. On the other hand, if there are blocks to which the MIP is applied around the regular block, it is assumed that the neighboring blocks to which the MIP is applied are in plane mode, and an MPM candidate list for the regular block may be generated.

[0142] II. Multiple transform selection (MTS)

[0143] When applying a transform in HEVC, DCT-II is used as a transform kernel (hereinafter, used interchangeably with a transform type) to transform the residual signal of a transform block (TB). However, in order to apply a more suitable transform technique depending on the diversity of residual signal characteristics, multiple transform selection (MTS) can be utilized. MTS determines one or two optimal types among multiple transform types, and then transforms the block according to the determined transform type. For example, in VVC, as shown in Table 3, in addition to DCT-II, two other transform types DCT-VIII and DST-VII are added so that the residual signal can be transformed in various ways.

[0144] [Table 3]

[0145]

[0146] Here, the basis functions constitute the transform matrix that defines each transform type. In the following, DCT-II, DCT-VIII and DST-VII will be used interchangeably with DCT2, DCT8 and DST7, respectively.

[0147] The flag for MTS may be determined at the CU level, and the transform type may be signaled in various ways depending on whether the prediction mode is intra / inter prediction and whether the prediction mode is in the horizontal / vertical direction, as shown in Table 4.

[0148] [Table 4]

[0149]

[0150] First, if MTS_CU_flag is 0, DCT2 transform type is applied in horizontal and vertical directions. On the other hand, if MTS_CU_flag is not 0, different transform types may be applied in horizontal and vertical directions according to a combination of MTS_Hor_flag and MTS_Ver_flag.

[0151] On the other hand, the MTS index (mts_idx) may be derived according to Table 4. For example, if MTS_CU_flag is 0, mts_idx may be set to 0. In addition, if MTS_CU_flag is 1, the MTS index may be set in the order of Table 4 according to a combination of MTS_Hor_flag and MTS_Ver_flag. For example, if MTS_Hor_flag=0 and MTS_Ver_flag=0, mts_idx may be set to 1; if MTS_Hor_flag=1 and MTS_Ver_flag=1, mts_idx may be set to 4.

[0152] III. Feedforward Network

[0153] A neural network based on deep learning consists of multiple neurons and edges connecting neurons, such as Figure 7 As shown in the example of . The input layer and output layer of the neural network are composed of a single layer, and the hidden layer may include one or more layers. A hidden layer includes one or more hidden units (or hidden nodes). In addition, each edge has a different weight. In addition, an activation function can be used in the process of propagating output values ​​from one layer to the next layer. Typical activation functions include sigmoid function, tangent hyperbolic function and rectified linear unit (Relu) function.

[0154] like Figure 7 As shown in the example of , a two-layer neural network model with a basic forward network structure provides an output vector y by passing an input vector X through an intermediate hidden unit. In the feedforward network structure, the value generated at any node located in the output layer can be expressed as shown in Equation 7.

[0155] [Equation 7]

[0156]

[0157] At this time, W 1 and W 2 is a weight matrix corresponding to the edge connecting the input vector to the hidden unit and the edge connecting the hidden unit to the output vector. ji (1) and w kj(2) Represents W 1 and W 2 Elements of y k Represents the elements of the output vector y. 0 and z 0 represents the bias units of the input layer and the hidden layer, respectively. Therefore, D represents the dimension of the input vector X, M represents the number of hidden units, and K represents the dimension of the output vector y. In addition, h and σ are activation functions applied to the hidden layer and the output layer, respectively. On the other hand, the activation function is not necessarily applied to the output of the hidden layer and the output layer. If the activation function is not used, Equation 7 includes the same as W 1 and W 2 The operation corresponding to the weight matrix multiplication between .

[0158] The process of calculating the weights of the edges based on the training data and labels is called training. Training can usually be performed using a stochastic gradient descent (SGD) algorithm based on a back-propagation algorithm. The process of calculating the output in the feed-forward direction using the weights (which are the parameters calculated based on the training) is called inference or testing.

[0159] The following embodiments are described with reference to a video decoding apparatus, but they may also be performed by a video encoding apparatus as described above.

[0160] IV. Embodiments according to the present invention

[0161] Hereinafter, a combination of one or more parameters used in the compression process of the current block is defined as meta information, and a method of using the meta information for video encoding / decoding will be described. The present embodiment includes a method of generating meta information, a method of generating a meta information feature vector using the meta information, and a method of encoding / decoding the current block using the meta information feature vector.

[0162] Figure 8 A video decoding apparatus using meta information according to an embodiment of the present invention is shown.

[0163] In order to decode the current block using the meta information, the video decoding apparatus may include a meta information generator 810 , a meta information feature vector generator 820 , and a decoding executor 830 .

[0164] As described above, the meta information generator 810 generates a combination of one or more parameters used in the compression process of the current block as meta information. The compression parameters consist of syntax elements and their derivatives, including block size, block structure, reference samples, intra prediction mode, inter prediction mode, motion vector and quantization parameter. On the other hand, the compression parameters can be used for both encoding and decoding of the current block.

[0165] The meta information feature vector generator 820 inputs the meta information into the deep learning-based extraction model to generate a meta information feature vector of size K×1. Here, the extraction model as a neural network may adopt a feedforward network as described above. At this time, the feedforward network may be utilized without the activation function. The meta information feature vector may be composed of the results of a mapping table utilized in an existing VVC, a mode according to a block coding condition, and the like.

[0166] The decoding executor 830 uses the meta information feature vector to decode the current block. For example, the decoding executor 830 may perform the operation of the predictor 540. Alternatively, the decoding executor 830 may perform the operation of the inverse transformer 530. In other words, the decoding executor 830 may be any component within the video decoding device that can use the meta information feature vector to decode the current block.

[0167] In addition, it is also possible to utilize, for example, a component in a decoding path within a video encoding device. Figure 8 Generation and utilization of the meta information shown. For example, the decoding executor 830 may perform the operation of the predictor 120 within the video encoding device. Alternatively, the decoding executor 830 may perform the operation of the inverse transformer 165 within the video encoding device.

[0168] According to the present embodiment, compared to the method of determining the encoding mode and index based on the mapping table utilized in the existing VVC, the existing process can be replaced to allow utilization of more diverse compression parameters without involving a mapping table.

[0169] For example, Fig. 9 As shown in the top example of , the MIP prediction in the existing VVC technology derives the index idx as shown in Equation 4, and then selects the set S idx , the set S idx Including the matrix and the offset vector according to the derived index. At this time, as shown in Equation 4, the idx value can be derived according to the block size. However, if different idx are set based on the block structure, reference samples, quantization parameters, and other parameters that can be utilized in the intra prediction process, the number of cases may increase significantly. At this time, as Fig. 9 As shown in the bottom example of , this embodiment can efficiently handle the above-mentioned various situations.

[0170] In addition, the present embodiment can replace the mapping table used in the MIP. For example, as shown in Table 1 and Table 2, the present embodiment can replace the mapping relationship between the MIP mode and the intra-frame prediction mode based on the size of the block to which the MIP is applied. In other words, although only the size of the block to which the MIP mode is applied is used in Table 1 and Table 2, the present embodiment can realize the mapping relationship by using more diverse compression parameters.

[0171] In addition, the present embodiment can generate an intra prediction mode that replaces the MIP mode. The present embodiment can use a feed-forward network to map the meta information to a replacement index repIdx. Here, the meta information is a reference sample of the current block, and repIdx can indicate the value of the index repIdx according to the MIP mode. Figure 3a One of the 67 intra prediction modes of the example. Later, when intra prediction is performed on the block, the intra prediction mode mapped to the replaced index can be used to derive the MPM. For example, if a luminance block located on the left or upper side of the current block is predicted using MIP, the video decoding device can apply the replaced index of the corresponding luminance block to the process of constructing the MPM list.

[0172] In another example, the present embodiment may replace the MTS selection process of the VVC, such as Fig.10 As shown in the top example of . For example, Table 4 shows the mapping relationship related to the transformation kernel selection in VVC. In other words, it is checked whether the inter prediction and intra prediction modes are selected, and depending on whether the transformation is applied in the horizontal direction and the vertical direction, one of DCT2, DST7 and DCT8 can be selected as the transformation kernel. On the other hand, as Fig.10 As shown in the bottom example of , by applying this embodiment, a mapping relationship between compression parameters and transformation kernels can be established by utilizing more diverse compression parameters.

[0173] The application of the present embodiment to MIP and MTS is merely an example; the present embodiment can be widely applied to mapping tables used in existing VVC, mode settings according to block coding conditions, and the like.

[0174] On the other hand, the extraction model in the meta information feature vector generator 820 can be pre-trained by using the training meta information and the corresponding labels. The label can be, for example, the idx in the MIP or the transform kernel in the MTS as described above. During the training process, the weight of the extraction model can be updated using a loss function based on the difference between the output generated by the extraction model and the label. Since the video encoding device passes the weight to the video decoding device, the extraction model can be shared between the video encoding device and the video decoding device.

[0175] Hereinafter, a method for generating meta information will be described. The meta information represents the video compression parameters used in the video decoding process as a vector. At this time, the video compression parameters include parameters decoded at a higher level, such as VPS, SPS, adaptive parameter set (APS), video picture level, and video slice level decoded from the bitstream. In addition, the video compression parameters may include all or part of the information related to the size, partition information, intra-frame prediction, inter-frame prediction, motion vector, etc. of the video block decoded at the current block level.

[0176] For example, an example of expressing the compression parameter as a vector will be described. In other words, the video decoding apparatus can generate meta information by expressing the compression parameter as a vector. Hereinafter, a vector represents a combination of one or more bits.

[0177] First, all block structures used in the video decoding process are represented as vectors. For example, when using 2N×2N, 2N×N, N×2N, and N×N block structures, each block structure is represented as a vector of size 2, such as 00, 01, 10, and 11. In VVC, blocks are partitioned based on quadtree (QT), binary tree (BT), and ternary tree (TT) structures. Meta information represents all block structures available to video encoding devices and video decoding devices as vectors.

[0178] In addition, the size of the block determined during the video decoding process can be represented as a vector. The width and height of the block can be used, and the condition on the width / height can also be used. For example, the case where one of the width and height of the block is greater than or equal to 16 can be represented by 0, and all other cases can be represented by 1. Alternatively, the case where the width and height of the block are the same can be represented by 0, and the case where the width and height are different can be represented by 1.

[0179] In addition, the quantization parameter (QP) used in the video decoding process is represented as a vector. For example, when the quantization parameter is a single number in the range of 0 to Q-1 (where Q is a natural number), a vector with a size of floor(log 2 Q)+1, and the quantization parameter can be mapped to one of the vectors. Here, the floor function rounds the real number to the smallest natural number. In another example, instead of expressing the quantization parameter as a vector of size floor(log 2 QP=27 is represented by a vector of size 3. For example, if the range of Q is 64, then QP=27 is represented by a vector of size 011.

[0180] In addition, the direction of the intra prediction mode used in the video decoding process is represented as a vector. For example, when the direction is a single number belonging to the range of 0 to P-1 (where P is a natural number), a floor(log 2P)+1, and the intra prediction mode can be mapped to one of the vectors. In another example, instead of representing the direction of the intra prediction mode as a vector of size floor(log 2 P)+1, the range of P can be partitioned into equal segments, and the partitioned segments can be represented as vectors. Then, the direction of the intra-frame prediction mode can be mapped to one of the vectors. For example, when the range is partitioned into 8 equal parts, the range of P is partitioned into [0, P / 8-1], [P / 8, 2P / 8-1], ..., [7P / 8, P-1], and each segment is represented by a vector of size 3. For example, if the range of P is 64, the intra-frame prediction mode 12 is represented by the vector 001.

[0181] In addition, the motion vector used in the video decoding process is expressed as a vector. For example, when the motion vector is represented by a vector having 6 elements, MV = (m0, m1, m2, m3, m4, m5, m6). Here, m0 indicates whether the motion vector is in the x direction or in the y direction. For example, when m0 is 0, it indicates that the motion vector is in the x direction (i.e., a motion vector closer to the x-axis, where the absolute value of x is greater than or equal to the absolute value of y); when m0 is 1, it indicates that the motion vector is in the y direction (i.e., a motion vector closer to the y-axis, where the absolute value of y is greater than the absolute value of x). m1 is used to determine the resolution of the motion vector. For example, when m1 is 0, the motion vector has an integer precision resolution; when m1 is 1, the motion vector has a fractional precision resolution.

[0182] In addition, m2 indicates the reference picture index of the motion vector. m3 indicates the reference picture list of the motion vector. For example, when m3 is 0, it indicates that the reference picture list is the L0 list; when m3 is 1, it indicates that the reference picture list is the L1 list. m4 indicates the sign of the motion vector. In other words, m4 can be set to 0 or 1 based on the sign of the motion vector. m5 indicates the size of the motion vector.

[0183] Among the vector elements of the meta information for the motion vector, m0, m1, m3, and m4 may be represented by 1 bit, and m2 and m5 may be represented by a plurality of bits.

[0184] In another example, the compression parameters parsed during the video decoding process can be used as meta information without modification. In other words, the video decoding device can use the encoding mode, flags, index information and derived parameters used in the decoding process as meta information without modification.

[0185] In case of intra prediction, the decoded directional prediction mode p is used as meta information.

[0186] Whether intra prediction or inter prediction is applied to the current block is indicated by 0 or 1.

[0187] When intra prediction is applied and an intra sub-partition (ISP) mode is used, ISP mode information is used as meta information.

[0188] For example, if the quantization parameter is 24, 24 is used as the meta information.

[0189] If a specific index within the filter parameter set A of the in-loop filter is used for decoding, the corresponding index is used as meta-information.

[0190] If the MTS mode is used for transformation, the MTS index or the flag of the MTS is used as meta information.

[0191] Hereinafter, an example of generating a meta information feature vector from meta information will be described.

[0192] For example, the video decoding apparatus generates the meta information input vector X by concatenating one or more pieces of meta information. For example, in the above example, if the current block size is 32×32, the structure is set to 2N×2N, and the vector 00 is selected. In addition, since one of the width and height of the block is greater than or equal to 16, the third file is selected as 0, and therefore, the final input meta information vector can be represented as 000. For example, in Figure 7 In the example, you can input 0, 0, 0 into x 1 、x 2 and x 3 .

[0193] The video decoding device may input the meta information vector into a neural network to generate a meta information feature vector y. y may be used to select a mode during the encoding and decoding process, such as in Fig. 9 and Fig.10 , or to express a query in a mapping table. For example, Fig. 9 In the example, if y is 0, then from the MIP set S 0 , S 1 and S 2 Select S 0 In addition, Fig.10 In the example of , if y is 00, the kernel in the horizontal and vertical directions can be selected as DCT2.

[0194] In another example, as described above, the video decoding device may utilize the parsed compression parameters as meta information without modification. For example, if the quantization parameter is 27, the length of the current block is 16, and the intra prediction modes are 12, 27, 16, and 12 may be input to the Figure 7 In the example of x1 、x 2 and x 3 .

[0195] In another example, in addition to inputting the meta information, the video decoding device may input the neighboring reference pixels of the current block into the neural network to generate the meta information feature vector. Figure 7 In the example of x 1 、x 2 and x 3 , the W reference pixel located on the upper side of the current block can be input to the W input node among the remaining input nodes. In addition, the H reference pixel located on the left side of the current block can be input to the H input node.

[0196] In the following, we will use Fig.11 and Fig.12 The diagram of FIG. 1 describes a method for encoding / decoding a current block based on meta information.

[0197] Fig.11 A method of encoding a current block performed by a video encoding apparatus according to an embodiment of the present invention is illustrated.

[0198] The video encoding apparatus obtains compression parameters for a current block S1100.

[0199] Here, the compression parameters include those parameters configured taking into account rate-distortion optimization at a higher level of the current block. In addition, the compression parameters may include information determined taking into account rate-distortion optimization at the level of the current block. The information may include all or part of the information related to the block structure of the current block, the size of the current block, partition information, intra-frame prediction, inter-frame prediction, and motion vector.

[0200] The video encoding apparatus generates meta information by combining one or more of the compression parameters S1102. Here, the meta information may include each compression parameter without modification, or may include a vector generated from each compression parameter.

[0201] The video encoding apparatus inputs the meta information into a deep learning-based extraction model to generate a meta information feature vector S1104.

[0202] The video encoding device may generate an input vector by concatenating one or more vectors of each compression parameter, and may input the generated input vector into the extraction model. Here, the extraction model may be a feedforward network as described above. The meta-information feature vector may be composed of the results of a mapping table utilized in an existing VVC, a mode according to a block coding condition, and the like.

[0203] On the other hand, the extraction model can be pre-trained by using the training meta information and the corresponding label. The label can be, for example, idx in MIP. Since the video encoding device passes the weight to the video decoding device, the extraction model can be shared between the video encoding device and the video decoding device.

[0204] The video encoding apparatus performs encoding processing on the current block by using the meta-information feature vector S1106.

[0205] For example, when encoding processing is performed based on MIP, the meta information includes a block structure of the current block, a size of the current block, a reference sample of the current block, a quantization parameter, and parameters related to intra prediction; and the meta information feature vector may be an idx indicating one of the sets. At this time, the set includes matrices and vectors for application of MIP.

[0206] In addition, an intra-frame prediction mode that replaces the MIP mode can be generated. Here, the meta information can be a reference sample of the current block, and the meta information feature vector can be composed of a replacement index repIdx indicating the intra-frame prediction mode. Later, when intra-frame prediction is performed on the block, the intra-frame prediction mode mapped to the replaced index can be used to derive the MPM. For example, if a luminance block located on the left or upper side of the current block is predicted using MIP, the video encoding device can apply the replaced index of the corresponding luminance block to the process of constructing the MPM list.

[0207] Fig.12 A method of decoding a current block performed by a video decoding apparatus according to an embodiment of the present invention is shown.

[0208] The video decoding apparatus decodes compression parameters for a current block from a bitstream S1200.

[0209] Here, the compression parameters include those parameters decoded at a higher level of the current block. In addition, the compression parameters may include information decoded at the level of the current block. At this time, the decoded information may include all or part of the information related to the block structure of the current block, the size of the current block, the reference sample of the current block, the partition information, the intra prediction, the inter prediction, and the motion vector of the current block.

[0210] The video decoding apparatus generates meta information by combining one or more of the compression parameters S1202. Here, the meta information may include each compression parameter without modification, or may include a vector generated from each compression parameter.

[0211] The video decoding apparatus inputs the meta information into a deep learning-based extraction model to generate a meta information feature vector S1204.

[0212] The video decoding device may generate an input vector by concatenating one or more vectors of each compression parameter, and may input the generated input vector into the extraction model. Here, the extraction model may be a feedforward network as described above. The meta-information feature vector may be composed of the results of a mapping table utilized in an existing VVC, a mode according to a block coding condition, and the like.

[0213] The video decoding apparatus performs a decoding process on the current block by using the meta-information feature vector S1206.

[0214] For example, when decoding is performed based on MIP, the meta information includes a block structure of the current block, a size of the current block, a reference sample of the current block, a quantization parameter, and parameters related to intra prediction; and the meta information feature vector may be an idx indicating one of the sets. At this time, the set includes matrices and vectors for application of the MIP.

[0215] In addition, an intra-frame prediction mode that replaces the MIP mode can be generated. Here, the meta information can be a reference sample of the current block, and the meta information feature vector can be composed of a replacement index repIdx indicating the intra-frame prediction mode. Later, when intra-frame prediction is performed on the block, the intra-frame prediction mode mapped to the replaced index can be used to derive the MPM. For example, if a luminance block located on the left or upper side of the current block is predicted using MIP, the video decoding device can apply the replaced index of the corresponding luminance block to the process of constructing the MPM list.

[0216] Although the steps in the various flowcharts are described as being performed sequentially, these steps merely illustrate the technical ideas of some embodiments of the present invention. Therefore, a person of ordinary skill in the art to which the present invention belongs can perform the steps by changing the order described in the various figures or by performing two or more steps in parallel. Therefore, the steps in the various flowcharts are not limited to the order shown in the order in which they occur.

[0217] It should be understood that the above description presents illustrative embodiments that can be implemented in various other ways. The functions described in some embodiments can be implemented by hardware, software, firmware and / or their combination. It should also be understood that the functional components described in the present invention are marked as "... units" to highlight the possibility of their independent implementation.

[0218] On the other hand, the various methods or functions described in some embodiments can be implemented as instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. Non-volatile recording media can include various types of recording devices that store data in a form that is readable by a computer system. For example, non-volatile recording media can include storage media, such as erasable programmable read-only memory (EPROM), flash drives, optical disk drives, magnetic hard drives, and solid-state drives (SSDs), etc.

[0219] Although exemplary embodiments of the present invention are described for illustrative purposes, it will be appreciated by those skilled in the art that various modifications, additions and substitutions may be made without departing from the spirit and scope of the present invention. Therefore, embodiments of the present invention are described for simplicity and clarity. The scope of the technical ideas of the embodiments of the present invention are not limited by the examples. Accordingly, it will be appreciated by those skilled in the art that the scope of the present invention should not be limited by the embodiments clearly described above, but by the claims and their equivalents.

[0220] Reference numerals

[0221] 530: Inverter

[0222] 540: Predictor

[0223] 810: Meta Information Generator

[0224] 820: Meta-information feature vector generator

[0225] 830: Decoding executor.

[0226] CROSS-REFERENCE TO RELATED APPLICATIONS

[0227] This application claims priority to and the benefit of Korean Patent Application No. 10-2022-0075990, filed on June 22, 2022, and Korean Patent Application No. 10-2023-0074954, filed on June 12, 2023, which are hereby incorporated by reference in their entirety.

Claims

1. A method for reconstructing a current block performed by a video decoding device, the method comprising: decoding compression parameters for a current block from a bitstream; generating meta-information by combining one or more of the compression parameters, wherein the meta-information comprises each compression parameter without modification or comprises a meta-information vector generated from each compression parameter; generating a feature vector of the meta-information by inputting the meta-information into a deep learning-based extraction model; and A decoding process is performed on the current block by using the feature vector.

2. The method according to claim 1, wherein: The compression parameters include parameters decoded at a higher level of the current block and information decoded at the level of the current block, wherein the decoded information includes all or part of the information related to the block structure of the current block, the size of the current block, partition information, intra-frame prediction, inter-frame prediction and motion vector.

3. The method according to claim 1, wherein: Generated meta information includes: When the compression parameter is the size of the current block, the meta information vector is generated based on the width and height of the current block or a condition related to the width and height.

4. The method according to claim 1, wherein: Generated meta information includes: When the compression parameter is a quantization parameter and the quantization parameter is a single number belonging to the range of 0 to Q-1, a vector of size floor(log2Q)+1 is generated and one of the vectors is mapped to a meta information vector of the quantization parameter, where Q is a natural number.

5. The method according to claim 1, wherein: Generated meta information includes: When the compression parameter is a quantization parameter, vectors corresponding to sections obtained by partitioning the range of the quantization parameter are generated, and one of the vectors is mapped to a meta information vector of the quantization parameter.

6. The method according to claim 1, wherein: Generated meta information includes: When the compression parameter is the direction of the intra prediction mode and the direction is a single number ranging from 0 to P-1, a vector of size floor(log2P)+1 is generated and one of the vectors is mapped to the meta-information vector of the direction, where P is a natural number.

7. The method according to claim 1, wherein: Generated meta information includes: When the compression parameter is the direction of the intra prediction mode, vectors corresponding to sections obtained by partitioning the range of the intra prediction mode are generated, and one of the vectors is mapped to a meta information vector of the direction.

8. The method according to claim 1, wherein: Generated meta information includes: When the compression parameter is a motion vector, the motion vector is represented by a meta information vector, Among them, the constituent elements of the meta-information vector include the direction of the motion vector, the resolution of the motion vector, the reference picture index of the motion vector, the reference picture list of the motion vector, the sign of the motion vector and the size of the motion vector.

9. The method according to claim 1, wherein: Generated meta information includes: Without modification, the intra prediction mode, whether intra prediction or inter prediction is applied, information of intra subpartitioning (ISP) mode, quantization parameter, multiple transform selection (MTS) index, flag for MTS, or index of parameter set for in-loop filter are used as meta-information.

10. The method according to claim 1, wherein: The feature vectors for generating meta information include: An input vector is generated by combining one or more of the meta information vectors and inputting the input vector to an extraction model to generate a feature vector.

11. The method according to claim 10, wherein: The feature vectors for generating meta information include: In addition to the input vector, a feature vector is generated by inputting neighboring reference pixels of the current block into the extraction model.

12. The method according to claim 1, wherein: When decoding is performed based on matrix-weighted intra-frame prediction, the meta-information includes a block structure of a current block, a size of a current block, a reference sample of a current block, a quantization parameter, and parameters related to intra-frame prediction; and the feature vector is an index indicating one of a set, wherein the set includes matrices and vectors for intra-frame prediction applying matrix weighting.

13. The method according to claim 1, wherein: When a decoding process is performed based on matrix-weighted intra prediction, the meta information includes a reference sample of the current block, and the feature vector is a replacement index indicating one of the intra prediction modes.

14. The method according to claim 13, wherein: The decoding process for the current block includes: When decoding processing is performed based on intra prediction of the current block and a luminance block located on the left or above the current block is predicted according to matrix-weighted intra prediction, a replacement index of the corresponding luminance block is applied to construct a most probable mode (MPM) list of the current block.

15. A method for encoding a current block performed by a video encoding device, the method comprising: Get compression parameters for the current block; generating meta-information by combining one or more of the compression parameters, wherein the meta-information comprises each compression parameter without modification or comprises a meta-information vector generated from each compression parameter; Generating a feature vector of the meta-information by inputting the meta-information into a deep learning-based extraction model; as well as An encoding process is performed on the current block by using the feature vector.

16. The method according to claim 15, wherein: The compression parameters include parameters configured at a higher level of the current block and information determined at the level of the current block, wherein the determined information includes all or part of the information related to the block structure of the current block, the size of the current block, partition information, intra-frame prediction, inter-frame prediction and motion vector.

17. A computer-readable recording medium storing a bit stream generated by a video encoding method, the video encoding method comprising: Get compression parameters for the current block; generating meta-information by combining one or more of the compression parameters, wherein the meta-information comprises each compression parameter without modification or comprises a meta-information vector generated from each compression parameter; generating a feature vector of the meta-information by inputting the meta-information into a deep learning-based extraction model; and An encoding process is performed on the current block by using the feature vector.

Images