Video encoding / decoding method and device
By determining the inverse transformation method based on the number of non-zero coefficients of the inverse quantization block, image encoding and decoding are optimized, the problem of increasing the amount of high-definition image data is solved, and efficient image transmission and storage is achieved.
Patent Information
- Application Number
- CN202380030687.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-16
- Filing Date
- 2023-01-27
- Publication Date
- 2024-11-12
AI Technical Summary
With the increasing demand for high-definition and ultra-high-definition images, the amount of data brought about by the high resolution and high quality of traditional image data has increased, resulting in an increase in transmission and storage costs, making it difficult for the existing technology to effectively compress and transmit high resolution. Stereo image content.
By obtaining the number of non-zero coefficients of the inverse quantization block, the inverse transformation method of the inverse quantization block is determined, and the inverse transformation is performed according to the method to optimize the image encoding and decoding process and reduce the computational complexity and transmission time.
Efficient image encoding and decoding is realized, reducing transmission and storage costs, while saving runtime without reducing encoding performance, especially in full-frame and random access conditions, which greatly reduce the average decoding time.
Smart Images

Figure CN118947113A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and device for image encoding / decoding, and more particularly to a method and device for image encoding / decoding that performs inverse transformation using linearity. Background Art
[0002] Recently, the demand for high-resolution and high-quality images such as HD (High Definition) images and UHD (Ultra High Definition) images has been increasing in various application fields. As the resolution and quality of image data are improved, the amount of data will increase significantly compared to traditional image data. Therefore, when the image data is transmitted using traditional wired or wireless broadband circuits or stored using traditional storage media, the transmission cost and storage cost will also increase. In order to solve the above problems caused by the high resolution and high quality of image data, efficient image compression technology can be used.
[0003] As image compression technology, there are many technologies, such as inter-frame prediction technology that predicts the pixel values contained in the current image based on the previous or next images of the current image, intra-frame prediction technology that predicts the pixel values contained in the current image using the pixel information in the current image, and entropy coding technology that assigns shorter codes to values with higher frequency of occurrence and longer codes to values with lower frequency of occurrence. By using the image compression technology as described above, image data can be effectively compressed and then transmitted or stored.
[0004] In addition, as the demand for high-resolution images increases, the demand for 3D image content as a new image service is also increasing. For this reason, video compression technology for efficiently providing high-resolution and ultra-high-resolution 3D image content is being actively discussed. Summary of the invention
[0005] Technical issues
[0006] An object of the present invention is to provide a method and apparatus for image encoding / decoding that performs inverse transformation using linearity.
[0007] In addition, an object of the present invention is to provide a storage medium for storing a bit stream generated by the image encoding method or device according to the present invention.
[0008] The technical issues to be achieved by the present invention are not limited to the technical issues mentioned in the above content. Those having general knowledge in the technical field to which the present invention belongs will be able to further clearly understand other technical issues not mentioned through the following description.
[0009] Technical Solution
[0010] In the present disclosure, a video decoding method is provided, comprising: a step of obtaining the number of non-zero coefficients of an inverse quantization block; a step of determining an inverse transformation method of the inverse quantization block according to the number of non-zero coefficients; and a step of performing an inverse transformation of the inverse quantization block according to the determined inverse transformation method.
[0011] In one embodiment, the step of determining the inverse transformation method of the inverse quantized block may include: a step of comparing the number of the non-zero coefficients with a specific critical value; and a step of determining the inverse transformation method of the inverse quantized block based on the comparison result.
[0012] In one embodiment, the step of determining the inverse transformation method of the inverse quantized block may include: a step of determining the number of multiplication operations required for the linear inverse transformation from the number of non-zero coefficients; a step of comparing the number of multiplication operations with a specific critical value; and a step of determining the inverse transformation method of the inverse quantized block based on the comparison result.
[0013] In one embodiment, the number of multiplication operations may be determined based on the number of non-zero coefficients and the size of the inverse quantization block.
[0014] In one embodiment, the specific threshold value may be determined based on the size of the inverse quantization block.
[0015] In one embodiment, it is characterized in that: the video decoding method may also include: a step of determining a vertical kernel and a horizontal kernel for the inverse quantization block; the specific critical value may be determined based on the size of the vertical kernel, the horizontal kernel and the inverse quantization block.
[0016] In one embodiment, it is characterized in that the vertical kernel and the horizontal kernel can be determined from at least one of discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII).
[0017] In one embodiment, the vertical kernel and the horizontal kernel may be determined based on the size of the inverse quantized block and the prediction method used for the inverse quantized block.
[0018] In one embodiment, the inverse transform method of the inverse quantized block may be determined based on the image type of the inverse quantized block.
[0019] In one embodiment, it is characterized in that: the step of determining the inverse transformation method of the inverse quantized block may include: when the image type of the inverse quantized block is an All Intra (AI) type or a Random Access (RA) type, according to the number of the non-zero coefficients, the step of determining whether to apply a linear inverse transformation to the inverse quantized block.
[0020] In one embodiment, it is characterized in that: the step of determining the inverse transformation method of the inverse quantized block may include: when the image type of the inverse quantized block is not an intra-frame (AI) type or a random access (RA) type, the step of determining not to apply a linear inverse transformation to the inverse quantized block.
[0021] In one embodiment, it is characterized in that: the inverse transformation method of the inverse quantized block can be determined based on the quantization parameter applicable to the inverse quantization of the inverse quantized block.
[0022] In one embodiment, it is characterized in that it can include: when the quantization parameter is greater than a critical quantization parameter value, according to the number of the non-zero coefficients, determining whether to apply a linear inverse transform to the inverse quantized block.
[0023] In one embodiment, it is characterized in that: when the quantization parameter is less than a critical quantization parameter value, it is determined not to apply an inverse linear transform to the inverse quantized block.
[0024] In one embodiment, it is characterized in that: the video decoding method may also include: a step of obtaining linear inverse transformation permission information for indicating whether linear inverse transformation is allowed from a parameter set; in the step of determining the inverse transformation method of the inverse quantization block, when the linear inverse transformation permission information indicates that the linear inverse transformation is allowed, it can be determined whether the inverse transformation method of the inverse quantization block is a linear inverse transformation method.
[0025] In one embodiment, it is characterized in that: the parameter set can be at least one of a video parameter set, a sequence parameter set, an image parameter set and an adaptation parameter set.
[0026] In one embodiment, the inverse transformation method of the inverse quantized block may be determined based on the color component of the inverse quantized block.
[0027] In one embodiment, it is characterized in that: the step of performing inverse transformation of the inverse quantized block according to the determined inverse transformation method may include: when the inverse transformation method is a linear inverse transformation, the step of dividing the inverse quantized block into a plurality of sub-blocks each containing only one non-zero coefficient and the remaining coefficients being zero coefficients; the step of performing inverse transformation on each of the plurality of sub-blocks; and the step of obtaining an inverse transformation block of the inverse quantized block based on the plurality of element blocks of each inverse transformation.
[0028] In the present disclosure, a video encoding method is provided, comprising: a step of encoding a block and inverse quantizing the encoded block; a step of obtaining the number of non-zero coefficients of the inverse quantized block; a step of determining an inverse transform method of the inverse quantized block according to the number of non-zero coefficients; a step of performing an inverse transform of the inverse quantized block according to the determined inverse transform method; and a step of reconstructing a block using the inverse transform block and encoding other blocks based on the reconstructed block.
[0029] In the present disclosure, a computer-readable storage medium is provided, characterized in that: as a computer-readable storage medium storing a bit stream related to an encoded video, the storage medium includes a bit stream generated by a video encoding method, and the video encoding method includes: a step of encoding a block and inverse quantizing the encoded block; a step of obtaining the number of non-zero coefficients of the inverse quantized block; a step of determining an inverse transform method of the inverse quantized block according to the number of non-zero coefficients; a step of performing an inverse transform of the inverse quantized block according to the determined inverse transform method; and a step of reconstructing a block using the inverse transform block and encoding other blocks based on the reconstructed block.
[0030] Effects of the Invention
[0031] In the present invention, a method and apparatus for image encoding / decoding that performs inverse transformation using linearity can be provided.
[0032] In addition, in the present invention, a method and an apparatus for transmitting or storing a bit stream generated by the image encoding method / apparatus according to the present invention can be provided.
[0033] Furthermore, in the present invention, a computer-readable recording medium for storing a bit stream generated by the image encoding method / apparatus according to the present invention can be provided.
[0034] Furthermore, in the present invention, the image data can be effectively encoded and decoded by the image encoding method / device according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is an example diagram that schematically illustrates the structure of a video encoding device.
[0036] Figure 2 This is an exemplary diagram illustrating an embodiment of a prediction unit of a video encoding device.
[0037] Figure 3 The diagram shows an example of representing a block as a sub-block consisting of one non-zero coefficient.
[0038] Figure 4 An embodiment of an inverse transform method that allows inverse transforms utilizing separate linearities is illustrated.
[0039] Figure 5 The diagram shows a scanning method for rearranging coefficients in an inverse quantized block into a 1-dimensional vector.
[0040] Figure 6 The figure shows an example of rearranging coefficients in a 4×4 block into a one-dimensional vector by horizontal scanning.
[0041] Figure 7 The diagram shows an example of separating a 1-dimensional vector into sub-vectors.
[0042] Figure 8 The figure shows an example of performing inverse transformation on each sub-vector.
[0043] Fig. 9 An embodiment of a video decoding method that can utilize an inverse linear transform method is provided.
[0044] Fig.10 An embodiment of a video encoding method that can utilize an inverse linear transform method is provided.
[0045] Fig.11 The figure shows the magnitude response of a 4-tap discrete cosine transform (4-tap DCT) interpolation filter and an 8-tap discrete sine transform (8-tap DST) interpolation filter at 16 / 32-pixel positions.
[0046] Fig.12 An embodiment of coefficients of an 8-tap discrete sine transform (DST) interpolation filter is illustrated.
[0047] Fig.13The diagram shows the magnitude responses of a 4-tap DCT interpolation filter, a 4-tap Gaussian interpolation filter, an 8-tap DCT interpolation filter, and an 8-tap Gaussian interpolation filter at 16 / 32-pixel positions.
[0048] Fig.14 An embodiment of a method for selecting an interpolation filter using frequency information is illustrated.
[0049] Fig.15 as well as Fig.16 The coefficients of an 8-tap discrete cosine transform interpolation filter and an embodiment of an 8-tap smoothing interpolation filter are respectively shown.
[0050] Fig.17 An embodiment of an interpolation filter selected according to a boundary correlation threshold is illustrated. DETAILED DESCRIPTION
[0051] In the present disclosure, a video decoding method is provided, comprising: a step of obtaining the number of non-zero coefficients of an inverse quantization block; a step of determining an inverse transformation method of the inverse quantization block according to the number of non-zero coefficients; and a step of performing an inverse transformation of the inverse quantization block according to the determined inverse transformation method.
[0052] Next, the embodiments according to the present invention will be described in detail with reference to the drawings of this specification so that a person having general knowledge of the technical field to which the present invention belongs can easily implement the present invention. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. In addition, in order to clearly describe the present invention, parts that are not related to the description are omitted in the drawings, and similar parts are assigned similar figure numbers throughout the specification.
[0053] Throughout the specification, when it is stated that a certain part is “connected” to another part, it includes not only a case of being directly connected to another part, but also a case of being electrically connected with another element interposed therebetween.
[0054] Furthermore, throughout the specification, when a certain part is described as “including” a certain constituent element, unless otherwise clearly described to the contrary, it does not mean that other constituent elements are excluded but rather means that other constituent elements may also be included.
[0055] In addition, when describing different components, terms such as "1" and "2" may be used, but the components are not limited by the terms. The terms are only used to distinguish one component from other components.
[0056] In addition, in the embodiments related to the apparatus and method described in this specification, a part of the configuration in the apparatus or a part of the steps in the method may be omitted. In addition, the order of a part of the configuration in the apparatus or a part of the steps in the method may be changed. In addition, other configurations or other steps may be inserted into a part of the configuration in the apparatus or a part of the steps in the method.
[0057] Furthermore, a part of the configuration or a part of the steps in the first embodiment according to the present invention may be added to the second embodiment according to the present invention, or may replace a part of the configuration or a part of the steps in the second embodiment.
[0058] Moreover, the components included in the embodiments of the present invention are illustrated separately only to show different specific functions, and do not mean that each component is composed of hardware or a software component unit separated from each other. That is, although each component is listed and recorded for the convenience of explanation, at least two components in each component can be combined into one component, or one component can be divided into multiple components and make them perform corresponding functions. The combined embodiments and divided embodiments of each component as described above are also included in the scope of the claims of the present invention without departing from the essence of the present invention.
[0059] First, the terms used in this application will be briefly explained as follows.
[0060] The decoding device (Video Decoding Apparatus) described in the following content can be included in devices such as civilian security cameras, civilian security systems, military security cameras, military security systems, personal computers (PC, Personal Computer), notebook computers, portable multimedia players (PMP, Portable Multimedia Player), wireless communication terminals (Wireless Communication Terminal), smart phones (Smart Phones), television (TV) application services and service servers and other service terminals. It can be equipped with various devices such as communication modems for communicating with various devices such as user terminals, wired and wireless communication networks, etc., a memory for storing various programs and data for performing image decoding or performing inter-screen or intra-screen prediction for decoding, and a microprocessor for performing calculations and control by executing programs.
[0061] In addition, the image encoded into a bitstream by the encoder can be transmitted to the image decoding device in real time or non-real time through a wired wireless communication network such as the Internet, a short-range wireless communication network, a wireless local area network, a wireless broadband access service (WiBro) network, a mobile communication network, or through various communication interfaces such as cables, a universal serial bus (USB, Universal Serial Bus), and reconstructed into an image for playback through decoding. Alternatively, the bitstream generated by the encoder can be stored in a memory. The above-mentioned memory may include a volatile memory and a non-volatile memory. In this specification, the memory may be represented as a storage medium storing a bitstream.
[0062] Generally, a video can be composed of a series of images (Picture), and each image can be divided into coding units (coding unit) such as blocks (Block). In addition, a person having general knowledge of the technical field to which the present embodiment belongs should understand that the term image recorded in the following content can be replaced by other terms with the same meaning such as image (Image), frame (Frame). In addition, a person having general knowledge of the technical field to which the present embodiment belongs should understand that the term coding unit recorded in the following content can also be replaced by other terms with the same meaning such as unit block, block, etc.
[0063] Next, the embodiments according to the present invention will be described in more detail with reference to the accompanying drawings. In the process of describing the present invention, repeated descriptions of the same components will be omitted.
[0064] Figure 1This is an example diagram that schematically illustrates the structure of a video encoding device.
[0065] The image encoding device 100 may include an image segmentation unit 101, an intra-frame prediction unit 102, an inter-frame prediction unit 103, a subtraction unit 104, a transformation unit 105, a quantization unit 106, an entropy encoding unit 107, an inverse quantization unit 108, an inverse transformation unit 109, an addition unit 110, a filtering unit 111 and a memory 112.
[0066] In each device, the rate distortion cost (RD-Cost) can be compared to select the most appropriate information. The rate distortion cost (RD-Cost) refers to the cost value calculated by using the distortion information between the original block and the reconstructed block and the amount of bits generated when the prediction mode is transmitted. In order to calculate the cost value, the sum of absolute difference (SAD), the sum of absolute transformed difference (SATD), the sum of square error (SSE), etc. can be used.
[0067] exist Figure 1 In order to represent different special functions in the image encoding device, each component is illustrated separately, but this does not mean that each component is composed of separate hardware or a software unit. That is, although each component is listed for the convenience of explanation, at least two components of each component can be combined into one component, and one component can be divided into multiple components to perform corresponding functions. The embodiments in which the components are integrated and the embodiments in which they are separated as described above are included in the scope of the claims of the present invention without departing from the essence of the present invention.
[0068] In addition, some of the components are not necessary components for performing essential functions in the present invention, but are only optional components for improving their performance. The present invention may include only necessary components for realizing the essence of the present invention other than the components only for improving its performance, and the structure including only necessary components other than the optional components only for improving its performance is also included in the scope of the claims of the present invention.
[0069] The image segmentation unit 100 can divide the input image into at least one block. At this time, the input image can have multiple forms and sizes such as images, strips, parallel blocks, fragments, etc. The block can refer to a coding unit (CU), a prediction unit (PU) or a transform unit (TU). The above segmentation can be performed based on at least one of a quad tree, a binary tree and a ternary tree. The quad tree is a method of dividing a superordinate block into four subordinate blocks whose width and height are both half of the superordinate block. The binary tree is a method of dividing a superordinate block into two subordinate blocks whose width or height is half of the superordinate block. The ternary tree is a method of dividing a superordinate block into three subordinate blocks based on either the width or the height. By performing segmentation based on the binary tree and the ternary tree as described above, the block can have not only a square shape, but also a non-square shape.
[0070] The prediction units 102 and 103 may include an inter-picture prediction unit 103 for performing inter-picture prediction and an intra-picture prediction unit 102 for performing intra-picture prediction. After deciding whether to use inter-picture prediction or intra-picture prediction for a prediction unit, specific information based on each prediction method (e.g., intra-picture prediction mode, motion vector, reference image, etc.) may be determined. At this time, the processing unit for performing prediction may be different from the processing unit for determining the prediction method and specific content. For example, the prediction method and prediction mode may be determined according to the prediction unit, and the prediction may be performed according to the transform unit.
[0071] The residual value (residual block) between the generated prediction block and the original block can be input to the transform unit 105. In addition, information such as prediction mode information and motion vector information used when performing prediction can be transmitted to the decoder after being encoded by the entropy encoding unit 107 together with the residual value. In the case of using a specific encoding mode, the prediction block can be transmitted to the decoder after directly encoding the original block instead of generating the prediction block through the prediction units 102 and 103.
[0072] The intra-picture prediction unit 102 can generate a prediction block based on pixel information in the current image, i.e., reference pixel information around the current block. When the prediction mode of the surrounding blocks of the current block that needs to perform intra-picture prediction is inter-picture prediction, the reference pixels contained in the surrounding blocks applicable to inter-picture prediction can be replaced with reference pixels in other surrounding blocks applicable to intra-picture prediction. That is, when the reference pixels are unavailable, the unavailable reference pixel information can be replaced with at least one reference pixel among the available reference pixels for use.
[0073] In intra-picture prediction, the prediction mode may include a directional prediction mode using reference pixel information according to a prediction direction and a non-directional mode not using directional information. The mode used to predict brightness information and the mode used to predict color difference information may be different from each other. When predicting color difference information, the intra-picture prediction mode information used in the process of predicting brightness information or the predicted brightness signal information may be used.
[0074] The intra prediction unit 102 may include an adaptive intra smoothing (AIS) filter, a reference pixel interpolation unit, and a mean value (DC) filter. The adaptive intra smoothing (AIS) filter is a filter for filtering the reference pixels of the current block, and the applicability of the filter may be adaptively determined according to the prediction mode of the current prediction unit. When the prediction mode of the current block is a mode in which the adaptive intra smoothing (AIS) filter is not performed, the adaptive intra smoothing (AIS) filter may not be applied.
[0075] When the intra-frame prediction mode of the prediction unit is a mode for performing intra-frame prediction based on pixel values interpolated from reference pixels, the reference pixel interpolation unit of the intra-frame prediction unit 102 may generate reference pixels at fractional unit positions by interpolating reference pixels. When the prediction mode of the current prediction unit is a prediction mode for generating a prediction block without interpolating reference pixels, the reference pixels may not be interpolated. When the prediction mode of the current block is a mean (DC) mode, a mean (DC) filter may generate a prediction block by filtering.
[0076] The inter-picture prediction unit 103 generates a prediction block using the reconstructed reference image and motion information stored in the memory 112. The motion information may include, for example, a motion vector, a reference picture index, a list 1 prediction flag, and a list 0 prediction flag.
[0077] Furthermore, a residual block including the prediction unit generated in the prediction units 102 and 103 and the difference value between the prediction unit and the original block, i.e., residual value information, may be generated. The generated residual block may be input to the transformation unit 105 for transformation.
[0078] The inter-picture prediction unit 103 may derive a prediction block based on information of at least one of the previous picture or the next picture of the current picture. In addition, the prediction block of the current block may also be derived based on information of a part of the domain that has been encoded in the current picture. According to one embodiment of the present invention, the inter-picture prediction unit 103 may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.
[0079] In the reference image interpolation unit, the reference image information can be received from the memory 112 and pixel information of an integer pixel or less can be generated in the reference image. For the luminance pixel, an 8-tap interpolation filter based on discrete cosine transform (DCT-basedInterpolation Filter) can be used in which the filter coefficient is changed in order to generate pixel information of an integer pixel or less in 1 / 4 pixel units. For the chrominance pixel, a 4-tap interpolation filter based on discrete cosine transform (DCT-basedInterpolation Filter) can be used in which the filter coefficient is changed in order to generate pixel information of an integer pixel or less in 1 / 8 pixel units.
[0080] The motion prediction unit can perform motion prediction based on the reference image interpolated by the reference image interpolation unit. As a method for calculating the motion vector, various methods such as the full search-based block matching algorithm (FBMA), the three-step search algorithm (TSS), the new three-step search algorithm (NTS), etc. can be used. The motion vector can be based on the interpolated pixel and have a motion vector value of 1 / 2 or 1 / 4 pixel units. In the motion prediction unit, the prediction block of the current block can be predicted by different motion prediction methods. As a motion prediction method, various methods such as the skip method, the merge method, the advanced motion vector prediction (AMVP) method, etc. can be used.
[0081] The hair cutting operation unit 104 generates a residual block of the current block by performing an addition operation on the block currently to be encoded and the prediction block generated by the intra-frame prediction unit 102 or the inter-frame prediction unit 103 .
[0082] The transform unit 105 may transform the residual block containing the residual data using a transform method such as discrete cosine transform (DCT), discrete sine transform (DST), Karhunen Loeve transform (KLT), etc. At this time, the transform method may be determined based on the intra-frame prediction mode of the prediction unit used to generate the residual block. For example, discrete cosine transform (DCT) may be used in the horizontal direction and discrete sine transform (DST) may be used in the vertical direction according to the intra-frame prediction mode. Alternatively, different transform techniques may be used in the horizontal and vertical directions according to the aspect ratio, size, etc. of the current block.
[0083] The quantization unit 106 can quantize the value transformed into the frequency domain by the transformation unit 105. The quantization coefficient can be changed according to the block or the importance of the image. The value calculated by the quantization unit 106 can be provided to the inverse quantization unit 108 and the entropy coding unit 107.
[0084] The transform unit 105 and / or the quantization unit 106 may be selectively included in the image encoding device 100. That is, the image encoding device 100 may perform at least one of transform or quantization on the residual data of the residual block, or may skip transform and quantization and encode the residual block. Even if one of the transform or quantization is not performed in the image encoding device 100, or both the transform and quantization are not performed, the block input to the entropy encoding unit 107 is generally referred to as a transform block.
[0085] The entropy coding unit 107 performs entropy coding on the input data. Entropy coding can use a variety of coding methods such as Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC).
[0086] The entropy coding unit 107 can encode various information such as coefficient information of the transform block, block type information, prediction mode information, partition unit information, prediction unit information, transmission unit information, motion vector information, reference frame information, block interpolation information, filtering information, etc. The coefficients of the transform block can be encoded in sub-block units within the transform block.
[0087] In order to encode the coefficients of the transformed block, various syntax elements may be encoded, such as Last_sig, a syntax element for indicating the position of the first non-zero coefficient in the reverse scanning order, Coded_sub_blk_flag, a flag for indicating whether the sub-block contains at least one non-zero coefficient, Sig_Coeff_flag, a flag for indicating whether it is a non-zero coefficient, Abs_greater1_flag, a flag for indicating whether the absolute value of the coefficient is greater than 1, Abs_greater2_flag, a flag for indicating whether the absolute value of the coefficient is greater than 2, and Sign_flag, a flag for indicating the sign of the coefficient. The remaining value of the coefficient that is not encoded by the above syntax elements may be encoded by the syntax element remaining_coeff.
[0088] In the inverse quantization unit 108 and the inverse transformation unit 109, the value quantized in the quantization unit 106 is inversely quantized, and the value transformed in the transformation unit 105 is inversely transformed. The residual value generated in the inverse quantization unit 108 and the inverse transformation unit 109 can be merged with the prediction unit predicted by the motion estimation unit, the motion compensation unit and the intra-frame prediction unit 102 included in the prediction unit 102 and 103 to generate a reconstructed block. The adder 110 generates a reconstructed block by adding the prediction block generated in the prediction unit 102 and 103 and the residual block generated by the inverse transformation unit 109.
[0089] The filter unit 111 may include at least one of a deblocking filter, an offset correction unit, and an adaptive loop filter (ALF, AdaptiveLoop Filter).
[0090] The deblocking filter can remove block distortion caused by the boundaries between blocks from the reconstructed difference. In order to determine whether deblocking needs to be performed, it can be determined whether the deblocking filter needs to be applied to the current block based on the pixels contained in several columns or rows contained in the block. When the deblocking filter is applied to the block, a strong filter (Strong Filter) or a weak filter (Weak Filter) can be applied according to the required deblocking filter strength. In addition, when the deblocking filter is applied, the horizontal filtering and the vertical filtering can be processed in parallel when performing vertical filtering and horizontal filtering.
[0091] The offset correction unit can correct the offset between the image after deblocking and the original image in pixel units. In order to correct the offset of a specific image, a method of dividing the pixels contained in the image into a certain number of domains, determining the domains that need to be offset and applying the offset to the corresponding domains can be used, or a method of applying the offset is used under the premise of considering the edge information of each pixel.
[0092] Adaptive loop filter (ALF) can be performed based on the value of comparing the filtered reconstructed image and the original image. After dividing the pixels contained in the image into specified groups, it can be determined that a filter in the corresponding group needs to be applied, so that different filtering is performed in each group. For information related to whether the adaptive loop filter (ALF) is applicable, the luminance signal can be transmitted according to each coding unit (CodingUnit, CU), and the shape and filter coefficient of the adaptive loop filter (ALF) applied can be changed according to each block. In addition, regardless of the characteristics of the applicable object block, the same form (fixed form) of the adaptive loop filter (ALF) can be applied.
[0093] The memory 112 may store the reconstructed block or image calculated by the filter unit 111 , and the stored reconstructed block or image may be provided to the prediction units 102 and 103 when performing inter-picture prediction.
[0094] Figure 2 This is an exemplary diagram illustrating an embodiment of a prediction unit of a video encoding device.
[0095] In the case where the prediction mode of the current block is the intra-frame prediction mode, the intra-frame prediction unit 201 can generate reference pixels by deriving reference pixels from the periphery of the current block and filtering the reference pixels. The reference pixels are determined using the reconstructed pixels around the current block. In the case where a portion of the reconstructed pixels cannot be used or there are no reconstructed pixels around the current block, the available reference pixels can be filled into the unusable domain, or filled with the intermediate values in the range of values that the pixels can have. After all the reference pixels are derived, the reference pixels can be filtered using an adaptive intra-frame smoothing (AIS) filter.
[0096] The intra-picture prediction mode search unit 202 may determine one of M intra-picture prediction modes, where M represents the total number of intra-picture prediction modes. The intra-picture prediction modes include directional prediction modes and non-directional prediction modes.
[0097] The prediction block may be generated using the determined prediction mode and the filtered reference pixels. The intra-picture prediction mode with the lowest cost may be selected by comparing the rate-distortion costs (RD-Cost) of different intra-picture prediction modes.
[0098] The inter-picture prediction unit 203 can be divided into a merge candidate exploration unit 204 and an advanced motion vector prediction (AMVP) candidate exploration unit 206 according to the method of deriving motion information. The merge candidate exploration unit 204 sets the reference block using inter-picture prediction in the reconstructed block around the current block as a merge candidate. The merge candidates are derived using the same method in the encoding / decoding device and the same number is used. The number of merge candidates can be transmitted from the encoding device to the decoding device, or a pre-agreed number is used. In the case where the agreed number of merge candidates cannot be derived from the reconstructed reference blocks around the current block, the motion information of the block at the same position as the current block in the image different from the current image can be used as the merge candidate. Alternatively, insufficient merge candidates can be derived by combining the motion information in the historical direction and the motion information in the future direction based on the current image. Alternatively, the block at the same position in other reference images can be set as the merge candidate.
[0099] The advanced motion vector prediction (AMVP) candidate search unit 206 may determine the motion information of the current block through the motion estimation unit 207. The motion estimation unit 207 searches for a prediction block similar to the current block from the reconstructed image.
[0100] When performing inter-picture prediction, the motion information of the current block may be determined by using one of the merge candidate search unit 204 and the advanced motion vector prediction (AMVP) candidate search unit 206, and the motion compensation unit 208 generates a prediction block based on the determined motion information.
[0101] In hybrid block-based video coding, transform plays an important role in energy compression. Transform coding refers to concentrating energy in low frequency bands by converting residual data in the spatial domain into frequency domain data. Considering that discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII) are linear transforms, inverse transforms that use the linearity of transforms to reduce the number of calculations can be used in video encoding and decoding. By applying the proposed inverse transform that uses linearity to video encoding and decoding, running time can be saved without reducing encoding performance. In particular, the average decoding time can be greatly reduced under All Intra (AI) and Random Access (RA) conditions.
[0102] In hybrid block-based video coding, the residual signal in the spatial domain obtained after intra / inter prediction is transformed into a residual signal in the frequency domain. Through an effective transformation method, more energy can be concentrated in the low-frequency components of the residual signal in the frequency domain. The Karhunen-Loeve transform (KLT) is a very efficient transformation method in terms of data inverse correlation and compression. However, because the complexity of the Karhunen-Loeve transform (KLT) to calculate the eigen vector equivalent to the signal-dependent covariance matrix is high and there is usually no faster calculation algorithm, it is not used in actual transform coding.
[0103] Because Discrete Cosine Transform-II (DCT-II) provides an excellent approximation of Karunan-Loewy Transform (KLT) under the 1st level Markov condition, the video coding standard uses Discrete Cosine Transform-II (DCT-II) instead of Karunan-Loewy Transform (KLT). However, due to the various characteristics of images and videos, Discrete Cosine Transform-II (DCT-II) is not always the best transform in terms of energy compression and inverse correlation. In order to solve the above problems, alternative transform methods such as Discrete Cosine Transform-II / Discrete Sine Transform-III (DCT-II / DST-VIII) and Enhanced Multiple Transform (EMT) for video coding can be used.
[0104] In addition, in some cases, a 1-level Gauss-Markov model associated with the image signal can be used to make the discrete sine transform-VII (DST-II) better approximate the Karonen-Loewe transform (KLT). Therefore, the video codec can be set to use a discrete cosine transform-II (DCT-II) based transform for 4×4, 8×8, 16×16 and 32×32 prediction residual blocks, and an alternative transform based on discrete sine transform-VII (DST-VII) for 4×4 intra-frame prediction residual blocks.
[0105] Recently, with the increasing demand for high-quality and high-resolution images and the development of services such as video streaming, more efficient image compression technology is needed. In order to improve the efficiency of transformation, joint separable transform and non-separable transform can be used.
[0106] The enhanced multiple transform (EMT) using the separable property of the transform can select one of the horizontal and vertical transforms with the best coding efficiency from the pre-defined horizontal and vertical transforms or discrete cosine transform-II (DCT-II), discrete cosine transform-V (DCT-V), discrete cosine transform-VIII (DCT-VIII), discrete sine transform-I (DST-I) and discrete sine transform-II (DST-II). In addition, a non-separable secondary transform can be applied as a secondary transform after the enhanced multiple transform (EMT).
[0107] Transforms can be roughly divided into two processes: level 1 transform and level 2 transform. The simplified enhanced multiple transform (EMT) applied to the prediction residual signal can also be called multiple transform selection (MTS). In addition to discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII) can also be used as transforms of multiple transform selection (MTS). However, discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII) can only be applied to luminance blocks.
[0108] The maximum transform size of applicable transforms can be set to 64 × 64. Discrete cosine transform-II (DCT-II) can be applied to transform blocks of size 4 × 4 to 64 × 64, while discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII) can be applied to transform blocks of size 4 × 4 to 32 × 32.
[0109] The transformation of large blocks can be used for high-resolution video, but it will lead to increased computational complexity. In order to solve the problems mentioned above, in the transformation of large blocks, the high-frequency transformation coefficients can be processed as 0 (zero out). For example, in the case of a 64-point discrete cosine transform-II (DCT-II) transform, only the initial 32 low-frequency coefficients can be maintained and the remaining high-frequency coefficients can be processed as 0 (zero out). In addition, in the case of a 32-point discrete cosine transform-VII / VII (DST-II), 16 low-frequency coefficients can be maintained and the remaining high-frequency coefficients can be processed as 0 (zero out). The zeroing (zeroing out) described above can also be used in last coefficient position coding (last coefficient position coding) and coefficient group scanning (coefficient group scanning).
[0110] Level 2 transform refers to an additional transform process after level 1 transform. In one embodiment, a low frequency non-separable transform (LFNST) can be used in a video codec. The low frequency non-separable transform (LFNST) can be applied to a region of interest (ROI) of a basic transform coefficient. The region of interest (ROI) can be the upper left low frequency domain. When the low frequency non-separable transform (LFNST) is applied, all basic transform coefficients except the region of interest (ROI) will become 0, and the output of the low frequency non-separable transform (LFNST) will be additionally quantized and entropy encoded.
[0111] The definitions of 1-dimensional (1-D) N-point transformation and its inverse transformation are shown in the following Mathematical Formulas 1 and 2.
[0112]
Mathematical formula 1
[0113]
[0114]
Mathematical formula 2
[0115]
[0116] Among them, F(u) is the N-point transformed signal, and p(x) is the original signal. In addition, v u,x is the basic element of the vu basic vector of size N×1 associated with each u in discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII). Where u,x=(0,1,2,…,N-1). v associated with discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII) u , the definitions of x are shown in Mathematical Formula 3, Mathematical Formula 4 and Mathematical Formula 5 respectively.
[0117]
Mathematical formula 3
[0118]
[0119]
Mathematical formula 4
[0120]
[0121]
Mathematical formula 5
[0122]
[0123] In the present disclosure, an inverse transform using separable linear properties is proposed to reduce computational complexity. The proposed inverse transform method can be applied to the basic transform and basic inverse transform of the encoder and decoder. In the inverse transform step of the encoder and decoder, the inverse quantized transform coefficients after the low-frequency non-separable transform (LFNST) will be input to the 2-dimensional (2-D) inverse transform. In most video codecs, the 2-D transform and inverse transform can be implemented as a separable transform by applying the 1-D inverse transform of Mathematical Formula 2 to each row and column in order to reduce computational complexity. The separable inverse transform for non-square block sizes is represented by Mathematical Formula 6.
[0124]
Mathematical formula 6
[0125] X′=B T YA
[0126] Among them, X' is the (n×m) inverse transform block, Y is the (n×m) inverse quantization transform block, A is the (m×m) transform block, and B T is an (n×n) transform. Where n and m are the height and width of the block, respectively. Through the quantization and inverse quantization process, when the quantization coefficient is large, most of the transform coefficients will become 0. In the case where Y is composed of N non-zero coefficients, Y can be expressed as the sum of N sub-blocks of the same size as Y with only one non-zero coefficient as shown in Mathematical Formula 7. Where yi represents the i-th sub-block of Y.
[0127]
Mathematical formula 7
[0128] Y=y0+y1+…y N-1
[0129] Figure 3 An example of expressing a 4×4 block consisting of 3 non-zero coefficients as a plurality of sub-blocks using Mathematical Formula 7 is provided. Figure 3 In FIG. 3 , the 4×4 block Y 300 includes 3 non-zero coefficients. Therefore, the 4×4 block Y 300 may be divided into 3 sub-blocks y0 302, y1 304, and y2 306 each including only one non-zero coefficient. Furthermore, the sub-blocks y0 302, y1 304, and y2 306 may be inversely transformed respectively, and the results generated by the inverse transformation may be accumulated using the linearity of the inverse transformation, thereby performing an inverse transformation of the block.
[0130] Discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII), and discrete cosine transform-VIII (DCT-VIII) have linear characteristics as shown in Math. 8.
[0131]
Mathematical formula 8
[0132] T(αx+βy)=αT(x)+βT(y)
[0133] Wherein, T(*) refers to the transformation, x and y are the inputs of the transformation, and α and β are arbitrary scalar values. Based on equations 7 and 8, the inverse transformation can be expressed as equation 9.
[0134]
Mathematical formula 9
[0135]
[0136] Assuming that the non-zero transform coefficient is located at the (i, j)th element of Y, B T ylA,0≤l≤N-1 can be obtained by using the transformation B T and the basis vectors of A are expressed as Formula 10. In Formula 10, yl is a matrix whose non-zero coefficients are located at the (i, j)th element and whose transformation coefficients of the remaining elements are zero.
[0137]
Mathematical formula 10
[0138]
[0139] X i,j The non-zero transform coefficient of the (i, j)th element of the inverse quantized transform block of size (n*m). i For transformation B T The i-th basis vector of iis the i-th basis vector of transformation A. By calculating B from equation 10 T y l , we can get mathematical formula 11.
[0140]
Mathematical formula 11
[0141]
[0142]
Mathematical formula 12
[0143]
[0144] When applying the proposed (n×m) inverse transform to 1 non-zero coefficient, the number of multiplication operations is n+(n×m). Therefore, the total number of multiplications for the inverse transform using linearity for a (n×m) transform block with N non-zero coefficients is calculated by Equation 13.
[0145]
Mathematical formula 13
[0146] N×(n+(n×m))
[0147] Therefore, only when the number of non-zero coefficients is small, in order to reduce the computational complexity in the inverse transform, the total number of multiplication operations in Mathematical Formula 13 can be applied to the high-speed inverse transform of the inverse quantized block.
[0148] Whether to perform the inverse transform using the existing method with separable characteristics or the proposed method with separable linear characteristics is determined based on a critical value obtained by comparing the number of multiplication operations of discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII), and discrete cosine transform-VIII (DCT-VIII) with mathematical formula 13. The critical value is calculated in advance based on the maximum number of non-zero coefficients in the inverse quantization transform block. Among them, in each block size, N×(n+(n×m)) does not exceed the number of multiplication operations of the inverse transform.
[0149] The proposed method is as follows Figure 4 The inverse transform is performed in the manner shown. First, before the inverse transform process, the number of non-zero coefficients of the inverse quantized transform block Y is calculated. Second, when the number of non-zero coefficients does not exceed a critical value, the proposed inverse transform using separation linearity is performed. Otherwise, other types of odd-even decomposition will perform inverse transforms as described below.
[0150] The transformation can be implemented using a fast method or direct matrix multiplication. When using direct array multiplication, the number of multiplications for a 1DN point transformation is N^2. In addition to discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII), fast algorithms can be used to implement them. Discrete cosine transform-II (DCT-II) uses a fast algorithm that exploits the symmetric and antisymmetric properties of discrete cosine transform-II (DCT-II). The even basis vectors of discrete cosine transform-II (DCT-II) are symmetric, while the odd basis vectors are antisymmetric. Regarding the even and odd parts of the N-point input, the partial set matrix obtained from the even columns and odd columns of each inverse transform matrix is used for calculation, and then addition and subtraction operations are performed between the even and odd parts to generate N-point outputs. The fast method is also called a partial butterfly structure.
[0151] Discrete Sine Transform-VII (DST-VII) and Discrete Cosine Transform-VIII (DCT-VIII) can be used as basic transform solutions. Fast methods related to Discrete Sine Transform-VII (DST-VII) and Discrete Cosine Transform-VIII (DCT-VIII) use functions inherited from Discrete Sine Transform-VII (DST-VII) and Discrete Cosine Transform-VIII (DCT-VIII) in order to reduce the number of operations. There are three useful functions for reducing the number of calculations in the Discrete Sine Transform-VII (DST-VII) and Discrete Cosine Transform-VIII (DCT-VIII) transform arrays. First, contain N elements regardless of sign changes. Second, contain only N elements regardless of sign changes. Third, a portion of the transform vector other than 0 contains only a single element when ignoring sign changes.
[0152]
Table 1
[0153]
[0154]
[0155]
Table 2
[0156]
[0157] Table 1 shows the number of multiplication operations required in each (n*m) block size when both the horizontal and vertical kernels are discrete cosine transform-II (DCT-II). Table 2 shows the number of multiplication operations required in each n*m block size when the horizontal and vertical transforms are a combination of discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VII), namely (DST-VII, DST-VII), (DST-VII, DCT-VIII), (DCT-VIII, DST-VII) and (DCT-VIII, DCT-VIII).
[0158]
Table 3
[0159]
[0160]
Table 4
[0161]
[0162]
[0163] Table 3 shows the critical values related to the number of non-zero coefficients in various block sizes (n*m) when the horizontal and vertical kernels are discrete cosine transform-II / discrete cosine transform-II (DCT-II / DCT-II) or other combinations of kernels.
[0164] Table 4 shows the critical values related to the number of non-zero coefficients in each block size (n*m) when the horizontal and vertical kernels are combinations of discrete sine transform-VII (DST-VII / ) and discrete cosine transform-VIII (DCT-VII) (DST-VII / DST-VII, DST-VII / DCT-VIII, DCT-VIII / DST-VII, DCT-VIII / DCT-VIII).
[0165] The critical value of each n*m block determined by comparing the number of multiplication operations in Tables 1 and 2 with the number of multiplication operations calculated a priori in Mathematical Formula 8 is determined by the combination of horizontal and vertical transforms in Tables 3 and 4. Table 3 shows the critical value related to the number of non-zero coefficients in each n*m block when the horizontal and vertical kernels are discrete cosine transform-II / discrete cosine transform-II (DCT-II / DCT-II) or other combinations. Table 4 shows the critical value representing the number of non-zero coefficients in each n*m block when the horizontal and vertical kernels are a combination of discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII). For example, when the horizontal and vertical transforms are a combination of discrete sine transform-VII / discrete cosine transform-VIII (DST-VII / DCT-VIII) and the number of non-zero coefficients indicated in bold in the 8*8 block of Table 4 is less than 14, the inverse transform can be performed by the inverse transform proposed in the present disclosure.
[0166] The average selection ratio of the proposal method related to the Y component can gradually increase with the increase of the QP value. In addition, the average selection ratio of the proposal method related to the Y component can gradually increase with the increase of the QP value. The above result occurs because the higher the QP value, the smaller the number of non-zero coefficients in the quantization process.
[0167] The proposed inverse transform with linearity can be implemented in both the encoder and the decoder. According to the separable transform of one embodiment, since 16-bit precision is used after the vertical and horizontal transforms, inconsistency between the encoder and the decoder may occur when the proposed linear transform is only applied to the decoder. Since the complexity of the decoder is very simple compared to the encoder, the average decoding time can be shortened more effectively than the shortening of the encoding time of the random access configuration.
[0168] Finally, by applying the proposed inverse transform utilizing linearity with separable properties to the encoder as well as the decoder, the running time can be saved while maintaining the encoding performance compared to when it is applied to the decoder.
[0169] For video coding, as described above, a low frequency non-separable transform (LFNST) can be performed as a secondary transform. The secondary transform refers to a transform performed in addition to the primary transform. In order to perform the secondary transform, the primary transformed coefficients represented by a 2-dimensional matrix can be rearranged into a 1-dimensional vector. In addition, the secondary transform can be performed by direct matrix multiplication based on the rearranged 1-dimensional vector and a non-separable kernel.
[0170] In one embodiment, by utilizing the linearity of the transformation to separate the 1-dimensional vector into sub-vectors having only one non-zero coefficient, and performing the transformation on each of the sub-vectors using direct matrix multiplication, the amount of computation can be reduced compared to when direct matrix multiplication is applied to the 1-dimensional vector. The following mathematical formula 14 represents a forward two-level transformation formula, and mathematical formula 15 represents an inverse two-level transformation formula.
[0171]
Mathematical formula 14
[0172]
[0173]
Mathematical formula 15
[0174]
[0175] When direct matrix multiplication is applied to a 1-dimensional vector, a total of N×R multiplications and (N-1)×R additions are required in the forward transform. In addition, a total of R×N multiplications and (R-1)×N additions are required in the inverse transform. In the forward transform, N represents the size of the input vector, and R represents the size of the output vector. In addition, in the inverse transform, R represents the size of the input vector, and N represents the size of the output vector.
[0176] In the case of utilizing the linearity of transformation, when there are n (forward: 0≤n≤N, reverse: 0≤n≤R) non-zero coefficients in the 1-dimensional vector as the input of the transformation, a total of R×n multiplications and R×(n-1) additions are required in the forward transformation. In addition, N×n multiplications and N×(n-1) additions are required in the reverse transformation. Therefore, by applying the transformation method utilizing the linearity of transformation to the 1-dimensional vector with fewer non-zero coefficients, the amount of calculation can be greatly reduced.
[0177] Thus, not only in the case of separable transform, but also in the case of non-separable transform, the amount of computation can be greatly reduced by applying the linearity-based transform. The non-separable transform method using linearity can be applied not only to secondary transform, but also to primary transform.
[0178] Figure 5 The diagram shows a scanning method for rearranging coefficients in an inverse quantized block into a one-dimensional vector. The coefficients in a block inversely quantized by the block scanning method can be rearranged into a one-dimensional vector. Figure 5 Starting from the left side of the diagram, diagonal scanning, horizontal scanning, and vertical scanning are introduced. Figure 6 The figure shows an example of rearranging coefficients in a 4×4 block into a one-dimensional vector using horizontal scanning. Figure 5 as well as Figure 6 As shown, for a 2D non-separable transformation, a 2D input matrix is rearranged into a 1D input vector. In addition, the 2D non-separable transformation is realized by dividing the rearranged 1D input vector into a plurality of sub-vectors each containing only one non-zero coefficient, and multiplying each of the plurality of sub-vectors by a 2D non-separable kernel, and then accumulating the results of each multiplication according to the linearity of the transformation.
[0179] The 1-dimensional vector may be separated into sub-vectors having only one non-zero coefficient, and the inverse transform may be performed on each sub-vector. Figure 7 The diagram shows an example of separating a 1-dimensional vector into sub-vectors. Figure 8 The figure shows an example of performing inverse transformation on each sub-vector.
[0180] Through Figure 8 The final inverse transformation vector can be generated by accumulating all the vectors based on the inverse transformation results related to each sub-vector. The 1-dimensional vector, i.e., the final inverse transformation vector, can be rearranged into a 2-dimensional block according to the block scanning method used in the transformation.
[0181] Fig. 9 An embodiment of a video decoding method that can utilize an inverse linear transform method is provided.
[0182] In step 902, the number of non-zero coefficients of the inverse quantized block is obtained.
[0183] In step 904, an inverse transform method of the inverse quantized block is determined according to the number of the non-zero coefficients.
[0184] In one embodiment, the number of the non-zero coefficients may be compared with a specific threshold value, and the inverse transform method of the inverse quantized block may be determined based on the comparison result.
[0185] In one embodiment, the number of multiplication operations required for the inverse linear transform can be determined from the number of non-zero coefficients. In addition, the number of multiplication operations can be compared with a specific threshold value, and the inverse transform method of the inverse quantized block can be determined based on the comparison result. The number of multiplication operations can be determined based on the number of non-zero coefficients and / or the size of the inverse quantized block.
[0186] In the embodiment, the specific threshold may be determined based on the size of the inverse quantization block.
[0187] In one embodiment, a vertical kernel and a horizontal kernel applicable to the inverse quantization block may be determined. In addition, the specific threshold may be determined based on the size of the vertical kernel, the horizontal kernel and / or the inverse quantization block. The vertical kernel and the horizontal kernel may be determined from at least one of discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII) and discrete cosine transform-VIII (DCT-VIII). In addition, the vertical kernel and the horizontal kernel may be determined based on the size of the inverse quantization block and the prediction method used for the inverse quantization block.
[0188] In one embodiment, the inverse transform method of the inverse quantized block may be determined based on the image type of the inverse quantized block. For example, when the image type of the inverse quantized block is an All Intra (AI) type or a Random Access (RA) type, it may be determined whether to apply a linear inverse transform to the inverse quantized block based on the number of non-zero coefficients. On the contrary, when the image type of the inverse quantized block is not an All Intra (AI) type or a Random Access (RA) type, it may be determined not to apply a linear inverse transform to the inverse quantized block.
[0189] In one embodiment, the inverse transform method of the inverse quantized block may be determined based on a quantization parameter for inverse quantization of the inverse quantized block. When the quantization parameter is greater than a critical quantization parameter value, it may be determined whether to apply a linear inverse transform to the inverse quantized block according to the number of non-zero coefficients. Conversely, when the quantization parameter is less than a critical quantization parameter value, it may be determined not to apply a linear inverse transform to the inverse quantized block.
[0190] In one embodiment, linear inverse transformation permission information indicating whether linear inverse transformation is allowed can be obtained from the parameter set. In addition, when the linear inverse transformation permission information indicates that the linear inverse transformation is allowed, it can be determined whether the inverse transformation method of the inverse quantization block is a linear inverse transformation method. On the contrary, when the linear inverse transformation permission information indicates that the linear inverse transformation is not allowed, it can be determined that the inverse quantization block is inversely transformed by an inverse transformation method other than a linear inverse transformation method. The parameter set can be at least one of a video parameter set, a sequence parameter set, an image parameter set, and an adaptation parameter set.
[0191] In one embodiment, the inverse transform method of the inverse quantized block may be determined based on the color component of the inverse quantized block.
[0192] In step 906, the inverse transform of the inverse quantized block is performed according to the determined inverse transform method. In the case where the inverse transform method is a linear inverse transform method, the inverse transform of the inverse quantized block may be performed as follows.
[0193] First, the inverse quantized block can be divided into a plurality of sub-blocks each containing only one non-zero coefficient and the remaining coefficients being zero coefficients. Next, an inverse transform can be performed on each of the plurality of sub-blocks. An inverse transform block of the inverse quantized block can be obtained based on the plurality of element blocks of each inverse transform.
[0194] Fig.10 An embodiment of a video encoding method that can utilize an inverse linear transform method is provided.
[0195] In the video encoding method, after encoding a block of an image, in order to encode other blocks of the image or other images, it will be stored in a decoded picture buffer after decoding. Therefore, by applying an appropriate inverse transformation method to the video encoding method, the speed of video encoding can be increased. The video encoding method using the linear inverse transformation method can be implemented as described below.
[0196] In step 1002, a block may be encoded, and the encoded block may be inverse quantized.
[0197] In step 1004, the number of non-zero coefficients of the inverse quantized block may be obtained.
[0198] In step 1006, an inverse transform method of the inverse quantized block may be determined according to the number of the non-zero coefficients.
[0199] In step 1008, an inverse transform of the inverse quantized block may be performed according to the determined inverse transform method.
[0200] In steps 1004 to 1008 , the configurations related to steps 902 to 906 may be applied.
[0201] In step 1010, the block may be reconstructed using the inverse transformed block, and other blocks may be encoded based on the reconstructed block.
[0202] Can provide storage through Fig.10 A method for encoding a video generates a bit stream on a computer-readable storage medium. Fig.10 The bit stream generated by the video encoding method can be stored in a computer-readable storage medium. Fig.10 The bit stream generated by the video encoding method can be transmitted from the video encoding device to the video decoding device.
[0203] A bitstream associated with video data stored in a computer-readable storage medium may be obtained by Fig. 9 In addition, the bit stream transmitted from the video encoding device to the video decoding device can be decoded by Fig.10 Decode the video using the video decoding method.
[0204] 8-tap discrete sine transform (8-tap DST) interpolation filter
[0205] The existing method applies a 4-tap discrete cosine transform (4-tap DCT) interpolation filter to all blocks regardless of the size of the block. In the present disclosure, in order to generate reference samples related to fractional angles in intra-picture prediction, a method of applying an 8-tap discrete sine transform (8-tap DST) interpolation filter to 4×4, 4×n and n×4 blocks is proposed.
[0206] For higher resolution A-type sequences, an 8-tap discrete sine transform (8-tap DST) interpolation filter is applied to 4×4 blocks instead of a 4-tap discrete cosine transform (4-tap DCT) interpolation filter, and for relatively lower resolution B, C, and D-type sequences, an 8-tap discrete sine transform (8-tap DST) interpolation filter is applied to 4×n blocks and n×4 blocks (n=4, 8, 16, 32, 64) instead of a 4-tap discrete cosine transform (4-tap DCT) interpolation filter.
[0207] The 8-tap discrete sine transform (8-tap DST) interpolation filter coefficients are induced by discrete sine transform-VII (DST-VII) and inverse discrete sine transform-VII (IDST-VII, Inverse DST-VII), and Table 5 shows the coefficients at specific 16 / 32-pixel positions in the 1 / 32 interpolation filter coefficients.
[0208]
Table 5
[0209]
[0210] Interpolation filter analysis
[0211] Fig.11 The figure shows the magnitude response of the 4-tap discrete cosine transform (4-tap DCT) interpolation filter and the 8-tap discrete sine transform (8-tap DST) interpolation filter at the 16 / 32-pixel position. Fig.11 In FIG. 5 , the x-axis represents the frequency (Frequency) normalized to a value between 0 and 1, and the y-axis represents the magnitude response (Magnitude response).
[0212] exist Fig.11 In the figure, the blue graph shows the magnitude response associated with the existing 4-tap discrete cosine transform (4-tap DCT) interpolation filter, while the red graph shows the magnitude response of the proposed method, i.e., the 8-tap discrete sine transform (8-tap DCT) interpolation filter. Fig.11 As shown, it can be confirmed that the low frequency responses of the two interpolation filters are similar, but the 8-tap discrete sine transform (8-tap DST) interpolation filter has a higher frequency response than the 4-tap discrete cosine transform (4-tap DCT) interpolation filter.
[0213] Fig.12 The coefficients of an 8-tap discrete sine transform (DST) interpolation filter are illustrated.
[0214] The coefficient induction method of the 8-tap discrete sine transform (DST) interpolation filter is shown in the following mathematical formula 16. In mathematical formula 16, formula (3) can be induced by substituting formula (1) into formula (2).
[0215]
Mathematical formula 16
[0216]
[0217] In the present disclosure, a method is proposed in which, for blocks with nTbS of 2, an 8-tap discrete cosine transform (8-tap DCT) interpolation filter is used instead of a 4-tap discrete cosine transform (4-tap DCT) interpolation filter, and for blocks with nTbS of 5 or more, an 8-tap Gaussian interpolation filter is used instead of a 4-tap Gaussian interpolation filter.
[0218] Table 6 shows the coefficients at specific 16 / 32-pixel positions in the 1 / 32 pixel 8-tap discrete sine transform (8-tap DCT) interpolation filter coefficients. Table 7 shows the coefficients at specific 16 / 32-pixel positions in the 1 / 32 pixel 8-tap Gaussian interpolation filter coefficients.
[0219]
Table 6
[0220]
[0221]
Table 7
[0222]
[0223]
[0224] Fig.13 The diagram shows the magnitude response at 16 / 32-pixel positions of a 4-tap DCT interpolation filter, a 4-tap Gaussian interpolation filter, an 8-tap DCT interpolation filter, and an 8-tap Gaussian interpolation filter. The x-axis represents the frequency (Frequency) normalized to a value between 0 and 1, and the y-axis represents the magnitude response (Magnitude response).
[0225] By comparing the blue graph, i.e., the 4-tap discrete cosine transform (4-tap DCT) interpolation filter, and the yellow graph, i.e., the 8-tap discrete sine transform (8-tap DCT) interpolation filter, it can be confirmed that the 8-tap discrete sine transform (8-tap DCT) interpolation filter has a high frequency response compared to the 4-tap discrete sine transform (4-tap DCT) interpolation filter. In addition, by comparing the red graph, i.e., the 4-tap Gaussian (4-tap Gaussian) interpolation filter, and the purple graph, i.e., the 8-tap Gaussian (8-tap Gaussian) interpolation filter, it can be confirmed that the 8-tap Gaussian (8-tap Gaussian) interpolation filter has a low frequency response compared to the 4-tap Gaussian (4-tap Gaussian) interpolation filter.
[0226] In the present disclosure, the performance can be improved by adding an 8-tap discrete sine transform (8-tap DCT) interpolation filter and an 8-tap Gaussian interpolation filter to the 4-tap discrete sine transform (4-tap DCT) interpolation filter and the 4-tap Gaussian interpolation filter used when generating reference samples in intra-picture prediction according to the block size and directional mode. In the present disclosure, a discrete cosine transform 8-tap (DCT 8-tap) interpolation filter (4×4, 4×8, 8×4, ...) can be applied to a block with nTbS of 2, and a Gaussian 8-tap (Gaussian 8-tap) interpolation filter (32×32, 32×64, 64×32, 64×64) can be applied to a block with nTbS of 5 or more.
[0227] Frequency-based AdaptiveInterpolation Filter in Intra Prediction
[0228] An 8-tap Discrete Cosine Transform-based interpolation filter (DCT-IF) and an 8-tap Smoothing interpolation filter (SIF) are used to replace the 4-tap DCT-IF and 4-tap SIF currently used in VVC intra-picture prediction. Since the 8-tap DCT-IF has a high frequency characteristic compared to the 4-tap DCT-IF, and the 8-tap SIF has a low frequency characteristic compared to the 4-tap SIF, the type of the 8-tap interpolation filter is selected according to the characteristics of the block.
[0229] The size of the block and the frequency characteristics of the reference samples are used to determine the characteristics of the block and select the type of interpolation filter used in the corresponding block.
[0230] The characteristic used is that the smaller the block size, the lower the correlation and the more high frequencies, while the larger the block size, the higher the correlation and the more low frequencies.
[0231] The frequency characteristics of the reference samples can be calculated by applying a transform to the reference samples of the block using discrete cosine transform-II (DCT-II). Depending on the intra-picture prediction mode, the upper reference samples are used in the vertical direction, and the left reference samples are used in the horizontal direction. The higher the percentage of high frequency energy, the higher the high frequency characteristics of the block.
[0232] The frequency characteristics of the block are determined by comparing the percentage of high frequency energy with a threshold based on the block size, and an interpolation filter is selected that needs to be applied to the block.
[0233] According to the frequency information, a strong high pass filter (HPF), i.e., an 8-tap discrete cosine transform interpolation filter (8-tap DCT-IF), is applied to blocks with more high frequencies, and a strong low pass filter (LPF), i.e., an 8-tap smoothing interpolation filter (8-tap SIF), is applied to blocks with more low frequencies.
[0234] According to the characteristic that the smaller the block, the lower the correlation, and the frequency information, a strong high-pass filter (HPF) is applied to the block with more high frequencies. When the block size is small, a strong high-pass filter (HPF), i.e., an 8-tap discrete cosine transform interpolation filter (8-tap DCT-IF), is applied. When the high frequencies are less, a low-pass filter (LPF), i.e., a 4-tap smoothing interpolation filter (4-tap SIF), is applied.
[0235] The method of applying a strong low-pass filter (LPF) to blocks with more low frequencies is based on the characteristic that the larger the block, the higher the correlation, and the frequency information. When the block size is large, a strong low-pass filter (LPF), i.e., an 8-tap smoothing interpolation filter (8-tap SIF), is applied. When there are more high frequencies, a weak low-pass filter (HPF), i.e., a 4-tap discrete cosine transform interpolation filter (4-tap DCT-IF), is applied.
[0236] Example of calculating the percentage of high frequency energy
[0237] In the case of the horizontal intra-screen mode, N is the height of the block, and in the case of the vertical intra-screen mode, N is the width of the block. The value of N may be smaller or larger when a smaller number of reference samples are used or when a larger number of reference samples are used. X represents a reference sample. In the case described above, the high frequency region uses reference samples of 1 / 4 length of N, and the length of the region may be reduced or increased when a smaller number of reference samples are used to calculate the high frequency or when a larger number of reference samples are used. Equation 17 shows a method for calculating the proportion of high frequency energy.
[0238]
Mathematical formula 17
[0239]
[0240] Fig.14 An example of a method of selecting an interpolation filter using frequency information is shown in the figure.
[0241] In the case where the percentage of high frequency energy of a block with nTbS of 2 is less than a threshold, a 4-tap smoothing interpolation filter (4-tap SIF) is applied, and in other cases, an 8-tap discrete cosine transform interpolation filter (8-tap DCT-IF) is applied. In the case where the percentage of high frequency energy of a block with nTbS of 5 or more is less than a threshold, an 8-tap smoothing interpolation filter (8-tap SIF) is applied, and in other cases, a 4-tap discrete cosine transform interpolation filter (4-tap DCT-IF) is applied.
[0242] Fig.15 as well as Fig.16 Coefficients of an 8-tap discrete cosine transform interpolation filter and an 8-tap smoothing interpolation filter are shown in the figure respectively.
[0243] The present disclosure can improve coding efficiency by calculating a critical value based on correlation, high_freq_ratio and block size (nTbS) for each image. For all block sizes, using only a boundary correlation threshold when selecting the filter length can also improve performance. The correlation threshold can be applied independently to the long / short tap discrete cosine transform interpolation filter (long / short tap DCT-IF) and the smoothing interpolation filter (SIF), or it can be applied together with the high_freq_ratio. Fig.17 An embodiment of an interpolation filter selected according to a boundary correlation threshold is illustrated.
[0244] The various embodiments of the present disclosure are not intended to list all possible combinations, but are merely intended to illustrate representative forms of the present disclosure. Matters described in the various embodiments may be applied independently or in combination of two or more.
[0245] In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. When implemented by hardware, it may be implemented by one or more application specific integrated circuits (ACICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general processors, controllers, microcontrollers, and microprocessors.
[0246] The scope of the present disclosure includes software or device-executable instructions (such as operating systems, applications, firmware, programs, etc.) that can be used to perform actions in methods applicable to various different embodiments on a device or computer, as well as non-transitory computer-readable media (non-transitory computer-readable medium) that can be executed by a device or computer and stores the above-mentioned software or instructions.
Claims
1. A video decoding method, comprising: The step of obtaining the number of non-zero coefficients of the inverse quantized block; The step of determining an inverse transform method of the inverse quantized block according to the number of the non-zero coefficients; as well as, According to the determined inverse transform method, the step of inverse transforming the inverse quantized block is performed.
2. The video decoding method according to claim 1, The step of determining the inverse transformation method of the inverse quantized block comprises: The step of comparing the number of said non-zero coefficients with a specific critical value; as well as, The step of determining an inverse transformation method of the inverse quantized block based on the comparison result.
3. The video decoding method according to claim 1, The step of determining the inverse transformation method of the inverse quantized block comprises: The step of determining, from the number of non-zero coefficients, the number of multiplication operations required for an inverse linear transform; The step of comparing the multiplication number with a specific threshold value; as well as, The step of determining an inverse transformation method of the inverse quantized block based on the comparison result.
4. The video decoding method according to claim 3, characterized in that: The number of multiplication operations, It is determined based on the number of the non-zero coefficients and the size of the inverse quantization block.
5. The video decoding method according to any one of claims 2 and 3, characterized in that: The specific threshold is determined based on the size of the inverse quantization block.
6. The video decoding method according to any one of claims 2 and 3, characterized in that: The video decoding method further includes: The step of determining a vertical kernel and a horizontal kernel applicable to the inverse quantized block; The specific threshold is determined based on the size of the vertical kernel, the crystal box, and the inverse quantization block.
7. The video decoding method according to claim 6, characterized in that: The vertical kernel and the horizontal kernel are determined from at least one of discrete cosine transform-II (DCT-II), discrete sine transform-VII (DST-VII), and discrete cosine transform-VIII (DCT-VIII).
8. The video decoding method according to claim 6, characterized in that: The vertical core and the horizontal core, It is determined based on the size of the inverse quantization block and the prediction method applicable to the inverse quantization block.
9. The video decoding method according to claim 1, characterized in that: An inverse transform method of the inverse quantized block is determined based on the image type of the inverse quantized block.
10. The video decoding method according to claim 9, characterized in that: The step of determining the inverse transformation method of the inverse quantized block includes: When the image type of the dequantized block is an All Intra (AI) type or a Random Access (RA) type, a step of determining whether to apply a linear inverse transform to the dequantized block is performed according to the number of the non-zero coefficients.
11. The video decoding method according to claim 9, characterized in that: The step of determining the inverse transform method of the inverse quantized block, When the picture type of the dequantized block is not an All Intra (AI) type or a Random Access (RA) type, it is determined not to apply an inverse linear transform to the dequantized block.
12. The video decoding method according to claim 1, characterized in that: An inverse transform method for the inverse quantized block is determined based on a quantization parameter applied to the inverse quantized block.
13. The video decoding method according to claim 12, characterized in that: include: When the quantization parameter is greater than a critical quantization parameter value, determining whether to apply an inverse linear transform to the inverse quantized block is performed according to the number of the non-zero coefficients.
14. The video decoding method according to claim 12, characterized in that: When the quantization parameter is less than a critical quantization parameter value, it is determined not to apply an inverse linear transform to the inverse quantized block.
15. The video decoding method according to claim 1, characterized in that: The video decoding method further includes: A step of acquiring, from a parameter set, linear inverse transformation permission information indicating whether linear inverse transformation is permitted; The step of determining the inverse transform method of the inverse quantized block, When the linear inverse transform permission information indicates that the linear inverse transform is permitted, it is determined whether the inverse transform method of the inverse quantized block is a linear inverse transform method.
16. The video decoding method according to claim 15, characterized in that: The parameter set, is at least one of a video parameter set, a sequence parameter set, a picture parameter set, and an adaptation parameter set.
17. The video decoding method according to claim 1, characterized in that: An inverse transform method of the inverse quantized block is determined based on the color component of the inverse quantized block.
18. The video decoding method according to claim 1, characterized in that: The step of performing inverse transformation of the inverse quantized block according to the determined inverse transformation method comprises: In the case where the inverse transform method is a linear inverse transform, The step of dividing the inverse quantized block into a plurality of sub-blocks each including only one non-zero coefficient and the remaining coefficients being zero coefficients; For each of the plurality of sub-blocks, performing an inverse transform step; and The step of obtaining an inverse transformed block of the inverse quantized block based on the multiple element blocks of each inverse transform.
19. A video encoding method, comprising: A step of encoding a block and performing inverse quantization on the encoded block; A step of obtaining the number of non-zero coefficients of the inverse quantized block; The step of determining an inverse transform method of the inverse quantized block according to the number of the non-zero coefficients; According to the determined inverse transform method, performing the step of inverse transforming the inverse quantized block; as well as, The step of reconstructing a block using the inverse transform block, and encoding other blocks based on the reconstructed block.
20. A computer-readable storage medium, characterized in that: A computer-readable storage medium storing a bit stream related to a coded video, including a bit stream generated by a video coding method, The video encoding method comprises: A step of encoding a block and performing inverse quantization on the encoded block; A step of obtaining the number of non-zero coefficients of the inverse quantized block; The step of determining an inverse transform method of the inverse quantized block according to the number of the non-zero coefficients; According to the determined inverse transform method, performing an inverse transform step of the inverse quantized block; as well as, The step of reconstructing a block using the inverse transform block, and encoding other blocks based on the reconstructed block.
