Method and apparatus for processing image

Abstract

An image filtering method according to an embodiment of the present disclosure comprises: obtaining at least one transform block by transforming an image data unit to be filtered at least one time in units of blocks having a predetermined size; maintaining a transform coefficient equal to or greater than a predetermined threshold value among transform coefficients in the at least one transform block and replacing a transform coefficient smaller than the predetermined threshold value with a predetermined value to obtain at least one substitution transform block; inversely transforming the at least one substitution transform block to obtain at least one inverse transform block; and performing filtering by determining a pixel value of a pixel at a predetermined position by using an inverse transform value in the at least one inverse transform block.

Description

Image processing method and device

[0001] The present disclosure relates to the field of image encoding and decoding. More specifically, the present disclosure relates to an image processing method and device that performs filtering on image data to remove artifacts present in the image data during image encoding / decoding.

[0002] With the development and widespread adoption of hardware capable of playing and storing high-resolution or high-definition video content, the need for codecs that effectively encode or decode this content is increasing. Recently, methods for effectively compressing such high-definition or high-definition video content have been implemented. Representative examples of codecs include High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC).

[0003] In HEVC, a picture is divided into one or more tiles / slices, which are then divided into multiple CTUs (Coding Tree Units). VVC can first divide a picture into multiple sub-pictures. A sub-picture is defined as a group of rectangular slices and was added to VVC to support the function of partially independently encoding / decoding and transmitting a picture. A sub-picture can be divided into tiles / slices, similar to HEVC. VVC also adds a new picture division structure called brick. Bricks are created by horizontally dividing tiles and are the basic unit of parallel processing. In order to process higher resolution images than HEVC, VVC uses a CTU (Coding Tree Unit) with a maximum size of 256x256, which is 16 times larger than HEVC, as the basic unit of encoding / decoding.

[0004] VVC incorporates various new technologies that build on HEVC's foundation to enhance encoding efficiency. For example, to enhance the image quality of restored images, VVC utilizes three filters, including the Adaptive Loop Filter (ALF), and in-loop filtering such as Luma Mapping with Chroma Scaling (LMCS), in addition to the Deblocking Filter (DF) and Sample Adaptive Offset (SAO) employed in HEVC. This not only improves subjective image quality but also enhances encoding efficiency.

[0005] In addition, video codecs such as VVC and AV1 support prediction using diagonal prediction blocks for efficient image compression. However, although the energy of the residual derived through diagonal prediction is significantly reduced compared to rectangular prediction, the residual goes through a rectangular-based transformation and restoration process, so the restored image cannot clearly restore the edges of the content, and ringing artifacts may occur at the diagonal edges. Ringing artifacts not only degrade the image quality of the restored image, but also significantly reduce the compression efficiency.

[0006] The present disclosure provides an image processing method and device that performs filtering to reduce ringing artifacts that occur during image encoding / decoding.

[0007] According to embodiments of the present disclosure, an image processing method and device are disclosed that perform nonlinear filtering by, after transforming image data to be filtered, setting a transform coefficient less than a predetermined threshold value to a predetermined value, performing inverse transformation again to obtain at least one inverse transform value corresponding to a pixel at a predetermined position, and determining a pixel value of a pixel at a predetermined position as a representative value determined using the at least one inverse transform value.

[0008] According to embodiments of the present disclosure, ringing artifacts present in image data can be removed during image encoding / decoding, and image compression efficiency can be improved.

[0009] FIG. 1 is a block diagram illustrating an encoding device according to one embodiment of the present disclosure.

[0010] FIG. 2 is a block diagram illustrating a decryption device according to one embodiment of the present disclosure.

[0011] FIG. 3 is a block diagram of an image processing device according to one embodiment of the present disclosure.

[0012] FIGS. 4A to 4C are diagrams illustrating a process of moving a block unit of a predetermined size in which transformation, transformation coefficient setting, and inverse transformation are performed according to one embodiment of the present disclosure.

[0013] Figures 5a to 5e are drawings showing pixels of image data units included in a block unit according to movement of the block unit.

[0014] Figure 6a is a diagram illustrating the first transformation block.

[0015] FIG. 6b is a diagram illustrating a second transformation block generated based on the first transformation block of FIG. 6a.

[0016] Figures 7a and 7b are diagrams illustrating the correspondence between pixels of an image data unit and pixels within an inverse transformation block.

[0017] FIG. 8 shows a flowchart of an image processing method according to one embodiment of the present disclosure.

[0018] An image processing method according to one embodiment of the present disclosure determines whether filtering is applied to a block unit of a predetermined size within an image data unit, stores pixel values ​​of pixels included in a block unit to which filtering is not applied according to the determination result for each pixel position within the image data unit, transforms pixels included in a block unit to which filtering is applied according to the determination result to obtain at least one first transform block, maintains transform coefficients greater than a predetermined threshold among transform coefficients within the at least one first transform block and sets transform coefficients less than the predetermined threshold to a predetermined value to obtain at least one second transform block, inversely transforms the at least one second transform block to obtain at least one inverse transform block, and stores inverse transform values ​​included in the inverse transform block according to each pixel position within the image data unit, and determines pixel values ​​according to each pixel position of the image data unit using at least one of the pixel values ​​stored according to each pixel position and the inverse transform value.

[0019] An image processing device according to one embodiment of the present disclosure includes: a determination unit that determines whether filtering is applied to a block unit of a predetermined size within an image data unit and stores pixel values ​​of pixels included in a block unit to which filtering is not applied for each pixel position within the image data unit; a transformation unit that transforms pixels included in a block unit to which filtering is applied based on the determination result of the determination unit to obtain at least one first transformation block; a transformation coefficient setting unit that maintains transformation coefficients greater than a predetermined threshold value among transformation coefficients within the at least one first transformation block and sets transformation coefficients less than the predetermined threshold value to a predetermined value to obtain at least one second transformation block; an inverse transformation unit that inversely transforms the at least one second transformation block to obtain at least one inverse transformation block and stores inverse transformation values ​​included in the inverse transformation block for each pixel position within the image data unit; and a representative value determination unit that performs filtering by determining pixel values ​​for each pixel position of the image data unit using at least one of the pixel values ​​stored for each pixel position and the inverse transformation value.

[0020] A computer program for an image processing method according to one embodiment of the present disclosure can be recorded on a computer-readable recording medium.

[0021] Hereinafter, a video encoding and decoding method and device according to an embodiment of the present disclosure will be described in detail with reference to the attached drawings.

[0022] The video is a sequence of pictures, and the method and device for encoding and decoding a video according to an embodiment of the present disclosure are described in detail with reference to the attached drawings.

[0023] A video is a sequence of pictures, and each picture is the target of encoding and decoding. A picture refers to a field or a frame, and these can be used interchangeably. For example, if the video is an interlaced video, one frame is divided into an odd (or odd, top) field and an even (or even, bottom) field, and each field is composed of one picture unit and can be encoded or decoded. If the video is a progressive scan video, one frame is composed of a picture and can be encoded or decoded. A slice / tile is a coding unit that constitutes a part of a picture during encoding, and one picture can include one or more slices / tiles. In addition, a slice / tile can include one or more CTUs (coding tree units). One picture can be composed of one or more tile groups. One tile group can include one or more tiles. A brick can represent a rectangular area of ​​CTU rows within a tile within a picture. A tile may contain one or more bricks. A brick may represent a rectangular area of ​​CTU rows within the tile. A tile may be divided into multiple bricks, each of which may contain one or more CTU rows within the tile. A tile that is not divided into multiple bricks may also be treated as a brick.

[0024] A pixel or pel is the smallest unit that constitutes an image, and the terms pixel and sample can be used interchangeably. A sample can generally represent a pixel or a pixel value, or it can represent only a pixel or pixel value of the luminance component, or only a pixel or pixel value of the chroma component.

[0025] In addition, the unit is used to mean a basic unit of image processing or a specific location of a picture, and represents an image area including at least one of a luminance component and a chrominance component. Specifically, the unit may be used as a concept including a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), and a transform unit (TU). In the present disclosure, a unit may be referred to as a unit. In addition, a block represents an image area including a specific component among luminance components and chrominance components, and an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. Here, the terms unit, block, partition, unit, signal, and area may be used interchangeably.

[0026] FIG. 1 is a block diagram of an encoding device according to one embodiment of the present disclosure, and is intended to explain the configuration and operation of a video encoder for encoding a video.

[0027] Referring to FIG. 1, a video encoder (100) may be configured to include a transformation unit (110), a quantization unit (120), an inverse quantization unit (130), an inverse transformation unit (140), a filtering unit (150), a prediction unit (160), a DPB (Decoded Picture Buffer, 170), and an entropy coding unit (180).

[0028] The conversion unit (110) converts the residual signal, which is the difference between the input video signal and the prediction signal generated by the prediction unit (160), to obtain a conversion coefficient value.

[0029] For example, the Discrete Cosine Transform (DCT), the Discrete Sine Transform (DST), or the Wavelet Transform can be used.

[0030] The transform kernel used for transforming the residual block may be a transform kernel having separable vertical and horizontal transform properties. In this case, the transform for the residual block may be performed separately as vertical and horizontal transforms. For example, the encoder may perform a vertical transform by applying the transform kernel in the vertical direction of the residual block. Alternatively, the encoder may perform a horizontal transform by applying the transform kernel in the horizontal direction of the residual block.

[0031] Meanwhile, the transform kernel may be used as a term referring to a set of parameters used for transforming the residual signal, such as a transform matrix, a transform array, a transform function, or a transform, and may be any one of a plurality of available kernels, and transform kernels based on different transform types may be used for each of the vertical transform and the horizontal transform.

[0032] The transformation coefficients may be distributed such that higher coefficients are distributed toward the upper left corner of the block, and coefficients closer to '0' are distributed toward the lower right corner of the block. Furthermore, as the current block size increases, there is a possibility that many '0' coefficients exist in the lower right area. To reduce the transformation complexity of large blocks, only the upper left area can be left, and the remaining areas can be reset to '0'.

[0033] Additionally, error signals may exist only in some regions of a coding block, in which case the conversion process may be performed only on some arbitrary regions. For example, in a block of size 2Nx2N, error signals may exist only in the first 2NxN block, in which case the conversion process may be performed only on the first 2NxN block, but the conversion process may not be performed on the second 2NxN block and may not be encoded or decoded.

[0034] The encoder may perform additional transformations before the transform coefficients are quantized. The transformation method described above may be referred to as a primary transform, and the additional transformation may be referred to as a secondary transform.

[0035] The secondary transform can be optional for each residual block, for example, the encoder can improve coding efficiency by performing the secondary transform for areas where it is difficult to concentrate energy in the low-frequency region using only the primary transform.

[0036] Specifically, a secondary transformation may be additionally performed on blocks in which residual values ​​appear significantly in directions other than the horizontal or vertical direction of the residual block, and the secondary transformation may not be performed separately into vertical transformation and horizontal transformation, unlike the primary transformation. Such a secondary transformation may be referred to as a low frequency non-separable transform (LFNST).

[0037] The quantization unit (120) quantizes the transformation coefficient value output from the transformation unit (110).

[0038] In order to increase coding efficiency, a method is used in which, instead of coding the picture signal as is, a picture is predicted using an already coded area through a prediction unit (160), and a restored picture is obtained by adding a residual value between the original picture and the predicted picture to the predicted picture.

[0039] To avoid mismatches between the encoder and decoder, when performing prediction in the encoder, information available in the decoder must also be used, and for this purpose, the encoder can perform a process of reconstructing the current block that it has encoded.

[0040] The inverse quantization unit (130) inversely quantizes the transform coefficient values, and the inverse transformation unit (140) restores the residual values ​​using the inverse quantized transform coefficient values.

[0041] The filtering unit (150) performs filtering operations using a deblocking filter, a sample adaptive offset (SAO), an adaptive loop filter (ALF), etc. to improve the quality of the restored picture and enhance encoding efficiency.

[0042] In addition, the filtering unit (150) according to one embodiment of the present disclosure may further include a nonlinear filtering unit to remove a ringing artifact. The nonlinear filtering unit may perform a filtering operation in which, after converting image data to be filtered, a conversion coefficient less than a predetermined threshold value is replaced with zero and inversely transformed again to obtain at least one inverse transformation value corresponding to a pixel at a predetermined position, and the pixel value of the pixel at the predetermined position is determined as a representative value using the at least one inverse transformation value.

[0043] In addition, the nonlinear filtering unit according to one embodiment of the present disclosure may be positioned in the filtering unit (150) of FIG. 1 as one of the in-loop filters, such as a deblocking filter, SAO, or ALF, or may be positioned at the output terminal of the filtering unit (150) to filter again the output value of the in-loop filter, such as a deblocking filter, SAO, or ALF. In addition, the nonlinear filtering unit may be positioned between in-loop filters, such as a deblocking filter, SAO, or ALF, to filter the output value after one or more in-loop filters, such as a deblocking filter, SAO, or ALF, are applied, or may be positioned at the output terminal of the inverse transform unit (140) to apply filtering to the residual value, which is the output value of the inverse transform unit (140). The operation of the nonlinear filtering unit according to one embodiment of the present disclosure will be described later.

[0044] A picture that has undergone an in-loop filtering process by the filtering unit (150) can be stored in the DPB (170) to be used as a reference picture.

[0045] The prediction unit (160) includes an intra / IBC prediction unit (161) and an inter prediction unit (165). The intra / IBC prediction unit (161) performs intra prediction and IBC prediction within the current picture, and the inter prediction unit (165) performs inter prediction to predict the current picture using a reference picture stored in the DPB (170).

[0046] The intra / IBC prediction unit (161) performs intra prediction from restored areas within the current picture and transmits intra encoding information to the entropy coding unit (180). Here, the intra encoding information may include at least one of an intra prediction mode, an MPM (Most Probable Mode) flag, an MPM index, and information about a reference sample.

[0047] Additionally, the intra / IBC prediction unit (161) can perform IBC (Intra BLock Copy) prediction from restored samples within the current picture and transmit IBC encoding information to the entropy coding unit (180).

[0048] In this case, the intra / IBC prediction unit (161) obtains a block vector value indicating a reference area used for prediction of the current area by referring to a specific area within the current picture, and can perform IBC prediction using the obtained block vector value.

[0049] The intra / IBC prediction unit (161) can transmit IBC encoding information including at least one of the size information of the reference area and block vector information (index information for block vector prediction of the current block within the motion candidate list, block vector difference information) to the entropy coding unit (160).

[0050] The inter prediction unit (165) refers to a specific area of ​​the restored reference picture to find the part most similar to the current area, obtains a motion vector value which is the distance between the areas, and transmits motion information (reference direction indication information (L0 prediction, L1 prediction, bidirectional prediction), reference picture index, motion vector information, etc.) for the obtained reference area to the entropy coding unit (180).

[0051] Additionally, the inter prediction unit (165) performs motion compensation using motion information to generate a prediction block for the current block, and transmits inter encoding information including motion information for the reference area to the entropy coding unit (180).

[0052] Meanwhile, the quantized transform coefficients in the form of a two-dimensional array can be rearranged into a one-dimensional array for entropy coding.

[0053] The method of scanning the quantized transform coefficients can be determined according to the size of the transform block and the intra prediction mode, and diagonal, vertical, and horizontal scans can be applied, and the scan information can be signaled on a block-by-block basis or derived from the decoder according to a set rule.

[0054] The entropy coding unit (180) generates a bitstream by entropy coding information representing quantized transform coefficients, intra-coding information, and inter-coding information, and for this purpose, a variable length coding (VLC) method and an arithmetic coding method can be used.

[0055] Meanwhile, in the process of encoding an input image, the video encoder (100) divides the input image into one or more coding tree units (CTUs), and can recursively divide one CTU again according to a quad tree (QT) structure or a multi type tree (MTT) structure.

[0056] Based on the final encoding unit that is no longer split, the video encoder (100) can perform prediction, transformation, and restoration processes for the encoding unit. Prediction of the encoding unit can be performed using a prediction unit (PU), and transformation can be performed using a transformation unit (TU). The division structure of the CTU can be determined by selecting a division structure that has the optimal RD cost from the RD (Rate Distortion) perspective.

[0057] The bitstream generated from the video encoder (100) is encapsulated into NAL (Network Abstraction Layer) units as basic units.

[0058] NAL units are divided into VCL (Video Coding Layer) NAL units containing video data and non-VCL NAL units containing parameter information for decoding video data, and various types of VCL or non-VCL NAL units may exist.

[0059] A NAL unit consists of NAL header information and data, a Raw Byte Sequence Payload (RBSP). The NAL header information includes summary information about the RBSP. The RBSP of a VCL NAL unit contains an integer number of encoded coding tree units.

[0060] In order to decode a bitstream in a decoder, the bitstream must first be divided into NAL units, and then each divided NAL unit must be decoded. Meanwhile, the information required for decoding the bitstream can be transmitted as part of a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), etc.

[0061] Meanwhile, the configuration and operation of the encoder described with reference to FIG. 1 are according to one embodiment of the present disclosure, and some configurations may be omitted or additional configurations may be added as needed.

[0062] FIG. 2 is a block diagram of a decoding device according to one embodiment of the present disclosure, and is intended to explain the configuration and operation of a video decoder for encoding video.

[0063] Referring to FIG. 2, a video decoder (200) may be configured to include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transformation unit (230), a filtering unit (240), a prediction unit (260), and a DPB (Decoded Picture Buffer, 270).

[0064] The entropy decoding unit (210) entropy decodes the bitstream to extract transform coefficient information, intra-coding information, inter-coding information, etc. for each region.

[0065] For example, the entropy decoding unit (210) can obtain a binary code for transform coefficient information of a specific area from a bitstream, and can obtain a quantized transform coefficient by de-binarizing the binary code.

[0066] The inverse quantization unit (220) inversely quantizes the quantized transform coefficients, and the inverse transformation unit (230) restores the residual value using the inverse quantized transform coefficients.

[0067] Meanwhile, the residual value obtained from the inverse transformation unit (230) is combined with the predicted value obtained from the prediction unit (260) to restore the original pixel value.

[0068] The filtering unit (240) performs a filtering operation using a deblocking filter, a sample adaptive offset, an adaptive loop filter, etc. to improve the image quality of the restored picture, and the filtered picture can be output or stored in the DPB (270) to be used as a reference picture for the next picture.

[0069] In addition, the filtering unit (240) according to one embodiment of the present disclosure may further include a nonlinear filtering unit to remove a ringing artifact. The nonlinear filtering unit may perform a filtering operation of, after transforming image data to be filtered, replacing a transform coefficient less than a predetermined threshold value with zero and performing inverse transform again to obtain at least one inverse transform value corresponding to a pixel at a predetermined position, and determining a pixel value of a pixel at a predetermined position as a representative value determined using the at least one inverse transform value.

[0070] In addition, the nonlinear filtering unit according to one embodiment of the present disclosure may be positioned in the filtering unit (240) of FIG. 2 as one of the in-loop filters, such as a deblocking filter, SAO, or ALF, or may be positioned at the output terminal of the filtering unit (240) to filter again the output value of the in-loop filter, such as a deblocking filter, SAO, or ALF. In addition, the nonlinear filtering unit may be positioned between in-loop filters, such as a deblocking filter, SAO, or ALF, to filter the output value after one or more in-loop filters, such as a deblocking filter, SAO, or ALF, are applied, or may be positioned at the output terminal of the inverse transform unit (230) to apply filtering to the residual value before the predicted value by the prediction unit (260) is added. The operation of the nonlinear filtering unit according to one embodiment of the present disclosure will be described later.

[0071] The prediction unit (260) includes an intra / IBC prediction unit (261) and an inter prediction unit (265), and generates a prediction picture by utilizing the encoding type decoded through the entropy decoding unit (210), the transform coefficient for each region, and the intra / inter encoding information.

[0072] To restore the current block being decoded, the decoded region of the current picture or other pictures containing the current block can be used. A picture (or tile / slice) that performs intra prediction or intra IBC prediction using only the current picture for restoration is called an intra picture or I picture (or tile / slice), and a picture (or tile / slice) that can perform all of intra prediction, inter prediction, and intra IBC prediction is called an inter picture (or tile / slice).

[0073] Meanwhile, a picture (or tile / slice) that uses at most one motion vector and reference picture index to predict sample values ​​of each block among inter-pictures (or tiles / slices) is called a predictive picture or P-picture (or tile / slice), and a picture (or tile / slice) that uses at most two motion vectors and reference picture indices is called a bi-predictive picture or B-picture (or tile / slice).

[0074] That is, a P picture (or tile / slice) uses at most one motion information set to predict each block, and a B picture (or tile / slice) uses at most two motion information sets to predict each block. Here, a motion information set may include one or more motion vectors and one reference picture index.

[0075] The intra / IBC prediction unit (261) generates a prediction block using intra encoding information and restored samples within the current picture, and the intra encoding information may include at least one of an intra prediction mode, an MPM (Most Probable Mode) flag, and an MPM index.

[0076] The intra / IBC prediction unit (261) can predict sample values ​​of the current block using restored samples located on the left and / or upper side of the current block as reference samples.

[0077] For example, the reference samples may be samples adjacent to the left boundary of the current block and / or samples adjacent to the upper boundary, and may be samples located on a line within a preset distance from the left boundary of the current block among samples of neighboring blocks of the current block and / or samples located on a line within a preset distance from the upper boundary of the current block. In this case, the neighboring blocks of the current block may include at least one of a left (L) block, an upper (A) block, a below left (BL) block, an above right (AR) block, or an above left (AL) block adjacent to the current block.

[0078] Additionally, the intra / IBC prediction unit (261) can restore the current region by referring to a specific region including restored samples within the current picture. To this end, the intra / IBC prediction unit (261) can perform IBC prediction using IBC encoding information obtained from the entropy decoding unit (210), and the IBC encoding information can include block vector information.

[0079] The inter prediction unit (265) generates a prediction block using the reference picture and inter encoding information stored in the DPB (270), and the inter encoding information may include a set of motion information (reference picture index, motion vector information, etc.) of the current block for the reference block.

[0080] Meanwhile, inter prediction can include L0 prediction, L1 prediction, and bi-prediction.

[0081] L0 prediction refers to prediction using a single reference picture included in the L0 picture list, and L1 prediction refers to prediction using a single reference picture included in the L1 picture list. For this, a set of motion information (e.g., motion vector and reference picture index) may be required.

[0082] In the biprediction method, up to two reference regions can be used. These two reference regions may exist in the same reference picture or in different pictures. Accordingly, in the two sets of motion information used in the biprediction method, two motion vectors may correspond to the same reference picture index or to different reference picture indices.

[0083] At this time, the reference pictures are pictures that are located temporally before or after the current picture, and may be completed pictures that have already been restored, and the two reference areas used in the pair prediction method may be areas selected from the L0 picture list and the L1 picture list, respectively.

[0084] The inter prediction unit (265) can obtain a reference block of the current block using a motion vector and a reference picture index, and the reference block exists in a reference picture corresponding to the reference picture index.

[0085] Additionally, the sample values ​​of a block specified by a motion vector or their interpolated values ​​can be used as a predictor of the current block. For motion prediction with subpel-level pixel accuracy, an 8-tap interpolation filter can be used for the luminance signal, and a 4-tap interpolation filter can be used for the chrominance signal.

[0086] Meanwhile, the configuration and operation of the decoder described with reference to FIG. 2 are according to one embodiment of the present disclosure, and some configurations may be omitted or added as needed, and the decoder may decode an image by performing the reverse process of the encoding method of the encoder described above.

[0087] Hereinafter, with reference to FIGS. 3 to 8, an image processing method and device for performing a nonlinear filtering operation according to embodiments of the present disclosure will be described in detail.

[0088] FIG. 3 is a block diagram of an image processing device according to one embodiment of the present disclosure.

[0089] An image processing device (300) according to an embodiment of the present disclosure is a filtering device that performs a nonlinear filtering operation. When applied to an encoding step like the video encoder (100) of FIG. 1, it may be positioned in the filtering unit (150) of the video encoder (100) as an in-loop filter, such as a deblocking filter, SAO, or ALF, or may be positioned at the output terminal of the filtering unit (150) to filter again the output value of the in-loop filter, such as a deblocking filter, SAO, or ALF. In addition, an image processing device according to an embodiment of the present disclosure may be positioned between in-loop filters, such as a deblocking filter, SAO, or ALF, to filter an output value after one or more in-loop filters, such as a deblocking filter, SAO, or ALF, are applied, or may be positioned at the output terminal of the inverse transform unit (140) to apply filtering to a residual value, which is an output value of the inverse transform unit (140).

[0090] In addition, the image processing device (300) according to one embodiment of the present disclosure, when applied to the decoding step like the video decoder (200) of FIG. 2, may be positioned in the filtering unit (240) of FIG. 2 as one of the in-loop filters, such as a deblocking filter, SAO, or ALF, or may be positioned at the output terminal of the filtering unit (240) to filter again the output value of the in-loop filter, such as a deblocking filter, SAO, or ALF. In addition, the image processing device (300) according to one embodiment of the present disclosure may be positioned between in-loop filters, such as a deblocking filter, SAO, or ALF, to filter the output value after one or more in-loop filters, such as a deblocking filter, SAO, or ALF, are applied, or may be positioned at the output terminal of the inverse transform unit (230) to apply filtering to the residual value before the prediction value by the prediction unit (260) is added.

[0091] The location where filtering according to one embodiment of the present disclosure is applied may be determined explicitly or implicitly. Here, the location where filtering is applied may indicate which input data is to be filtered. For example, the image processing method according to one embodiment of the present disclosure may be applied to a restored residual obtained after inverse quantization and inverse transformation, to a restored value obtained by adding a predicted value to the restored residual, or to a filtered value to which one or more in-loop filters have been applied.

[0092] If the position where filtering is applied is determined implicitly, the video encoder (100) and the video decoder (200) can apply an image processing method to the determined position according to a rule determined in advance. If the position where filtering is applied is explicitly determined, the video encoder (100) can signal information about the position where filtering is applied to the video decoder (200) by including it in the bitstream. The information about the position where filtering is applied can be included in one of the units of the SPS (Sequence Parameter Set), PPS (Picture Parameter Set), PH (Picture Header), SH (Slice Header), tile, coding block, prediction block, transform block, sub-block unit, and user-specified block unit of the bitstream.

[0093] The video data unit to which filtering is applied may be one of the following units: a sequence, a frame, a slice, a tile, a coding block, a prediction block, a transform block, and a sub-block. The video data unit to which filtering is applied may be determined explicitly or implicitly. If the video data unit to which filtering is applied is determined implicitly, the video data unit may be determined according to a rule set in advance in the video encoder (100) and the video decoder (200). If the video data unit is explicitly determined, the video encoder (100) may include information about the video data unit to which filtering is applied in the bitstream and signal it to the video decoder (200).

[0094] Whether to apply filtering to a video data unit can be explicitly or implicitly determined for one of the following units: a sequence, a frame, a slice, a tile, a coding block, a prediction block, a transform block, and a sub-block. When explicitly signaling whether to apply filtering to a video data unit, the video encoder (100) can signal to the video decoder (200) information indicating whether to apply filtering to the video data unit in one of the following units: a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), a Picture Header (PH), a Slice Header (SH), a tile, a coding block, a prediction block, a transform block, a sub-block unit, and a user-specified block unit. The video encoder (100) can determine the video data unit to which filtering will be applied and whether to apply filtering based on the RDO cost, and can signal to the video decoder (200) information on the determined video data unit and information on whether to apply filtering by including them in a bitstream.

[0095] When it is implicitly determined whether to apply filtering to a video data unit, the video encoder (100) and the video decoder (200) can determine whether to apply filtering to the video data unit based on at least one of the number of zero or non-zero coefficients of quantized coefficients, the Min or Max value of the quantized coefficients, position information of the zero or non-zero coefficients of the quantized coefficients, position information of the first non-zero transform coefficient according to the scan order among the transform coefficients in the transform block, size information of the image data unit, width and height ratio information of the image data unit, QP (Quantization Parameter) information applied to the image data unit, luminance / chrominance channel information of the image data unit, and prediction mode information of the image data unit.

[0096] In addition, whether to apply filtering can be determined in units of blocks of a predetermined size into which the video data unit is divided. Whether to apply filtering in units of blocks of a predetermined size can be explicitly or implicitly determined for one unit among a sequence, a frame, a slice, a tile, a coding block, a prediction block, a transform block, a sub-block unit, and a block unit specified by the user. When signaling whether to explicitly apply filtering to units of blocks of a predetermined size, the video encoder (100) can signal to the video decoder (200) information indicating whether to apply filtering to a block unit included in the video data unit in one unit among a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), a Picture Header (PH), a Slice Header (SH), a tile, a coding block, a prediction block, a transform block, a sub-block unit, and a block unit specified by the user. The video encoder (100) can determine the block unit to which filtering is to be applied and whether to apply filtering based on the RDO cost, and can include information about the determined block unit and information about whether to apply filtering in the bitstream and signal it to the video decoder (200).

[0097] Referring to FIG. 3, an image processing device (300) according to an embodiment of the present disclosure includes a judgment unit (305), a transformation unit (310), a transformation coefficient setting unit (320), an inverse transformation unit (330), and a representative value determination unit (340). The image processing device (300) may include at least one processor and a memory that stores instructions to be performed by at least one processor. The transformation unit (310), the transformation coefficient setting unit (320), the inverse transformation unit (330), and the representative value determination unit (340) may be implemented as separate hardware from at least one processor, or may be modules operated by at least one processor.

[0098] The determination unit (305) determines whether filtering is applied to a block unit of a predetermined size within an image data unit. As described above, the determination unit (305) may explicitly or implicitly determine whether to apply filtering to an image data unit. The image data unit may be one of a coding block such as a CTB or a CB, a prediction block, a transform block, a sub-block, and a user-specified data unit. A block unit is a block of a predetermined size obtained by arbitrarily dividing an image data unit, and the block units may be set to overlap or not overlap within the image data unit. The determination unit (305) may determine whether filtering is applied to each block unit by moving the block units in a zigzag direction from left to right and from top to bottom in that order, starting with a block unit including an upper-left pixel within the image data unit.

[0099] When the size of a block unit of a predetermined size is NxN (N is a natural number), a predetermined pixel unit for moving a block of a predetermined size can be one of 1 to N pixel units. For example, when a block of a predetermined size is a block of a 4x4 size, the 4x4 block can be moved in a zigzag direction from left to right and from top to left in one pixel unit of 1 to 4 pixel units. The determination unit (305) can determine whether to apply filtering to pixels included in the 4x4 block while starting with a 4x4 block including an upper left pixel in an image data unit and moving the 4x4 block in a zigzag direction by 1 to 4 pixel units.

[0100] When moving a block of a given size in a zigzag direction, the pixel units used to move from left to right and the pixel units used to move from top to bottom may be identical. For example, a block of a given size may be moved by pixel units of the same size determined from (1 to N) when moving from left to right and from top to bottom.

[0101] When moving a block of a given size in a zigzag direction, the pixel units used to move from left to right and the pixel units used to move from top to bottom may be different. For example, a block of a given size may be moved from top to bottom in N pixel units, that is, block size units, and from left to right in pixel units determined from 1 to (N-1) pixel units, or a block of a given size may be moved from top to bottom in pixel units determined from 1 to (N-1) pixel units, and from left to right in N pixel units, that is, block size units.

[0102] The judgment unit (305) stores the pixel values ​​of pixels in block units to which filtering is determined not to be applied, by pixel position. For block units to which filtering is not applied, the transformation, transformation coefficient setting process, and inverse transformation process described below are skipped, and the pixel values ​​of pixels included in the current block unit are stored by pixel position in a buffer, etc. Here, the pixel position can be determined based on the image data unit. For example, if the size of the image data unit is MxN (M, N are natural numbers), the pixel position within the MxN image data unit can be determined as (x, y) (x is an integer from 0 to M-1, y is an integer from 0 to N-1), and the corresponding pixel value of the current block to which filtering is not applied can be stored by (x, y) position.

[0103] If it is determined by the judgment result of the judgment unit (305) that filtering is to be applied to the current block unit, the transformation unit (310) can obtain a first transformation block by transforming pixels included in the current block unit to which filtering is to be applied. At least one of DCT2, DCT5, DCT8, DST1, DST4, DS7, Hadamard transformation, and wavelet transformation can be used as the transformation method.

[0104] The conversion unit (310) starts with a block unit of a predetermined size including the upper left pixel of the image data unit, and moves the block units of the predetermined size in a zigzag direction from left to right and from top to bottom in the predetermined pixel unit, thereby converting the pixels included in the block unit to which filtering is applied, thereby obtaining at least one first conversion block. If the determination result of the determination unit (305) is that filtering is not applied to the current block unit, the conversion unit (310) can skip the conversion process for the current block unit. If the conversion process is skipped, the process of obtaining the second conversion block by setting the conversion coefficient less than a predetermined threshold value in the first conversion block described below to a predetermined value and the process of inversely transforming the second conversion block can also be skipped. As described above, the pixel values ​​of the pixels of the block unit to which filtering is not applied can be stored by pixel position.

[0105] It moves in units of determined pixels, and can move in units of N pixels from left to right, i.e. in units of block size.

[0106] The conversion unit (310) can perform conversion on a block unit to which filtering is applied while moving the block unit until it reaches a block unit including the lower right pixel of the image data unit.

[0107] The conversion unit (310) moves block units of a predetermined size, skips the conversion process for block units to which filtering is not applied, and performs conversion only for block units to which filtering is determined to be applied. Therefore, depending on the pixel position of the image data unit, conversion may not be performed on pixels included in block units to which filtering is not applied and may be skipped, and conversion may be performed only for pixels included in block units to which filtering is applied. As a block unit of a predetermined size is moved, conversion may be performed for pixels at positions overlapping with a previous block unit as many times as the number of overlapping block units. For example, when filtering is applied to all of a plurality of block units that overlap each other due to movement, conversion may be performed for pixels existing at positions where the plurality of block units overlap as many times as the number of overlapping block units.

[0108] When a plurality of block units to which filtering is applied are included in an image data unit, the transformation unit (310) can generate a plurality of first transformation blocks for the image data unit. For example, when the image data unit is 16x16 and filtering is applied to all 4x4 block units while moving the 4x4 block units in a zigzag direction by 1 pixel unit, the 4x4 block unit can move 12 pixels to the right with the left boundary of the image data as the center, so a total of 13 transformations (1 transformation at the initial left boundary + 12 transformations according to the 12 movements) can be performed as the 4x4 block unit moves from left to right. Similarly, since a 4x4 block unit can be shifted downward by 12 pixels centered on the upper boundary of the image data, a total of 13 transformations (1 transformation at the initial upper boundary + 12 transformations due to the 12 movements) can be performed as the 4x4 block unit is shifted from upper to lower. Therefore, as in the example described above, if the image data unit is 16x16 and transformations are performed for all 4x4 block units while shifting the 4x4 block unit by 1 pixel, a maximum of 169 (=13*13) transformations can be performed, and a maximum of 169 first transformation blocks can be generated for the 16x16 image data unit. If the boundary pixels of the image data are expanded through padding, etc., the number of transformations increases further.

[0109] If the size of a block unit of a predetermined size is NxN (N is a natural number), and the transformation is performed by moving the block unit in a zigzag direction by N pixel units, the block units of the predetermined size do not overlap, so the transformation can be performed at most once for all pixels within the image data unit. As described above, if it is determined that filtering is not applied to a block unit, the transformation process for the block unit can be skipped.

[0110] A predetermined pixel unit for moving a block unit of a predetermined size may be determined according to a rule commonly predetermined by the video encoder (100) and the video decoder (200), or the movement information of the block unit applied by the video encoder (100) may be included in the bitstream and signaled to the video decoder (200).

[0111] The transformation coefficient setting unit (320) can obtain a second transformation block by comparing the transformation coefficients within the first transformation block with a predetermined threshold value, and maintaining the transformation coefficients within the first transformation block that are higher than the predetermined threshold value and setting the transformation coefficients below the predetermined threshold value to a predetermined value. For example, the transformation coefficient setting unit (320) can set the transformation coefficients within at least one first transformation block that are lower than the predetermined threshold value to zero.

[0112] The predetermined threshold value may be a value predetermined in the video encoder (100) and the video decoder (200). In addition, the predetermined threshold value may be a value predetermined based on at least one of the size of the image data unit, the size of the block unit, the width and height ratio of the image data unit or the block unit, QP information of the image data unit or the block unit, luminance / chrominance information of the image data unit or the block unit, and prediction mode information of the image data unit or the block unit. The predetermined threshold value may be a value derived using one or more of the various additional information described above. The predetermined threshold value may be determined in the form of a table for each combination of one or more of the various additional information described above. The above-described additional information is exemplary, and the additional information used to derive the predetermined threshold value is not limited to the above-described examples.

[0113] In addition, predetermined threshold information used in the video encoder (100) may be included in the bitstream and explicitly transmitted to the video decoder (200). The video encoder (100) may transmit the predetermined threshold information based on the data unit to which the predetermined threshold information is applied. For example, the video encoder (100) may transmit the predetermined threshold information in at least one unit from among a sequence, a frame, a slice, a tile, a coding block, a prediction block, a transform block, a sub-block, a block unit, and a predetermined block unit specified by a user.

[0114] The inverse transform unit (330) can obtain at least one inverse transform block by inversely transforming at least one second transform block in which a transform coefficient less than a predetermined threshold value within a first transform block is set to a predetermined value. As described above, since a block unit of a predetermined size is moved in a zigzag direction within an image data unit in units of predetermined pixels, a maximum of (number of movements + 1) inverse transform blocks can be obtained.

[0115] The inverse transform unit (330) can store the inverse transform values ​​included in the inverse transform block by pixel position. Here, the pixel position can be determined based on the image data unit. For example, if the size of the image data unit is MxN (M, N are natural numbers), the pixel position within the MxN image data unit can be determined as (x, y) (x is an integer from 0 to M-1, y is an integer from 0 to N-1), and the corresponding inverse transform value can be stored for each (x, y) position. If filtering is applied to all overlapping block units, multiple inverse transform values ​​can be stored for pixel positions included in the overlapping positions of the block units.

[0116] The representative value determination unit (340) performs filtering by determining the pixel value of a pixel at a predetermined position in an image data unit using at least one of the pixel value and the inverse transformation value stored for each pixel position. One or more accumulated values ​​may be accumulated and stored for each pixel position in the image data unit. As described above, the pixel values ​​of pixels included in a block unit to which filtering is not applied are stored as original pixel values, and the inverse transformation value is stored for pixels included in a block unit to which filtering is applied. Therefore, depending on whether filtering is applied to the pixel position and block unit, only the original pixel value, the original pixel value and the inverse transformation values, or only the inverse transformation values ​​may be accumulated and stored for each pixel position in the image data unit.

[0117] The representative value determination unit (340) can perform filtering by determining the representative value of the accumulated value stored for each pixel position within the image data unit as the pixel value of the pixel at a predetermined position.

[0118] For example, the representative value determination unit (340) may determine one of the accumulated average, weighted average, Min value, Max value, and median for each pixel location as the representative value. The number of accumulated values ​​accumulated for a given pixel location is M (M is an integer), and each of the accumulated values ​​accumulated and stored for a given pixel location is IT. i When (i is an integer from 0 to (M-1)), the representative value determination unit (340) can obtain the average value of the accumulated values ​​according to the following mathematical expression 1.

[0119]

[0120] If a certain weight is applied to M accumulated values, α i When this is said, the representative value determination unit (340) can obtain a weighted average value according to the following mathematical expression 2.

[0121]

[0122] Representative value determination unit (340) Min value is IT i The minimum value among the values, the Max value is IT i It can be determined as the maximum value among the values. The representative value determination unit (340) is IT i The median can be determined by arranging the values ​​in order of size and selecting the value in the middle.

[0123] The representative value determination unit (340) can perform filtering by determining the corresponding representative value as the final pixel value for a pixel at a predetermined location within an image data unit.

[0124] Meanwhile, the representative value determination unit (340) can perform a division operation by deriving a numerator and a denominator in the process of calculating the average value according to mathematical expression 1 or the weighted average value according to mathematical expression 2. The representative value determination unit (340) can perform the operation according to mathematical expression 1 and mathematical expression 2 using a shift operation and a modular operation. In mathematical expressions 1 and 2, the values ​​of the numerator and denominator can both be determined as values ​​of the power of 2, and in this case, mathematical expressions 1 and 2 can be calculated through a shift operation. The shift operation and the modular operation used in the operation process can be tabulated and used in the video encoder (100) and the video decoder (200).

[0125] With reference to FIGS. 4A to 7B, specific examples of image processing operations according to one embodiment of the present disclosure are described.

[0126] FIGS. 4A to 4C are diagrams illustrating a process of moving a block unit of a predetermined size in which transformation, transformation coefficient setting, and inverse transformation are performed according to one embodiment of the present disclosure.

[0127] Referring to FIGS. 4A to 4C, the image processing device (300) determines whether to apply filtering to the block unit (410) of the predetermined size while moving the block unit (410) of the predetermined size in a zigzag direction from left to right and from top to bottom in the image data unit (400) in units of predetermined pixels, and if filtering is applied to the block unit (410), performs the aforementioned transformation, transformation coefficient setting, and inverse transformation operations on the pixels included in the block unit (410). For example, as illustrated in FIG. 4A, the image processing device (300) transforms, sets transformation coefficients, and inversely transforms the pixels included in the block unit (410) of the predetermined size including the upper left pixel of the image data unit (400) to obtain inverse transformation values ​​for the pixels included in the block unit (410) of the predetermined size. The image processing device (300) stores the obtained inverse transformation values ​​for each position of the pixels included in the block unit (410) of the predetermined size. As described above, if filtering is not applied to a block unit (410) of a predetermined size, the image processing device (300) stores the pixel value of each pixel included in the block unit (410) of the predetermined size by pixel position.

[0128] And, as illustrated in FIG. 4b, the image processing device (300) moves a block unit (410) of a predetermined size to the right by a predetermined pixel unit (dx), and again determines whether filtering is applied to the moved block unit (410) of the predetermined size, and if it is determined that filtering is applied to the moved block unit (410) of the predetermined size, the transformation, transformation coefficient setting, and inverse transformation processes are performed again for the pixels included in the moved block unit (410) of the predetermined size to obtain inverse transformation values ​​for the pixels included in the moved block unit (410) of the predetermined size. If filtering is not applied to the moved block unit (410) of the predetermined size in FIG. 4b, the image processing device (300) stores the pixel value of each pixel included in the block unit (410) of the predetermined size for each pixel position.

[0129] The image processing device (300) stores the inverse transformation values ​​obtained for each position of pixels included in a block unit (410) of a predetermined size to which filtering is applied. In this way, the process of determining whether filtering is applied to the moved block unit while moving the block unit (410) of the predetermined size to the right, and performing transformation, transformation coefficient setting, and inverse transformation on the pixels included in the moved block unit when it is determined that filtering is applied is performed each time the block unit (410) of the predetermined size is moved. Depending on whether filtering is applied, the original pixel values ​​or inverse transformation values ​​included in the moved block unit (410) of the predetermined size can be stored for each pixel position.

[0130] As illustrated in FIG. 4c, as a result of moving a block unit (410) of a predetermined size, if the right boundary of the block unit (410) of the predetermined size reaches the right boundary of the image data unit (400), the block unit (410) of the predetermined size is moved downward by a predetermined pixel unit (dy) while touching the left boundary of the image data unit (400) again to a block unit (420) of the predetermined size at a position. Thereafter, the image processing device (300) may repeatedly perform a judgment on whether the aforementioned filtering is applied, a conversion, a conversion coefficient setting, an inverse conversion, and the process while moving the block unit (420) of the predetermined size by a predetermined pixel unit (dx) again to the right.

[0131] A block unit (410, 420) of a predetermined size can be moved within a range that does not exceed the boundary of the image data unit (400). The image processing device (300) can repeatedly perform the process of determining whether filtering is applied, converting, setting conversion coefficients, and inverse conversion while moving the block unit in a zigzag direction until processing of the block unit including the lower right pixel of the image data unit (400) is completed.

[0132] Without being limited thereto, the image processing device (300) may perform padding on the boundary of the image data unit (400) using surrounding pixels and perform the aforementioned operation including the padded area. Various boundary extension methods may be applied as the padding method. For example, the boundary may be extended by copying the pixel values ​​of the boundary area outside the boundary.

[0133] FIGS. 5A to 5E are drawings showing pixels of image data units included in a block unit according to movement of the block unit, FIGS. 6A to 6B are drawings illustrating a process of setting a transformation coefficient within a transformation block by a transformation coefficient setting unit (320), and FIGS. 7A to 7B are drawings illustrating a correspondence relationship between pixels of an image data unit and pixels within an inverse transformation block output by an inverse transformation unit (330).

[0134] In FIGS. 5A to 5E, it is assumed that a block unit is a 4x4 block, an image data unit (500) is a 7x7 block, and a block unit moves in units of 1 pixel. The image processing device (300) moves 4x4 block units in a zigzag direction from left to right and from top to bottom within the image data unit (500) in units of 1 pixel, and determines whether the aforementioned filtering is applied to pixels included in the 4x4 block units, and if it is determined that filtering is applied, performs a transformation process for the corresponding block unit, sets a transformation coefficient, and performs an inverse transformation process. In addition, the image processing device (300) stores the original pixel values ​​for each pixel position for block units to which filtering is not applied, and if filtering is applied, stores the inverse transformation values ​​of the inverse transformation block in a buffer for each pixel position.

[0135] Referring to FIG. 5A, the determination unit (305) determines whether filtering is applied to a 4x4 block unit (510). If it is determined that filtering is not applied to the 4x4 block unit (510), the determination unit (305) stores the original pixel value for each pixel location. That is, it stores the value of R(x,y) for the (x,y) location. If the determination result of the determination unit (305) determines that filtering is applied to the 4x4 block unit (510), the transformation unit (310) performs transformation on 16 pixels (R(0,0) to R(3,3)) included in the 4x4 block unit (510) to obtain a first transformation block, and the transformation coefficient setting unit (320) sets the transformation coefficients in the first transformation block that are less than a predetermined threshold value to a predetermined value, for example, zero (0), to obtain a second transformation block having a size of 4x4.

[0136] Fig. 6a illustrates a first transform block output by the transform unit (310). The transform coefficient setting unit (320) compares the transform coefficients in the 4x4 first transform block with a predetermined threshold value, maintains the transform coefficients above the predetermined threshold value as is, and sets the transform coefficients below the predetermined threshold value to a predetermined value. For example, assume that the predetermined threshold value is 15 and the set value is 0. In this case, as in the second transform block illustrated in Fig. 6b, the transform coefficient setting unit (320) sets the transform coefficients in the first transform block having a value less than 15 to 0 to obtain the second transform block.

[0137] Referring back to FIG. 5A, the inverse transform unit (330) inversely transforms the second transform block to obtain an inverse transform block of size 4x4. When the transformation, transform coefficient setting, and inverse transform processes are performed for the 4x4 block unit (510), 16 inverse transform values ​​corresponding to each of the 16 pixels (R(0,0) to R(3,3)) included in the 4x4 block unit (510) can be obtained. The image processing device (300) stores the inverse transform values ​​obtained for the 4x4 block unit (510) in a buffer for each pixel position.

[0138] Referring to FIG. 7a, the inverse transform block (710) output through the transformation, transformation coefficient setting, and inverse transform processes for a 4x4 block unit (510) includes 16 inverse transform values ​​(IT1(0,0) to IT1(3,3)). The inverse transform value IT1(x,y) at the (x,y) position is the inverse transform value corresponding to the pixel at the (x,y) position included in the 4x4 block unit (510). In this way, the image processing device (300) can obtain 16 inverse transform values ​​(IT1(0,0) to IT1(3,3)) corresponding to each of the 16 pixels (R(0,0) to R(3,3)) included in the 4x4 block unit (510), and can store the inverse transform values ​​IT1(x,y) obtained for each (x,y) position in a buffer. That is, the image processing device (300) stores the value of IT1(x,y) corresponding to the R(x,y) pixel as the inverse transformation value for the pixel at the (x,y) location.

[0139] Referring to FIG. 5b, after the processing for the 4x4 block unit (510) illustrated in FIG. 5a is completed, the image processing device (300) moves the 4x4 block unit (510) by 1 pixel to the right as illustrated in FIG. 5b. The determination unit (305) determines whether filtering is applied to the moved 4x4 block unit (520). If it is determined that filtering is not applied to the 4x4 block unit (520), the determination unit (305) stores the original pixel value for each pixel position. That is, it stores the value of R(x,y) for the (x,y) position. If, as a result of the determination of the determination unit (305), it is determined that filtering is applied to the 4x4 block unit (520), the process of transforming, setting transform coefficients, inverse transforming, and storing the inverse transform values ​​for each position of pixels included in the block unit is performed. When the transformation, transformation coefficient setting, and inverse transformation processes are performed for a 4x4 block unit (520), 16 inverse transformation values ​​corresponding to each of the 16 pixels (R(1,0) to R(4,3)) included in the 4x4 block unit (520) can be obtained. The image processing device (300) stores the inverse transformation values ​​obtained for the 4x4 block unit (520) in a buffer for each pixel position.

[0140] Referring to FIG. 7b, the inverse transform block (720) output through the transformation, transformation coefficient setting, and inverse transform processes for a 4x4 block unit (520) includes 16 inverse transform values ​​(IT2(1,0) to IT2(4,3)). The inverse transform value IT2(x,y) at the (x,y) position is the inverse transform value corresponding to the pixel at the (x,y) position included in the 4x4 block unit (520). In this way, the image processing device (300) can obtain 16 inverse transform values ​​(IT2(1,0) to IT2(4,3)) corresponding to each of the 16 pixels (R(1,0) to R(4,3)) included in the 4x4 block unit (520), and can store the inverse transform values ​​IT2(x,y) obtained for each (x,y) position. That is, the image processing device (300) stores the value of IT2(x,y) corresponding to the R(x,y) pixel as the inverse transformation value for the pixel at the (x,y) location.

[0141] For pixels existing at overlapping positions of 4x4 block units (510) and 4x4 block units (520), two accumulated values ​​can be obtained. As described above, pixel values ​​of pixels included in block units to which filtering is not applied are stored as original pixel values, and inverse transformed values ​​are stored for pixels included in block units to which filtering is applied. Therefore, depending on whether filtering is applied to the pixel position and block unit, only the original pixel value, the original pixel value and inverse transformed values, or only the inverse transformed values ​​can be accumulated and stored at each pixel position of the image data unit.

[0142] For example, assuming that processing is completed up to the block units (510, 520) illustrated in FIGS. 5A and 5B and that filtering is applied to all of the block units (510, 520), IT1(0,0), IT1(0,1), IT1(0,2), IT1(0,3), IT2(5,0), IT2(5,1), IT2(5,2), IT2(5,3) are stored for each pixel position of (0,0), (0,1), (0,2), (0,3), (5,0), (5,1), (5,2), (5,3) where the block units (510, 520) do not overlap. For each of the overlapping locations of block units (510, 520), (1,0), (1,1), (1,2), (1,3), (2,0), (2,1), (2,2), (2,3), (3,0), (3,1), (3,2), (3,3), two inverse transform values ​​are stored. For example, for the pixel location (1,0), two inverse transform values ​​of {IT1(1,0), IT2(1,0)} are stored in the buffer.

[0143] As another example, assuming that filtering is not applied to all of the block units (510, 520) illustrated in FIGS. 5A and 5B, the original pixel values ​​R(0,0), R(0,1), R(0,2), R(0,3), R(5,0), R(5,1), R(5,2), R(5,3) are stored for each pixel position of (0,0), (0,1), (0,2), (0,3), (5,0), (5,1), (5,2), (5,3) where the block units (510, 520) do not overlap. For each of the overlapping locations of block units (510, 520), (1,0), (1,1), (1,2), (1,3), (2,0), (2,1), (2,2), (2,3), (3,0), (3,1), (3,2), (3,3), two original pixel values ​​are accumulated and stored in the buffer. For example, for the pixel location (1,0), two original pixel values ​​of {R(1,0), R(1,0)} are accumulated and stored in the buffer.

[0144] As another example, assuming that filtering is applied to the block unit (510) illustrated in FIG. 5a to generate an inverse transformation block and that filtering is not applied to the block unit (520) illustrated in FIG. 5b, among the pixel positions (0,0), (0,1), (0,2), (0,3), (5,0), (5,1), (5,2), (5,3) that are not overlapping pixel positions of the block units (510, 520), IT1(0,0), IT1(0,1), IT1(0,2), IT1(0,3) are stored in the buffer for each pixel position (0,0), (0,1), (0,2), (0,3) included in the block unit (510) to which filtering is applied, and R(5,0), R(5,1), IT1(0,2), IT1(0,3) are stored for each pixel position (5,0), (5,1), (5,2), (5,3) included in the block unit (520) to which filtering is not applied. R(5,2) and R(5,3) are stored in the buffer respectively. For the locations (1,0), (1,1), (1,2), (1,3), (2,0), (2,1), (2,2), (2,3), (3,0), (3,1), (3,2), (3,3), where the block units (510, 520) overlap, the original pixel values ​​and the inverse transformed values ​​are accumulated and stored in the buffer respectively. For example, for the pixel location (1,0), two accumulated values ​​of {IT1(1,0), R(1,0)} are accumulated and stored in the buffer.

[0145] As another example, assuming that filtering is not applied to the block unit (510) illustrated in FIG. 5a, and filtering is applied to the block unit (520) illustrated in FIG. 5b, among the pixel positions (0,0), (0,1), (0,2), (0,3), (5,0), (5,1), (5,2), (5,3) where the block units (510, 520) do not overlap, the original pixel values ​​R(0,0), R(0,1), R(0,2), R(0,3) for each pixel position (0,0), (0,1), (0,2), (0,3) included in the block unit (510) to which filtering is not applied are stored in the buffer, and the inverse values ​​IT2(5,0), IT2(5,1), IT2(5,0), IT2(5,1), IT2(5,1), IT2(5,0), IT2(5,1), IT2(5,1), IT2(5,0), IT2(5,1), IT2(5,1), IT2(5,2), IT2(5,3 ...3), IT2(5,0), IT2(5,1), IT2(5,2), IT2(5,3), IT2(5,3), IT2(5,3), IT2(5,3), IT2(5,3), IT2(5,3), IT2(5,3), IT2 IT2(5,2) and IT2(5,3) are stored in the buffer respectively. For the locations (1,0), (1,1), (1,2), (1,3), (2,0), (2,1), (2,2), (2,3), (3,0), (3,1), (3,2), (3,3), where the block units (510, 520) overlap, the inverse transformation value and the original pixel value are accumulated and stored in the buffer respectively. For example, for the pixel location (1,0), two accumulated values ​​of {R(1,0), IT2(1,0)} are accumulated and stored in the buffer.

[0146] The process of determining whether filtering is applied, setting the conversion coefficient, and performing the inverse conversion process for the corresponding block unit when it is determined that filtering has been applied, as described with reference to the aforementioned FIGS. 5a and 5b, can be similarly performed for the block units (530, 540, 550) illustrated in FIGS. 5c to 5e.

[0147] Referring to FIG. 5c, after the processing process for the 4x4 block unit (520) illustrated in FIG. 5b is completed, the image processing device (300) moves the 4x4 block unit (520) by 1 pixel to the right as illustrated in FIG. 5c, and the determination unit (305) determines whether filtering is applied to the moved 4x4 block unit (530). If it is determined that filtering is not applied to the 4x4 block unit (530), the determination unit (305) stores the original pixel value for each pixel position. If, as a result of the determination of the determination unit (305), it is determined that filtering is applied to the 4x4 block unit (530), the process of transforming again, setting the transformation coefficient, performing inverse transformation, and storing the inverse transformation value for each position of the pixels included in the block unit is performed for the pixels included in the 4x4 block unit (530).

[0148] Also, referring to FIG. 5d, after the processing process for the 4x4 block unit (530) illustrated in FIG. 5c is completed, the image processing device (300) moves the 4x4 block unit (530) by 1 pixel to the right as illustrated in FIG. 5d, and the determination unit (305) determines whether filtering is applied to the moved 4x4 block unit (540). If it is determined that filtering is not applied to the 4x4 block unit (540), the determination unit (305) stores the original pixel value for each pixel position. If, as a result of the determination of the determination unit (305), it is determined that filtering is applied to the 4x4 block unit (540), the process of transforming again, setting the transformation coefficient, performing inverse transformation, and storing the inverse transformation value for each position of the pixels included in the block unit is performed for the pixels included in the 4x4 block unit (540).

[0149] In addition, referring to FIG. 5e, after the image processing device (300) completes the processing process for the 4x4 block unit (540) that touches the right border of the image data unit (500) illustrated in FIG. 5d, the 4x4 block unit (540) is moved downward by 1 pixel based on the upper left pixel of R(0,0) as illustrated in FIG. 5e, and the determination unit (305) determines whether filtering is applied to the moved 4x4 block unit (550), and if it is determined that filtering is not applied to the 4x4 block unit (550), the determination unit (305) stores the original pixel value for each pixel position. If, as a result of the judgment of the judgment unit (305), it is determined that filtering is applied to a 4x4 block unit (550), a process of re-converting, setting the conversion coefficient, inverse transforming, and storing the inverse transform values ​​for each position of the pixels included in the block unit is performed for the pixels included in the 4x4 block unit (550).

[0150] The image processing device (300) can repeatedly perform the process of determining whether to apply filtering while moving the 4x4 block unit in a zigzag direction until the processing for the 4x4 block unit including the lower right pixel R(6,6) of the image data unit (500) is completed, and performing transformation, setting transformation coefficients, inverse transformation, and storing the inverse transformation value for each position of the pixels included in the block unit for the block unit to which filtering is applied.

[0151] As described above, since a block unit of a predetermined size is moved in a zigzag direction in units of predetermined pixels within an image data unit, a plurality of inverse transformation blocks can be obtained at most (the number of times a block unit of a predetermined size is moved + 1). In addition, depending on the pixel position within the image data unit, a plurality of inverse transformation values ​​can be obtained for pixels existing at positions where the moved block units overlap.

[0152] The representative value determination unit (340) can perform filtering by determining the representative value obtained by using the accumulated values ​​accumulated for a predetermined pixel position as the pixel value of the pixel at the predetermined position. As described above, the representative value determination unit (340) can determine one of the average value, the weighted average value, the Min value, the Max value, and the median value of the accumulated values ​​accumulated for the pixel position as the representative value. For example, if two inverse transformation values ​​of IT1 (1,0) and IT2 (1,0) are accumulated and stored for R (1,0), the representative value determination unit (340) determines one of the average value, the weighted average value, the Min value, and the Max value of the two inverse transformation values ​​of IT1 (1,0) and IT2 (1,0) as the representative value, and finally determines the determined representative value as the pixel value of R (1,0), thereby performing filtering for the R (1,0) pixel. Such a filtering process can be performed for all pixels within an image data unit.

[0153] FIG. 8 shows a flowchart of an image processing method according to one embodiment of the present disclosure.

[0154] Referring to FIG. 8, the judgment unit (305) determines whether filtering is applied to a block unit of a predetermined size within an image data unit (S801).

[0155] The judgment unit (305) stores pixel values ​​of pixels in a block unit that are judged not to be filtered, by pixel location (S805).

[0156] If it is determined by the judgment result of the judgment unit (305) that filtering is to be applied to the current block unit, the conversion unit (310) can obtain a first conversion block by converting pixels included in the current block unit to which filtering is applied (S810). The conversion unit (310) can obtain at least one first conversion block by starting the conversion for a block unit of a predetermined size including the upper left pixel of the image data unit, and converting pixels included in the block unit of a predetermined size to which filtering is applied while moving the block unit of the predetermined size in a zigzag direction from left to right and from top to bottom. The conversion unit (310) can perform the conversion for the block unit to which filtering is applied while moving the block unit of the predetermined size until the processing for the block unit including the lower right pixel of the image data unit is completed.

[0157] The transformation coefficient setting unit (320) can obtain at least one second transformation block by comparing the transformation coefficients within the first transformation block with a predetermined threshold value, and maintaining the transformation coefficients within the transformation block that are higher than the predetermined threshold value and setting the transformation coefficients below the predetermined threshold value to a predetermined value (S820). For example, the transformation coefficient setting unit (320) can set the transformation coefficients within the at least one first transformation block that are lower than the predetermined threshold value to zero.

[0158] The inverse transform unit (330) can obtain at least one inverse transform block by inversely transforming at least one second transform block in which a transform coefficient less than a predetermined threshold value is set to a predetermined value (S830).

[0159] The representative value determination unit (340) performs filtering by determining the pixel value of a pixel at a predetermined position in an image data unit using at least one of the pixel values ​​and inverse transformation values ​​stored for each pixel position (S840). One or more accumulated values ​​may be accumulated and stored for each pixel position in an image data unit. The representative value determination unit (340) may perform filtering by determining the representative value of the accumulated values ​​stored for each pixel position in an image data unit as the pixel value of the pixel at the predetermined position. The representative value determination unit (340) may determine one of the average value, the weighted average value, the Min value, the Max value, and the median value of the accumulations for each pixel position as the representative value.

[0160] The methods described herein may be performed by a processor of a video encoder or video decoder. Furthermore, the encoder may generate a bitstream that is decoded by a video signal processing method, and the bitstream generated by the encoder may be stored in a computer-readable, non-transitory storage medium (recording medium).

[0161] The embodiments of the present invention described above may be implemented through various means. For example, the embodiments of the present invention may be implemented using hardware, firmware, software, or a combination thereof.

[0162] Some embodiments may also be implemented in the form of a computer-executable storage medium containing computer-executable instructions, such as program modules executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media.

[0163] Additionally, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes other data, such as computer-readable instructions, data structures, or program modules, in a modulated data signal, or other transport mechanism, and includes any information delivery media.

[0164] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be implemented by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. In the image processing method, A step of determining whether filtering is applied to a block unit of a predetermined size within an image data unit; A step of storing pixel values ​​of pixels included in a block unit to which filtering is not applied based on the above judgment result by pixel position within the image data unit; A step of obtaining at least one first transformation block by transforming pixels included in a block unit to which filtering is applied based on the above judgment result; A step of obtaining at least one second transform block by maintaining transform coefficients higher than a predetermined threshold among transform coefficients in at least one first transform block and setting transform coefficients lower than the predetermined threshold to a predetermined value; A step of obtaining at least one inverse transform block by inversely transforming at least one second transform block and storing inverse transform values ​​included in the inverse transform block for each pixel position within the image data unit; and An image processing method comprising a step of performing filtering by determining a pixel value for each pixel position of the image data unit using at least one of the pixel values ​​stored for each pixel position and the inverse transformation value.

2. In paragraph 1, The step to determine whether the above filtering is applied is An image processing method, which starts with a block unit including an upper left pixel within the above image data unit, and determines whether the filtering is applied to each block unit while moving the block unit in a zigzag direction from left to right and from top to bottom in a predetermined pixel unit.

3. In paragraph 2, An image processing method, characterized in that when the size of the block unit of the above predetermined size is NxN (N is a natural number), the above predetermined pixel unit is one of 1 to N pixel units.

4. In paragraph 1, The step of obtaining at least one second transformation block is An image processing method for replacing transform coefficients in at least one of the first transform blocks with zeros that are less than a predetermined threshold value.

5. In paragraph 1, The steps to perform the above filtering are An image processing method for determining a representative value of an accumulated value including at least one of the pixel value and the inverse transformation value stored for each pixel position within the image data unit as the pixel value of the pixel position.

6. In paragraph 5, The above representative value is An image processing method, wherein the image is determined using one of the average, weighted average, Min value, Max value, and median of accumulated values ​​for the above pixel locations.

7. In paragraph 6, The number of accumulated values ​​for the above pixel location is M (M is an integer), and each of the accumulated values ​​accumulated and stored for the above pixel location is IT i (where i is an integer from 0 to (M-1)), The above average value is and, A predetermined weight applied to the above M accumulated values ​​is α i When you say, The above weighted average is and, The above Min value is the IT i The minimum value among the values, the Max value is the IT i A method of filtering an image that is the maximum value among the values.

8. In paragraph 1, A method for filtering an image, characterized in that the above filtering is performed on one of a restored value obtained by adding a residual value and a predicted value, a residual value, and a filtered data value after at least one other filtering method is applied.

9. In paragraph 1, An image processing method, characterized in that the image data unit is one of a coding block, a prediction block, a transformation block, a sub-block, and a user-specified data unit.

10. In paragraph 1, The step to determine whether the above filtering is applied is Whether to apply filtering to a block unit is determined based on information on whether to apply filtering to each block unit included in the bitstream, or An image processing method for determining whether to apply filtering to a block unit by using at least one of the size of the image data unit, the size of the block unit, the horizontal and vertical ratio of the image data unit, the horizontal and vertical ratio of the block unit, QP information of the image data unit, QP information of the block unit, luminance / chrominance channel information of the image data unit, luminance / chrominance channel information of the block unit, prediction mode information of the image data unit, and prediction mode information of the block unit.

11. In paragraph 1, The step to determine whether the above filtering is applied is Further comprising a step of determining whether to apply filtering to the above image data unit, An image processing method characterized in that, when it is determined that filtering is applied to the above image data unit, it is further determined whether filtering is applied to a block unit of a predetermined size within the image data unit.

12. In paragraph 11, An image processing method, characterized in that whether to apply filtering to the above image data unit is explicitly or implicitly determined for one of a sequence, frame, slice, tile, coding block, prediction block, transform block, and sub-block unit.

13. In paragraph 12, If it is implicitly determined whether filtering will be applied to the above image data unit, An image processing method for determining whether to apply filtering to an image data unit based on at least one of the number of zero or non-zero coefficients of a quantized coefficient, a Min or Max value of the quantized coefficient, position information of the zero or non-zero coefficients of the quantized coefficient, position information of the first non-zero transform coefficient according to the scan order among the transform coefficients in the transform block, size information of the image data unit, width and height ratio information of the image data unit, QP (Quantization Parameter) information applied to the image data unit, luminance / chrominance channel information of the image data unit, and prediction mode information of the image data unit.

14. In paragraph 1, The above threshold value is An image processing method, characterized in that the value is predetermined based on at least one of the size of the image data unit, the size of the block unit, the width and height ratio of the image data unit or the block unit, QP information of the image data unit or the block unit, luminance / chrominance information of the image data unit or the block unit, and prediction mode information of the image data unit or the block unit.

15. In paragraph 1, An image processing method, characterized in that the above transformation is performed using at least one of DCT2, DCT5, DCT8, DST1, DST4, DS7, Hadamard transformation, and wavelet transformation.

16. In the image processing device, A judgment unit that determines whether filtering is applied to a block unit of a predetermined size within an image data unit and stores pixel values ​​of pixels included in a block unit to which filtering is not applied by pixel position within the image data unit; A transformation unit that obtains at least one first transformation block by transforming pixels included in a block unit to which filtering is applied as a result of the judgment of the above judgment unit; A transform coefficient setting unit that obtains at least one second transform block by maintaining transform coefficients higher than a predetermined threshold value among transform coefficients in at least one first transform block and setting transform coefficients lower than the predetermined threshold value to a predetermined value; An inverse transform unit that obtains at least one inverse transform block by inversely transforming at least one second transform block and stores inverse transform values ​​included in the inverse transform block for each pixel position within the image data unit; and An image processing device including a representative value determination unit that performs filtering by determining a pixel value for each pixel position of the image data unit using at least one of the pixel values ​​stored for each pixel position and the inverse transformation value.

17. A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 15 on a computer.

Images