Encoder, decoder and corresponding method for adaptive de-blocking filtering

Abstract

The present application provides a decoding method, wherein the decoding comprises decoding or encoding, and the method comprises: determining whether to use combined inter-intra prediction (CIIP) to predict a current coding unit; and setting a boundary strength of a boundary of the current coding unit to a first value after determining to use CIIP to predict the current coding unit.

Description

[0001] This application is a divisional application. The original application has the application number 202080005810.X and the original application date is January 16, 2020. The entire contents of the original application are incorporated herein by reference.

[0002] Cross-referencing related applications

[0003] This application claims priority to U.S. Provisional Application 62 / 793,840, filed January 17, 2019, with the United States Patent and Trademark Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0004] The embodiments of this application generally relate to the field of image processing, and more specifically to encoders, decoders and corresponding methods for adaptive deblocking filtering. Background Technology

[0005] Video decoding (video encoding and video decoding) is widely used in digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat and video conferencing, DVD and Blu-ray discs, video content capture and editing systems, and security applications for portable cameras.

[0006] Even with shorter videos, a large amount of video data needs to be described, which can be challenging when the data needs to be transmitted over bandwidth-constrained communication networks or otherwise. Therefore, video data is typically compressed before transmission over modern telecommunications networks. Video size can also be an issue when storing video on storage devices due to potentially limited memory resources. Video compression devices typically use software and / or hardware at the source side to decode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device used to decode the video data. Given limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression techniques that can increase compression ratios with minimal impact on image quality. Summary of the Invention

[0007] This application provides the encoding and decoding methods and apparatus described in the independent claims.

[0008] The above and other objectives are achieved through the subject matter claimed in the independent claims. Other implementations are presented in the dependent claims, the specification, and the drawings.

[0009] One embodiment of the present invention is a decoding method, wherein the decoding includes decoding or encoding, and the method includes: determining whether the current decoding unit uses combined inter-intra-frame prediction (CIIP); and after determining that the current decoding unit uses CIIP for prediction, setting the boundary strength of the boundary of the current decoding unit to a first value.

[0010] The first value can be in the range of 1 to 2. Specifically, the first value can be 2. Alternatively, the first value can be 1. In the latter case, the method may further include: incrementing the first value by 1 when one of the following conditions is met:

[0011] - At least one of the current decoding unit and the adjacent decoding unit adjacent to the boundary of the current decoding unit has non-zero transformation coefficients;

[0012] - The absolute difference between the motion vectors of the current decoding unit and the adjacent decoding unit is greater than or equal to an integer sample;

[0013] - The current decoding unit and the adjacent decoding unit are predicted based on different reference images;

[0014] - The number of motion vectors used to predict the current decoding unit and the adjacent decoding units are different.

[0015] The method may further include: when the boundary of the current decoding unit is a horizontal edge, determining whether the adjacent decoding units adjacent to the boundary of the current decoding unit are in different coding tree units (CTUs).

[0016] The method may further include setting the boundary strength of the boundary of a sub-decoding unit to a second value, wherein the current decoding unit includes at least two sub-decoding units, and the boundary of the sub-decoding unit is the boundary between the at least two sub-decoding units. Specifically, the second value may be 1. When the boundary of the sub-decoding unit is an edge of a transform unit, the second value may be equal to the first value. When the boundary of the sub-decoding unit is not an edge of a transform unit, the second value may be different from the first value.

[0017] In the above embodiments, the method may further include: determining whether the boundary of the current decoding unit is aligned with an 8×8 grid; and after determining that the boundary of the current decoding unit is not aligned with the 8×8 grid, setting the boundary strength of the boundary of the current decoding unit to 0.

[0018] The method may further include: determining whether the boundary of the sub-decoding unit is aligned with a sub-grid, wherein the sub-grid is a 4×4 grid or an 8×8 grid; and after determining that the boundary of the sub-decoding unit is not aligned with the sub-grid, setting the boundary strength of the boundary of the sub-decoding unit to 0.

[0019] In the above embodiments, the method may further include: deblocking the boundary of the luminance component when the boundary strength is greater than 0. The method may further include: deblocking the boundary of the chrominance component when the boundary strength is greater than 1.

[0020] In the above embodiments, when CIIP is used to predict the current decoding unit, the current decoding unit can be regarded as a decoding unit using intra-frame prediction during deblocking.

[0021] Another embodiment of the present invention is an encoder, including processing circuitry for performing the methods provided in any of the above embodiments.

[0022] Another embodiment of the present invention is a decoder, including processing circuitry for performing the methods provided in any of the above embodiments.

[0023] Another embodiment of the present invention is a computer program product including instructions, wherein when a computer executes the program, the computer performs the method provided in any of the above embodiments.

[0024] Another embodiment of the present invention is a decoder, comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing instructions executed by the one or more processors, wherein, when the one or more processors execute the instructions, the decoder performs the method provided in any of the above embodiments.

[0025] Another embodiment of the present invention is an encoder, comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing instructions executed by the one or more processors, wherein, when the one or more processors execute the instructions, the encoder performs the method provided in any of the above embodiments.

[0026] The present invention also provides the following aspects, starting from one.

[0027] According to a first aspect, the present invention relates to a decoding method, wherein the decoding includes decoding or encoding, the method comprising: determining whether a current decoding unit (or decoding block) uses combined inter-frame-intra-frame prediction; when the current decoding unit uses combined inter-frame-intra-frame prediction,

[0028] Set the boundary strength (Bs) of the current decoding unit to a first value;

[0029] Set the boundary strength (Bs) of the boundary of the sub-decoding unit (or sub-block, or sub-part) to a second value, wherein the current decoding unit includes at least two sub-decoding units, and the boundary of the sub-decoding unit is the boundary between the at least two sub-decoding units.

[0030] The first value can be 2. The second value can be 1. The first value can be the same as or different from the second value. When the boundary of the sub-decoding unit is the boundary (or edge) of the transform unit, the first value can be the same as the second value. When the boundary of the sub-decoding unit is not the boundary (or edge) of the transform unit, the first value can be different from the second value.

[0031] The method may further include: performing deblocking when the Bs value of the luminance component is greater than 0; or performing deblocking when the Bs value of the chrominance component is greater than 1, wherein the Bs value is one of the first value or the second value.

[0032] When the current decoding unit (or block) is predicted using combined inter-frame-intra-frame prediction, the current decoding unit can be considered as a unit using intra-frame prediction during deblocking.

[0033] According to a second aspect, the present invention relates to an encoder including processing circuitry for performing any of the methods provided in the first aspect.

[0034] According to a third aspect, the present invention relates to a decoder including processing circuitry for performing any of the methods provided in the first aspect.

[0035] According to a fourth aspect, the present invention relates to a computer program product comprising program code for performing any of the methods provided in the first aspect.

[0036] According to a fifth aspect, the present invention relates to a decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processors and storing a program executed by the processors, wherein, when the processors execute the program, the decoder performs the method provided in the first aspect.

[0037] According to a sixth aspect, the present invention relates to an encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processors and storing a program executed by the processors, wherein, when the processors execute the program, the encoder performs the method provided in the first aspect.

[0038] According to a seventh aspect, the present invention relates to a decoding method, wherein the decoding includes decoding or encoding, the decoding method comprising: determining whether at least one of two blocks is a block predicted using CIIP (or MH), wherein the two blocks include a first block (block Q) and a second block (block P), and the two blocks are associated with a boundary; when at least one of the two blocks is a block predicted using CIIP, setting the boundary strength (Bs) of the boundary to a first value; and when neither of the two blocks is a block predicted using CIIP, setting the boundary strength (Bs) of the boundary to a second value.

[0039] The method provided in the seventh aspect of the present invention can be executed by the apparatus provided in the eighth aspect of the present invention. Other features and implementations of the apparatus provided in the eighth aspect of the present invention correspond to the features and implementations of the method provided in the seventh aspect of the present invention.

[0040] According to a ninth aspect, the present invention relates to an apparatus for decoding a video stream, comprising a processor and a memory. The memory stores instructions that cause the processor to perform the method provided in the seventh aspect.

[0041] According to a tenth aspect, the present invention relates to an apparatus for encoding a video stream, comprising a processor and a memory. The memory stores instructions that cause the processor to perform the method provided in the seventh aspect.

[0042] According to an eleventh aspect, a computer-readable storage medium is provided storing instructions that, when executed, cause one or more processors to decode video data. The instructions cause the one or more processors to perform the method provided in the seventh aspect or any possible embodiment of the seventh aspect.

[0043] According to the twelfth aspect, the present invention relates to a computer program including program code, which, when run in a computer, performs the method provided in the seventh aspect or any possible embodiment of the seventh aspect.

[0044] The accompanying drawings and the following description illustrate details of one or more embodiments. Other features, objects, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0045] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and illustrations. In the drawings:

[0046] Figure 1AA block diagram illustrating an example of a video decoding system for implementing embodiments of the present invention;

[0047] Figure 1B A block diagram of another example of a video decoding system for implementing embodiments of the present invention;

[0048] Figure 2 A block diagram illustrating an example of a video encoder used to implement embodiments of the present invention;

[0049] Figure 3 This is a block diagram of an exemplary structure for implementing a video decoder according to an embodiment of the present invention;

[0050] Figure 4 A block diagram of an example encoding or decoding device;

[0051] Figure 5 A block diagram of another example of an encoding or decoding device;

[0052] Figure 6 A diagram showing the boundaries of exemplary sub-blocks of a decoding unit;

[0053] Figure 7 Another diagram showing the exemplary sub-block boundary of a decoding unit;

[0054] Figure 8 A diagram illustrating the division of the decoding unit into four transformation units;

[0055] Figure 9 A diagram illustrating the division of a CIIP block into multiple transformation units;

[0056] Figure 10 An example of using a deblocking filter for samples in a sub-part;

[0057] Figure 11 A graph representing the boundary neighboring blocks;

[0058] Figure 12 A flowchart illustrating the derivation of the boundary strength of a boundary provided in one embodiment of the present invention;

[0059] Figure 13 A flowchart for deriving the boundary strength of the boundary provided by the prior art;

[0060] Figure 14 A flowchart for deriving the boundary strength of a boundary provided in another embodiment of the present invention;

[0061] Figure 15 A flowchart illustrating the derivation of the boundary strength of a boundary provided in yet another embodiment of the present invention;

[0062] Figure 16A graph showing the edges of sub-blocks within the decoding unit relative to the 8×8 sample grid starting from the top left sample of the CU;

[0063] Figure 17 A graph showing the edges of sub-blocks within the decoding unit relative to an 8×8 sample grid that does not start from the top left sample of the CU;

[0064] Figure 18 This is a graph showing the edges of sub-blocks within the decoding unit relative to the 4×4 sample grid.

[0065] In the following text, unless otherwise expressly stated, the same reference numerals refer to the same or at least functionally equivalent features. Detailed Implementation

[0066] In the following description, reference is made to the accompanying drawings, which form part of this invention and illustrate specific aspects of embodiments of the invention or from which specific aspects of embodiments of the invention may be used. It should be understood that embodiments of the invention may be used in other aspects and may include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the invention is defined by the appended claims.

[0067] For example, it should be understood that the disclosure of the described method is equally applicable to corresponding devices or systems for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more of multiple steps respectively), even if the one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific apparatus is described according to one or more units (e.g., functional units), the corresponding method may include a step to implement the function of one or more units (e.g., one step implements the function of one or more units, or multiple steps implement the function of one or more of multiple units respectively), even if the one or more steps are not explicitly described or illustrated in the drawings. Furthermore, it should be understood that, unless otherwise stated, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0068] Video decoding generally refers to the processing of a sequence of images that constitute a video or video sequence. In the field of video decoding, the terms "frame" and "picture / image" can be used synonymously. Video decoding (or generally referred to as decoding) consists of two parts: video encoding and video decoding. Video encoding is performed on the source side and typically involves processing (e.g., compressing) the raw video image to reduce the amount of data required to represent the video image (thus enabling more efficient storage and / or transmission). Video decoding is performed on the destination side and typically involves inverse processing relative to the encoder to reconstruct the video image. The "decoding" of the video image (or generally referred to as an image) involved in the embodiments should be understood as involving the "encoding" or "decoding" of the video image or corresponding video sequence. The combination of the encoding and decoding parts is also called encoding and decoding (CODEC).

[0069] In lossless video decoding, the original video image can be reconstructed, meaning the reconstructed video image has the same quality as the original (assuming no transmission loss or other data loss occurs during storage or transmission). In lossy video decoding, further compression, such as quantization, is used to reduce the amount of data required to represent the video image. However, the decoder cannot completely reconstruct the video image, meaning the quality of the reconstructed video image is lower or worse than the original video image.

[0070] Several video decoding standards belong to the "lossy hybrid video codec" group (i.e., combining spatial and temporal prediction in the sample domain with 2D transform decoding in the transform domain for applying quantization). Each image in a video sequence is typically divided into a set of non-overlapping blocks, and decoding is usually performed at the block level. In other words, the encoder typically processes the video at the block (video block) level, i.e., encoding, for example, generating prediction blocks through spatial (intra-frame) prediction and / or temporal (inter-frame) prediction; subtracting the prediction blocks from the current block (the block currently being processed / to be processed) to obtain residual blocks; transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be sent (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Furthermore, the encoder repeats the decoding processing steps, such that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructions for processing (i.e., decoding) subsequent blocks.

[0071] In the following embodiments, according to Figures 1A to 3 The video decoding system 10, video encoder 20, and video decoder 30 are described.

[0072] Figure 1AFor illustrative purposes, an exemplary decoding system 10 is shown, such as a video decoding system 10 (or simply decoding system 10) that can utilize the techniques of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent examples of devices that can be used to perform various techniques according to the various examples described in this application.

[0073] like Figure 1A As shown, the decoding system 10 includes a source device 12, for example, the source device 12 is used to provide encoded image data 21 to the destination device 14 for decoding encoded image data 13.

[0074] The source device 12 includes an encoder 20 and may additionally (optionally) include an image source 16, a preprocessor (or preprocessing unit) 18 (e.g., an image preprocessor 18), and a communication interface or communication unit 22.

[0075] Image source 16 may include or may be any type of image capture device, such as a camera for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any other type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images) and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory / storage for storing any of the aforementioned images.

[0076] To distinguish between the processing performed by the preprocessor 18 and the preprocessing unit 18, the image or image data 17 may also be referred to as the raw image or raw image data 17.

[0077] The preprocessor 18 can be used to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0078] Video encoder 20 can be used to receive preprocessed image data 19 and provide encoded image data 21 (e.g., hereinafter referred to as...). Figure 2 (Further details)

[0079] The communication interface 22 in the source device 12 can be used to receive encoded image data 21 and send the encoded image data 21 (or any other processed version thereof) to another device (e.g., destination device 14) or any other device via the communication channel 13 for storage or direct reconstruction.

[0080] Destination device 14 includes decoder 30 (e.g., video decoder 30) and may additionally (i.e., optionally) include communication interface or communication unit 28, post-processor 32 (or post-processing unit 32) and display device 34.

[0081] The communication interface 28 of the destination device 14 can be used to receive encoded image data 21 (or any other processed version thereof), for example, directly from the source device 12 or any other source (e.g., a storage device such as an encoded image data storage device), and provide the encoded image data 21 to the decoder 30.

[0082] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14, or via any type of network (e.g., wired or wireless networks or any combination thereof, or any type of private and public network), or any combination thereof.

[0083] The communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a data packet, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it over a communication link or communication network.

[0084] The communication interface 28 corresponding to the communication interface 22 can be used to receive transmitted data and process the transmitted data through any type of corresponding transmission decoding or processing and / or decapsulation method to obtain encoded image data 21.

[0085] Both communication interface 22 and communication interface 28 can be configured as unidirectional communication interfaces (e.g., Figure 1A The communication channel 13 (indicated by the arrow pointing from source device 12 to destination device 14) or a bidirectional communication interface can be used to send and receive messages, such as establishing connections, acknowledging and interacting with any other information related to the communication link and / or data transmission (such as encoded image data transmission).

[0086] Decoder 30 can be used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (e.g., according to the following description). Figure 3 or Figure 5 (Further details)

[0087] The post-processor 32 of the destination device 14 can be used to post-process the decoded image data 31 (also known as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (such as post-processed image 33). The post-processing performed by the post-processing unit 32 may include one or more of color format conversion (e.g., from YCbCr to RGB), color correction, trimming, or resampling, or any other processing, such as preparing the decoded image data 31 for display by a display device 34, etc.

[0088] The display device 34 of the destination device 14 can be used to receive post-processed image data 33 to display the image to a user or viewer. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display or monitor. The display can be a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0089] although Figure 1A The source device 12 and destination device 14 are shown as separate devices. However, in embodiments, the device may also include the functions of both the source device 12 and the destination device 14, i.e., the source device 12 or its corresponding function and the destination device 14 or its corresponding function. In such embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0090] Based on the description, it is obvious to the technicians that... Figure 1A The presence and (precise) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0091] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30 can be transmitted via, for example, Figure 1BThe processing circuitry shown can be implemented using one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video decoding processors, or any combination thereof. Encoder 20 can be implemented by processing circuitry 46 to reflect the combination... Figure 2 The encoder 20 describes various modules and / or any other encoder system or subsystem described herein. The decoder 30 may be implemented by processing circuitry 46 to embody the combination of... Figure 3 The decoder 30 describes various modules and / or any other decoder system or subsystem described herein. The processing circuitry can be used to perform the various operations described below. Figure 5 As shown, if the technology is partially implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable medium, and can use one or more processors to execute the instructions in the hardware to perform the technology of the present invention. The video encoder 20 and video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as... Figure 1B As shown.

[0092] Figure 1B The video decoding system 40 shown includes processing circuitry that implements both the video encoder 20 and the video decoder 30. Furthermore, one or more imaging devices 41 (such as a camera for capturing real-world images), an antenna 42, one or more memories 44, one or more processors 43, and / or a display device 45 (such as the display device 34 described above) may be provided as part of the video decoding system 40.

[0093] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or tablet computer, mobile phone, smartphone, tablet / tabletcomputer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (such as a content service server or content distribution server), broadcast receiver device, broadcast transmitter device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 can be configured for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0094] In some cases, Figure 1A The video decoding system 10 shown is merely an example, and the technology of this application can be applied to video decoding systems (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, transmitted over a network, etc. The video encoding device may encode data and store it in memory, and / or the video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve data from memory and decode it.

[0095] For ease of description, this document refers to the High-Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC) (next-generation video decoding standard) reference software developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that the embodiments of this invention are not limited to HEVC or VVC.

[0096] Encoders and Encoding Methods

[0097] Figure 2 This is a schematic block diagram of an exemplary video encoder 20 used to implement the technology of this application. Figure 2 In the example, the video encoder 20 includes an input terminal 201 (or input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a (line) buffer 216, a loop filtering unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal 272 (or output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0098] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 can form the forward signal path of the encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, (line) buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 can form the reverse signal path of the video encoder 20. The reverse signal path of the video encoder 20 is related to the decoder (see...). Figure 3 The signal path of the video decoder 30 in the video encoder 20 corresponds to that of the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244 and intra-frame prediction unit 254 also constitute the "built-in decoder" of the video encoder 20.

[0099] Image and image partitioning (images and blocks)

[0100] Encoder 20 can be used to receive image 17 (or image data 17) via input terminal 201, for example, an image in an image sequence forming a video or video sequence. The received image or image data can also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be encoded (especially in video decoding, in order to distinguish the current image from other images (e.g., previously encoded and / or decoded images of the same video sequence (i.e., a video sequence that also includes the current image)).

[0101] A (digital) image is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. Samples in the array can also be called pixels (short for image elements). The size and / or resolution of an image are defined by the number of samples in the array or image along the horizontal and vertical directions (or axes). Color is typically represented using three color components; that is, the image can be represented as an array of three samples or includes three sample arrays. In RGB format or color space, the image includes corresponding red, green, and blue sample arrays. However, in video decoding, each pixel is typically represented by a luminance and chrominance format or in a color space, for example, YCbCr, including the luminance component represented by Y (sometimes also L) and the two chrominance components represented by Cb and Cr. The luminance component Y represents the brightness or grayscale intensity (e.g., both are the same in a grayscale image), while the two chrominance components Cb and Cr represent the chrominance or color information components. Therefore, an image in YCbCr format consists of a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). An RGB format image can be converted to YCbCr format, and vice versa. This process is also known as color transformation or conversion. If the image is monochrome, it may only include the luminance sample array. Accordingly, the image can be, for example, a luminance sample array in black and white format or a luminance sample array and two corresponding chrominance sample arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0102] Embodiments of the video encoder 20 may include an image segmentation unit ( Figure 2 (Not shown in the image), the image partitioning unit is used to divide image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), or coding tree blocks (CTBs), or coding tree units (CTUs) (according to H.265 / HEVC and VVC). The image partitioning unit can be used to apply the same block size and a corresponding grid defining the block size to all images in a video sequence, or to vary the block size between images or subsets or groups of images and divide each image into corresponding blocks.

[0103] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, such as one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be encoded.

[0104] Similar to image 17, image block 203 is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values ​​(sample values); however, the size of image block 203 is smaller than that of image 17. In other words, depending on the applied color format, block 203 may include one sample array (e.g., a luminance array in monochrome, or a luminance or chrominance array in color) or three sample arrays (e.g., one luminance array and two chrominance arrays in color) or any other number and / or type of array. The number of samples in the horizontal and vertical directions (or axes) of block 203 determines its size. Therefore, a block may include M×N (M columns × N rows) sample arrays, or M×N transform coefficient arrays, etc.

[0105] Figure 2 The embodiment of the video encoder 20 shown can be used to encode the image 17 block by block, for example, to encode and predict in blocks 203.

[0106] Figure 2 The embodiment of the video encoder 20 shown can also be used to divide and / or encode an image using slices (also known as video slices), wherein one or more slices (typically non-overlapping) can be used to divide or encode an image, and each slice may include one or more blocks (e.g., CTUs).

[0107] Figure 2 The embodiment of the video encoder 20 shown can also be used to divide and / or encode an image using tile groups (also known as video tile groups) and / or tiles (also known as video tiles), wherein one or more tile groups (typically non-overlapping) can be used to divide or encode an image, each tile group may include one or more blocks (e.g., CTUs) or one or more tiles, wherein each tile may be a shape such as a rectangle and may include one or more blocks (e.g., CTUs), such as complete or partial tiles.

[0108] Residual calculation

[0109] The residual calculation unit 204 can be used to calculate the residual block 205 (also called residual 205) based on the image block 203 and the prediction block 265 (the prediction block 265 is described in detail below) in the following manner: the residual block 205 in the sample domain is obtained by subtracting the sample value of the prediction block 265 from the sample value of the image block 203 on a sample-by-sample (pixel-by-pixel) basis.

[0110] Transformation

[0111] The transformation processing unit 206 can be used to perform transformations such as discrete cosine transform (DCT) or discrete sine transform (DST) on the sample values ​​of the residual block 205 to obtain the transformation coefficients 207 in the transform domain. The transformation coefficients 207 can also be called transformation residual coefficients, representing the residual block 205 in the transform domain.

[0112] Transform processing unit 206 can be used for integer approximations of DCT / DST, such as those specified for H.265 / HEVC. This integer approximation is typically scaled by a certain factor compared to orthogonal DCT transforms. Other scaling factors are used during the transform process to maintain the norm of the residual blocks after both the forward and inverse transforms. These scaling factors are usually selected based on certain constraints, such as the scaling factor being a power of 2 used for shift operations, the bit depth of the transform coefficients, and a trade-off between accuracy and implementation cost. For example, a specific scaling factor is specified for the inverse transform (and the corresponding inverse transform on the video decoder 30 side) via inverse transform processing unit 212, etc.; correspondingly, a corresponding scaling factor can be specified for the forward transform on the encoder 20 side via transform processing unit 206, etc.

[0113] An embodiment of the video encoder 20 (specifically the transform processing unit 206) can be used to encode or compress output transform parameters (e.g., one or more types of transforms) directly or through the entropy encoding unit 270, such that, for example, the video decoder 30 can receive and use the transform parameters for decoding.

[0114] Quantification

[0115] Quantization unit 208 can be used to quantize transform coefficient 207 using scalar quantization or vector quantization to obtain quantization coefficient 209. Quantization coefficient 209 can also be called quantization transform coefficient 209 or quantization residual coefficient 209.

[0116] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different degrees of scaling can be used to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be represented by the quantization parameter (QP). For example, the quantization parameter can be an index of a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include division by the quantization step size, while corresponding and / or dequantization performed by the dequantization unit 210, etc., may include multiplication by the quantization step size. The quantization step size can be determined using the quantization parameter according to embodiments of some standards such as HEVC. Typically, the quantization step size can be calculated using a fixed-point approximation of an equation including division based on the quantization parameter. Quantization and dequantization can introduce additional scaling factors to recover the norm of the residual block. This scaling may modify the norm of the residual block due to the use of scaling in the fixed-point approximation of the equations for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0117] An embodiment of the video encoder 20 (specifically the quantization unit 208) can be used to encode output quantization parameters (QPs) directly or through the entropy coding unit 270, such that a video decoder 30 can receive and use the quantization parameters for decoding.

[0118] Inverse Quantization

[0119] The dequantization unit 210 is used to dequantize the quantization coefficients using the quantization unit 208 by using the inverse process of the quantization scheme used by the quantization unit 208 according to or using the same quantization step size as the quantization unit 208, to obtain the dequantization coefficients 211. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, corresponding to the transform coefficients 207, but due to the loss caused by quantization, the dequantization coefficients 211 are usually different from the transform coefficients.

[0120] Inverse Transformation

[0121] The inverse transform processing unit 212 is used to perform the inverse transform of the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the reconstructed residual block 213 (or the corresponding dequantization coefficients 213) in the pixel domain. The reconstructed residual block 213 can also be called the transform block 213.

[0122] reconstruction

[0123] The reconstruction unit 214 (e.g., adder or summer 214) is used to add the transform block 213 (i.e., reconstruction residual block 213) to the prediction block 265 to obtain the reconstruction block 215 in the pixel domain in such a way as: for example, adding the sample values ​​of the reconstruction residual block 213 and the sample values ​​of the prediction block 265 one sample at a time.

[0124] Filtering

[0125] Loop filtering unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or typically to filter the reconstructed samples to obtain the filtered samples. The loop filtering unit can be used to smooth pixel transitions or improve video quality. Loop filtering unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening filter, a smoothing filter, or a cooperative filter, or any combination thereof. Although loop filtering unit 220 in... Figure 2 The filter is shown as an in-loop filter, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filter block 221 can also be called the filter reconstruction block 221.

[0126] An embodiment of the video encoder 20 (specifically the loop filter unit 220) can be used to encode output loop filter parameters (such as sample adaptive offset information) directly or through the entropy coding unit 270, so that, for example, the decoder 30 can receive and use the same loop filter parameters or the corresponding loop filter for decoding.

[0127] Decoding image buffer

[0128] The decoded picture buffer (DPB) 230 can be a memory that stores reference images or reference image data typically used by the video encoder 20 to encode video data. The DPB 230 can be composed of any of a variety of storage devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The decoded picture buffer (DPB) 230 can be used to store one or more filter blocks 221. The decoded picture buffer 230 can also be used to store other previously filtered blocks (e.g., previously reconstructed and filtered blocks 221) of the same current image or different images (e.g., previously reconstructed images), and can provide the complete previously reconstructed (i.e., decoded) image (and corresponding reference blocks and samples) and / or partially reconstructed current image (and corresponding reference blocks and samples) for inter-frame prediction, etc. For example, when the reconstructed block 215 is not filtered by the loop filter unit 220, the decoded picture buffer (DPB) 230 can also be used to store one or more unfiltered reconstructed blocks 215, or typically store unfiltered reconstructed samples, or any other unprocessed versions of the reconstructed blocks or reconstructed samples.

[0129] Pattern selection (partitioning and prediction)

[0130] The mode selection unit 260 includes a partitioning unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, for receiving or obtaining raw image data (such as raw block 203 (current block 203 of current image 17)) and reconstructed image data (such as filtered and / or unfiltered reconstructed samples or reconstructed blocks of the same (current) image and / or one or more previously decoded images) from the decoded image buffer 230 or other buffers (e.g., line buffer 216). The reconstructed image data is used as reference image data for inter-frame prediction or intra-frame prediction to obtain prediction block 265 or prediction value 265. Specifically, the reference sample 217 of the line buffer 216 can be used by the intra-frame prediction unit 254.

[0131] The mode selection unit 260 can be used to determine or select the partitioning type for the current block prediction mode (including no partitioning) and prediction mode (e.g., intra-frame or inter-frame prediction mode), and generate the corresponding prediction block 265 to calculate the residual block 205 and reconstruct the reconstructed block 215.

[0132] Embodiments of the mode selection unit 260 can be used to select partitioning and prediction modes (e.g., from prediction modes supported by or available to the mode selection unit 260), which provide the best match or minimum residual (minimum residual implies better compression in transmission or storage), or provide minimum indication overhead (minimum indication overhead implies better compression in transmission or storage), or consider or balance both. The mode selection unit 260 can be used to determine the partitioning and prediction modes based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum rate distortion. In this context, terms such as "best," "minimum," and "optimal" do not necessarily refer to "best," "minimum," or "optimal" overall; they can also refer to meeting termination or selection criteria. For example, values ​​exceeding or falling below a threshold or other constraints may lead to a "suboptimal selection," but this reduces complexity and processing time.

[0133] In other words, partitioning unit 262 can be used to divide block 203 into smaller partition blocks or sub-blocks (forming blocks again), for example, using quad-tree (QT) partitioning, binary-tree (BT) partitioning, or triple-tree (TT) partitioning, or any combination thereof, iteratively, and to predict for each partition block or sub-block, wherein the mode selection includes selecting the tree structure of partition block 203 and applying the prediction mode to each partition block or sub-block.

[0134] The following describes in detail the partitioning (e.g., partitioning unit 262) and prediction processing (performed by inter-frame prediction unit 244 and intra-frame prediction unit 254) performed by the exemplary video encoder 20.

[0135] Division

[0136] Partitioning unit 262 can divide (or divide) the current block 203 into smaller partitions, such as smaller blocks of square or rectangular size. These smaller blocks (also called sub-blocks) can be further divided into even smaller partitions. This is also called tree partitioning or hierarchical tree partitioning, where the root block at root level 0 (level 0, depth 0) can be recursively partitioned, for example, into blocks at two or more lower tree levels, such as nodes at tree level 1 (level 1, depth 1). These blocks can be further partitioned into blocks at two or more lower levels, such as blocks at tree level 2 (level 2, depth 2), etc., until the partitioning ends, for example, because a termination criterion is met, such as reaching the maximum tree depth or minimum block size. Blocks that are not further partitioned are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary-tree (BT), a tree divided into three parts is called a ternary-tree (TT), and a tree divided into four parts is called a quad-tree (QT).

[0137] As stated above, the term "block" as used herein can be a portion of an image, particularly a square or rectangular portion. For example, in conjunction with HEVC and VVC, a block can be or correspond to a coding tree unit (CTU), coding unit (CU), prediction unit (PU), or transform unit (TU), and / or correspond to a corresponding block, such as a coding tree block (CTB), coding block (CB), transform block (TB), or prediction block (PB).

[0138] For example, a coding tree unit (CTU) can be or include one CTB of luminance samples and two corresponding CTBs of chrominance samples from an image with three sample arrays, or a monochrome image or an image with samples decoded using three independent color planes and a syntax structure for decoding the samples. Correspondingly, a coding tree block (CTB) can be N×N sample blocks, where N can be set to a value to divide the components into multiple CTBs; this is called partitioning. Similarly, a coding unit (CU) can be or include one coding block of luminance samples and two corresponding coding blocks of chrominance samples from an image with three sample arrays, or a monochrome image or an image with samples decoded using three independent color planes and a syntax structure for decoding the samples. Correspondingly, a coding block (CB) can be M×N sample blocks, where M and N can be set to a value to divide the CTB into multiple coding blocks; this is called partitioning.

[0139] In some embodiments, such as according to HEVC, a coding tree unit (CTU) can be divided into multiple CUs using a quadtree structure represented as a decoding tree. At the CU level, it is decided whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to decode the image region. Each CU can be further divided into one, two, or four PUs depending on the PU partitioning type. The same prediction process is used within a PU, and relevant information is sent to the decoder based on the PU. After obtaining residual blocks using the prediction process according to the PU partitioning type, the CU can be divided into transform units (TUs) according to another quadtree structure similar to the one used for the CU.

[0140] In an embodiment, the decoding blocks are partitioned, for example, according to the latest video decoding standard currently being developed, known as Versatile Video Coding (VVC), using a combined quad-tree and binary tree (QTBT) partitioning. In the QTBT block structure, the CU can be a square or a rectangle. For example, the coding tree unit (CTU) is first partitioned using a quad-tree structure. The leaf nodes of the quad-tree are further partitioned using a binary or ternary (triple) tree structure. The partitioning of the leaf nodes is called the coding unit (CU), and this partitioning is used for prediction and transform processing without any further partitioning. This means that in the QTBT decoding block structure, the CU, PU, ​​and TU have the same block size. Simultaneously, multiple partitioning methods, such as ternary tree partitioning, can be combined with the QTBT block structure.

[0141] In one example, the mode selection unit 260 of the video encoder 20 can be used to perform any combination of the partitioning techniques described herein.

[0142] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) prediction mode set. The prediction mode set may include intra-frame prediction modes and / or inter-frame prediction modes.

[0143] Intra-frame prediction

[0144] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined for VVC.

[0145] Intra-prediction unit 254 is used to generate (intra) prediction block 265 using reconstructed samples of neighboring blocks in the same current image, based on intra-prediction modes in the intra-prediction mode set.

[0146] Intra-prediction unit 254 (or typically mode selection unit 260) can also be used to output intra-prediction parameters (or typically information of the selected intra-prediction mode of the indicator block) to entropy coding unit 270 in the form of syntax element 266, so as to be included in the encoded image data 21, so that video decoder 30 and the like can receive and use the prediction parameters for decoding.

[0147] Inter-frame prediction

[0148] The set of (possible) inter-frame prediction modes depends on the available reference image (i.e., a previously at least partially decoded image stored in the DPB 230) and other inter-frame prediction parameters, such as whether the entire reference image or only a portion of the reference image (e.g., a search window region near the current block) is used to search for the best matching reference block, and / or, for example, whether pixel interpolation (such as half-pixel and / or quarter-pixel interpolation) is used.

[0149] In addition to the prediction modes mentioned above, skip mode and / or direct mode can also be used.

[0150] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (neither of which is in the same unit). Figure 2(As shown in the figure). The motion estimation unit can be used to receive or acquire image block 203 (current image block 203 of current image 17) and decoded image 231, or at least one or more previously reconstructed blocks, such as reconstructed blocks of one or more previously decoded images 231, for motion estimation. For example, the video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form the image sequence that forms the video sequence.

[0151] Encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple previously decoded images, and provide the offset (spatial offset) between the position (x-coordinate, y-coordinate) of the reference image (or reference image index) and / or the position of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also called the motion vector (MV).

[0152] The motion compensation unit can be used to acquire (e.g., receive) inter-frame prediction parameters and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain (inter-frame) prediction blocks 265. Motion compensation performed by the motion compensation unit may include extracting or generating prediction blocks based on motion / block vectors determined by motion estimation, and may also include performing interpolation with sub-pixel precision. Interpolation filtering can generate samples of additional pixels from samples of known pixels, potentially increasing the number of candidate prediction blocks available for decoding the image block. Once the motion vector of the PU for the current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0153] The motion compensation unit can also generate syntax elements associated with blocks and video stripes for use by the video decoder 30 when decoding image blocks of the video stripes. In addition to stripes and corresponding syntax elements, or as alternatives to stripes and corresponding syntax elements, tile groups and / or tiles and corresponding syntax elements can also be generated and / or used.

[0154] Entropy coding

[0155] For example, entropy coding unit 270 is used to apply entropy coding algorithms or schemes (e.g., variable length coding (VLC), context adaptive VLC (CAVLC), arithmetic coding, binarization, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) or bypass entropy coding algorithms or schemes (without compression) to quantization coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements to obtain encoded image data 21 that can be output through output terminal 272 in the form of encoded bitstream 21, such that, for example, video decoder 30 can receive and use these parameters for decoding. The encoded bitstream 21 can be sent to video decoder 30, or stored in memory for subsequent transmission or retrieval by video decoder 30.

[0156] Other structural variations of the video encoder 20 can be used to encode video streams. For example, a non-transform-based encoder 20 can directly quantize the residual signals of certain blocks or frames without the transform processing unit 206. In another implementation, the quantization unit 208 and the inverse quantization unit 210 in the encoder 20 can be combined into a single unit.

[0157] Decoder and Decoding Method

[0158] Figure 3 An example of a video decoder 30 for implementing the technology of this application is shown. The video decoder 30 is used to receive encoded image data 21 (e.g., encoded bitstream 21) encoded by encoder 20, for example, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks of encoded video stripes (and / or chunks or blocks) and associated syntax elements.

[0159] exist Figure 3In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a (row) buffer 316, a loop filter 320, a decoded picture buffer (DPB) 330, a mode selection unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform actions typically associated with… Figure 2 The video encoder 20 describes the encoding process as the opposite of the decoding process.

[0160] As described for encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, (line) buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 also constitute the "built-in decoder" of video encoder 20. Accordingly, the function of inverse quantization unit 310 can be the same as that of inverse quantization unit 210; the function of inverse transform processing unit 312 can be the same as that of inverse transform processing unit 212; the function of reconstruction unit 314 can be the same as that of reconstruction unit 214; the function of (line) buffer 316 can be the same as that of (line) buffer 216, used to provide reference sample 317 to intra-frame prediction unit 354; the function of loop filter 320 can be the same as that of loop filter 220; and the function of decoded picture buffer 330 can be the same as that of decoded picture buffer 230. Therefore, the descriptions of the corresponding units and functions of video encoder 20 are correspondingly applicable to the corresponding units and functions of video decoder 30.

[0161] Entropy Decoding

[0162] Entropy decoding unit 304 is used to parse bitstream 21 (or typically encoded image data 21) and, for example, to perform entropy decoding on the encoded image data 21 to obtain quantization coefficients 309 and / or decoded decoding parameters 366, such as inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. Entropy decoding unit 304 can be used to employ a decoding algorithm or scheme corresponding to the encoding scheme described for entropy coding unit 270 in encoder 20. Entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to mode selection unit 360, and to provide other parameters to other units of decoder 30. Video decoder 300 can receive video strip-level and / or video block-level syntax elements. In addition to stripes and corresponding syntax elements, or as alternatives to stripes and corresponding syntax elements, it can also receive and / or use chunk groups and / or chunks and corresponding syntax elements.

[0163] Inverse Quantization

[0164] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304, etc.), and dequantize the decoded quantization coefficients 309 according to the quantization parameters to obtain dequantization coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include determining the degree of quantization using the quantization parameters determined by the video encoder 20 for each video block in the video strip (or block or block group), and similarly determining the degree of dequantization to be used.

[0165] Inverse Transformation

[0166] The inverse transform processing unit 312 can be used to receive the dequantized coefficients 311, also known as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain the reconstructed residual block 313 in the sample domain. The reconstructed residual block 313 can also be referred to as transform block 313. The transform can be an inverse transform, such as inverse DCT, inverse DST, inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 can also be used to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsing and / or decoding by the entropy decoding unit 304, etc.) to determine the transform to be applied to the dequantized coefficients 311.

[0167] reconstruction

[0168] The reconstruction unit 314 (e.g., adder or summer 314) can be used to add the reconstruction residual block 313 to the prediction block 365 by adding the sample values ​​of the reconstruction residual block 313 to the sample values ​​of the prediction block 365, so as to obtain the reconstruction block 315 in the sample domain.

[0169] Filtering

[0170] Loop filtering unit 320 (in or after the decoding loop) is used to filter the reconstructed block 315 to obtain filtered block 321, to smooth pixel transitions or otherwise improve video quality. Loop filtering unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening filter, a smoothing filter, or a cooperative filter, or any combination thereof. Although loop filtering unit 320 is in... Figure 3 The loop filter unit 320 is shown as an in-loop filter, but in other configurations, it can be implemented as a post-loop filter.

[0171] Decoding image buffer

[0172] Then, the decoded video block 321 of the image is stored in the decoded image buffer 330, which stores the decoded image 331 as a reference image. These reference images are used for subsequent motion compensation of other images and / or for output or display respectively.

[0173] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., to present to the user or allow the user to view it.

[0174] predict

[0175] The function of the inter-frame prediction unit 344 can be the same as that of the inter-frame prediction unit 244 (especially the motion compensation unit), and the function of the intra-frame prediction unit 354 can be the same as that of the intra-frame prediction unit 254. It determines the partitioning or segmentation based on the partitioning and / or prediction parameters or corresponding information received from the encoded image data 21 (e.g., parsing and / or decoding by the entropy decoding unit 304, etc.) and performs the prediction. The mode selection unit 360 can be used to perform prediction (intra-frame or inter-frame prediction) for each block based on the reconstructed image, block, or corresponding sample (filtered or unfiltered) to obtain the prediction block 365. Furthermore, the reference sample 317 of the line buffer 316 can be used by the intra-frame prediction unit 354.

[0176] When encoding a video strip or image into an intra-coded (I) strip, the intra-prediction unit 354 of the mode selection unit 360 generates a prediction block 365 of the image blocks of the current video strip based on the indicated intra-prediction mode and data from the previous decoded block of the current image. When encoding a video strip or image into an inter-coded (i.e., B or P) strip, the inter-prediction unit 344 (e.g., a motion compensation unit) of the mode selection unit 360 generates a prediction block 365 of the video blocks of the current video strip based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct the reference image lists using a default construction technique based on the reference images stored in the DPB 330: List 0 and List 1. In addition to stripes (e.g., video stripes) or as an alternative to stripes, the same or similar methods may be applied to or by embodiments using chunk groups (e.g., video chunk groups) and / or chunks (e.g., video chunks), for example, video may be decoded using I, P, or B chunk groups and / or chunks.

[0177] The mode selection unit 360 is used to determine prediction information for video / image blocks of the current video strip by parsing motion vectors and other syntax elements, and to generate prediction blocks for the decoded current video blocks using the prediction information. For example, the mode selection unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction), inter-frame prediction stripe type (e.g., B-strip, P-strip, or GPB-strip), construction information of one or more reference image lists for the stripe, motion vectors of each inter-frame coded video block of the stripe, inter-frame prediction state of each inter-frame coded video block of the stripe, and other information to decode video blocks within the current video stripe. In addition to stripes (e.g., video stripes) or as an alternative to stripes, the same or similar methods can be applied to or applied by embodiments using chunk groups (e.g., video chunk groups) and / or chunks (e.g., video chunks), for example, video can be decoded using I, P, or B chunk groups and / or chunks.

[0178] Figure 3 The embodiment of the video decoder 30 shown can be used to divide and / or decode an image using strips (also known as video strips), wherein an image can be divided or decoded using one or more strips (typically non-overlapping), and each strip can include one or more blocks (e.g., CTUs).

[0179] Figure 3The embodiment of the video decoder 30 shown can be used to divide and / or decode an image using chunk groups (also known as video chunk groups) and / or chunks (also known as video chunks), wherein one or more chunk groups (typically non-overlapping) can be used to divide or decode an image, each chunk group may include one or more blocks (e.g., CTUs) or one or more chunks, wherein each chunk may be rectangular and may include one or more blocks (e.g., CTUs), such as complete or partial chunks.

[0180] Other variations of the video decoder 30 can be used to decode the encoded image data 21. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly dequantize the residual signal in certain blocks or frames without the inverse transform processing unit 312. In another implementation, the dequantization unit 310 and the inverse transform processing unit 312 can be combined into a single unit in the video decoder 30.

[0181] It should be understood that in encoder 20 and decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, the processing result of interpolation filtering, motion vector derivation, or loop filtering can be further calculated, such as clipping or shifting.

[0182] It should be noted that further calculations can be performed on the derived motion vector of the current block (including but not limited to the control point motion vector in affine mode, sub-block motion vectors in affine mode, planar mode, and ATMVP mode, and time motion vectors). For example, the value of the motion vector can be restricted to a predefined range based on the number of bits used to represent it. If the number of bits used to represent the motion vector is bitDepth, the range is from –2^(bitDepth–1) to 2^(bitDepth–1)–1, where “^” represents exponentiation. For example, if bitDepth is set to 16, the range is –32768–32767; if bitDepth is set to 18, the range is –131072–131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is restricted such that the maximum difference between the integer parts of the MVs of the four 4×4 sub-blocks does not exceed N pixels, such as not exceeding 1 pixel.

[0183] The following description provides two methods for limiting motion vectors based on bitDepth.

[0184] Method 1: Remove the most significant bit (MSB) of the overflow using the following operation:

[0185] ux = (mvx+2) bitDepth ) % 2 bitDepth (1)

[0186] mvx = (ux ≥ 2) bitDepth–1 ) ? (ux – 2 bitDepth ): ux(2)

[0187] uy = (mvy+2) bitDepth ) % 2 bitDepth (3)

[0188] mvy = (uy ≥ 2) bitDepth–1 ) ? (uy – 2 bitDepth ): uy(4)

[0189] Where mvx is the horizontal component of the motion vector of the image block or sub-block; mvy is the vertical component of the motion vector of the image block or sub-block; ux and uy represent the corresponding intermediate values.

[0190] For example, if the value of mvx is -32769, then the value obtained after using formulas (1) and (2) is 32767. In computer systems, decimal numbers are stored in two's complement form. The two's complement of -32769 is 1,0111,1111,1111,1111 (17 bits). If the MSB is discarded, the resulting two's complement is 0111,1111,1111,1111 (decimal 32767), which is the same as the output obtained after using formulas (1) and (2).

[0191] ux = (mvpx + mvdx + 2) bitDepth ) % 2 bitDepth (5)

[0192] mvx = (ux ≥ 2) bitDepth–1 ) ? (ux – 2 bitDepth ): ux(6)

[0193] uy = (mvpy + mvdy +2) bitDepth ) % 2 bitDepth (7)

[0194] mvy = (uy ≥ 2) bitDepth–1 ) ? (uy – 2 bitDepth ): uy(8)

[0195] These operations can be applied during the summation of the motion vector prediction value mvp and the motion vector difference mvd, as shown in equations (5) to (8).

[0196] Method 2: Correct the value to remove the overflowing MSB:

[0197] vx = Clip3(–2 bitDepth–1 , 2 bitDepth–1 –1, vx)

[0198] vy = Clip3(–2 bitDepth–1 , 2 bitDepth–1 –1, vy)

[0199] Where vx is the horizontal component of the motion vector of the image block or sub-block; vy is the vertical component of the motion vector of the image block or sub-block; x, y, and z correspond to the three input values ​​of the MV correction process, and the function Clip3 is defined as follows:

[0200] Clip3(x, y, z) =

[0201] Figure 4 This is a schematic diagram of a video decoding device 400 provided for an embodiment of the present invention. The video decoding device 400 is suitable for implementing the disclosed embodiments described below. In one embodiment, the video decoding device 400 may be a decoder (such as...) Figure 1A Video decoder 30) or encoder (such as Figure 1A (video encoder 20).

[0202] The video decoding device 400 may include: an input port 410 and a receiving unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitting unit (Tx) 440 and an output port 450 for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 410, receiving unit 420, transmitting unit 440, and output port 450, serving as input or output points for optical or electrical signals.

[0203] Processor 430 can be implemented in hardware and software. Processor 430 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 can communicate with ingress port 410, receiver 420, transmitter 440, egress port 450, and memory 460. Processor 430 may include decoding module 470. Decoding module 470 implements the disclosed embodiments described above and below. For example, decoding module 470 is used to implement, process, prepare, or provide various decoding operations. Therefore, including decoding module 470 significantly improves the functionality of video decoding device 400, enabling transitions between different states of video decoding device 400. Alternatively, decoding module 470 can be implemented as instructions stored in memory 460 and executed by processor 430.

[0204] Memory 460 may include one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store programs when an executable program is selected, and to store instructions and data read during program execution. For example, memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0205] Figure 5 A simplified block diagram of the apparatus 500 provided for an exemplary embodiment, wherein the apparatus 500 can be used as Figure 1A The source device 12 and the destination device 14 are either or both of them.

[0206] The processor 502 in device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation may be implemented using a single processor, such as the processor 502 shown in the figure, using more than one processor can improve speed and efficiency.

[0207] In one implementation, the memory 504 in device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by processor 502 via bus 512. Memory 504 may also include an operating system 508 and an application program 510, wherein application program 510 includes at least one program that allows processor 502 to execute the methods described herein. For example, application program 510 may include applications 1 to N, and may also include a video decoding application that executes the methods described herein.

[0208] The device 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element capable of sensing touch input. The display 518 may be coupled to the processor 502 via a bus 512.

[0209] Although bus 512 of device 500 is shown here as a single bus, there can be multiple buses 512. Furthermore, auxiliary memory 514 can be directly coupled to other components in device 500 or accessible via a network, and can include a single integrated unit (such as a memory card) or multiple units (such as multiple memory cards). Additionally, image sensing device 520 and / or sound sensing device 522 can be included in device 500. Therefore, device 500 can be implemented in a variety of configurations.

[0210] Combined Inter-Intra Prediction (CIIP)

[0211] Traditionally, coding units are predicted either intra-frame (using reference samples from the same image) or inter-frame (using reference samples from other images). Combined inter-intra-frame prediction combines these two methods. Therefore, it is sometimes called multi-hypothesis (MH) prediction. When combined inter-intra-frame prediction is enabled, intra-frame and inter-frame prediction samples are used in a weighted manner, and the final prediction result is derived as a weighted average of the prediction samples.

[0212] The CIIP flag indicates that the block uses combined inter-frame-intra-frame predictive decoding.

[0213] A block decoded using CIIP can be further divided into several sub-blocks, such as... Figure 6 and Figure 7As shown. In one example, the sub-blocks are derived by dividing the block into horizontal sub-blocks (i.e., by dividing it along the vertical direction), where each sub-block has the same width as the original block but only ¼ of the original block's height.

[0214] In one example, the sub-blocks are derived by dividing the block into vertical sub-blocks (i.e., by dividing it along the horizontal direction), where each sub-block has the same height as the original block but only ¼ of the width of the original block.

[0215] CIIP introduces block artifacts because it incorporates results obtained using intra-frame prediction, which typically has more residual signal. Block artifacts appear not only at the boundaries of CIIP blocks but also at the edges of sub-blocks within CIIP blocks, such as... Figure 6 The vertical sub-block edges A, B, and C within the CIIP block can be used to identify the corresponding horizontal sub-block edges. To remove block artifacts, the sub-block edges within the CIIP block can be deblocked, such as... Figure 7 As shown.

[0216] Although block artifacts can appear at both the CIIP block boundary and the edges of sub-blocks within the CIIP block, the distortion caused by these two boundaries may differ, thus requiring different boundary strengths.

[0217] Sub-block edges may be caused by CIIP itself. For example, if the intra-prediction mode of a CIIP block is horizontal, then using... Figure 6 The vertical division shown creates three sub-block edges.

[0218] However, sub-block edges can also be caused by limitations in the size of the transform unit (TU). In the Versatile Video Coding Test Model 3.0 (VTM3.0), the maximum TU size is 64×64 samples. If the coding unit (CU) is 128×128 samples, it is divided into 4 TUs, thus generating 4 TU boundaries, such as... Figure 8 As shown. Therefore, a transformation is used with a 64×64 granularity. Deblocking is required for the TU boundaries represented by the dashed lines.

[0219] Furthermore, when using specific decoding tools (such as sub-block transform), TU edges may appear within the CU processed by CIIP prediction, such as... Figure 9 As shown, the decoding unit using CIIP decoding can be further divided into multiple transformation units. Therefore, Figure 9 The TU boundaries highlighted by dashed lines represent the internal TU boundaries within the CIIP cell. These internal TU edges within the CIIP cell also require deblocking.

[0220] In the remainder of the instruction manual, the following terms shall be used:

[0221] CIIP block: A decoding block that uses CIIP for prediction.

[0222] Intra-frame block: A decoded block that uses intra-frame prediction instead of CIIP for prediction.

[0223] Inter-frame block: A decoded block that uses inter-frame prediction instead of CIIP for prediction.

[0224] Deblocking filters and boundary strength

[0225] Video decoding schemes such as HEVC and VVC are designed based on the successful principles of block-based hybrid video decoding. Using this principle, the image is first divided into blocks, and then each block is predicted via intra-frame or inter-frame prediction. These blocks are decoded relative to neighboring blocks, and these blocks have some degree of similarity to the original signal. Since the decoded blocks only approximate the original signal, differences between approximations can lead to discontinuities at predicted block boundaries and transformed block boundaries. Deblocking filters reduce these discontinuities.

[0226] Figure 10 An example of using a deblocking filter on samples from a sub-part. If the sub-part size is less than 8 samples in a direction orthogonal to the deblocking direction, a weak filter is used that uses only 3 samples and modifies 1 sample during the decision-making process.

[0227] Bitstream information such as prediction modes and motion vectors is used when deciding whether to filter block boundaries. Some decoding conditions are more likely to produce strong block artifacts, which are represented by boundary strength (Bs or BS) variables. These boundary strength variables are assigned to each block boundary and are determined as shown in Table 1.

[0228] Table 1

[0229] condition Bs At least one neighboring block is an intra-frame block 2 At least one neighboring block has non-zero transformation coefficients 1 The absolute difference between the vertical or horizontal components of the motion vector belonging to a neighboring block is greater than or equal to an integer brightness sample. 1 Motion prediction in neighboring blocks may refer to different reference images or have different numbers of motion vectors. 1 otherwise 0

[0230] For the luma component, deblocking is applied only to block boundaries where Bs is greater than 0; for the chroma component, deblocking is applied only to block boundaries where Bs is greater than 1. A larger Bs value allows for enhanced filtering by using a higher correction parameter value. The Bs derivation condition reflects the probability of the strongest block artifacts appearing at the intra-frame prediction block boundaries.

[0231] Typically, the two adjacent blocks at a boundary are labeled P and Q, such as... Figure 11 As shown in the figure. This figure depicts the case of a vertical boundary. If the horizontal boundary is considered, then... Figure 11 Rotate 90 degrees clockwise, where P is the upper block and Q is the lower block.

[0232] Construction of the list of most likely patterns

[0233] A Most Probable Mode (MPM) list is used for intra-block mode decoding to improve decoding efficiency. Since there are many intra-block modes (e.g., 35 in HEVC and 67 in VVC), the intra-block mode of the current block is not directly indicated. Instead, a list of most probable modes for the current block is constructed based on the intra-block prediction modes of its neighboring blocks. Because the intra-block mode of the current block is related to its neighboring blocks, the MPM list typically provides good predictions as its name (most probable mode list) suggests. Therefore, the intra-block mode of the current block has a very high probability of being in its MPM list. Thus, only the index of the MPM list is used to deduce the intra-block mode of the current block. The length of the MPM list is much smaller than the total number of intra-block modes (e.g., a 3-MPM list is used in HEVC and a 6-MPM list in VVC). Therefore, fewer bits are required to decode the intra-block mode. A flag (mpm_flag) is used to indicate whether the intra-block mode of the current block belongs to its MPM list. If true, the intra-mode of the current block can be indexed using the MPM list. Otherwise, the intra-mode is indicated directly using the binarized code. In both VVC and HEVC, the MPM list is constructed based on the left and upper neighbor blocks of the current block. When the left and upper neighbor blocks of the current block cannot be used for prediction, the default mode list is used.

[0234] Motion vector prediction

[0235] Motion vector prediction is a technique used for decoding motion data. Motion vectors typically have two components, x and y, representing horizontal and vertical motion, respectively. The motion vector of the current block is usually associated with the motion vectors of neighboring blocks in the current or previously encoded image. This is because neighboring blocks may correspond to the same moving object with similar motion, and the object's motion is unlikely to change abruptly over time. Therefore, using motion vectors from neighboring blocks as predictions reduces the magnitude of the indicated motion vector differences. Motion vector predictions (MVPs) are typically derived from already decoded motion vectors in spatially and / or temporally adjacent blocks within a co-located image.

[0236] If CIIP is used to predict a block, the final predicted sample for that block is partially based on intra-predicted samples. Because intra-prediction is also involved, the residual and transform coefficients are typically larger compared to inter-blocks (MVD, fusion, skipping). Therefore, there will be more discontinuities at the boundaries when these multi-hypothesis (MH or MHITRA) blocks (i.e., CIIP blocks) are adjacent to other blocks. In HEVC and VVC, a strong deblocking filter is used on the boundary when intra-prediction is performed on either of the two neighboring blocks at the boundary, where the Boundary Strong (BS) parameter is set to 2 (strongest).

[0237] However, VTM 3.0 does not account for block artifacts that may be caused by blocks predicted via CIIP. Boundary strength derivation still treats blocks using CIIP as inter-frame blocks. In some cases, this approach may result in poor subjective and objective quality.

[0238] Embodiments of the present invention provide several alternatives for merging CIIP blocks to improve deblocking filters, wherein the boundary strength derivation of a particular boundary is affected by the CIIP block.

[0239] The reference document for Universal Video Decoding (Draft 3) is defined as VVC Draft 3.0 and can be obtained through the following link:

[0240] http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 12_Macao / wg11 / JVET-L1001-v3.zip.

[0241] Example 1:

[0242] For a boundary with two edges (where each edge's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be determined as follows.

[0243] ●as Figure 12 As shown, if at least one of the P and Q blocks is a block using CIIP (MHIntra Prediction), then the boundary strength parameter of that boundary is set to a first value. For example, the first value can be equal to 2.

[0244] ● If neither P-block nor Q-block uses CIIP for prediction, and intra-frame prediction is used to predict at least one of P-block and Q-block, then the boundary strength is determined to be equal to 2.

[0245] ● If neither P-block nor Q-block uses CIIP for prediction, and inter-frame prediction is used for both P-block and Q-block prediction, then the boundary strength is determined to be less than 2. The exact value of the boundary strength is determined through evaluation of other conditions. The derivation of the boundary strength is as follows: Figure 12 As shown, and follow the instructions in Table 1.

[0246] More specifically, the boundary strength is determined to be equal to 1 if at least one of the P-blocks and Q-blocks has non-zero transform coefficients. Similarly, the boundary strength is determined to be equal to 1 if the inter-frame prediction of the P-blocks and Q-blocks uses different reference images, or if the number of motion vectors of the P-blocks and Q-blocks is different. Furthermore, the boundary strength is determined to be equal to 1 if, for at least one of the horizontal and vertical components of the motion vectors, the absolute difference between the motion vectors of the P-blocks and Q-blocks is greater than or equal to an integer luminance sample.

[0247] ● Figure 13 This example demonstrates a comparison with methods specified in the VVC or ITU-H.265 video decoding standards. For cases where neither P-block nor Q-block prediction uses CIIP, the boundary strength is determined according to this embodiment, which is consistent with... Figure 13 This corresponds to the known methods.

[0248] ●Based on the boundary strength determined above, the pixel samples contained in the P block and Q block are filtered using deblocking filtering.

[0249] Example 2:

[0250] like Figure 14 As shown, for a boundary with two edges (where each edge's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can also be derived as follows.

[0251] ● If at least one of the P-blocks and Q-blocks is a block that uses intra-frame prediction, then the boundary strength is set to 2.

[0252] ● Otherwise, if at least one of the P and Q blocks is a block using CIIP (MHIntra prediction), the boundary strength of that boundary is set to a first value, such as 1.

[0253] ● Otherwise, if at least one of the P-blocks and Q-blocks has non-zero transformation coefficients, the boundary strength of that boundary is set to a second value, such as 1. The first and second values ​​can be different.

[0254] ● Otherwise, if the absolute difference between the motion vectors belonging to the P block and the Q block is greater than or equal to an integer brightness sample, the boundary strength of that boundary is set to a second value, such as 1.

[0255] ● Otherwise, if the motion prediction references in neighboring blocks are different reference images or the number of motion vectors is different, the boundary strength of the boundary is set to a second value, such as 1.

[0256] ●Otherwise, set the boundary strength of the boundary to 0.

[0257] ●Based on the determined boundary strength, the pixel samples contained in the P-block and Q-block are filtered using a deblocking filter.

[0258] Example 3:

[0259] like Figure 15 As shown, for a boundary with two edges (where each edge's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can also be derived as follows.

[0260] ● If intra-frame prediction is used instead of CIIP to predict at least one of the P-blocks and Q-blocks, the boundary strength is set to 2. It is possible to predict the P-block using intra-frame prediction instead of multiple hypothesis (MH or MHIntra) (i.e., CIIP) prediction, and the Q-block using any prediction function, and vice versa.

[0261] ● If inter-frame prediction or CIIP is used to predict P-blocks and Q-blocks (which may be the case where: P-block is an inter-frame block and Q-block is an inter-frame block; or, P-block is an inter-frame block and Q-block is a CIIP block; or, P-block is a CIIP block and Q-block is an inter-frame block; or, P-block is a CIIP block and Q-block is a CIIP block), then the following conditions apply:

[0262] If at least one of the P-blocks and Q-blocks has a non-zero transformation coefficient, then the boundary strength parameter of the boundary is set to 1.

[0263] Otherwise (if P and Q blocks have no non-zero transform coefficients), if P and Q blocks are predicted based on different reference images, or if the number of motion vectors used to predict P and Q blocks is not equal, then the boundary strength of the boundary is set to 1.

[0264] Otherwise (if P and Q blocks have no non-zero transform coefficients, and P and Q blocks are predicted based on one or more of the same reference images, and the number of motion vectors used to predict P and Q blocks is the same), if the absolute difference between the motion vectors used to predict P and Q blocks is greater than or equal to an integer brightness sample, then the boundary strength of the boundary is set to 1.

[0265] Otherwise (if all three conditions above are evaluated as false), the boundary strength of the boundary is set to 0.

[0266] ●Next, if at least one of the P and Q blocks is a block using CIIP, the boundary strength is modified as follows.

[0267] If the boundary strength is not equal to a predefined first value (in one example, the predefined first value is equal to 2), then the boundary strength is increased by a predefined second value (in one example, the predefined second value is equal to 1).

[0268] ●Based on the determined boundary strength, the pixel samples contained in the P-block and Q-block are filtered using a deblocking filter.

[0269] Example 4:

[0270] For a boundary with two sides (P and Q, as described in the VVC draft 3.0 provided in the above reference document), the boundary strength can be derived as follows:

[0271] ●If the boundary is a horizontal boundary, and P and Q belong to different CTUs, then:

[0272] ○ If block Q is a block using CIIP, then set the boundary strength to 2.

[0273] Otherwise, such as Figure 13 As shown, the boundary strength is derived according to the definition in VVC draft 3.0 provided in the above reference document.

[0274] ●Otherwise:

[0275] If at least one of the P and Q blocks is a block using CIIP, then the boundary strength of the boundary is set to 2.

[0276] Otherwise, such as Figure 13 As shown, the boundary strength of this boundary is derived according to the definition in VVC draft 3.0 provided in the above reference document.

[0277] Example 5:

[0278] For a boundary with two sides (where each side's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be determined according to this embodiment as follows.

[0279] ● If at least one of the P-blocks or Q-blocks is predicted using intra-frame prediction instead of CIIP, the boundary strength is set to 2. It is possible to predict the P-block using intra-frame prediction instead of multiple hypothesis (CIIP) prediction, and the Q-block using any prediction function, and vice versa.

[0280] ● If inter-frame prediction or CIIP is used to predict P-blocks and Q-blocks (which may be the case where: P-block is an inter-frame block and Q-block is an inter-frame block; or, P-block is an inter-frame block and Q-block is a CIIP block; or, P-block is a CIIP block and Q-block is an inter-frame block; or, P-block is a CIIP block and Q-block is a CIIP block), then the following cases are applicable:

[0281] ○ If the boundary is a horizontal boundary, and P and Q are located in two different CTUs, then:

[0282] ■ If CIIP is used to predict Q blocks (where Q blocks are represented as blocks located below P blocks), the boundary strength of the boundary is set to 1.

[0283] ■ Otherwise (if block Q is not predicted using CIIP), the boundary strength of the boundary is set to 1 if at least one of the neighboring blocks P and Q has non-zero transformation coefficients.

[0284] ■ Otherwise, if the absolute difference between the motion vectors used to predict the P block and the Q block is greater than or equal to an integer brightness sample, the boundary strength of that boundary is set to 1.

[0285] ■ Otherwise, if motion compensation predictions for neighboring blocks P and Q are based on different reference images, or if the number of motion vectors used to predict blocks P and Q is unequal, then the boundary strength of the boundary is set to 1. The order of the last two conditions can be reversed, such as... Figure 13 As shown.

[0286] ■Otherwise, set the boundary strength of the boundary to 0.

[0287] Otherwise (if the boundary is a vertical boundary, or if block P and block Q are contained within the same CTU):

[0288] ■ If at least one of the P-blocks and Q-blocks is predicted using CIIP, then the boundary strength parameter of the boundary is set to 1.

[0289] ■ Otherwise, if at least one of the neighboring blocks P and Q has a non-zero transformation coefficient, the boundary strength of the boundary is set to 1.

[0290] ■ Otherwise, if the absolute difference between the motion vectors used to predict blocks P and Q is greater than or equal to an integer brightness sample, the boundary strength of that boundary is set to 1.

[0291] ■ Otherwise, if motion compensation predictions for neighboring blocks P and Q are based on different reference images, or if the number of motion vectors used to predict blocks P and Q is unequal, then the boundary strength of the boundary is set to 1. The order of the last two conditions can be reversed, such as... Figure 13 As shown.

[0292] ■Otherwise, set the boundary strength of the boundary to 0.

[0293] ●Based on the determined boundary strength, the pixel samples contained in blocks P and Q are filtered using a deblocking filter.

[0294] Benefits of this embodiment:

[0295] Deblocking filters with moderate strength (boundary strength equal to 1) are used to deblock blocks predicted using multiple hypothesis prediction (i.e., CIIP).

[0296] If CIIP is used to predict blocks, a first prediction is obtained using inter-frame prediction, and a second prediction is obtained using intra-frame prediction. These predictions are then combined. Since the final prediction includes the intra-frame prediction component, block artifacts may exist at the boundaries of blocks predicted by CIIP. To mitigate this problem, according to the present invention, the boundary strength is set to 1 to ensure filtering of block edges predicted using CIIP.

[0297] Furthermore, this invention reduces the required row memory as follows. Row memory is defined as the memory required to store information corresponding to the upper CTU row, which is needed when processing adjacent lower CTU rows. For example, to filter the horizontal boundary between two CTU rows, the prediction mode information (intra-frame prediction / inter-frame prediction / multiple hypothesis (CIIP) prediction) of the upper CTU row needs to be stored in row memory. Since the three states (intra-frame prediction / inter-frame prediction / multiple hypothesis (CIIP) prediction) can describe the prediction mode of a block, the row memory requirement can be defined as 2 bits per block.

[0298] However, according to the present invention, if the block (P block in the embodiment) belongs to the upper CTU line, the deblocking operation only requires information about whether the block is predicted by inter-frame prediction or intra-frame prediction (therefore, there are only 2 states, which can be stored using 1 bit per block).

[0299] The reasons are as follows:

[0300] If the boundary between P block and Q block is a horizontal boundary, and Q block and P block belong to two different CTUs (in all embodiments, Q block is the block below P block), then information about whether to use CIIP to predict P block is not used when determining the boundary strength. Since intra-frame prediction and CIIP are mutually exclusive, checking the first condition of embodiment 5 above is sufficient to determine whether to predict P block using intra-frame prediction. Therefore, there is no need to store any information about whether to use CIIP to predict P block.

[0301] This invention, implemented in hardware, allows the prediction mode of a P-block to be temporarily changed to inter-frame prediction (when the P-block is predicted via CIIP), and the boundary strength can be determined based on the changed prediction mode. Afterwards (once the boundary strength is determined), the prediction mode can be changed back to CIIP. It should be noted that the hardware implementation is not limited to the method described herein (changing the prediction mode of the P-block at the CTU boundary). As an example only, according to this invention, information about whether the P-block is predicted via CIIP is not required when determining the boundary strength (at the horizontal CTU boundary).

[0302] Therefore, according to the present invention, the required row memory is reduced from 2 bits per block to 1 bit per block. It should be noted that the total row memory required to be implemented in hardware is directly proportional to the image width and inversely proportional to the minimum block width.

[0303] Example 6:

[0304] For a boundary with two sides (where each side's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be determined according to this embodiment as follows.

[0305] ●Firstly, according to Figure 13 The boundary strength of the boundary is determined using the methods specified in the VVC draft 3.0 or ITU-H.265 video decoding standard provided in the above-mentioned reference documents.

[0306] ●If the boundary is a horizontal boundary, and P and Q are located in two different CTUs, then:

[0307] ○ If CIIP is used to predict block Q, the boundary strength is modified as follows:

[0308] ■If the boundary strength is not equal to 2, then increase the boundary strength by 1.

[0309] ●Otherwise (if the boundary is a vertical boundary, or if block P and block Q are contained within the same CTU):

[0310] ○ If CIIP is used to predict at least one block in block P or block Q, the boundary strength is adjusted as follows:

[0311] ■If the boundary strength is not equal to 2, then increase the boundary strength by 1.

[0312] ●Based on the determined boundary strength, the pixel samples contained in blocks P and Q are filtered using a deblocking filter.

[0313] Example 7:

[0314] This embodiment is a variation of embodiment 4.

[0315] For a boundary with two sides (where each side's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be derived as follows based on this embodiment.

[0316] ●If the boundary is a horizontal boundary, and blocks P and Q are located in different CTUs, then:

[0317] If CIIP is used to predict Q blocks (where Q blocks are represented as blocks located below P blocks), then the boundary strength is set to 2.

[0318] If CIIP is not used to predict Q blocks, and intra-frame prediction is used to predict at least one of P blocks or Q blocks, then the boundary strength is determined to be equal to 2.

[0319] ○ If CIIP is not used to predict block Q, and inter-frame prediction is used to predict blocks P and Q, then the boundary strength is determined to be less than 2. The exact value of the boundary strength can be determined by evaluating other conditions, such as... Figure 13 As shown.

[0320] ●Otherwise (if the boundary is a vertical boundary, or if block P and block Q are contained within the same CTU):

[0321] If CIIP is used to predict at least one of the P-blocks or Q-blocks, the boundary strength of the boundary is set to 2.

[0322] ○ If neither P-block nor Q-block uses CIIP for prediction, and intra-frame prediction is used to predict at least one of the P-block or Q-block, then the boundary strength is determined to be equal to 2.

[0323] ○ If neither P-block nor Q-block uses CIIP for prediction, and both P-block and Q-block use inter-frame prediction, then the boundary strength is determined to be less than 2. The exact value of the boundary strength can be determined through other evaluation criteria, such as... Figure 13 As shown.

[0324] ●Based on the determined boundary strength, the pixel samples contained in the P-block and Q-block are filtered using a deblocking filter.

[0325] Benefits of this embodiment:

[0326] Deblocking filters with moderate strength (boundary strength equal to 1) are used to deblock blocks predicted using multiple hypothesis prediction (CIIP).

[0327] If CIIP is used to predict blocks, a first prediction is obtained using inter-frame prediction, and a second prediction is obtained using intra-frame prediction. These predictions are then combined. Since the final prediction includes the intra-frame prediction component, block artifacts may exist at the boundaries of blocks predicted by CIIP. To mitigate this problem, according to the present invention, the boundary strength is set to 2 to ensure filtering of block edges predicted using CIIP.

[0328] Furthermore, this invention reduces the required row memory as follows. Row memory is defined as the memory required to store information corresponding to the upper CTU row, which is needed when processing adjacent lower CTU rows. For example, to filter the horizontal boundary between two CTU rows, the prediction mode information (intra-frame prediction / inter-frame prediction / multiple hypothesis (CIIP) prediction) of the upper CTU row needs to be stored in row memory. Since the three states (intra-frame prediction / inter-frame prediction / multiple hypothesis (CIIP) prediction) can describe the prediction mode of a block, the row memory requirement can be defined as 2 bits per block.

[0329] However, according to the present invention, if the block (P block in the embodiment) belongs to the upper CTU line, the deblocking operation only requires information about whether the block is predicted by inter-frame prediction or intra-frame prediction (therefore, there are only 2 states, which can be stored using 1 bit per block).

[0330] The reasons are as follows:

[0331] If the boundary between P block and Q block is a horizontal boundary, and Q block and P block belong to two different CTUs (in all embodiments, Q block is the block located below P block), then information about whether to use CIIP to predict P block is not used when determining the boundary strength. Therefore, this information does not need to be stored. Since intra-frame prediction and CIIP are mutually exclusive, checking the first condition of embodiment 7 above is sufficient to determine whether to predict P block by intra-frame prediction.

[0332] This invention, implemented in hardware, allows the prediction mode of a P-block to be temporarily changed to inter-frame prediction (when the P-block is predicted via CIIP), and the boundary strength can be determined based on the changed prediction mode. Afterwards (once the boundary strength is determined), the prediction mode can be changed back to CIIP. It should be noted that the hardware implementation is not limited to the method described herein (changing the prediction mode of the P-block at the CTU boundary). As an example only, according to this invention, information about whether the P-block is predicted via CIIP is not required when determining the boundary strength (at the horizontal CTU boundary).

[0333] Therefore, according to the present invention, the required row memory is reduced from 2 bits per block to 1 bit per block. It should be noted that the total row memory required to be implemented in hardware is directly proportional to the image width and inversely proportional to the minimum block width.

[0334] Example 8:

[0335] This embodiment is a variation of embodiment 6.

[0336] For a boundary with two sides (where each side's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be determined according to this embodiment as follows.

[0337] ● If at least one of the blocks P and Q is predicted using intra-frame prediction instead of CIIP, then the boundary strength is set to 2. It is possible for the P block to be predicted using intra-frame prediction instead of multiple hypothesis prediction (CIIP), and the Q block to be predicted using any prediction function, and vice versa.

[0338] ● If inter-frame prediction or CIIP is used to predict P-blocks and Q-blocks (which may be the case where: P-block is an inter-frame block and Q-block is an inter-frame block; or, P-block is an inter-frame block and Q-block is a CIIP block; or, P-block is a CIIP block and Q-block is an inter-frame block; or, P-block is a CIIP block and Q-block is a CIIP block), then the following cases are applicable:

[0339] If at least one of blocks P and Q has non-zero transformation coefficients, then the boundary strength of the boundary is set to 1.

[0340] Otherwise (if blocks P and Q have no non-zero transform coefficients), if the absolute difference between the motion vectors used to predict blocks P and Q is greater than or equal to an integer brightness sample, the boundary strength of that boundary is set to 1.

[0341] Otherwise (if blocks P and Q have no non-zero transform coefficients and the absolute difference between motion vectors is less than one integer brightness sample), if blocks P and Q are predicted based on different reference images, or if the number of motion vectors used to predict blocks P and Q is unequal, then the boundary strength of that boundary is set to 1. The order of the last two conditions can be reversed, such as... Figure 13 As shown.

[0342] Otherwise (if all three conditions above are false), set the boundary strength of the boundary to 0.

[0343] ●If the boundary is a horizontal boundary, and P and Q are located in two different CTUs, then:

[0344] ○ If CIIP is used to predict block Q, the determined boundary strength is modified as follows:

[0345] ■If the boundary strength is not equal to 2, then increase the boundary strength by 1.

[0346] ●If the boundary is a vertical boundary, or if block P and block Q are contained within the same CTU:

[0347] ○ If CIIP is used to predict at least one of blocks P and Q, the boundary strength is adjusted as follows:

[0348] ■If the boundary strength is not equal to 2, then increase the boundary strength by 1.

[0349] ●Based on the determined boundary strength, the pixel samples contained in blocks P and Q are filtered using a deblocking filter.

[0350] Benefits of this embodiment:

[0351] Deblocking filters with moderate strength (boundary strength equal to 1) are used to deblock blocks predicted using multiple hypothesis prediction (i.e., CIIP).

[0352] If CIIP is used to predict blocks, a first prediction is obtained using inter-frame prediction, and a second prediction is obtained using intra-frame prediction. These predictions are then combined. Since the final prediction includes the intra-frame prediction component, block artifacts may exist at the boundaries of blocks predicted by CIIP. To mitigate this problem, according to the present invention, the boundary strength is increased by 1 to ensure filtering of block edges predicted using CIIP.

[0353] Furthermore, this invention reduces the required row memory as follows. Row memory is defined as the memory required to store information corresponding to the upper CTU row, which is needed when processing adjacent lower CTU rows. For example, to filter the horizontal boundary between two CTU rows, the prediction mode information (intra-frame prediction / inter-frame prediction / multiple hypothesis (CIIP) prediction) of the upper CTU row needs to be stored in row memory. Since the three states (intra-frame prediction / inter-frame prediction / multiple hypothesis (CIIP) prediction) can describe the prediction mode of a block, the row memory requirement can be defined as 2 bits per block.

[0354] However, according to the present invention, if the block (P block in the embodiment) belongs to the upper CTU line, the deblocking operation only requires information about whether the block is predicted by inter-frame prediction or intra-frame prediction (therefore, there are only 2 states, which can be stored using 1 bit per block).

[0355] The reasons are as follows:

[0356] If the boundary between P block and Q block is a horizontal boundary, and P block and Q block belong to two different CTUs (in all embodiments, Q block is the block located below P block), then information about whether to use CIIP to predict P block is not used when determining the boundary strength. Therefore, this information does not need to be stored. Since intra-frame prediction and CIIP are mutually exclusive, checking the first condition of embodiment 8 above is sufficient to determine whether to predict P block by intra-frame prediction.

[0357] This invention, implemented in hardware, allows the prediction mode of a P-block to be temporarily changed to inter-frame prediction (when the P-block is predicted via CIIP), and the boundary strength can be determined based on the changed prediction mode. Afterwards (once the boundary strength is determined), the prediction mode can be changed back to CIIP. It should be noted that the hardware implementation is not limited to the method described herein (changing the prediction mode of the P-block at the CTU boundary). As an example only, according to this invention, information about whether the P-block is predicted via CIIP is not required when determining the boundary strength (at the horizontal CTU boundary).

[0358] Therefore, according to the present invention, the required row memory is reduced from 2 bits per block to 1 bit per block. It should be noted that the total row memory required to be implemented in hardware is directly proportional to the image width and inversely proportional to the minimum block width.

[0359] It should be noted that, according to all the above embodiments, if CIIP is used to predict blocks, the first prediction is obtained by using inter-frame prediction, the second prediction is obtained by using intra-frame prediction, and then these predictions are combined.

[0360] The above embodiments demonstrate that CIIP blocks are treated as intra-frame blocks to varying degrees during deblocking filtering. Embodiments 1, 2, and 3 employ three different strategies to adjust the boundary strength of the boundaries.

[0361] Example 1 treats CIIP blocks entirely as intra-blocks. Therefore, the conditions for setting Bs to 2 are the same as in Table 1.

[0362] Example 2 assumes that the distortion caused by CIIP blocks is not as high as that caused by intra-frame blocks. Therefore, when a CIIP block is detected within the boundary, Bs is considered to be 1.

[0363] Example 3 treats the CIIP block portion as an intra-frame block; if at least one neighboring block at the boundary is a CIIP block, then Bs is incremented by 1. If already used... Figure 13 If the conventional derivation strategy derives Bs as 2, then Bs remains unchanged.

[0364] Figure 11 The derivation of Bs in the VVC draft 3.0 provided in the aforementioned reference document is shown. Figure 12 , Figure 14 and Figure 15 The variations in the Bs derivation of Examples 1, 2 and 3 are described respectively.

[0365] It should be noted that Examples 1 and 2 not only reduce potential distortion but also simplify the processing logic. In Examples 1 and 2, as long as the P block or Q block is a CIIP block, it is no longer necessary to check the coefficients and motion vectors, thereby shortening the latency of condition checks.

[0366] Examples 4, 5, and 6 are variations of Examples 1, 2, and 3, respectively, considering row buffer memory. The main change in Examples 1, 2, and 3 is that the CIIP block inspection is asymmetrical when the P-block and Q-block are located in different CTUs and the edges are horizontal. That is, the P-side block (i.e., the upper part) is not inspected, only the Q-side block (i.e., the lower part) is inspected. In this way, no additional row buffer memory is allocated to store the CIIP flags of the P-side block located in other CTUs.

[0367] In addition to the six embodiments described above, an additional characteristic of CIIP blocks is that CIIP blocks are not always considered intra-blocks. In one example, when searching for motion vector prediction values ​​for the current block, if the neighboring blocks of the current block are CIIP blocks, then the motion vectors of these CIIP blocks can be considered as motion vector prediction values. In this case, the inter-frame prediction information of the CIIP blocks is used, and therefore the CIIP blocks are no longer considered intra-blocks. In another example, when constructing the MPM list for intra-blocks, it can be assumed that the neighboring CIIP blocks of the current block do not contain intra-frame information. Therefore, when checking whether these CIIP blocks are available for the construction of the MPM list for the current block, they are marked as unavailable. It should be noted that the CIIP blocks mentioned in this paragraph are not limited to CIIP blocks used to determine the Bs value of the deblocking filter.

[0368] In addition to the six embodiments described above, an additional feature of CIIP blocks is that MH blocks are always considered intra-blocks. In one example, when searching for motion vector prediction values ​​for the current block, if the neighboring blocks of the current block are CIIP blocks, the motion vectors of these CIIP blocks are excluded from the motion vector prediction values. In this case, the inter-frame prediction information of CIIP blocks is not used, and the CIIP blocks are considered intra-blocks. In another example, when constructing the MPM list for intra-blocks, it can be assumed that the neighboring CIIP blocks of the current block contain intra-frame information. Therefore, when checking whether these CIIP blocks are available for the construction of the MPM list for the current block, they are marked as available. It should be noted that the CIIP blocks mentioned in this paragraph are not limited to CIIP blocks used to determine the Bs value of the deblocking filter.

[0369] Example 9:

[0370] In one example, the boundary strength (Bs) of the CIIP block boundary can be set to a value of 2, but the boundary strength of the boundaries of the sub-blocks inside the CIIP block can be set to a value of 1. When the boundaries of the sub-blocks are not aligned with the 8×8 sample grid, the boundary strength of these edges can be set to a value of 0. Figure 16 or Figure 17 An 8×8 grid is shown, where, Figure 16 The 8×8 sample grid is shown, starting from the top left sample of the CU. Figure 17 The 8×8 sample grid is shown, which does not start from the top left sample of the CU.

[0371] In another example, the boundary strength of an edge can be determined as follows.

[0372] For a boundary with two edges (where each edge's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be derived from this example as follows:

[0373] ●If the boundary is a horizontal boundary, and blocks P and Q are located in different CTUs, then:

[0374] If CIIP is used to predict Q blocks (where Q blocks are represented as blocks located below P blocks), then the boundary strength is set to 2.

[0375] If CIIP is not used to predict Q blocks, and intra-frame prediction is used to predict at least one of P blocks or Q blocks, then the boundary strength is determined to be equal to 2.

[0376] If CIIP is not used to predict Q blocks, and inter-frame prediction is used to predict P and Q blocks, then the boundary strength is determined to be less than 2. The exact value of the boundary strength can be determined through other evaluation criteria, such as... Figure 13 As shown.

[0377] ●Otherwise (if P and Q correspond to two sub-blocks within the CIIP block, i.e., if the target boundary is a sub-block boundary within the CIIP block):

[0378] ○ If the sub-block boundary is aligned with the 8×8 grid, set the boundary strength to a value of 1.

[0379] Otherwise (if the sub-block boundary is not aligned with the 8×8 grid), the boundary strength is set to a value of 0.

[0380] ●Otherwise (if the boundary is a vertical boundary, or if block P and block Q are contained within the same CTU, and P and Q are not within the same CIIP block):

[0381] If at least one of blocks P or Q is predicted using CIIP, then the boundary strength parameter of the boundary is set to 2.

[0382] ○ If neither P-block nor Q-block uses CIIP for prediction, and intra-frame prediction is used to predict at least one of the P-block or Q-block, then the boundary strength is determined to be equal to 2.

[0383] If neither P nor Q blocks use CIIP for prediction, and inter-frame prediction is used for both blocks P and Q, then the boundary strength is determined to be less than 2. The exact value of the boundary strength can be determined through other evaluation criteria, such as... Figure 13 As shown.

[0384] ●Based on the determined boundary strength, the pixel samples contained in blocks P and Q are filtered using a deblocking filter.

[0385] In another example, the boundary strength (Bs) of the CIIP block boundary can be set to a value of 2, but the boundary strength of the boundaries of the sub-blocks inside the CIIP block can be set to a value of 1. When the boundaries of the sub-blocks are not aligned with the 4×4 sample grid, the boundary strength of these edges can be set to a value of 0. Figure 18 A 4×4 grid is shown.

[0386] In another example, the boundary strength of an edge can be determined as follows.

[0387] For a boundary with two edges (where each edge's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be derived from this example as follows:

[0388] ●If the boundary is a horizontal boundary, and blocks P and Q are located in different CTUs, then:

[0389] If CIIP is used to predict Q blocks (where Q blocks are represented as blocks located below P blocks), then the boundary strength is set to 2.

[0390] If CIIP is not used to predict Q blocks, and intra-frame prediction is used to predict at least one of P blocks or Q blocks, then the boundary strength is determined to be equal to 2.

[0391] ○ If CIIP is not used to predict Q blocks, and inter-frame prediction is used to predict P and Q blocks, then the boundary strength is determined to be less than 2. The exact value of the boundary strength is determined through evaluation of other conditions, such as... Figure 13 As shown.

[0392] ●Otherwise (if P and Q correspond to two sub-blocks within the CIIP block, i.e., the target boundary is the sub-block boundary within the CIIP block):

[0393] ○ If the sub-block boundary is aligned with the 4×4 grid, set the boundary strength to a value of 1.

[0394] Otherwise (if the sub-block boundary is not aligned with the 4×4 grid), the boundary strength is set to a value of 0.

[0395] ●Otherwise (if the boundary is a vertical boundary, or if P block and Q block are contained within the same CTU, and P and Q are not within the same CIIP block):

[0396] If at least one of the P-blocks or Q-blocks is predicted using CIIP, then the boundary strength of the boundary is set to 2.

[0397] ○ If neither P-block nor Q-block uses CIIP for prediction, and intra-frame prediction is used to predict at least one of the P-block or Q-block, then the boundary strength is determined to be equal to 2.

[0398] ○ If neither P-block nor Q-block uses CIIP for prediction, and inter-frame prediction is used for both P-block and Q-block prediction, then the boundary strength is determined to be less than 2. The exact value of the boundary strength is determined through evaluation of other conditions, such as... Figure 13 As shown.

[0399] ●Based on the determined boundary strength, the pixel samples contained in blocks P and Q are filtered using a deblocking filter.

[0400] Example 10 (No row buffer limitation):

[0401] For a boundary with two sides (where each side's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be derived as follows based on this embodiment.

[0402] ● If CIIP is used to predict at least one of blocks P or Q, and blocks P and Q are not in the same CIIP block, then the boundary strength of the boundary is set to 2.

[0403] ●If both blocks P and Q are predicted using CIIP, and blocks P and Q are located within the same CIIP block, then:

[0404] ○ If the sub-block boundary is aligned with the 8×8 grid, set the boundary strength to a value of 1.

[0405] Otherwise (if the sub-block boundary is not aligned with the 8×8 grid), the boundary strength is set to a value of 0.

[0406] ● If neither P-block nor Q-block uses CIIP for prediction, and intra-frame prediction is used to predict at least one of the P-block or Q-block, then the boundary strength is determined to be equal to 2.

[0407] ● If neither P-block nor Q-block uses CIIP for prediction, and inter-frame prediction is used for both P-block and Q-block prediction, then the boundary strength is determined to be less than 2. The exact value of the boundary strength can be determined through other evaluation criteria, such as... Figure 13 As shown.

[0408] Based on the determined boundary strength, a deblocking filter is used to filter the pixel samples contained in blocks P and Q.

[0409] Example 11 (No row buffer limitation, CIIP aligned with 8×8 grid):

[0410] For a boundary with two sides (where each side's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be derived as follows based on this embodiment.

[0411] ● If CIIP is used to predict at least one of blocks P or Q, and blocks P and Q are not in the same CIIP block, and the boundary is aligned with an 8×8 grid, then the boundary strength of the boundary is set to 2.

[0412] ●If both blocks P and Q are predicted using CIIP, and blocks P and Q are located within the same CIIP block, then:

[0413] ○ If the sub-block boundary is aligned with the 8×8 grid, set the boundary strength to a value of 1.

[0414] Otherwise (if the sub-block boundary is not aligned with the 8×8 grid), the boundary strength is set to a value of 0.

[0415] ● If neither P-block nor Q-block uses CIIP for prediction, and intra-frame prediction is used to predict at least one of the P-block or Q-block, then the boundary strength is determined to be equal to 2.

[0416] ● If neither P-block nor Q-block uses CIIP for prediction, and inter-frame prediction is used for both P-block and Q-block prediction, then the boundary strength is determined to be less than 2. The exact value of the boundary strength can be determined through other evaluation criteria, such as... Figure 13 As shown.

[0417] ●Based on the determined boundary strength, the pixel samples contained in blocks P and Q are filtered using a deblocking filter.

[0418] Example 12 (no row buffer restrictions, TU size restrictions, CIIP boundaries also aligned with the 8×8 grid):

[0419] In one example, the boundary strength (Bs) of the CIIP block boundary can be set to a value of 2, but the boundary strength of the boundaries of sub-blocks within the CIIP block can be set to a value of 1, except for sub-blocks constrained by TU size (e.g., Figure 8(As shown). If the boundary is a TU boundary and its adjacent P and Q blocks belong to the same CIIP block, then the boundary strength of the boundary is set to 2. When the boundaries of sub-blocks or CIIP blocks (CIIP block size can be less than 8×8) are not aligned with the 8×8 sample grid, the boundary strength of these edges can be set to a value of 0. Figure 16 or Figure 17 An 8×8 grid is shown, where, Figure 16 The 8×8 sample grid is shown, starting from the top left sample of the CU. Figure 17 The 8×8 sample grid is shown, which does not start from the top left sample of the CU.

[0420] For a boundary with two edges (where each edge's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be derived from this example as follows:

[0421] ● If CIIP is used to predict at least one of blocks P or Q, and blocks P and Q are not in the same CIIP block, and the boundary is aligned with an 8×8 grid, then the boundary strength of the boundary is set to 2.

[0422] ●If both blocks P and Q are predicted using CIIP, and blocks P and Q are located within the same CIIP block, then:

[0423] ○ If the sub-block boundary is aligned with an 8×8 grid, the boundary strength is set to 2 when the size of at least one block in sub-block P or Q is equal to the maximum allowable TU size.

[0424] ○ If the sub-block boundary is aligned with an 8×8 grid, the boundary strength is set to 1 when the size of either sub-block P or Q is not equal to the maximum allowable TU size.

[0425] Otherwise (if the sub-block boundary is not aligned with the 8×8 grid), the boundary strength is set to a value of 0.

[0426] ● If neither P-block nor Q-block uses CIIP for prediction, and intra-frame prediction is used to predict at least one of the P-block or Q-block, then the boundary strength is determined to be equal to 2.

[0427] ● If neither P-block nor Q-block uses CIIP for prediction, and inter-frame prediction is used for both P-block and Q-block prediction, then the boundary strength is determined to be less than 2. The exact value of the boundary strength can be determined through other evaluation criteria, such as... Figure 13 As shown.

[0428] Based on the determined boundary strength, a deblocking filter is used to filter the pixel samples contained in blocks P and Q.

[0429] Example 13 (No row buffer limitation, TU size limitation, CIIP sub-block boundaries are only aligned with the 8×8 grid):

[0430] In one example, the boundary strength (Bs) of the CIIP block boundary can be set to a value of 2, but the boundary strength of the boundaries of sub-blocks within the CIIP block can be set to a value of 1, except for sub-blocks constrained by TU size (e.g., Figure 8 (As shown). If the boundary is a TU boundary and its adjacent P and Q blocks belong to the same CIIP block, then the boundary strength of the boundary is set to 2. When the boundaries of sub-blocks of a CIIP block are not aligned with the 8×8 sample mesh, the boundary strength of these edges can be set to a value of 0. Figure 16 or Figure 17 An 8×8 grid is shown, where, Figure 16 The 8×8 sample grid is shown, starting from the top left sample of the CU. Figure 17 The 8×8 sample grid is shown, which does not start from the top left sample of the CU.

[0431] For a boundary with two edges (where each edge's spatially adjacent blocks are denoted as blocks P and Q), the boundary strength can be derived from this example as follows:

[0432] ● If CIIP is used to predict at least one of blocks P or Q, and blocks P and Q are not in the same CIIP block, then the boundary strength of the boundary is set to 2.

[0433] ●If both blocks P and Q are predicted using CIIP, and blocks P and Q are located within the same CIIP block, then:

[0434] ○ If the sub-block boundary is aligned with an 8×8 grid, the boundary strength is set to 2 when the size of at least one block in sub-block P or Q is equal to the maximum allowable TU size.

[0435] ○ If the sub-block boundary is aligned with an 8×8 grid, the boundary strength is set to 1 when the size of either sub-block P or Q is not equal to the maximum allowable TU size.

[0436] Otherwise (if the sub-block boundary is not aligned with the 8×8 grid), the boundary strength is set to a value of 0.

[0437] ● If neither P-block nor Q-block uses CIIP for prediction, and intra-frame prediction is used to predict at least one of the P-block or Q-block, then the boundary strength is determined to be equal to 2.

[0438] ● If neither P-block nor Q-block uses CIIP for prediction, and inter-frame prediction is used for both P-block and Q-block prediction, then the boundary strength is determined to be less than 2. The exact value of the boundary strength is determined through other evaluation criteria, such as... Figure 13 As shown.

[0439] Based on the determined boundary strength, a deblocking filter is used to filter the pixel samples contained in blocks P and Q.

[0440] Example 14 (TU transform edge, CIIP sub-blocks are aligned only with 8×8 grid):

[0441] In one example, the boundary strength (Bs) of the CIIP block boundary or the boundary of the transform unit can be set to a value of 2, but the boundary strength of the boundaries of sub-blocks within the CIIP block can be set to a value of 1. When the boundaries of sub-blocks, transform units, or CIIP blocks are not aligned with the 8×8 sample mesh, the boundary strength of these edges can be set to a value of 0. For example, an 8×8 mesh can be like this... Figure 16 or Figure 17 As shown, where, Figure 16 The 8×8 sample grid is shown, starting from the top left sample of the CU. Figure 17 An 8×8 sample grid is shown that does not start from the top left sample of the CU.

[0442] For a boundary with two sides (where each side's spatially adjacent blocks are denoted as P-blocks and Q-blocks, and the boundary is aligned with an 8×8 grid), the boundary strength can be derived as follows based on this embodiment.

[0443] ● If intra-frame prediction is used to predict at least one of the P-blocks or Q-blocks, the boundary strength is set to 2.

[0444] ● Otherwise, if CIIP is used to predict at least one of blocks P or Q, and blocks P and Q are not in the same CIIP block, then the boundary strength parameter of the boundary is set to 2.

[0445] ● Otherwise, if both blocks P and Q are predicted using CIIP, and blocks P and Q are within the same CIIP block (i.e., at sub-block boundaries), then:

[0446] If the sub-block boundary is aligned with the transform unit, set the boundary strength of the sub-block boundary to the value 2.

[0447] Otherwise, set the boundary strength of the sub-block boundary to the value 1.

[0448] ● If neither P-block nor Q-block uses CIIP for prediction, and inter-frame prediction is used for both P-block and Q-block prediction, then the boundary strength is determined to be less than 2. The exact value of the boundary strength is determined through other evaluation criteria, such as... Figure 13 As shown.

[0449] Based on the determined boundary strength, a deblocking filter is used to filter the pixel samples contained in blocks P and Q.

[0450] Example 15:

[0451] In another example, the process of determining the boundary strength according to the present invention can be described in the pseudocode language used in VVC draft 3.0 provided in the aforementioned reference document, as follows:

[0452] The input to this process is:

[0453] - Image sample array recPicture;

[0454] - Position (xCb, yCb) represents the top-left sample of the current coding block relative to the top-left sample of the current image;

[0455] - The variable nCbW represents the width of the current coded block;

[0456] - The variable nCbH represents the height of the current coded block;

[0457] - The variable edgeType indicates whether to filter vertical edges (EDGE_VER) or horizontal edges (EDGE_HOR);

[0458] ‒-Two-dimensional (nCbW)×(nCbH) array edgeFlags.

[0459] The output of this process is a two-dimensional (nCbW)×(nCbH) array bS, representing the boundary filter strength.

[0460] variable xD i yD j The derivation of xN and yN is as follows:

[0461] - If edgeType equals EDGE_VER, then xD i Set to (i<<3), and set yD j Set it to (j<<2), set xN to Max(0, (nCbW / 8) – 1), and set yN to (nCbH / 4) – 1.

[0462] ‒-Otherwise (edgeType equals EDGE_HOR), xD i Set it to (i<<2), and set yD j Set it to (j<<3), set xN to (nCbW / 4) – 1, and set yN to Max(0, (nCbH / 8) – 1).

[0463] For xD of i = 0……xN i and yD of j = 0……yN j Applicable to the following situations:

[0464] ‒-If edgeFlags[ xD i ][ yD j If ] equals 0, then the variable bS[ xD] will be set to 0. i ][ yD j Set it to 0.

[0465] ‒-Otherwise, the following applies:

[0466] - The sample values ​​p0 and q0 are derived as follows:

[0467] - If edgeType equals EDGE_VER, then set p0 to recPicture[xCb + xD] i – 1 ][yCb + yD j Set q0 to recPicture[xCb + xD] i ][ yCb + yD j ].

[0468] ‒- Otherwise (edgeType equals EDGE_HOR), set p0 to recPicture[xCb + xD] i ][ yCb +yD j – 1], set q0 to recPicture [ xCb + xD i ][ yCb + yD j ].

[0469] The following derivation of the variable bS[ xD] is as follows. i ][ yD j ]:

[0470] - If sample p0 or q0 is in a coding block of a coding unit coded using intra-prediction mode, then bS[xD] will be used. i ][ yD j Set it to 2.

[0471] ‒- Otherwise, if the block edge is also a transformed block edge, and the ciip_flag of sample p0 or q0 is equal to 1, then bS[xD] will be... i ][ yD j Set it to 2.

[0472] ‒- Otherwise, if the ciip_flag of sample p0 or q0 is equal to 1, then bS[ xD i ][ yD j Set it to 1.

[0473] ‒- Otherwise, if the block edge is also a transform block edge, and sample p0 or q0 is located in a transform block containing one or more non-zero transform coefficient values, then bS[xD] will be... i ][ yD j Set it to 1.

[0474] ‒-Otherwise, if one or more of the following conditions are met, then bS[xD] will be... i ][ yD j Set to 1:

[0475] - Use a different reference image or a different number of motion vectors in the prediction of the coded sub-block containing sample p0 compared to the reference image or number of motion vectors used to predict the coded sub-block containing sample q0.

[0476] Note 1 - The reference images used for two coded sub-blocks are determined solely based on the referenced images, without considering whether the indexes in reference image list 0 or reference image list 1 are used for prediction, or whether the index positions within the reference image lists are different.

[0477] Note 2 - The number of motion vectors used to predict the top left sample coverage (xSb, ySb) of the encoded sub-block is equal to PredFlagL0[xSb][ySb] + PredFlagL1[xSb][ySb].

[0478] - Use a motion vector to predict the coded sub-block containing sample p0 and use a motion vector to predict the coded sub-block containing sample q0, and the absolute difference between the horizontal or vertical components of the motion vector used is greater than or equal to 4 quarter-luminance samples.

[0479] - Predict a coded subblock containing sample p0 using two motion vectors and two different reference images, and predict a coded subblock containing sample q0 using two motion vectors from two identical reference images, wherein the absolute difference between the horizontal or vertical components of the two motion vectors used when predicting two coded subblocks from the same reference image is greater than or equal to 4 quarter-luminance samples.

[0480] - Two motion vectors from the same reference image are used to predict a coded sub-block containing sample p0, and two motion vectors from the same reference image are used to predict a coded sub-block containing sample q0, and both satisfy the following condition:

[0481] - The absolute difference between the horizontal or vertical components of the motion vectors in List 0 used when predicting two coded subblocks is greater than or equal to four quarter-luminance samples; or, the absolute difference between the horizontal or vertical components of the motion vectors in List 1 used when predicting two coded subblocks is greater than or equal to four quarter-luminance samples.

[0482] - The absolute difference between the horizontal or vertical component of the List 0 motion vector used in predicting the coded subblock containing sample p0 and the List 1 motion vector used in predicting the coded subblock containing sample q0 is greater than or equal to 4 quarter-luminance samples; or, the absolute difference between the horizontal or vertical component of the List 1 motion vector used in predicting the coded subblock containing sample p0 and the List 0 motion vector used in predicting the coded subblock containing sample q0 is greater than or equal to 4 quarter-luminance samples.

[0483] ‒-Otherwise, set the variable bS[ xDi ][ yDj ] to 0.

[0484] Although embodiments of the present invention are described primarily in relation to video decoding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30 (correspondingly, system 10), as well as other embodiments described herein, can also be used for still image processing or decoding, i.e., processing or decoding a single image in video decoding independent of any previous or consecutive images. Typically, if image processing decoding is limited to a single image 17, only inter-frame prediction units 244 (encoder) and 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, partitioning 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0485] Embodiments of encoder 20 and decoder 30, and the functions described herein relating to encoder 20 and decoder 30, can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the various functions may be stored as one or more instructions or code in a computer-readable medium or transmitted via a communication medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium (such as a data storage medium), or a communication medium that includes any medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described herein. A computer program product may include a computer-readable medium.

[0486] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store required program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote resource using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the definition of medium includes coaxial cable, optical fiber, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. The disks and optical discs used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs utilize lasers to reproduce data optically. Combinations of the above items should also be included within the scope of computer-readable media.

[0487] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the various functions described herein can be provided within dedicated hardware and / or software modules for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be implemented entirely within one or more circuit or logic elements.

[0488] The technology of this invention can be implemented in a variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). This invention describes various components, modules, or units to emphasize functional aspects of the devices used to perform the disclosed technology, but these do not necessarily need to be implemented by different hardware units. In fact, as described above, various units can be combined with suitable software and / or firmware within codec hardware units, or provided as a collection of interoperable hardware units, including one or more processors as described above.

Claims

1. A decoding method, characterized in that, The decoding includes decoding or encoding, and the method includes: Determine whether the current decoding unit uses combined inter-frame-intra-frame prediction (CIIP) for prediction; When it is determined that the current decoding unit is using CIIP for prediction, the boundary strength of the boundary of the current decoding unit is set to a first value; and when it is determined that the current decoding unit is not using CIIP for prediction, it is determined whether the boundary of the current decoding unit is the boundary of a transform block and whether the transform block contains one or more non-zero transform coefficient values; when it is determined that the boundary of the current decoding unit is the boundary of the transform block and the transform block contains one or more non-zero transform coefficient values, the boundary strength of the boundary of the current decoding unit is set to a second value. Based on the boundary strength of the current decoding unit's boundary, a filtered reconstruction block is obtained through filtering. The first value and the second value are different.

2. The method according to claim 1, characterized in that, The second value is 1.

3. The method according to claim 1 or 2, characterized in that, The first value is 2.

4. The method according to claim 1, characterized in that, The first value is 1.

5. The method according to claim 4, characterized in that, Also includes: The first value is incremented by 1 when one of the following conditions is met: At least one of the current decoding unit and the adjacent decoding unit adjacent to the boundary of the current decoding unit has non-zero transformation coefficients; The absolute difference between the motion vectors of the current decoding unit and the adjacent decoding unit is used to predict that it is greater than or equal to an integer sample. The current decoding unit and the adjacent decoding units are predicted based on different reference images; or The number of motion vectors used to predict the current decoding unit and the adjacent decoding units are different.

6. The method according to claim 1 or 2, characterized in that, Also includes: When the boundary of the current decoding unit is a horizontal edge, determine whether the adjacent decoding units adjacent to the boundary of the current decoding unit are in different decoding tree units (CTUs).

7. The method according to claim 1 or 2, characterized in that, Also includes: Determine whether the boundary of the current decoding unit is aligned with an 8×8 grid; After determining that the boundary of the current decoding unit is not aligned with the 8×8 grid, the boundary strength of the boundary of the current decoding unit is set to 0.

8. The method according to claim 1 or 2, characterized in that, Also includes: When the boundary strength of the boundary is greater than 0, the boundary of the luminance component is deblocked.

9. The method according to claim 1 or 2, characterized in that, Also includes: When the boundary strength of the boundary is greater than 1, the boundary of the chromaticity component is deblocked.

10. The method according to claim 1 or 2, characterized in that, When the current decoding unit is predicted using CIIP, the current decoding unit is considered as a decoding unit using intra-frame prediction during deblocking.

11. The method according to claim 1 or 2, characterized in that, Also includes: Predict other decoding units based on the filtered reconstruction block.

12. An encoder (20), characterized in that, Includes processing circuitry for performing the method according to any one of claims 1 to 11.

13. A decoder (30), characterized in that, Includes processing circuitry for performing the method according to any one of claims 1 to 11.

14. A computer-readable storage medium including instructions, characterized in that, When the computer executes the instructions, the instructions cause the computer to perform the method according to any one of claims 1 to 11.

15. A decoder (30), characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to and storing instructions executable by the one or more processors, wherein, when the one or more processors execute the instructions, they cause the decoder to perform the method according to any one of claims 1 to 11.

16. An encoder (20), characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to and storing instructions executable by the one or more processors, wherein, when the one or more processors execute the instructions, the encoder performs the method according to any one of claims 1 to 11.

Images