Video processing methods, video processing devices, encoders, decoders, media, and computer programs

Abstract

The present invention provides a video processing method and corresponding apparatus for improving coding efficiency. [Solution] A video processing method comprising the steps of: initializing an HMVP list for a current CTU row when the current CTU is the starting CTU of the current CTU row; and processing the current CTU row based on the HMVP list. By performing this method, encoding efficiency and decoding efficiency are improved.

Description

Technical Field

[0001] Embodiments of the present application (disclosure) generally relate to the field of video coding, and more particularly to video processing methods, video processing apparatuses, encoders, decoders, media, and computer programs.

Background Art

[0002] Video coding (video encoding and video decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission via the Internet and mobile networks, real-time conversation applications such as video chat, video conferencing, DVD and Blu-ray (registered trademark) discs, video content acquisition systems and video content editing systems, and camcorders for security applications.

[0003] Since the development of block-based hybrid video coding techniques in the 1990 H.261 standard, new video coding technologies and tools have been developed, forming the basis for new video coding standards. Further video coding standards include MPEG-1 video, MPEG-2 video, ITU-T H.262 / MPEG-2, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 / Versatile Video Coding (VVC), and extensions to these standards, such as scalability and / or three-dimensional (3D) extensions. As video creation and use become increasingly ubiquitous, video traffic has become the greatest load on communication networks and data storage, and therefore, one of the goals of many video coding standards has been to achieve a reduction in bitrate compared to previous standards without sacrificing picture quality. Even the latest High Efficiency Video Coding (HEVC) can compress video to about twice the size of AVC without sacrificing quality, meaning there is still a need to compress video even further compared to HEVC. [Overview of the Initiative] [Means for solving the problem]

[0004] Embodiments of this application provide a video processing method and a corresponding apparatus for improving coding efficiency.

[0005] The aforementioned and other objectives are achieved by the subject matter of the independent claims. Further implementations are evident from the independent claims, the embodiments for carrying out the invention, and the drawings.

[0006] A first aspect of the present invention provides a video processing method comprising the steps of: initializing a history-based motion vector prediction (HMVP) list for a current coding tree unit (CTU) row, when the current CTU is the starting CTU of the current CTU row; and processing the current CTU row based on the HMVP list. The starting CTU is sometimes called the starting CTU and is the first CTU of the same CTU row being processed.

[0007] The HMVP list for the current CTU row is initialized at the start of processing the current CTU row, and the process of the current CTU row does not need to rely on the HMVP list of the previous CTU row, thereby potentially improving encoding and decoding efficiency.

[0008] Referring to the first embodiment, in the first possible implementation of the first embodiment, the number of candidate motion vectors in the initialized HMVP list is zero.

[0009] Referring to the first embodiment, or the aforementioned implementation method of the first embodiment, in the second possible implementation method of the first embodiment, the current CTU row belongs to a picture area consisting of multiple CTU rows, and the current CTU row is one of the multiple CTU rows, for example, the first (e.g., topmost) CTU row, the second CTU row, ..., and the last (e.g., bottommost) CTU row of the picture area.

[0010] Referring to the first embodiment or any one of the aforementioned implementations of the first embodiment, a third possible implementation of the first embodiment further includes the step of initializing an HMVP list for each of a plurality of CTU rows, excluding the current CTU row, wherein the HMVP lists for the plurality of CTU rows are identical or different. In other words, embodiments may additionally initialize HMVP lists for all other CTU rows in the picture area, i.e., initialize HMVP lists for all CTU rows in the picture area.

[0011] Referring to the first embodiment or any one of the aforementioned implementations of the first embodiment, a fourth possible implementation of the first embodiment includes the step of processing a current CTU row based on an HMVP list, the step of processing the current CTU of the current CTU row, the step of updating an initialized HMVP list based on the processed current CTU, and the step of processing a second CTU of the current CTU row based on the updated HMVP list.

[0012] Referring to the first aspect, or any one of the aforementioned implementations of the first aspect, in a fifth possible implementation of the first aspect, the HMVP list is updated according to the processed CTUs of the current CTU row.

[0013] Referring to the first embodiment, or any one of the aforementioned implementations of the first embodiment, in the sixth possible implementation of the first embodiment, the HMVP list for the current CTU row is initialized as follows, i.e., to empty the HMVP list for the current CTU row:

[0014] Referring to the first embodiment or any one of the aforementioned implementations of the first embodiment, in a seventh possible implementation of the first embodiment, the step of processing a current CTU row based on an HMVP list is a step of processing a current CTU row based on an HMVP list from a second CTU of the current CTU row, the second CTU being adjacent to the start CTU, and the step of processing is as follows:

[0015] Referring to the first embodiment, or any one of the aforementioned implementations of the first embodiment, in the eighth possible implementation of the first embodiment, multiple CTU rows are processed in wavefront parallel processing (WPP) mode.

[0016] Currently, the HMVP list for CTU lines is initialized at the start of processing of the current CTU line. Therefore, when combined with WPP mode, CTU lines of picture frames or picture areas can be processed simultaneously, thereby potentially improving encoding and decoding efficiency.

[0017] Referring to the first aspect, or any one of the aforementioned implementations of the first aspect, in the ninth possible implementation of the first aspect, the current CTU row begins processing when a particular CTU of the previous CTU row is processed (or processing of the current CTU row begins).

[0018] Referring to the first aspect, or any one of the aforementioned implementations of the first aspect, in the tenth possible implementation of the first aspect, the previous CTU row is a CTU row that is directly adjacent to the current CTU row and is currently above or above the current CTU row.

[0019] Referring to the ninth implementation method of the first embodiment or the tenth implementation method of the first embodiment, in the eleventh possible implementation form of the first embodiment, a particular CTU in the preceding CTU row is the second CTU in the preceding CTU row; or a particular CTU in the preceding CTU row is the first CTU in the preceding CTU row.

[0020] A second aspect of the present invention provides a video processing device comprising: an initialization unit configured to initialize a history-based motion vector prediction (HMVP) list for a current CTU row when the current coding tree unit (CTU) is the starting CTU of the current CTU row; and a processing unit configured to process the current CTU row based on the HMVP list.

[0021] Referring to the second aspect, in the first possible implementation of the second aspect, the number of candidate motion vectors in the initialized HMVP list is zero.

[0022] Referring to the second aspect, or the aforementioned implementation method of the second aspect, in the second possible implementation method of the second aspect, the current CTU row belongs to a picture area consisting of multiple CTU rows, and the current CTU row is one of the multiple CTU rows.

[0023] Referring to the second aspect, or any one of the aforementioned implementations of the second aspect, in a third possible implementation of the second aspect, the initialization unit is further configured to initialize an HMVP list for each of a plurality of CTU rows, except for the current CTU row, wherein the HMVP lists for the plurality of CTU rows are identical or different.

[0024] Referring to the second aspect, or any one of the aforementioned implementations of the second aspect, in a fourth possible implementation of the second aspect, the processing unit is further configured to process the current CTU of the current CTU row, update the initialized HMVP list based on the processed current CTU, and process the second CTU of the current CTU row based on the updated HMVP list.

[0025] Referring to the second aspect, or any one of the aforementioned implementations of the second aspect, in a fifth possible implementation of the second aspect, the HMVP list is updated according to the processed CTUs of the currently CTU row.

[0026] Referring to the second aspect, or any one of the aforementioned implementations of the second aspect, in a sixth possible implementation of the second aspect, the initialization unit is further configured to initialize the HMVP list for the current CTU row, namely, to empty the HMVP list for the current CTU row.

[0027] Referring to either the second aspect or any one of the aforementioned implementation methods of the second aspect, in a seventh possible implementation method of the second aspect, the processing unit processes the current CTU row as follows, that is, based on the HMVP list from the second CTU of the current CTU row, and the second CTU is adjacent to the start CTU and is further configured to process the current CTU row based on the HMVP list so as to perform the processing.

[0028] Referring to either the second aspect or any one of the aforementioned implementation methods of the second aspect, in an eighth possible implementation method of the second aspect, a plurality of CTU rows are processed in a wavefront parallel processing (WPP) mode.

[0029] Referring to either the second aspect or any one of the aforementioned implementation methods of the second aspect, in a ninth possible implementation method of the second aspect, the current CTU row begins to be processed (or the processing of the current CTU row starts) when a specific CTU of the previous CTU row is processed.

[0030] Referring to either the second aspect or any one of the aforementioned implementation methods of the second aspect, in a tenth possible implementation method of the second aspect, the previous CTU row is a CTU row that is directly adjacent to and above the current CTU row.

[0031] Referring to the ninth implementation method or the tenth implementation method of the second aspect, in an eleventh possible implementation method of the second aspect, the specific CTU of the previous CTU row is the second CTU of the previous CTU row; or the specific CTU of the previous CTU row is the first CTU of the previous CTU row.

[0032] The third aspect of the present invention is a coding method implemented by a decoding device, including the steps of constructing / initializing an HMVP list for the current CTU row, and processing the CTU of the current CTU row based on the constructed / initialized HMVP list, to provide a coding method. Referring to the third aspect, in the first possible implementation of the third aspect, the HMVP list for the current CTU row is constructed / initialized in the following manner: namely, to empty the HMVP list for the current CTU row and / or to set a default value for the HMVP list for the current CTU row and / or to construct / initialize the HMVP list for the current CTU row based on the HMVP list of the CTU of the previous CTU row.

[0033] Referencing the first possible implementation method of the third embodiment, the second possible implementation method of the third embodiment includes the step of setting a default value for the HMVP list for the current CTU row, which includes the step of populating the MV of the HMVP list as the MV of a single prediction method, wherein the MV of the single prediction method is either a zero-motion vector or not a zero-motion vector, and the reference picture includes the first reference picture in the L0 list, and / or the step of populating the MV of the HMVP list as the MV of a dual prediction method, wherein the MV of the dual prediction method is either a zero-motion vector or not a zero-motion vector, and the reference picture includes the first reference picture in the L0 list and the first reference picture in the L1 list.

[0034] Referring to the first possible implementation method of the third embodiment, in the third possible implementation method of the third embodiment, each identical picture can store a temporal HMVP list for each CTU row or for the entire picture, and the step of setting a default value for the HMVP list for the current CTU row includes the step of initializing / constructing the HMVP list for the current CTU row based on the temporal HMVP list.

[0035] Referring to the first possible implementation method of the third embodiment, in the fourth possible implementation method of the third embodiment, the previous CTU row is a CTU row that is directly adjacent to the current CTU row and is currently above the current CTU row.

[0036] Referring to the fourth possible implementation method of the third embodiment, in the fifth possible implementation method of the third embodiment, the CTU of the previous CTU row is the second CTU of the previous CTU row.

[0037] Referring to the fourth possible implementation method of the third embodiment, in the fifth possible implementation method of the third embodiment, the CTU of the previous CTU row is the first CTU of the previous CTU row.

[0038] A fourth aspect of the present invention provides a coding method implemented by an encoding device, comprising the steps of: constructing / initializing an HMVP list for the current CTU row; and processing the CTU of the current CTU row based on the constructed / initialized HMVP list.

[0039] Referring to the fourth aspect, in the first possible implementation of the fourth aspect, the HMVP list for the current CTU row is constructed / initialized in the following manner: namely, to empty the HMVP list for the current CTU row; and / or to set a default value for the HMVP list for the current CTU row; and / or to construct / initialize the HMVP list for the current CTU row based on the HMVP list of the CTU of the previous CTU row.

[0040] Referencing the first possible implementation method of the fourth embodiment, the second possible implementation method of the fourth embodiment includes the step of setting a default value for the HMVP list for the current CTU row, which includes the step of populating the MV of the HMVP list as the MV of a single prediction method, wherein the MV of the single prediction method is either a zero-motion vector or not a zero-motion vector, and the reference picture includes a first reference picture in the L0 list; and / or the step of populating the MV of the HMVP list as the MV of a dual prediction method, wherein the MV of the dual prediction method is either a zero-motion vector or not a zero-motion vector, and the reference picture includes a first reference picture in the L0 list and a first reference picture in the L1 list.

[0041] Referring to the first possible implementation method of the fourth embodiment, in the third possible implementation method of the fourth embodiment, each identical picture may store a temporal HMVP list for each CTU row or for the entire picture, and the step of setting a default value for the HMVP list for the current CTU row includes the step of initializing / constructing the HMVP list for the current CTU row based on the temporal HMVP list.

[0042] Referring to the first possible implementation method of the fourth aspect, in the fourth possible implementation method of the fourth aspect, the previous CTU row is a CTU row that is directly adjacent to the current CTU row and is currently above or above the current CTU row.

[0043] Referring to the fourth possible implementation method of the fourth aspect, in the fifth possible implementation method of the fourth aspect, the CTU in the previous CTU row is the second CTU in the previous CTU row.

[0044] Referring to the fourth possible implementation method of the fourth aspect, in the sixth possible implementation method of the fourth aspect, the CTU of the previous CTU row is the first CTU of the previous CTU row.

[0045] A fifth aspect of the present invention provides an encoder comprising a processing circuit for performing a method according to any one of the first aspect or an implementation of the first aspect, or according to any one of the third aspect or an implementation of the third aspect, or according to any one of the fourth aspect or an implementation of the fourth aspect. For example, the encoder may include an initialization circuit configured to initialize a history-based motion vector prediction (HMVP) list for a current CTU row when the current coding tree unit (CTU) is the starting CTU of the current CTU row, and a processing circuit configured to process the current CTU row based on the HMVP list.

[0046] A sixth aspect of the present invention provides a decoder comprising processing circuitry for performing a method according to any one of the first aspect or an implementation of the first aspect, or according to any one of the third aspect or an implementation of the third aspect, or according to any one of the fourth aspect or an implementation of the fourth aspect. For example, the decoder may include an initialization circuitry configured to initialize a history-based motion vector prediction (HMVP) list for a current CTU row when the current coding tree unit (CTU) is the starting CTU of the current CTU row, and a processing circuitry configured to process the current CTU row based on the HMVP list.

[0047] A seventh aspect of the present invention provides a computer program product comprising program code for executing a method according to any one of the first aspect or an implementation method of the first aspect, or according to any one of the third aspect or an implementation method of the third aspect, or according to any one of the fourth aspect or an implementation method of the fourth aspect.

[0048] An eighth aspect of the present invention provides a computer-readable storage medium that stores computer instructions, wherein when the computer instructions are executed by one or more processors, the computer instructions cause one or more processors to execute a method according to any one of the first aspect or implementation methods of the first aspect, or according to any one of the third aspect or implementation methods of the third aspect, or according to any one of the fourth aspect or implementation methods of the fourth aspect.

[0049] A ninth aspect of the present invention provides a decoder comprising one or more processors and a non-temporary computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the decoder is configured to perform, when the program is executed by the processor, a method according to any one of the first aspect or implementations of the first aspect, or a method according to any one of the third aspect or implementations of the third aspect, or a method according to any one of the fourth aspect or implementations of the fourth aspect.

[0050] A tenth aspect of the present invention provides an encoder comprising one or more processors and a non-temporary computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processor, configures the encoder to perform a method according to any one of the first aspect or implementations of the first aspect, or according to any one of the third aspect or implementations of the third aspect, or according to any one of the fourth aspect or implementations of the fourth aspect.

[0051] Embodiments of the present invention will be described in more detail below with reference to the accompanying figures and drawings. [Brief explanation of the drawing]

[0052] [Figure 1A]This is a block diagram showing an example of a video coding system configured to implement embodiments of the present invention. [Figure 1B] This is a block diagram showing another example of a video coding system configured to implement embodiments of the present invention. [Figure 2] This is a block diagram showing an example of a video encoder configured to implement an embodiment of the present invention. [Figure 3] This is a block diagram showing one exemplary structure of a video decoder configured to implement an embodiment of the present invention. [Figure 4] This is a block diagram showing an example of an encoding or decoding device. [Figure 5] This is a block diagram showing other examples of encoding or decoding devices. [Figure 6] This diagram shows the spatially adjacent block locations used in merging and AMVP candidate list construction. [Figure 7] This is a flowchart for decoding using the HMVP method. [Figure 8] This is the WPP processing order block. [Figure 9] This is a flowchart illustrating one exemplary operation of a video decoder according to one embodiment. [Figure 10] This is a flowchart illustrating one exemplary operation according to one embodiment. [Figure 11] An example of a video processing device. [Modes for carrying out the invention]

[0053] In the following, unless otherwise specified, identical reference numerals refer to the same or at least functionally equivalent features.

[0054] In the following description, reference will be made to the accompanying figures, which form part of this disclosure and, as an example, illustrate specific embodiments of the present invention or specific ways in which embodiments of the present invention may be used. Embodiments of the present invention may be used in other embodiments, and it will be understood that these may include structural or logical modifications not shown in these figures. The following detailed description should therefore not be understood as limiting, and the scope of the present invention is defined by the appended claims.

[0055] For example, disclosures relating to a method described may also apply to a corresponding device or system configured to perform that method, and vice versa. For example, when describing one or more specific method steps, the corresponding device may include one or more units, e.g., functional units, even if one or more units for performing the method steps described are not explicitly described or shown in the figures (e.g., one unit performs one or more steps, or multiple units each perform one or more of the multiple steps). On the other hand, for example, when describing a particular device based on one or more units, e.g., functional units, the corresponding method may include one step for performing the functionality of one or more units, even if one or more such steps are not explicitly described or shown in the figures (e.g., one step performs the functionality of one or more units, or multiple steps each perform the functionality of one or more of the multiple units). Furthermore, it is understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with each other unless otherwise expressly provided.

[0056] Video coding generally refers to the processing of a sequence of pictures that make up a video or video sequence. The terms “frame” or “image” may be used synonymously in the field of video coding instead of “picture.” As used in this application (or disclosure), video coding refers to either video encoding or video decoding. Video encoding generally occurs on the source side, including processing the original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding occurs on the destination side and generally involves reverse processing compared to the encoder to reconstruct the video picture. Embodiments referring to “coding” a video picture (or, as will be described later, generally a picture) should be understood to refer to either “encoding” or “decoding” a video sequence. The combination of the encoding and decoding portions is also called a CODEC (coding and decoding).

[0057] In lossless video coding, the original video picture is reconstructible; that is, the reconstructed video picture has the same quality as the original video picture (assuming there is no transmission loss or other data loss during storage or transmission). In lossy video coding, further compression is performed, for example by quantization, to reduce the amount of data representing the video picture, but these video pictures cannot be fully reconstructed in the decoder; that is, the quality of the reconstructed video picture is lower or worse than the quality of the original video picture.

[0058] Since H.261, several video coding standards belong to the group of “lossy hybrid video codecs” (i.e., combining spatial and temporal prediction in the sample region with 2D transform coding to apply quantization in the transform region). Each picture in a video sequence is generally divided into a set of non-overlapping blocks, and coding is generally performed at the block level. In other words, in an encoder, video is generally processed, i.e., encoded, at the block (video block) level by, for example, using spatial (intra-picture) and temporal (inter-picture) prediction to generate prediction blocks, subtracting the prediction blocks from the current block (the block currently being processed / will be processed) to obtain residual blocks, transforming the residual blocks, and quantizing the residual blocks in the transform region to reduce (compress) the amount of data that will be transmitted, while in a decoder, in contrast to an encoder, partial inverse processing is applied to the encoded or compressed blocks to reconstruct the current block for representation. Furthermore, the encoder duplicates the decoder processing loop so that both generate identical predictions (e.g., intra-predictions and inter-predictions) and / or reconstructions for processing subsequent blocks, i.e., for coding.

[0059] As used herein, the term “block” may refer to a portion of a picture or frame. For ease of explanation, embodiments of the present invention are described herein with reference to High Efficiency Video Coding (HEVC) or the standard software for Multipurpose Video Coding (VVC) developed by the ITU-T Video Coding Expert Group (VCEG) and the ISO / IEC Motion Picture Expert Group (MPEG) Joint Collaboration Team for Video Coding (JCT-VC). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC. Embodiments of the present invention may refer to CUs, PUs, and TUs. In HEVC, a CTU is divided into CUs by using a quadtree structure shown as a coding tree. The decision of whether to code a picture area using interpicture (time) prediction or intrapicture (spatial) prediction is made at the CU level. Each CU may be further divided into one, two, or four PUs, depending on the PU division type. Within a single PU, the same prediction process is applied, and relevant information is sent to the decoder on a PU basis. After obtaining residual blocks by applying a prediction process based on the PU partitioning type, the CU can be partitioned into transformation units (TUs) according to other quadtree structures similar to coding trees for the CUs. In the development of modern video compression technologies, quadtree and binary tree (QTBT) partitioning frames are used to partition coding blocks. In the QTBT block structure, the CU can have either a square or rectangular shape. For example, a coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. The binary tree leaf nodes are called coding units (CUs), and their segmentation is used for prediction and transformation processing without further partitioning. This means that CUs, PUs, and TUs have the same block size in the QTBT coding block structure. In parallel, polypartitioning, such as ternary tree partitioning, has also been proposed for use with the QTBT block structure.

[0060] Embodiments of the encoder 20, decoder 30, and coding system 10 will be described below with reference to Figures 1 to 3.

[0061] Figure 1A is a conceptual or schematic block diagram showing one exemplary coding system 10, for example, a video coding system 10 that may utilize the technology of this application (this disclosure). The encoder 20 (e.g., video encoder 20) and decoder 30 (e.g., video decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform the technology described in the various examples in this application. As shown in Figure 1A, the coding system 10 includes a source device 12 configured to provide encoded data 13, for example, an encoded picture 13, to a destination device 14, for example, in order to decode the encoded data 13.

[0062] The source device 12 includes an encoder 20 and may additionally, optionally, include a picture source 16, a preprocessing unit 18, for example, a picture preprocessing unit 18, and a communication interface or communication unit 22.

[0063] The picture source 16 may include, for example, any kind of picture capture device for capturing real-world pictures, and / or any kind of picture or comment generation device (in the case of screen content coding, some text on the screen will also be considered part of the picture or image to be encoded), for example, a computer graphics processor for generating computer video pictures, or any kind of device for acquiring and / or providing real-world pictures, computer video pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures).

[0064] A (digital) picture is a two-dimensional array or matrix of samples having intensity values, or can be considered as such. Samples within the array are sometimes called pixels (short for picture elements) or pels. The number of samples in the horizontal and vertical (or axis) directions of the array or picture defines the size and / or resolution of the picture. To represent color, generally three color components are employed; that is, a picture can or may contain a three-sample array. In the RGB format or color space, a picture contains corresponding red, green, and blue sample arrays. However, in video coding, each pixel is generally represented by a luminance / chromaticity format or color space, e.g., YCbCr, containing a luminance component represented by Y (sometimes L is used instead) and two chromaticity components represented by Cb and Cr. The luminance (or, short, luma) component Y represents brightness or density intensity (for example, as in a grayscale picture), and the two chromaticity (or, short, chroma) components Cb and Cr represent chromaticity or color information components. Therefore, a picture in YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chromaticity sample arrays of chromaticity values ​​(Cb and Cr). A picture in RGB format can be converted to or transformed to YCbCr format, and vice versa; this process is also called a color transformation or conversion. If a picture is monochrome, it may contain only a luminance sample array.

[0065] In monochromatic sampling, there is only one sample sequence, which is nominally considered a luma sequence.

[0066] In 4:2:0 sampling, each of the two chroma sequences has half the height and half the width of the luma sequence.

[0067] In 4:2:2 sampling, each of the two chroma sequences has the same height and half the width of the luma sequence.

[0068] In 4:4:4 sampling, the following applies depending on the value of separate_colour_plane_flag: - If separate_colour_plane_flag is equal to 0, each of the two chroma sequences has the same height and width as the luma sequence. - If not (separate_colour_plane_flag is equal to 1), the three color planes are processed separately as monochrome sampled pictures.

[0069] The picture source 16 (for example, the video source 16) may be, for example, a camera for capturing pictures, memory, for example, a picture memory containing or storing previously captured or generated pictures, and / or any kind of interface (internal or external) for acquiring or receiving pictures. The camera may be, for example, a local camera, or, for example, an integrated camera integrated within the source device, and the memory may be local memory, or, for example, an integrated memory integrated within the source device. The interface may be, for example, an external video source, for example, a camera, external memory, or an external picture generation device, for example, an external computer graphics processor, computer, or server, or an external interface for receiving pictures from an external picture capture device. The interface may be any kind of interface, for example, a wired interface or wireless interface, or an optical interface, by any characteristic or standardized interface protocol. The interface for acquiring picture data 17 may be the same interface as the communication interface 22, or a part of the communication interface 22.

[0070] To distinguish it from the preprocessing unit 18 and the processing performed by the preprocessing unit 18, the picture or picture data 17 (for example, video data 16) is sometimes referred to as the raw picture or raw picture data 17.

[0071] The preprocessing unit 18 is configured to receive (raw) picture data 17, perform preprocessing on the picture data 17, and obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessing unit 18 may include, for example, cropping, color format conversion (e.g., RGB to YCbCr), color correction, or noise reduction. It can be understood that the preprocessing unit 18 may be an optional component.

[0072] The encoder 20 (for example, a video encoder 20) is configured to receive preprocessed picture data 19 and provide encoded picture data 21 (further details are described below, for example, based on Figure 2 or Figure 4).

[0073] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and send it to another device, for example, the destination device 14 or any other device, for storage or direct reconstruction, or to process the encoded picture data 21 before storing the encoded data 13 and / or before sending the encoded data 13 to another device, for example, the destination device 14 or any other device, for decoding or storage.

[0074] The destination device 14 includes a decoder 30 (for example, a video decoder 30) and may additionally, i.e., optionally, include a communication interface or communication unit 28, a post-processing unit 32, and a display device 34.

[0075] The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 or encoded data 13, for example, directly from the source device 12 or from any other source, for example, a storage device, for example, an encoded picture data storage device.

[0076] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired network or a wireless network, or any combination thereof, or any type of private and public network, or any combination thereof.

[0077] The communication interface 22 may be configured to package the encoded picture data 21 into an appropriate format, such as a packet, for transmission over a communication link or communication network.

[0078] The communication interface 28, which is the counterpart to the communication interface 22, may be configured, for example, to depackage the encoded data 13 in order to obtain the encoded picture data 21.

[0079] Both communication interfaces 22 and 28 may be configured as one-way or two-way communication interfaces, as indicated by the arrows to the encoded picture data 13 in Figure 1A, pointing from source device 12 to destination device 14, and may be configured, for example, to set up a connection and send and receive messages to acknowledge and exchange any other information regarding the communication link and / or data transmission, such as the transmission of encoded picture data.

[0080] The decoder 30 is configured to receive the encoded picture data 21 and provide the decoded picture data 31 or the decoded picture 31 (further details are described below, for example, based on Figure 3 or Figure 5).

[0081] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), for example, the decoded picture 31, in order to obtain the post-processed picture data 33, for example, the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display, for example, by the display device 34.

[0082] The display device 34 of the destination device 14 is configured to receive picture data 33 that has been post-processed to display the picture, for example, to a user or viewer. The display device 34 may be, or may include, any type of display for representing the reconstructed picture, for example, an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0083] Figure 1A shows the source device 12 and the destination device 14 as separate devices, but the device embodiment may have both or both functionalities, the source device 12 or its corresponding functionality, and the destination device 14 or its corresponding functionality. In such embodiments, the source device 12 or its corresponding functionality and the destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0084] As will become apparent to those skilled in the art based on the description, the presence and (exact) separation of the functionality of different units or the functionality within the source device 12 and / or destination device 14, as shown in Figure 1A, may vary depending on the actual device and application.

[0085] The encoder 20 (e.g., video encoder 20) and the decoder 30 (e.g., video decoder 30) may each be implemented as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If these technologies are partially implemented in software, the device may store instructions for the software in a suitable, non-temporary computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the technologies of this disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered as one or more processors. The video encoder 20 and the video decoder 30 may each be contained within one or more encoders or decoders, and any of them may be integrated as part of a combined encoder / decoder (CODEC) within their respective devices.

[0086] Source device 12 is sometimes called a video coding device or video coding apparatus. Destination device 14 is sometimes called a video decoding device or video decoding apparatus. Source device 12 and destination device 14 may be examples of video coding devices or video coding apparatus.

[0087] The source device 12 and destination device 14 may comprise any of a wide range of devices, including any type of handheld or stationary device, such as a notebook computer or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content distribution server), a broadcast receiver device, a broadcast transmitter device, and may or may not use an operating system, or may use any type of operating system.

[0088] In some cases, the source device 12 and the destination device 14 may be equipped for wireless communication. Therefore, the source device 12 and the destination device 14 may be wireless communication devices.

[0089] In some cases, the video coding system 10 shown in Figure 1A is merely an example, and the technology of this application may be applied to video coding configurations (e.g., video coding or video decoding) that do not necessarily involve any data communication between the coding device and the decoding device. In other examples, data may be retrieved from local memory and streamed over a network. The video coding device can code the data and store it in memory, and / or the video decoding device can retrieve the data from memory and decode it. In some examples, coding and decoding are performed by devices that do not communicate with each other, but simply code the data into memory and / or retrieve the data from memory and decode it.

[0090] For each of the above examples described with reference to the video encoder 20, it should be understood that the video decoder 30 may be configured to perform the reverse process. With respect to the signaling of syntax elements, the video decoder 30 is configured to receive and parse such syntax elements and, accordingly, decode the associated video data. In some examples, the video encoder 20 may entropically encode one or more syntax elements into an encoded video bitstream. In such examples, the video decoder 30 may parse such syntax elements and, accordingly, decode the associated video data.

[0091] Figure 1B is an exemplary diagram of another exemplary video coding system 40, including the encoder 20 of Figure 2 and / or the decoder 30 of Figure 3, according to one exemplary embodiment. The system 40 may implement the techniques described in various examples in this application. In the shown implementation, the video coding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video coder implemented via the logic circuits 47 of a processing unit 46), an antenna 42, one or more processors 43, one or more memory stores 44, and / or a display device 45.

[0092] As illustrated, the imaging device 41, antenna 42, processing unit 46, logic circuit 47, video encoder 20, video decoder 30, processor 43, memory store 44, and / or display device 45 may be able to communicate with each other. Although both the video encoder 20 and video decoder 30 are shown as described above, the video coding system 40 may, in various examples, include only the video encoder 20 or only the video decoder 30.

[0093] As illustrated, in some examples, the video coding system 40 may include an antenna 42. The antenna 42 may be configured, for example, to transmit or receive an encoded bitstream of video data. Furthermore, in some examples, the video coding system 40 may include a display device 45. The display device 45 may be configured to present video data. As illustrated, in some examples, the logic circuits 47 may be implemented via a processing unit 46. The processing unit 46 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. The video coding system 40 may also include any processor 43, which may similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. In some examples, the logic circuits 47 may be implemented via hardware, video coding-specific hardware, etc., and the processor 43 may be implemented via general-purpose software, an operating system, etc. In addition, the memory store 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In an unrestricted example, the memory store 44 may be implemented by a cache memory. In some examples, the logic circuit 47 may access the memory store 44 (e.g., for the implementation of an image buffer). In other examples, the logic circuit 47 and / or the processing unit 46 may include a memory store (e.g., a cache) for the implementation of an image buffer or the like.

[0094] In some examples, the video encoder 20 implemented via logic circuits may include an image buffer (for example, via either a processing unit 46 or a memory store 44) and a graphics processing unit (for example, via processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video encoder 20 and / or any other encoder system or subsystem described herein, implemented via logic circuits 47 to perform various modules as described with respect to Figure 2. The logic circuits may be configured to perform various operations as described herein.

[0095] The video decoder 30 may be implemented in a manner similar to how it is implemented via logic circuits 47 to implement various modules, such as those described with respect to the decoder 30 in Figure 3 and / or any other decoder systems or subsystems described herein. In some examples, the video decoder 30 may be implemented via logic circuits and may include an image buffer (e.g., via either a processing unit 420 or a memory store 44) and a graphics processing unit (e.g., via a processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 and / or any other decoder systems or subsystems described herein, such as how it is implemented via logic circuits 47 to implement various modules, such as those described with respect to Figure 3.

[0096] In some examples, the antenna 42 of the video coding system 40 may be configured to receive an encoded bitstream of video data. As described herein, the encoded bitstream may include data, indicators, index values, mode selection data, etc., related to encoding video frames as described herein, such as data related to coding divisions (e.g., conversion coefficients or quantization conversion coefficients, any indicators (as described), and / or data defining coding divisions). The video coding system 40 may also include a video decoder 30 coupled to the antenna 42 and configured to decode the encoded bitstream. A display device 45 is configured to present video frames.

[0097] Encoder and encoding method Figure 2 shows a schematic / conceptual block diagram of one exemplary video encoder 20 configured to implement the technology of the present application. In the example of Figure 2, the video encoder 20 includes a residual calculation unit 204, a transformation unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transformation unit 212, a reconstruction unit 214, a buffer 216, a loop filter unit 220, a decoded picture buffer (DPB) 230, a prediction unit 260, and an entropy coding unit 270. The prediction unit 260 may include an inter-prediction unit 244, an intra-prediction unit 254, and a mode selection unit 262. The inter-prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 as shown in Figure 2 is sometimes called a hybrid video encoder or a video encoder with a hybrid video codec.

[0098] For example, the residual calculation unit 204, the transformation processing unit 206, the quantization unit 208, the prediction processing unit 260, and the entropy coding unit 270 form the forward signal path of the encoder 20, while, for example, the inverse quantization unit 210, the inverse transformation processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, and the prediction processing unit 260 form the backward signal path of the encoder, and the backward signal path of the encoder corresponds to the signal path of the decoder (see decoder 30 in Figure 3).

[0099] The encoder 20 is configured, for example, to receive a block 203 of picture 20 or picture 201, for example, a picture of a sequence of pictures that form a video or video sequence, via input 202. Picture block 203 may also be called the current picture block or the picture block to be coded, and picture 201 may be called the current picture or the picture to be coded (particularly in video coding to distinguish the current picture from other pictures, for example, pictures encoded and / or decoded before the same video sequence, i.e., a video sequence that also contains the current picture).

[0100] division Embodiments of encoder 20 may include a splitting unit (not shown in Figure 2) configured to divide picture 201 into multiple blocks, which are generally multiple non-overlapping blocks, such as block 203. The splitting unit may use the same block size and corresponding grid defining the block size for all pictures in the video sequence, or it may be configured to change the block size between pictures or between subsets or groups of pictures, dividing each picture into a corresponding block.

[0101] In one example, the prediction processing unit 260 of the video encoder 20 may be configured to perform any combination of the division techniques described above.

[0102] Like picture 201, block 203 is, in this case, a two-dimensional array or matrix of samples having intensity values ​​(sample values), although it is smaller in dimensions than picture 201, or can be considered a two-dimensional array or matrix. In other words, block 203 may contain, depending on the applied color format, for example, one sample array (e.g., one lumen array for monochrome picture 201), or three sample arrays (e.g., one lumen array and two chromen arrays for color picture 201), or any other number and / or type of arrays. The number of samples in the horizontal and vertical (or axis) directions of block 203 defines the size of block 203.

[0103] The encoder 20 shown in Figure 2 is configured to encode the picture 201 in blocks, and for example, encoding and prediction are performed for each block 203.

[0104] Residual calculation The residual calculation unit 204 is configured to calculate the residual block 205 based on the picture block 203 and the prediction block 265 (further details regarding the prediction block 265 will be described later), for example, by subtracting the sample value of the prediction block 265 from the sample value of the picture block 203 for each sample (for each pixel) in order to obtain the residual block 205 within the sample region.

[0105] conversion The transformation processing unit 206 is configured to apply a transformation, such as a discrete cosine transform (DCT) or discrete sine transform (DST), to the sample values ​​of the residual block 205 in order to obtain transformation coefficients 207 within the transformation domain. The transformation coefficients 207 are sometimes called transformation residual coefficients and represent the residual block 205 within the transformation domain.

[0106] The conversion processing unit 206 may be configured to apply an integer approximation of the DCT / DST, such as the conversion specified for HEVC / H.265. Compared to the orthogonal DCT conversion, such an integer approximation is generally scaled by a certain coefficient. An additional scaling coefficient is applied as part of the conversion process to maintain a reference for the residual blocks processed by the forward and inverse conversions. The scaling coefficient is generally selected based on certain constraints, such as the scaling coefficient being a power of 2 in the shift operation, the bit depth of the conversion coefficient, and the trade-off between precision and implementation cost. A particular scaling coefficient may be specified, for example, by the inverse conversion processing unit 212 in the decoder 30 for the inverse conversion (and, for example, by the inverse conversion processing unit 212 in the encoder 20 for the corresponding inverse conversion), and accordingly, for example, by the conversion processing unit 206 in the encoder 20, a corresponding scaling coefficient may be specified for the forward conversion.

[0107] Quantization The quantization unit 208 is configured to quantize the transformation coefficients 207 to obtain quantization transformation coefficients 209, for example, by applying scalar quantization or vector quantization. The quantization transformation coefficients 209 are sometimes called quantization residual coefficients 209. The quantization process may reduce the bit depth associated with some or all of the transformation coefficients 207. For example, n-bit transformation coefficients may be truncated to m-bit transformation coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, in the case of scalar quantization, different scalings may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. Applicable quantization step sizes may be indicated by the quantization parameter (QP). The quantization parameter may be, for example, an index to a predefined set of applicable quantization step sizes. For example, a small quantization parameter may correspond to fine quantization (small quantization step size), a large quantization parameter may correspond to coarse quantization (large quantization step size), or vice versa. Quantization may involve division by the quantization step size, and corresponding or inverse dequantization by inverse quantization may involve multiplication by the quantization step size, for example. Some standards, for example, embodiments by HEVC, may be configured to use the quantization parameter to determine the quantization step size. Generally, the quantization step size may be calculated based on the quantization parameter using a fixed-point approximation of the equation, including division. Additional scaling factors may be introduced for quantization and dequantization to restore the residual block criterion, and the residual block criterion may be modified by the scaling used in the fixed-point approximation of the equation for the quantization step size and quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization may be combined.Alternatively, a customized quantization table may be used, for example, signaling from the encoder to the decoder within the bitstream. Quantization is a lossy operation, and the loss increases with increasing quantization step size.

[0108] The inverse quantization unit 210 is configured to apply the inverse quantization of the quantization unit 208 to the quantization coefficients in order to obtain the dequantization coefficient 211, for example, by applying the inverse of the quantization scheme applied by the quantization unit 208, based on or using the same quantization step size as the quantization unit 208. The dequantization coefficient 211 is sometimes called the dequantized residual coefficient 211 and generally corresponds to the transformation coefficient 207, although it is not identical to the transformation coefficient due to the loss due to quantization.

[0109] The inverse transform processing unit 212 is configured to apply the inverse transform of the transform applied by the transform processing unit 206, for example, the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), in order to obtain the inverse transform block 213 within the sample region. The inverse transform block 213 is sometimes referred to as the inverse transform dequantized block 213 or the inverse transform residual block 213.

[0110] The reconstruction unit 214 (for example, an adder 214) is configured to add the inverse transformed block 213 (i.e., the reconstructed residual block 213) to the predictive block 265 by adding, for example, the sample values ​​of the reconstructed residual block 213 to the sample values ​​of the predictive block 265, thereby obtaining the reconstructed block 215 within the sample region.

[0111] An optional buffer unit 216 (or, for short, “buffer” 216), such as a line buffer 216, is configured to buffer or store the reconstructed block 215 and their respective sample values, for example, for intra-prediction. In a further embodiment, the encoder may be configured to use the unfiltered, reconstructed block and / or the respective sample values ​​stored in the buffer unit 216 for any kind of estimation and / or prediction, such as for intra-prediction.

[0112] Embodiments of the encoder 20 may be configured such that, for example, a buffer unit 216 is used to store the reconstructed block 215 not only for the intra-prediction 254 but also for the loop filter unit 220 (not shown in Figure 2), and / or, for example, the buffer unit 216 and the decoded picture buffer unit 230 form a single buffer. Further embodiments may be configured to use a filtered block 221 and / or a block or sample from the decoded picture buffer 230 (neither shown in Figure 2) as input or basis for the intra-prediction 254.

[0113] The loop filter unit 220 (or, for short, the "loop filter" 220) is configured to filter the reconstructed block 215 to obtain the filtered block 221, for example, to smooth pixel transitions or otherwise to improve video quality. The loop filter unit 220 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters, such as a bilateral filter, or an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a co-filter. The loop filter unit 220 is shown as an in-loop filter in Figure 2, but in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 is sometimes referred to as the filtered, reconstructed block 221. The decoded picture buffer 230 may store the reconstructed coding block after the loop filter unit 220 has performed a filtering operation on the reconstructed coding block.

[0114] Embodiments of the encoder 20 (each a loop filter unit 220) may be configured to output loop filter parameters (such as sample adaptive offset information) entropically encoded, for example, directly or via the entropy coding unit 270 or any other entropy coding unit, so that the decoder 30 can receive and apply the same loop filter parameters for decoding.

[0115] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use when encoding video data by the video encoder 20. The DPB 230 may be formed by any of a variety of memory devices, including dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), or resistive RAM (RRAM®), or other types of memory devices. The DPB 230 and buffer 216 may be provided by the same memory device or separate memory devices. In some examples, the decoded picture buffer (DPB) 230 is configured to store filtered blocks 221. The decoded picture buffer 230 may be further configured to store other previously filtered blocks of the same or different picture, for example, a previously reconstructed picture, such as a previously reconstructed and filtered block 221, so as to provide the fully reconstructed, i.e., decoded picture (and the corresponding reference blocks and samples) and / or partially reconstructed current picture (and the corresponding reference blocks and samples) for interpretation, for example. In some examples, if the reconstructed block 215 is reconstructed without in-loop filtering, the decoded picture buffer (DPB) 230 is configured to store the reconstructed block 215.

[0116] The prediction processing unit 260, sometimes called the block prediction processing unit 260, is configured to receive or acquire block 203 (the current block 203 of the current picture 201) and reconstructed picture data, for example, a reference sample of the same (current) picture, from buffer 216 and / or reference picture data 231 from one or more previously decoded pictures from the decoded picture buffer 230, and to process such data for prediction, i.e., to provide a predicted block 265 which may be an inter-predicted block 245 or an intra-predicted block 255.

[0117] The mode selection unit 262 may be configured to select a prediction mode (for example, intra-prediction mode or inter-prediction mode) and / or a corresponding prediction block 245 or 255 which will be used as the prediction block 265 for the calculation of the residual block 205 and for the reconstruction of the reconstructed block 215.

[0118] Embodiments of the mode selection unit 262 may be configured to select a prediction mode that, for example, from the prediction modes supported by the prediction processing unit 260, provides the best match, i.e., in other words, the minimum residual (where minimum residual means better compression for transmission or storage), or the minimum signaling overhead (where minimum signaling overhead means better compression for transmission or storage), or considers both, or balances both. The mode selection unit 262 may be configured to determine the prediction mode based on rate distortion optimization (RDO), i.e., to select a prediction mode that provides minimum rate distortion optimization, or which associated rate distortions at least satisfy the prediction mode selection criteria.

[0119] The following describes in more detail the prediction processing (e.g., by the prediction processing unit 260) and mode selection (e.g., by the mode selection unit 262) performed by one example encoder 20.

[0120] As described above, the encoder 20 is configured to determine or select the best or most optimal prediction mode from a set of (predetermined) prediction modes. The set of prediction modes may include, for example, an intra-prediction mode and / or an inter-prediction mode.

[0121] The set of intra-predictive modes may include 35 different intra-predictive modes, such as omnidirectional modes including DC (or average) mode and planar mode, or directional modes as defined in H.265, for example, or 67 different intra-predictive modes, such as omnidirectional modes including DC (or average) mode and planar mode, or directional modes as defined in H.266, which is currently under development.

[0122] The set of interpretation modes (or possible interpretation modes) depends on the available reference picture (i.e., a previous, at least partially decoded picture stored in DBP230), and other interpretation parameters, such as whether the entire reference picture is used to search for the best-matching reference block, or only a portion of the reference picture, such as the search window area around the current block's area, and / or whether pixel interpolation, such as half / semi-pel and / or quarter-pel interpolation, is applied or not.

[0123] In addition to the prediction modes described above, skip mode and / or direct mode may be applied.

[0124] The prediction processing unit 260 can be further configured to divide block 203 into smaller block segments or subblocks, for example, by iteratively using quadtree partitioning (QT), binary tree partitioning (BT), or ternary tree partitioning (TT), or any combination thereof, and to make predictions for each of the block segments or subblocks, and mode selection includes selecting the tree structure of the divided block 203 and the prediction mode to be applied to each of the block segments or subblocks.

[0125] The interpretation unit 244 may include a motion estimation (ME) unit (not shown in Figure 2) and a motion compensation (MC) unit (not shown in Figure 2). The motion estimation unit is configured to receive or acquire, for motion estimation, a picture block 203 (the current picture block 203 of the current picture 201) and a decoded picture 231, or at least one or more previously reconstructed blocks, for example, one or more other / different reconstructed blocks of a previously decoded picture 231. For example, a video sequence may include the current picture and a previously decoded picture 231, or in other words, the current picture and a previously decoded picture 231 may be, or may form part of a sequence of pictures that make up a video sequence.

[0126] The encoder 20 may be configured, for example, to select one reference block from multiple reference blocks of the same or different pictures of multiple other pictures, and to provide the motion estimation unit (not shown in Figure 2) as an interpretation parameter an offset (spatial offset) between the reference picture (or reference picture index, ...) and / or the position (x coordinate, y coordinate) of the reference block and the position of the current block. This offset is also called the motion vector (MV).

[0127] The motion compensation unit is configured to acquire interprediction parameters, for example, receive them, and perform interprediction based on or using the interprediction parameters to acquire interprediction blocks 245. Motion compensation performed by the motion compensation unit (not shown in Figure 2) may require interpolation to subpixel precision, in some cases, based on the motion / block vector determined by the motion estimation, to fetch or generate predictive blocks. Interpolation filtering may generate additional pixel samples from known pixel samples, thereby potentially increasing the number of candidate predictive blocks that can be used to code picture blocks. Now receiving the motion vector for the picture block PU, the motion compensation unit 246 may locate the predictive block pointed to by the motion vector in one of the reference picture lists. The motion compensation unit 246 may also generate block and syntax elements associated with the video slice for use by the video decoder 30 when decoding the picture blocks of the video slice.

[0128] The intra-prediction unit 254 is configured to receive, for example, the picture block 203 (current picture block) and one or more previously reconstructed blocks of the same picture, such as reconstructed neighboring blocks, for intra-prediction. The encoder 20 may be configured to select the intra-prediction mode from a plurality of (predetermined) intra-prediction modes, for example.

[0129] Embodiments of the encoder 20 may be configured to select an intra-prediction mode based on an optimization criterion, for example, minimum residual (e.g., an intra-prediction mode that provides a prediction block 255 that is most similar to the current picture block 203) or minimum rate distortion.

[0130] The intra-prediction unit 254 is further configured to determine an intra-prediction block 255 based on intra-prediction parameters, for example, a selected intra-prediction mode. In any case, after selecting an intra-prediction mode for a block, the intra-prediction unit 254 is also configured to provide the entropy coding unit 270 with intra-prediction parameters, i.e., information indicating the intra-prediction mode selected for that block. In one example, the intra-prediction unit 254 may be configured to perform any combination of intra-prediction techniques, which will be described later.

[0131] The entropy coding unit 270 is configured to obtain encoded picture data 21, which can be output by output 272, for example, in the form of an encoded bitstream 21, by applying an entropy coding algorithm or entropy coding scheme (e.g., variable-length coding (VLC) scheme, context-adaptive VLC scheme (CAVLC), arithmetic coding scheme, context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methodologies or techniques) individually or together (or not at all) to the quantization residual coefficients 209, inter-prediction parameters, intra-prediction parameters, and / or loop filter parameters. The encoded bitstream 21 may be transmitted to the video decoder 30, or it may be stored in an archive for later transmission or retrieval by the video decoder 30. The entropy coding unit 270 may be further configured to entropy code other syntax elements for the currently coded video slice.

[0132] Other structural variations of the video encoder 20 may be used to encode a video stream. For example, a non-conversion-based encoder 20 may directly quantize the residual signal for a given block or frame without a conversion processing unit 206. In other implementations, the encoder 20 may combine a quantization unit 208 and an inverse quantization unit 210 into a single unit.

[0133] Figure 3 shows an exemplary video decoder 30 configured to implement the technology of the present application. The video decoder 30 is configured to receive encoded picture data (e.g., encoded bitstream) 21, encoded by, for example, the encoder 20, in order to obtain a decoded picture 311. During the decoding process, the video decoder 30 receives from the video encoder 20 an encoded video bitstream and associated syntax elements representing video data, for example, picture blocks of an encoded video slice.

[0134] In the example in Figure 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transformation unit 312, a reconstruction unit 314 (e.g., an adder 314), a buffer 316, a loop filter 320, a decoded picture buffer 330, and a prediction unit 360. The prediction unit 360 may include an inter-prediction unit 344, an intra-prediction unit 354, and a mode selection unit 362. In some examples, the video decoder 30 may perform a decoding path that is generally the reverse of the encoding path described with respect to the video encoder 20 from Figure 2.

[0135] The entropy decoding unit 304 is configured to perform entropy decoding on the encoded picture data 21 to obtain, for example, the quantization coefficients 309 and / or the decoded coding parameters (not shown in Figure 3), such as any or all of the (decoded) inter-prediction parameters, intra-prediction parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 is further configured to transfer the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the prediction processing unit 360. The video decoder 30 may receive syntax elements at the video slice level and / or video block level.

[0136] The inverse quantization unit 310 may be functionally identical to the inverse quantization unit 210, the inverse transformation processing unit 312 may be functionally identical to the inverse transformation processing unit 212, the reconstruction unit 314 may be functionally identical to the reconstruction unit 214, the buffer 316 may be functionally identical to the buffer 216, the loop filter 320 may be functionally identical to the loop filter 220, and the decoded picture buffer 330 may be functionally identical to the decoded picture buffer 230.

[0137] The prediction processing unit 360 may include an inter-prediction unit 344 and an intra-prediction unit 354, the inter-prediction unit 344 may be functionally similar to the inter-prediction unit 244, and the intra-prediction unit 354 may be functionally similar to the intra-prediction unit 254. The prediction processing unit 360 is generally configured to perform block prediction and / or to obtain prediction blocks 365 from encoded data 21, and to receive or obtain (explicitly or implicitly) information regarding prediction relation parameters and / or selected prediction modes from, for example, an entropy decoding unit 304.

[0138] When a video slice is coded as an intra-coded (I) slice, the intra-prediction unit 354 of the prediction processing unit 360 is configured to generate a prediction block 365 for the picture block of the current video slice based on the signaled intra-prediction mode and data from blocks decoded before the current frame or current picture. When a video frame is coded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., motion compensation unit) of the prediction processing unit 360 is configured to create a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. In the case of inter-prediction, the prediction block may be created from one of the reference pictures in one of the reference picture lists. The video decoder 30 may construct reference frame lists, i.e., list 0 and list 1, using default construction techniques based on the reference pictures stored in the DPB 330.

[0139] The prediction processing unit 360 is configured to determine prediction information about the video blocks of the current video slice by parsing motion vectors and other syntax elements, and to use the prediction information to create prediction blocks for the current video blocks being decoded. For example, the prediction processing unit 360 uses some of the received syntax elements to determine the prediction mode used to code the video blocks of the video slice (e.g., intra-prediction or inter-prediction), the inter-prediction slice type (e.g., B-slice, P-slice, or GPB-slice), construction information about one or more of the reference picture lists for the slice, motion vectors for each intercoded video block of the slice, the inter-prediction state for each intercoded video block of the slice, and other information for decoding the video blocks in the current video slice.

[0140] The inverse quantization unit 310 is provided within the bitstream and is configured to inverse quantize, i.e., dequantize, the quantization conversion coefficients decoded by the entropy decoding unit 304. The inverse quantization process may include determining the degree of quantization, and similarly the degree of inverse quantization to be applied, using quantization parameters calculated by the video encoder 20 for each video block in the video slice.

[0141] The inverse transformation processing unit 312 is configured to apply inverse transformations, such as inverse DCT, inverse integer transformation, or conceptually similar inverse transformation processes, to the transformation coefficients in order to create residual blocks within the pixel region.

[0142] The reconstruction unit 314 (for example, an adder 314) is configured to add the inverse transformed block 313 (i.e., the reconstructed residual block 313) to the predictive block 365 by adding the sample values ​​of the reconstructed residual block 313 to the sample values ​​of the predictive block 365, thereby obtaining the reconstructed block 315 within the sample region.

[0143] The loop filter unit 320 is configured to filter the reconstructed block 315 (either during or after the coding loop) to obtain the filtered block 321, for example, to smooth pixel transitions or otherwise improve video quality. In one example, the loop filter unit 320 may be configured to perform any combination of filtering techniques described later. The loop filter unit 320 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters, such as a bilateral filter or an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a co-filter. The loop filter unit 320 is shown in Figure 3 as an in-loop filter, but in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0144] The decoded video block 321 within a given frame or picture is then stored in the decoded picture buffer 330, which stores a reference picture used for subsequent motion compensation.

[0145] The decoder 30 is configured to output the decoded picture 311, for example, via output 312, for display to the user or for viewing by the user.

[0146] Other variations of the video decoder 30 may be used to decode a compressed bitstream. For example, the decoder 30 may produce an output video stream without the loop filtering unit 320. For example, a non-transformation-based decoder 30 may directly dequantize the residual signal without the inverse transformation processing unit 312 for a given block or frame. In other implementations, the video decoder 30 may combine the inverse quantization unit 310 and the inverse transformation processing unit 312 into a single unit.

[0147] Figure 4 is a schematic diagram of a video coding device 400 according to one embodiment of the present disclosure. The video coding device 400 is suitable for implementing the disclosed embodiments as described herein. In one embodiment, the video coding device 400 may be a decoder such as the video decoder 30 in Figure 1A or an encoder such as the video encoder 20 in Figure 1A. In one embodiment, the video coding device 400 may be one or more components of the video decoder 30 in Figure 1A or the video encoder 20 in Figure 1A as described above.

[0148] The video coding device 400 includes an inlet port 410 and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an exit port 450 for transmitting data; and memory 460 for storing data. The video coding device 400 may also include optical-to-electrical (OE) and electric-to-optical (EO) components coupled to the inlet port 410, receiver unit 420, transmitter unit 440, and exit port 450 for optical or electrical signal exits or inlets.

[0149] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., a multi-core processor), FPGAs, ASICs, and DSPs. The processor 430 communicates with an inlet port 410, a receiver unit 420, a transmitter unit 440, an exit port 450, and memory 460. The processor 430 includes a coding module 470. The coding module 470 implements the embodiments disclosed above. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Including the coding module 470 therefore provides a considerable improvement to the functionality of the video coding device 400, causing the video coding device 400 to be converted into different states. Alternatively, the coding module 470 is implemented as instructions stored in memory 460 and executed by the processor 430.

[0150] Memory 460 includes one or more disks, tape drives, and solid drives and may be used as an overflow data storage device to store such programs when they are selected for execution, and to store instructions and data read during program execution. Memory 460 may be volatile and / or non-volatile and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random access memory (SRAM).

[0151] Figure 5 is a simplified block diagram of a device 500 that may be used as either or both of the source device 310 and destination device 320 from Figure 1, according to one exemplary embodiment. The device 500 may implement the technology of the present application as described above. The device 500 may be in the form of a computing system including multiple computing devices, or in the form of a single computing device, such as a mobile phone, tablet computer, laptop computer, notebook computer, or desktop computer.

[0152] The processor 502 within the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device, or multiple devices, that currently exist or will be developed in the future, that are capable of manipulating or processing information. The disclosed implementation can be practiced using a single processor, for example, processor 502, as illustrated, but advantages in speed and efficiency can be achieved by using two or more processors.

[0153] The memory 504 in the device 500 may, in one implementation, be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may contain code and data 506 accessed by the processor 502 using the bus 512. Memory 504 may further contain an operating system 508 and an application program 510, the application program 510 containing at least one program that allows the processor 502 to perform the method described herein. For example, the application program 510 may contain applications 1 through N, which further include video coding applications that perform the method described herein. The device 500 may also contain additional memory in the form of a secondary storage device 514, which may be, for example, a memory card used with a mobile computing device. Since video communication sessions can contain a considerable amount of information, these sessions may be stored in whole or in part in the secondary storage device 514 and loaded into memory 504 as needed for processing.

[0154] The device 500 may include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display combining a display with a touch-sensitive element capable of sensing touch input. The display 518 may be coupled to the processor 502 via a bus 512. In addition to, or instead of, the display 518, other output devices may be provided that allow a user to program or otherwise use the device 500. When the output device is a display or includes a display, the display may be implemented in a variety of ways, including by light-emitting diode (LED) displays such as liquid crystal displays (LCDs), cathode ray tube (CRT) displays, plasma displays, or organic LED (OLED) displays.

[0155] The device 500 may include, or communicate with, any other existing or future-developed image sensing device 520 capable of sensing images, such as a camera or an image of a user operating the device 500. The image sensing device 520 may be positioned so that it faces the user operating the device 500. In one example, the position and optical axis of the image sensing device 520 may be configured such that its field of view includes an area directly adjacent to the display 518 from which the display 518 is visible.

[0156] The device 500 may include, or communicate with, a voice sensing device 522, for example, a microphone, or any other voice sensing device that is currently existing or to be developed in the future and capable of sensing sound in the vicinity of the device 500. The voice sensing device 522 may be positioned so as to face the user operating the device 500 and may be configured to receive sound uttered by the user while the user is operating the device 500, for example, utterances or other speech.

[0157] Figure 5 shows the processor 502 and memory 504 of device 500 as integrated into a single unit, but other configurations may be available. The operation of processor 502 may be distributed across multiple machines (each machine having one or more processors) that can be coupled directly or over a local area network or other network. Memory 504 may be distributed across multiple machines, such as network-based memory or memory in multiple machines operating device 500. Although shown here as a single bus, the bus 512 of device 500 may consist of multiple buses. Furthermore, the secondary storage device 514 may be directly coupled to other components of device 500 or accessed over a network, and may include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Device 500 can therefore be implemented in a wide variety of configurations.

[0158] In VVC, the motion vectors of an interconnected block can be signaled in two ways: Advanced Motion Vector Prediction (AMVP) mode or Merge mode. In AMVP mode, the difference between the actual motion vector and the motion vector prediction (MVP), a reference index pointing to an AMVP candidate list, and an MVP index are signaled, where the reference index points to a reference picture from which a reference block is copied for motion compensation. In Merge mode, a merge index pointing to a merge candidate list is signaled, and all motion information associated with the merge candidate is inherited.

[0159] In both the AMVP candidate list and the merge candidate list, these are derived from coded blocks that are temporally or spatially close. More specifically, the merge candidate list is constructed by sequentially examining the following four types of merge MVP candidates: 1. As shown in Figure 6, a spatial merge candidate can be determined from five spatially adjacent blocks, namely blocks A0 and A1 located in the lower left corner, blocks B0 and B1 located in the upper right corner, and block B2 located in the upper left corner. 2. Time MVP (TMVP) merge candidate. 3. Combined bipredictive merging candidates. 4. Zero motion vector merging candidate.

[0160] The merge candidate list building process terminates when the number of available merge candidates reaches the signaled maximum number of possible merge candidates (e.g., 5 under typical test conditions). Note that the maximum number of possible merge candidates may differ under different conditions.

[0161] Similarly, in the case of an AMVP candidate list, three types of MVP candidates are examined in order: 1. Up to two spatial MVP candidates, one of which is determined from blocks B0, B1, and B2, as shown in Figure 6, and the other of which is determined from blocks A0 and A1, as shown in Figure 6. 2. Candidate for Time MVP (TMVP). 3. Zero MVP candidate.

[0162] The history-based motion vector prediction (HMVP) method was introduced by JVETK104 (accessible at http: / / phenix.it-sudparis.eu / jvet / ), which is an input document to the Joint Video Expert Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC29 / WG 11. Here, an HMVP candidate is defined as motion information from a previously coded block. A table with multiple HMVP candidates is maintained during the coding / decoding process. This table is emptied when a new slice is encountered. Whenever an intercoded block exists, the associated motion information is added to the last entry in the table as a new HMVP candidate. The overall coding flow, including the following, is shown in Figure 7.

[0163] Step 701: Load the HMVP candidates into the table.

[0164] Step 702: Decrypt the block with the HMVP candidate in the loaded table.

[0165] Step 703: When decrypting the block, update the table with the decrypted motion information.

[0166] Steps 701-703 may be performed in a circular fashion.

[0167] HMVP candidates can be used in the merge candidate list construction process. All HMVP candidates from the last entry to the first entry in the table are inserted after the TMVP candidates. Pruning may be applied to HMVP candidates. The merge candidate list construction process terminates when the total number of available merge candidates reaches the signaled maximum number of possible merge candidates.

[0168] The pruning operation involves identifying identical motion predictor candidates in a list and removing one of the identical candidates from the list.

[0169] Similarly, HMVP candidates may be used in the AMVP candidate list construction process. The motion vectors of the last K HMVP candidates in the table are inserted after the TMVP candidates. In some implementations, only HMVP candidates with the same reference picture as the AMVP target reference picture are used to construct the AMVP candidate list. Pruning may be applied to HMVP candidates.

[0170] To improve processing efficiency, a process called wavefront parallel processing (WPP) is introduced, where WPP mode allows rows of CTUs to be processed simultaneously. In WPP mode, each row of CTUs is processed relative to its preceding (directly adjacent) row of CTUs by using the delay of two consecutive CTUs. For example, referring to Figure 8, a picture frame or picture area consists of multiple rows of CTUs, and each thread (row) contains 11 CTUs, i.e., thread 1 contains CTU0 through CTU10, thread 2 contains CTU11 through CTU21, thread 3 contains CTU22 through CTU32, thread 4 contains CTU33 through 43, and so on. Therefore, in WPP mode, when the encoding / decoding process for CTU1 in thread 1 is completed, the encoding / decoding process for CTU11 in thread 2 may start; similarly, when the encoding / decoding process for CTU12 in thread 2 is completed, the encoding / decoding process for CTU22 in thread 3 may start; when the encoding / decoding process for CTU23 in thread 3 is completed, the encoding / decoding process for CTU33 in thread 4 may start; and when the encoding / decoding process for CTU34 in thread 4 is completed, the encoding / decoding process for CTU44 in thread 5 may start.

[0171] However, when WPP is combined with HMVP, as mentioned above, a single HMVP list is maintained, updated after each coding block is processed, and thus continues to be updated until the last CTU in the CTU line. Therefore, thread N cannot perform wavefront parallel processing because it must wait for the processing of the last CTU in the above CTU line to be completed.

[0172] Figure 9 is a flowchart illustrating one exemplary operation of a video decoder, such as the video decoder 30 of Figure 3, according to one embodiment of the present application. One or more structural elements of the video decoder 30, including the interpretation unit 344, may be configured to perform the art of Figure 9. In the example of Figure 9, the video decoder 30 may perform the following steps:

[0173] 901. At the start of processing a CTU row, a step is taken to build / initialize the HMVP list for the CTU row.

[0174] When the CTU to be processed is the first CTU (starting CTU) of the CTU row, an HMVP list for the CTU row is constructed or initialized, and therefore the first CTU of the CTU row may be processed based on the HMVP list for the CTU row.

[0175] When the method is an encoding method, the HMVP list for the CTU row can be constructed or initialized by the interprediction unit 344 in Figure 3. Alternatively, when the method is a decoding method, the HMVP list for the CTU row can be constructed or initialized by the interprediction unit 244 in Figure 2.

[0176] In one implementation, all CTU rows with different HMVP lists may be maintained for a picture frame. In another implementation, all CTU rows with different HMVP lists may be maintained for a picture area, where the picture area consists of multiple CTU rows, and the picture may be a slice, tile, or brick of VVC.

[0177] If a brick is a rectangular area of ​​a row of CTUs within a particular tile in a picture, then the tile may be divided into multiple bricks, each of which consists of one or more rows of CTUs within the tile. A tile that is not divided into multiple bricks is also called a brick. However, a brick that is a true subset of a tile is not called a tile.

[0178] Maintaining a different HMVP list for every CTU row means that a specific HMVP list may be maintained for a CTU row, but the candidates in different HMVP lists may be the same, for example, all candidates in one HMVP list are the same as the candidates in another HMVP list, and the candidates in one HMVP list may not be redundant; or the candidates in different HMVP lists may have overlap, for example, some of the candidates in one HMVP list are the same as some of the candidates in another HMVP list, and some of the candidates in that one HMVP list do not have identical candidates in the other HMVP list; or the candidates in different HMVP lists may be completely different, for example, none of the candidates in one HMVP list have identical candidates in the other HMVP list. Note that when all CTUs in a CTU row have been processed, the HMVP lists maintained for that CTU row may be freed, thus reducing storage requirements.

[0179] This disclosure provides the following methods for constructing / initializing an HMVP list:

[0180] Method 1: At the start of processing a CTU line, the corresponding HMVP list is either empty or set to a default value. The default value is a predetermined candidate known to both the encoder and the decoder.

[0181] For example, the corresponding HMVP list is populated with the following default MV: a) MV from a single prediction method, where MV can be a zero motion vector, the reference picture may include a first reference picture in the L0 list, and / or b) MV from the biprediction method, where MV may be a zero motion vector, and the reference picture may include a first reference picture in the L0 list and a first reference picture in the L1 list, and / or c) MVs of previously processed pictures according to the picture processing order. More specifically, MVs belonging to a previously processed picture and located in the spatial neighborhood of the current block when the current block position is overlaid on a previous picture. and / or d) MV of the temporal HMVP list, where each identical picture may store a temporal HMVP list for each CTU row or for the entire picture, and thus the temporal HMVP list may be used to build / initialize the HMVP list for the current CTU row.

[0182] Method 2: At the start of processing the current CTU row, the corresponding HMVP list is constructed / initialized based on the HMVP list of the second CTU of the previous CTU row, where the previous CTU row is the CTU row that is directly adjacent to the current CTU row and above the current CTU row.

[0183] Using Figure 8 as an example, when the current CTU row is the CTU row of thread 2, the previous CTU row was the CTU row of thread 1, and the second CTU of the previous row is CTU1; when the current CTU row is the CTU row of thread 3, the previous CTU row was the CTU row of thread 2, and the second CTU of the previous row is CTU12; when the current CTU row is the CTU row of thread 4, the previous CTU row was the CTU row of thread 3, and the second CTU of the previous row is CTU23; when the current CTU row is the CTU row of thread 5, the previous CTU row was the CTU row of thread 4, and the second CTU of the previous row is CTU34; when the current CTU row is the CTU row of thread 6, the previous CTU row was the CTU row of thread 5, and the second CTU of the previous row is CTU45.

[0184] Method 3: At the start of processing the current CTU row, the corresponding HMVP list is constructed / initialized based on the HMVP list of the first CTU of the previous CTU row, where the previous CTU row is the CTU row directly adjacent to the current CTU row and above the current CTU row.

[0185] Using Figure 8 as an example, when the current CTU row is the CTU row of thread 2, the previous CTU row was the CTU row of thread 1, and the first CTU of the previous row is CTU0; when the current CTU row is the CTU row of thread 3, the previous CTU row was the CTU row of thread 2, and the first CTU of the previous row is CTU11; when the current CTU row is the CTU row of thread 4, the previous CTU row was the CTU row of thread 3, and the first CTU of the previous row is CTU22; when the current CTU row is the CTU row of thread 5, the previous CTU row was the CTU row of thread 4, and the first CTU of the previous row is CTU33; when the current CTU row is the CTU row of thread 6, the previous CTU row was the CTU row of thread 5, and the first CTU of the previous row is CTU44.

[0186] According to methods 1 to 3, the processing of the current CTU line does not need to wait for the processing of the preceding CTU line to be completed, and therefore the processing efficiency of the current picture frame can be improved.

[0187] 902. Process the CTUs in the CTU row based on the constructed / initialized HMVP list.

[0188] The CTU processing may be interpretation processing performed during the decoding process, i.e., the CTU processing may be implemented by the interpretation unit 344 in Figure 3. Alternatively, the CTU processing may be interpretation processing performed during the coding process, i.e., the CTU processing may be implemented by the interpretation unit 244 in Figure 2.

[0189] It should be noted that the above method for constructing / initializing the HMVP list can also be used for normal HMVP processing without wavefronts, such as HMV processing without WPP. As a result, HMVP processing is identical regardless of whether WPP is applied, which reduces the need for additional logical implementation.

[0190] It should be noted that the process shown in Figure 9 may be an encoding process implemented by an encoder, such as the video encoder 20 in Figure 2, according to one embodiment of this application.

[0191] Furthermore, it should be noted that the above method for combining wavefront and HMVP-based predictions can also be used for intra-predictions. That is, the history intra-mode may be used, and the history table for each CTU row is initialized to default values.

[0192] For example, the initialization of the HMVP list for each CTU row in the intra prediction may be performed in a default intra mode, such as planar mode, DC mode, vertical mode, horizontal mode, mode 2 mode, VDIA mode, and DIA mode.

[0193] Figure 10 is a flowchart illustrating one exemplary operation of a video decoder or video encoder, such as the video decoder 30 of Figure 3 and the video encoder 20 of Figure 2, according to one embodiment of the present application. One or more structural elements of the video decoder 30 / encoder 20, including the interpretation unit 344 / interpretation unit 244, may be configured to perform the art of Figure 10. In the example of Figure 10, the video decoder 30 / video encoder 20 may perform the following steps:

[0194] Step 1010, initialize the HMVP list for the current CTU row, when the current CTU is the starting CTU of the current CTU row.

[0195] Currently, a CTU line can be any CTU line in a picture frame that consists of multiple CTU lines, or a picture area (which may be a part of a picture frame) can consist of multiple CTU lines. Also, currently, a CTU line can be any one of multiple CTU lines.

[0196] Whether the current CTU is the starting CTU (or first CTU) of the current CTU row can be determined based on the index of the current CTU. For example, as disclosed in Figure 8, each CTU has a unique index, and therefore, based on the index of the current CTU, it can be determined whether the current CTU is the first CTU of the current CTU row. For example, CTUs with indexes 0, 11, 22, 33, 44, or 55… are, respectively, the first CTU of the CTU row. Alternatively, using Figure 8 as an example, each CTU row contains 11 CTUs, i.e., the width of each CTU row is 11, and therefore, it can be determined whether the index of the CTU is divided by the width of the CTU row and the remainder is 0. If the remainder is 0, the corresponding CTU is the first CTU of the CTU row; otherwise, if the remainder is not 0, the corresponding CTU is not the first CTU of the CTU row. That is, if the index of CTU %CTU row width = 0, then CTU is the first CTU of the CTU row; otherwise, if the index of CTU %CTU row width ≠ 0, then CTU is not the first CTU of the CTU row. Note that when the process of a CTU row is from right to left, whether CTU is the starting CTU of the CTU row can be determined in a similar manner.

[0197] After the HMVP list is initialized, the number of candidate motion vectors in the initialized HMVP list is zero.

[0198] Initialization can be performed by emptying the HMVP list for the current CTU row, that is, by removing the contents of the HMVP list for the current CTU row, or in other words, by setting the number of candidates in the HMVP list for the current CTU row to zero.

[0199] In other implementations, this method may further include the following steps: namely, initializing an HMVP list for each of several CTU rows, excluding the current CTU row, wherein the HMVP lists for the several CTU rows are identical or different.

[0200] Initialization can be performed by setting default values ​​for the HMVP list for the current CTU row, or by initializing the HMVP list for the current CTU row based on the HMVP list of the previous CTU row's CTU, as described above.

[0201] Step 1020: Process the current CTU row based on the HMVP list.

[0202] The process may be an interpretation process, through which a predicted block may be obtained. Reconstruction may be performed based on the predicted block to obtain a reconstructed block, and finally, a decoded picture may be obtained based on the reconstructed block. Details of these processes are described above.

[0203] As shown in Figure 8, the current picture frame contains multiple CTU lines to improve coding / decoding efficiency, and multiple CTU lines may be processed in wavefront parallel processing (WPP) mode. That is, when a particular CTU of the previous CTU line is processed, the current CTU line begins to be processed (or processing of the current CTU line begins), where the previous CTU line is a CTU line that is directly adjacent to the current CTU line and above the current CTU line, and a particular CTU of the previous CTU line is the second CTU of the previous CTU line; or a particular CTU of the previous CTU line is the first CTU of the previous CTU line. Using Figure 8 as an example, when the current CTU line is thread 3, the previous CTU line is thread 2, and a particular CTU of the previous CTU line may be CTU12, that is, when CTU12 is processed, the decoder / encoder begins to process the CTU line of thread 3, that is, the decoder / encoder begins to process CTU22. Using Figure 8 as another example, when the current CTU line is thread 4, the previous CTU line was thread 3, and the specific CTU of the previous CTU line may be CTU23, that is, when CTU23 is processed, the decoder / encoder starts processing thread 4's CTU line, that is, the decoder / encoder starts processing CTU33.

[0204] In one implementation, the step of processing the current CTU row based on the HMVP list may include the steps of processing the current CTU of the current CTU row, updating the initialized HMVP list based on the processed current CTU, and processing the second CTU of the current CTU row based on the updated HMVP list.

[0205] Figure 11 is a block diagram showing an example of a video processing device 1100 configured to implement an embodiment of the present invention, the video processing device 1100 may be an encoder 20 or a decoder 30 as shown in Figure 11, and the device includes the following:

[0206] Initialization unit 1110 is configured to initialize the HMVP list for the current CTU row when the current CTU is the starting CTU (first CTU) of the current CTU row.

[0207] Details of the initialization performed by the initialization unit 1110 can be found in step 1010.

[0208] Processing unit 1120 is currently configured to process CTU rows based on the HMVP list.

[0209] Details of the processing performed by the processing unit 1120 can be found in step 1020.

[0210] The process may be an interpretation process, through which a predicted block may be obtained. Reconstruction may be performed based on the predicted block to obtain a reconstructed block, and finally, a decoded picture may be obtained based on the reconstructed block. Details of these processes are described above.

[0211] As shown in Figure 8, the current picture frame contains multiple CTU lines, and to improve coding / decoding efficiency, multiple CTU lines may be processed in WPP mode. That is, when a particular CTU of the previous CTU line is processed, the current CTU line begins to be processed, where the previous CTU line is a CTU line that is directly adjacent to the current CTU line and above the current CTU line, and the particular CTU of the previous CTU line is the second CTU of the previous CTU line; or, the particular CTU of the previous CTU line is the first CTU of the previous CTU line. Using Figure 8 as an example, when the current CTU line is thread 3, the previous CTU line is thread 2, and the particular CTU of the previous CTU line may be CTU12, that is, when CTU12 is processed, the decoder / encoder begins to process the CTU line of thread 3, that is, the decoder / encoder begins to process CTU22. Using Figure 8 as another example, when the current CTU line is thread 4, the previous CTU line was thread 3, and the specific CTU of the previous CTU line may be CTU23, that is, when CTU23 is processed, the decoder / encoder starts processing thread 4's CTU line, that is, the decoder / encoder starts processing CTU33.

[0212] This disclosure further discloses an encoder including processing circuitry for performing a video processing method or a coding method of this disclosure.

[0213] This disclosure further discloses a decoder including processing circuitry for performing a video processing method or a coding method of this disclosure.

[0214] This disclosure further discloses a computer program product that includes program code for performing a video processing method or a coding method of this disclosure.

[0215] This disclosure further discloses a computer-readable storage medium storing computer instructions, wherein, when executed by one or more processors, these instructions cause one or more processors to execute a video processing method or a coding method of this disclosure. The computer-readable storage medium is either non-temporary or temporary.

[0216] The Disclosure further discloses a decoder comprising one or more processors and a non-temporary computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures the decoder to perform a video processing method or a coding method of the Disclosure.

[0217] The Disclosure further discloses an encoder comprising one or more processors and a non-temporary computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures the encoder to perform a video processing method or a coding method of the Disclosure.

[0218] The initialization process for HMVP lists is described in the general slice data syntax of VVC (Joint Video Expert Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, Multipurpose Video Coding (Draft 6)), and Section 7.3.8.1 of VVC states the following:

[0219] [Table 1]

[0220] Here, j%BrickWidth[SliceBrickIdx[i]])==0 means that the CTU with index j is the starting CTU of the CTU row, and NumHmvpCand=0 means that the quantity of candidates in the HMVP list is set to 0, in other words, the HMVP list is empty.

[0221] The update process for the HMVP list is described in section 8.5.2.16 of VVC (Joint Video Expert Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, Multipurpose Video Coding (Draft 6)), which states the following: The inputs for this process are as follows: Luma motion vectors mvL0 and mvL1 with 1 / 16 fractional sample accuracy. Reference indices refIdxL0 and refIdxL1, Prediction list usage flags predFlagL0 and predFlagL1, Dual-prediction weighted index bcwIdx. The MVP candidate hMvpCand consists of the Luma motion vectors mvL0 and mvL1, reference indices refIdxL0 and refIdxL1, prediction list utilization flags predFlagL0 and predFlagL1, and the dual prediction weighted index bcwIdx. The candidate list HmvpCandList is modified using candidate hMvpCand in the following steps, in the order specified below: The variable `identicalCandExist` is set to equal to FALSE, and the variable `removeIdx` is set to equal to 0. If NumHmvpCand is greater than 0, for each index hMvpIdx with hMvpIdx=0..NumHmvpCand-1, the following steps are applied until identicalCandExist equals TRUE: When hMvpCand is equal to HmvpCandList[hMvpIdx], identicalCandExist is set to TRUE and removeIdx is set to hMvpIdx. The candidate list HmvpCandList will be updated as follows: If identicalCandExist is equal to TRUE, or NumHmvpCand is equal to 5, then the following applies: For each index i in the array i=(removeIdx+1)..(NumHmvpCand-1), HmvpCandList[i-1] is set to be equal to HmvpCandList[i]. HmvpCandList[NumHmvpCand-1] is set to be equal to mvCand. Otherwise (if identicalCandExist is equal to FALSE and NumHmvpCand is less than 5), the following applies: HmvpCandList[NumHmvpCand++] is set to be equal to mvCand.

[0222] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including any medium that facilitates the transfer of computer programs between them, for example, according to a communication protocol. Thus, the computer-readable medium may generally correspond to (1) non-transient tangible computer-readable storage media, or (2) communication media such as signal waves or carrier waves. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described herein. A computer program product may include computer-readable media.

[0223] Such computer-readable storage media may include, but are not limited to, any other media that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer, such as RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Also, any connection is appropriately called a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. However, it should be noted that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary, tangible storage media. As used herein, the terms "disk" and "disc" include Compact Disc (CD), LaserDisc® (disc), Optical Disc (disc), Digital Multipurpose Disc (disc) (DVD), Floppy Disk (disk), and Blu-ray® Disc (disc), where a Disk typically replicates data magnetically, and a Disc (disc) replicates data optically using a laser. Any combination of the above should also be included within the scope of computer-readable media.

[0224] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term “processor” as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the technologies described herein. In addition, in some embodiments, the functionalities described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated within a combined codec. Furthermore, these technologies may be fully implemented in one or more circuit or logic elements.

[0225] The technology of this disclosure can be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to highlight the functional aspects of a device configured to perform the disclosed technology, but do not necessarily require implementation by different hardware units. Rather, as described above, various units may be combined within a codec hardware unit, or may be provided by an interoperable hardware unit collection, including one or more processors as described above, along with suitable software and / or firmware. [Explanation of Symbols]

[0226] 10 Coding Systems 12 Source Devices 13 Encoded picture 14 Destination device 16 Picture Sources 17 Picture Data 18 Pre-processing unit 19 Preprocessed picture data 20 encoders 21 Encoded picture data 22 Communication Interfaces 28 Communication Interfaces 30 Decoders 31 Decrypted picture 32 Post-Processors 33 Post-processed pictures 34 Display Devices 40 Video Coding Systems 41 Imaging devices 42 Antennas 43 processors 44 Memory Store 45 Display Devices 46 Processing Units 47 Logic Circuits 110 Inverse Quantization Unit 112 Inverse Transform Processing Unit 114 Reconstruction Unit 116 buffers 120 Loop Filter 130 Decode picture buffer 131 Decrypted Picture 144 Interpretation Units 154 Intra Prediction Units 201 Pictures 202 inputs 203 Picture Block 204 Residual Calculation Unit 205 Residual Block 206 Conversion Processing Unit 207 Conversion coefficient 208 Quantization Units 209 Quantization coefficients 210 Inverse Quantization Unit 211 Dequantization coefficient 212 Inverse Transform Processing Unit 213 Inverse Transform Block 214 Reconstruction Unit 215 Reconstructed Blocks 216 buffers 220 Loop Filter 221 filtered blocks 230 Decoded picture buffer 231 Decrypted picture 244 Interpretation Units 245 Interpreted Blocks 254 Intra Prediction Units 255 Intra-predicted blocks 260 Prediction Processing Units 262 Mode Selection Unit 265 Prediction Blocks 270 Entropy Coding Units 272 Output 304 Entropy Decoding Unit 309 Quantization coefficient 310 Inverse Quantization Unit 311 Decrypted Picture 312 Inverse Transform Processing Unit 313 Inverse Transform Block 314 Reconstruction Unit 315 Reconstructed Blocks 316 buffers 320 Loop Filter 321 filtered blocks 330 Decode picture buffer 344 Interpretation Units 354 Intra Prediction Units 360 Predictive Processing Unit 362 Mode Selection Unit 365 Prediction Block 400 video coding devices 410 Entrance Port 420 Tx / Rx 430 processors 440 Tx / Rx 450 Exit Port 460 memory 470 Coding Modules 500 devices 502 Processors 504 memory 506 data 508 Operating Systems 510 Application Programs 512 Bus 514 Secondary storage device 518 displays 520 Image Sensing Devices 522 Voice-activated devices 1100 Video Processing Unit 1110 Initialization Unit 1120 Processing Units

Claims

1. A video processing method, Steps to get the current frame of the video, A step of dividing the current frame into one or more areas including the current area, wherein the current area includes one or more coding tree unit (CTU) rows, the one or more CTU rows include a first CTU row and a second CTU row, the first CTU row includes a plurality of CTUs in a first sequence, and the second CTU row includes a plurality of CTUs in a second sequence. The steps include obtaining the first CTU row and A step of detecting the first start CTU in the first sequence from among the plurality of CTUs, The steps include initializing a first history-based motion vector prediction (HMVP) list for the first CTU row in response to the detection of the first start CTU, The steps include obtaining a first reconstructed block by performing interpretation on the first CTU row based on the initialized first HMVP list, The steps include obtaining the second CTU row and A step of detecting the second start CTU in the second sequence from among the plurality of CTUs, The steps include initializing a second HMVP list for the second CTU row in response to the detection of the second start CTU, The steps include obtaining a second reconstructed block by performing interpretation on the second CTU row based on the initialized second HMVP list, The steps include obtaining a picture decoded based on the first reconstructed block and the second reconstructed block, and A video processing method that includes [this].

2. The step of initializing the first HMVP list is: Steps to set the quantity of candidate motion vectors in the first HMVP list to zero. The video processing method according to claim 1, further comprising:

3. The step of obtaining a first reconstructed block by performing interpretation processing on the first CTU row based on the first HMVP list is: A step of processing the first start CTU in the first sequence, The steps include updating the first HMVP list based on the processed first start CTU to obtain the first updated HMVP list, A step of processing a second CTU of the first CTU row based on the first updated HMVP list, wherein the second CTU is a CTU that follows the first start CTU in the first sequence. The video processing method according to claim 1, further comprising:

4. Steps to update the first updated HMVP list based on the processed second CTU of the first CTU row. The video processing method according to claim 3, further comprising:

5. The step of initializing the second HMVP list for the second CTU row is: Step to empty the second HMVP list mentioned above. The video processing method according to claim 1, further comprising:

6. The video processing method according to claim 1, wherein the one or more CTU rows are processed in wavefront parallel processing (WPP) mode.

7. The video processing method according to claim 6, wherein the second CTU row is processed after a particular CTU of the first CTU row has been processed.

8. video encoder, One or more processors, A first non-temporary computer-readable storage medium connected to one or more processors and storing a program for execution by one or more processors, A second non-temporary computer-readable storage medium connected to the first non-temporary computer-readable storage medium for storing a bitstream for transmission, Equipped with, When the program is executed by the one or more processors, the one or more processors will: To get the current frame of the video, The division of the current frame into one or more areas including the current area, wherein the current area includes one or more coding tree unit (CTU) rows, the one or more CTU rows include a first CTU row and a second CTU row, the first CTU row includes multiple CTUs in a first sequence, and the second CTU row includes multiple CTUs in a second sequence. Obtaining the aforementioned first CTU row, Among the plurality of CTUs, the first start CTU in the first sequence is detected, In response to detecting the first start CTU, a first history-based motion vector prediction (HMVP) list for the first CTU row is initialized, Obtaining a first reconstructed block by performing interpretation on the first CTU row based on the initialized first HMVP list, Obtaining a first residual block based on the reconstructed block described above, Obtaining the aforementioned second CTU row, Among the plurality of CTUs, the second start CTU in the second sequence is detected, In response to detecting the second start CTU, the second HMVP list for the second CTU row is initialized, Obtaining a second reconstructed block by performing interpretation on the second CTU row based on the initialized second HMVP list, Obtaining a second residual block based on the aforementioned second reconstructed block, The bitstream is obtained based on the first residual block and the second residual block. A video encoder that performs this operation.

9. In initializing the first HMVP list, when the program is executed by the one or more processors, it sends the following instructions to the one or more processors: Set the quantity of candidate motion vectors in the first HMVP list to zero. A video encoder according to claim 8, which causes the following to occur.

10. In obtaining a first reconstructed block by performing interpretation processing on the first CTU row based on the first HMVP list, the program, when executed by the one or more processors, to the one or more processors, Processing the first start CTU in the first sequence, The first HMVP list is updated based on the processed first start CTU to obtain the first updated HMVP list, Processing a second CTU of the first CTU row based on the first updated HMVP list, wherein the second CTU is a CTU that follows the first start CTU in the first sequence. A video encoder according to claim 8, which causes the following to occur.

11. When the program is executed by the one or more processors, the one or more processors will: Updating the first updated HMVP list based on the processed second CTU of the first CTU row. The video encoder according to claim 10, which further performs the following.

12. In initializing the second HMVP list for the second CTU row, the program, when executed by the one or more processors, sends to the one or more processors: Clear the second HMVP list mentioned above. A video encoder according to claim 8, which causes the following to occur.

13. The video encoder according to claim 8, wherein the one or more CTU rows are processed in wavefront parallel processing (WPP) mode.

14. The video encoder according to claim 13, wherein the second CTU row is processed after a particular CTU of the first CTU row has been processed.

15. A video decoder, One or more processors, A non-temporary computer-readable storage medium connected to one or more processors and storing a program for execution by one or more processors. Equipped with, When the program is executed by the one or more processors, the one or more processors will: To get the current frame of the video, The division of the current frame into one or more areas including the current area, wherein the current area includes one or more coding tree unit (CTU) rows, the one or more CTU rows include a first CTU row and a second CTU row, the first CTU row includes multiple CTUs in a first sequence, and the second CTU row includes multiple CTUs in a second sequence. Obtaining the aforementioned first CTU row, Among the plurality of CTUs, the first start CTU in the first sequence is detected, In response to detecting the first start CTU, a first history-based motion vector prediction (HMVP) list for the first CTU row is initialized, Obtaining a first reconstructed block by performing interpretation on the first CTU row based on the initialized first HMVP list, Obtaining the aforementioned second CTU row, Among the plurality of CTUs, the second start CTU in the second sequence is detected, In response to detecting the second start CTU, the second HMVP list for the second CTU row is initialized, Obtaining a second reconstructed block by performing interpretation on the second CTU row based on the initialized second HMVP list, Obtaining a decoded picture based on the first reconstructed block and the second reconstructed block A video decoder that performs this operation.

16. In initializing the first HMVP list, when the program is executed by the one or more processors, it sends the following instructions to the one or more processors: Set the quantity of candidate motion vectors in the first HMVP list to zero. A video decoder according to claim 15, which causes the following to occur.

17. In obtaining a first reconstructed block by performing interpretation processing on the first CTU row based on the first HMVP list, the program, when executed by the one or more processors, to the one or more processors, Processing the first start CTU in the first sequence, The first HMVP list is updated based on the processed first start CTU to obtain the first updated HMVP list, Processing a second CTU of the first CTU row based on the first updated HMVP list, wherein the second CTU is a CTU that follows the first start CTU in the first sequence. A video decoder according to claim 15, which causes the following to occur.

18. When the program is executed by the one or more processors, the one or more processors will: Updating the first updated HMVP list based on the processed second CTU of the first CTU row. The video decoder according to claim 17, which further performs the following:

19. In initializing the second HMVP list for the second CTU row, the program, when executed by the one or more processors, sends to the one or more processors: Clear the second HMVP list mentioned above. A video decoder according to claim 15, which causes the following to occur.

20. The video decoder according to claim 15, wherein the one or more CTU rows are processed in wavefront parallel processing (WPP) mode.

21. A receiver configured to receive a bitstream acquired by an encoder according to any one of claims 8 to 14, A storage device configured to store the aforementioned bitstream, A transmitter configured to transmit the bitstream and A transmitting device equipped with the following features.

Images