Video coding method and apparatus using history-based motion vector prediction

Abstract

To provide methods and systems of video coding using history-based motion vector prediction.SOLUTION: Processing with a video coder comprises: acquiring an encoded video bitstream including data associated with encoded pictures each including multiple rows of coding tree units (CTUs); and resetting a history-based motion vector predictor (HMVP) table for a current row of CTUs. While decoding the current row of CTUs, the processing comprises: maintaining a plurality of motion vector predictors in the HMVP table; extracting a prediction mode from the video bitstream; constructing a motion vector candidate list in accordance with the prediction mode and on the basis of, at least partially, the plurality of motion vector predictors in the HMVP table; selecting, from the list, a motion vector predictor for a current coding unit (CU); determining a motion vector on the basis of at least partially, the prediction mode and the motion vector predictor; and updating the HMVP table on the basis of the motion vector.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001]

[0001] This application relates generally to encoding and decoding video data, and more particularly to video coding methods and systems that use history-based motion vector prediction. [Background technology]

[0002]

[0002] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media playback devices, video game consoles, smartphones, video conferencing devices, and video streaming devices. Such electronic devices transmit, receive, encode, decode, and / or store digital video data by implementing video compression and decompression standards, such as MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10 Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC). Video compression generally involves spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data. In block-based video coding, a video frame is divided into one or more slices, each of which contains multiple video blocks, also known as coding tree units (CTUs). Each CTU may contain one coding unit (CU) or may be recursively divided into multiple smaller CUs until a predetermined minimum CU size is reached. Each CU (also called a leaf CU) contains one or more transform units (TUs) and one or more prediction units (PUs). Each CU may be coded in either intra mode, inter mode, or IBC mode. Video blocks in an intra-coded (I) slice in a video frame are coded using spatial prediction with respect to reference samples of neighboring blocks in the same video frame.Multiple video blocks within an inter-coded (P or B) slice in a video frame may use spatial prediction relative to reference samples of neighboring blocks within the same video frame, or may use temporal prediction relative to reference samples of another reference video frame in the past and / or future.

[0003]

[0003] Spatial or temporal prediction based on previously coded reference blocks (e.g., neighboring blocks) can be performed to obtain a predicted block for a current video block being coded. The process of locating the reference block may be performed using a block matching algorithm. Residual data indicating pixel differences between the current block being coded and the predicted block is called a residual block or prediction error. An inter-coded block is coded according to a motion vector pointing to a reference frame that forms the predicted block and a residual block. The process of determining a motion vector is generally called motion estimation. An intra-coded block is coded according to an intra-prediction mode and a residual block. For further compression, residual transform coefficients are obtained by transforming the residual block from the pixel domain to a transform domain (e.g., the frequency domain), and then these are quantized. The quantized transform coefficients are initially arranged in a two-dimensional array, which may be scanned to generate a one-dimensional vector of transform coefficients, which may then be entropy coded to obtain a video stream for further compression.

[0004]

[0004] The encoded video stream is then recorded on a computer-readable storage medium (e.g., flash memory) that can be accessed by another electronic device with digital video capabilities, or transmitted directly to the electronic device via wired or wireless connections. The electronic device then performs video decompression (the reverse process of video compression) by, for example, parsing the encoded bitstream to obtain syntax elements from the bitstream, reconstructing digital video data from the encoded bitstream into its original format based at least in part on the syntax elements obtained from the bitstream, and rendering the reconstructed digital video data on the electronic device's display.

[0005]

[0005] As digital video quality transitions from High Definition to 4Kx2K or 8Kx4K, the amount of video data to be encoded / decoded increases exponentially. There is a continuous effort to find ways to encode / decode video data more efficiently while maintaining the image quality of the decoded video data. Summary of the Invention

[0006]

[0006] This application relates to the encoding and decoding of video data, and more particularly, describes embodiments relating to systems and methods for parallel processing of video data during video encoding and decoding using history-based motion vector prediction.

[0007]

[0007] A video data decoding method according to a first aspect of the present application is implemented in a computing device having one or more processors and a memory storing a plurality of programs executed by the one or more processors. After obtaining a video bitstream, the computing device begins by extracting data associated with a plurality of coded pictures from the video bitstream. Each picture includes a plurality of coding tree unit (CTU) rows, and each CTU includes one or more coding units (CUs). Before starting to decode a first CU of a current CTU row to be decoded, the computing device resets a history-based motion vector predictor (HMVP) table. Then, while decoding the current CTU row, the computing device maintains a plurality of motion vector predictors in the HMVP table. Each motion vector predictor has been used to decode at least one CU. For a current CU of the current CTU row to be decoded, the computing device extracts a prediction mode from the video bitstream and constructs a motion vector candidate list based at least in part on the motion vector predictors in the HMVP table according to the prediction mode. After selecting a motion vector predictor from the motion vector candidate list, the computing device determines a motion vector based at least in part on the prediction mode and the selected motion vector predictor, decodes the current CU using the determined motion vector, and updates the HMVP table based on the determined motion vector.

[0008]

[0008] A computing device according to a second aspect of the present application includes one or more processors, a memory, and a plurality of programs stored in the memory, which, when executed by the one or more processors, cause the computing device to perform the above-described process.

[0009]

[0009] A non-transitory computer-readable recording medium according to a third aspect of the present application stores a plurality of programs for execution by a computing device having one or more processors, the programs causing the computing device to perform the above-described processes when executed by the one or more processors.

[0010] The accompanying drawings, which are included to provide a further understanding of the embodiments, are incorporated in and constitute a part of this specification, illustrate the described embodiments, and together with the description, serve to explain the underlying principles. Corresponding elements are designated with the same reference numerals. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an example of a video encoding and decoding system according to some embodiments of the present disclosure. [Figure 2]

[0012] FIG. 2 is a block diagram illustrating an example of a video encoder according to some embodiments of the present disclosure. [Figure 3]

[0013] FIG. 3 is a block diagram illustrating an example of a video decoder according to some embodiments of the present disclosure. [Figure 4A]

[0014] 4A-4D are block diagrams illustrating a method for recursively quadtree partitioning a frame into video blocks having different sizes in accordance with some embodiments of this disclosure. [Figure 4B] 4A-4D are block diagrams illustrating a method for recursively quadtree partitioning a frame into multiple video blocks having multiple different sizes in accordance with some embodiments of this disclosure. [Figure 4C] 4A-4D are block diagrams illustrating a method for recursively quadtree partitioning a frame into multiple video blocks having multiple different sizes in accordance with some embodiments of this disclosure. [Figure 4D] 4A-4D are block diagrams illustrating a method for recursively quadtree partitioning a frame into multiple video blocks having multiple different sizes in accordance with some embodiments of this disclosure. [Figure 5A]

[0015] FIG. 5A is a block diagram illustrating block locations that are spatially neighboring and temporally connected to a current CU to be coded, according to some embodiments of this disclosure. [Figure 5B]

[0016] FIG. 5B is a block diagram illustrating multi-threaded encoding of multiple rows of CTUs in a picture using wavefront parallel processing in accordance with some embodiments of this disclosure. [Figure 6]

[0017] FIG. 6 is a flowchart illustrating an example process by a video coder implementing a technique for constructing a candidate list of motion vector predictors in accordance with some embodiments of this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0018] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in an understanding of the subject matter described herein. However, it will be apparent to those skilled in the art that various modifications may be employed without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter described herein may be implemented in a variety of electronic devices with digital video capabilities.

[0013]

[0019] Figure 1 is a block diagram illustrating an example system 10 for encoding and decoding video blocks in parallel, in accordance with some embodiments of the present disclosure. As shown in Figure 1, system 10 includes a source device 12 that generates and encodes video data to be subsequently decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media playback devices, video game consoles, or video streaming devices. In some embodiments, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0014]

[0020] In some embodiments, destination device 14 may receive the encoded video data to be decoded over link 16. Link 16 may comprise any type of communication medium or device that transfers encoded video data from source device 12 to destination device 14. As one example, link 16 may comprise a communication medium that enables source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may comprise any wireless or wired communication medium, such as radio frequency (RF) or one or more physical transmission lines. The communication medium may be part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or other equipment used to facilitate communication from source device 12 to destination device 14.

[0015]

[0021] In some other embodiments, the encoded video data may be transmitted from output interface 22 to recording device 32. The encoded video data in recording device 32 may then be accessed by destination device 14 via input interface 28. Recording device 32 may include any of a variety of distributed or locally accessed data storage media for recording the encoded video data, such as a hard drive, Blu-ray Disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage media. As yet another example, recording device 32 may represent a file server or other intermediate storage device that may hold encoded video data generated by source device 12. Destination device 14 may access the recorded video data through streaming or download from recording device 32. A file server may be any type of computer capable of storing encoded video data and transmitting the encoded video data to destination device 14. Examples of file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data through any standard data connection suitable for accessing encoded video data recorded on a file server, including a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL or cable media), or a combination of both. The transmission of the encoded video data from recording device 32 may be a streaming transmission, a download transmission, or a combination of both.

[0016]

[0022] 1, source device 12 includes video source 18, video encoder 20, and output interface 22. Video source 18 may include, for example, a source such as a video capture device, such as a video camera, a video archive for storing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics for the source video. As an example, when video source 18 is a video camera in a security surveillance system, source device 12 and destination device 14 may comprise a camera phone or a video phone. However, the embodiments described herein may be applicable to video coding generally and may be applied to wireless and / or wired applications.

[0017]

[0023] Such captured, previously captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be transmitted directly to destination device 14 through output interface 22 of source device 12. Also (or alternatively), the encoded video data may be recorded to recording device 32 for decoding and / or playback for later access by destination device 14 or other devices. Output interface 22 may also include a modem and / or transmitter.

[0018]

[0024] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or a modem for receiving encoded video data over link 16. The encoded video data communicated over link 16 or provided by recording device 32 may include various syntax elements generated by video encoder 20 for use in decoding the video data at video decoder 30. Such syntax elements may be incorporated into the encoded video data transmitted over a communications medium, recorded on a recording medium, or recorded on a file server.

[0019]

[0025] In some embodiments, destination device 14 may include a display device 34, which may be an integrated display device or an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0020]

[0026] Video encoder 20 and video decoder 30 may operate in accordance with proprietary or industry standards, such as VVC, HEVC, MPEG-4 Part 10 AVC (Advanced Video Coding), or extensions of these standards. It should be understood that the present application is not limited to a particular video encoding / decoding standard and may be applicable to other video encoding / decoding standards. It is generally contemplated that video encoder 20 of source device 12 may be configured to encode video data in accordance with any current or future standard. Similarly, it is generally contemplated that video decoder 30 of destination device 14 may be configured to decode video data in accordance with any current or future standard.

[0021]

[0027] Video encoder 20 and video decoder 30 may each be implemented as any suitable encoding circuit, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or a combination thereof. If implemented partially in software, the electronic device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video encoding / decoding processes disclosed in this disclosure. Each of video encoder 20 and video decoder 30 may be incorporated into one or more encoders or decoders, either of which may be integrated into the respective device as part of a combined encoder / decoder (CODEC).

[0022]

[0028] 2 is a block diagram illustrating an example video encoder 20 according to some embodiments described herein. The video encoder 20 may perform intra-predictive and inter-predictive coding of video blocks within a video frame. Intra-predictive coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-predictive coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence.

[0023]

[0029] As shown in FIG. 2, video encoder 20 includes video data memory 40, prediction processor 41, decoded picture buffer (DPB) 64, adder 50, transform processor 52, quantizer 54, and entropy encoder 56. Prediction processor 41 includes motion estimation unit 42, motion compensation unit 44, segmentation unit 45, intra prediction processor 46, and intra block copy (BC) unit 48. In some embodiments, video encoder 20 further includes inverse quantization unit 58, inverse transform processor 60, and adder 62 for video block reconstruction. A deblocking filter (not shown) may be disposed between adder 62 and DPB 64 to remove block artifacts from the reconstructed video. In addition to the deblocking filter, an in-loop filter (not shown) may be used to filter the output of adder 62. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be divided into one or more fixed or programmable hardware units as shown.

[0024]

[0030] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from video source 18. DPB 64 is a buffer that stores reference video data used when encoding video data by video encoder 20. Video data memory 40 and DPB 64 may be comprised of any of a variety of memory devices. In various examples, video data memory 40 may be implemented on-chip with other components of video encoder 20 or may be implemented off-chip relative to the other components.

[0025]

[0031] As shown in FIG. 2, after receiving video data, a partitioning unit 45 in a prediction processor 41 partitions the video data into multiple video blocks. This partitioning may include dividing a video frame into multiple slices or larger coding units (CUs) according to a predetermined partitioning structure, such as a quadtree structure, associated with the video data. The video frame may be partitioned into multiple video blocks (or multiple sets of video blocks called tiles). The prediction processor 41 may select one of multiple selectable predictive coding modes, such as one of multiple intra-predictive coding modes or one of multiple inter-predictive coding modes, for the current video block based on an error result (e.g., coding rate and distortion level). The prediction processor 41 may provide the resulting intra-predictive or inter-predictive coded block to an adder 50 to generate a residual block and to an adder 62 to reconstruct a coded block that will later be used as part of a reference frame. The prediction processing unit 41 also supplies syntax elements such as motion vectors, intra-mode predictors, partition information, and other syntax information to the entropy coding unit 56.

[0026]

[0032] To select an appropriate intra-prediction coding mode for a current video block, intra-prediction unit 46 within prediction unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction unit 41 may perform inter-prediction coding of the current video block relative to one or more predictive blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple coding steps, for example, to select an appropriate coding mode for each block of video data.

[0027]

[0033] In some embodiments, motion estimation unit 42 determines the inter-prediction coding mode for a current video frame by generating a motion vector that indicates a displacement of a prediction unit (PU) of a video block in a current video frame relative to a prediction block in a reference frame according to a predetermined pattern in a sequence of video frames. Motion estimation, performed by motion estimation unit 42, is the process of generating a motion vector that estimates motion for a video block. The motion vector may, for example, indicate a displacement of a PU of a video block in a current video frame or picture relative to a prediction block in a reference frame (or other coding unit) for a current block to be coded in the current frame (or other coding unit). The predetermined pattern may designate multiple video frames in the sequence as P frames or B frames. Intra BC unit 48 may determine vectors, such as block vectors, for intra BC coding in a manner similar to how motion estimation unit 42 determines motion vectors for inter prediction, or may use motion estimation unit 42 to determine the block vectors.

[0028]

[0034] A prediction block is a block of a reference frame that is considered to closely correspond to a PU of a video block to be coded in terms of pixel differences, which may be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference measure. In some embodiments, video encoder 20 may calculate values ​​for sub-integer pixel locations of the reference frame stored in DPB 64. For example, video encoder 20 may interpolate values ​​for quarter-pixel, eighth-pixel, or other fractional pixel locations of the reference frame. Thus, motion estimator 42 may perform motion searches for whole pixel locations and fractional pixel locations to output motion vectors with fractional pixel precision.

[0029]

[0035] For a PU of a video block in an inter-predictively coded frame, the motion estimation unit 42 calculates a motion vector by comparing the position of the PU with the position of a prediction block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), where the first reference frame list or the second reference frame list identifies one or more reference frames stored in the DPB 64. The motion estimation unit 42 transfers the calculated motion vector to the motion compensation unit 44 and then to the entropy coding unit 56.

[0030]

[0036] Motion compensation is performed by motion compensation unit 44 and may include fetching or generating a predictive block based on the motion vector determined by motion estimation unit 42. Upon receiving a motion vector for a PU of a current video block, motion compensation unit 44 finds the predictive block pointed to by the motion vector in one of a plurality of reference frame lists, obtains the predictive block, and sends the predictive block to summer 50. Summer 50 then constructs a residual video block of pixel difference values ​​by subtracting pixel values ​​of the predictive block provided by motion compensation unit 44 from pixel values ​​of the current video block to be coded. The pixel difference values ​​that make up the residual video block may include luma difference components, chroma difference components, or both. Motion compensation unit 44 may also generate syntax elements associated with the video blocks of the video frames for use in decoding the video blocks of the video frames by video decoder 30. The syntax elements may include, for example, a syntax element defining the prediction vector used to identify the prediction block, any flag indicating the prediction mode, or any other syntax information described herein. It should be noted that the motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are shown separated for conceptual purposes.

[0031]

[0037] In some embodiments, the intra BC unit 48 may generate vectors and fetch predictive blocks in a manner similar to that described above with respect to the motion estimation unit 42 and motion compensation unit 44, except that the predictive block is in the same frame as the current block being coded, and the vector is considered to be a block vector with the opposite direction to the motion vector. In particular, the intra BC unit 48 may determine an intra prediction mode to use to code the current block. In some examples, the intra BC unit 48 may code the current block using various intra prediction modes, e.g., in separate coding steps, and evaluate their performance using a rate-distortion analysis. The intra BC unit 48 may then select an appropriate intra prediction mode from the various evaluated intra prediction modes to generate the intra mode predictor to use. For example, the intra BC unit 48 may calculate rate-distortion values ​​using a rate-distortion analysis for the various evaluated intra prediction modes, and select the intra prediction mode with the best rate-distortion performance from the evaluated modes as the appropriate intra prediction mode to use. Rate-distortion analysis generally determines the bitrate (i.e., number of bits) used to generate a coded block, along with the amount of distortion (or error) between the coded block and the original block that was coded to generate the coded block. The intra BC unit 48 calculates a ratio from the distortion to determine which intra prediction mode exhibits the highest rate-distortion value for the block, and calculates rates for the various coded blocks.

[0032]

[0038] In other examples, the intra BC unit 48 may use all or part of the motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction in accordance with embodiments described herein. In either case, for intra block copying, the predictive block may be any block that is considered to closely correspond to the block being coded in terms of pixel differences, and may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference measures. Identifying the predictive block may then involve calculating values ​​for sub-integer pixel locations.

[0033]

[0039] Whether the predictive block is derived from the same frame according to intra prediction or from a different frame according to inter prediction, video encoder 20 may construct a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded to construct pixel difference values. The pixel difference values ​​that make up the residual video block may include differences in luma components and differences in chroma components.

[0034]

[0040] The intra-prediction unit 46 may perform intra-prediction of the current video block as an alternative to the inter-prediction performed by the motion estimation unit 42 and motion compensation unit 44 or the intra-block copy prediction performed by the intra-BC unit 48, as described above. In particular, the intra-prediction unit 46 may determine an intra-prediction mode to use to encode the current block. To do so, the intra-prediction unit 46 may encode the current block using various intra-prediction modes, e.g., in separate encoding steps, and the intra-prediction unit 46 (or, in some examples, a mode selection unit) may select an appropriate intra-prediction mode to use from among the evaluated intra-prediction modes. The intra-prediction unit 46 may provide information indicating the selected intra-prediction mode for the block to the entropy coder 56. The entropy coder 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0035]

[0041] After prediction processor 41 determines a predictive block for a current video block, either inter- or intra-predictively, adder 50 constructs a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be assembled into one or more transform units (TUs) and provided to transform processor 52. Transform processor 52 converts the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0036]

[0042] The transform processor 52 forwards the residual transform coefficients to a quantizer 54, which quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth for some or all of the coefficients. The degree of quantization may be changed by adjusting a quantization parameter. In some examples, the quantizer 54 may then perform a scan on a matrix containing the quantized transform coefficients. Alternatively, the entropy encoder 56 may perform the scan.

[0037]

[0043] Following quantization, entropy coding unit 56 entropy codes the quantized transform coefficients to obtain a video bitstream, using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy coding (PIPE), or other entropy coding methodology or technique. The coded bitstream may then be transmitted to video decoder 30 or recorded on recording device 32 for later transmission to or retrieval from video decoder 30. Entropy coding unit 56 may also entropy code motion vectors and other syntax elements associated with the current video frame being coded.

[0038]

[0044] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate reference blocks for predicting other video blocks. As described above, motion compensation unit 44 may generate a motion-compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​used for motion compensation.

[0039]

[0045] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block that is stored in DPB 64. The reference block may then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a prediction block for performing inter prediction on other video blocks in subsequent video frames.

[0040]

[0046] 3 is a block diagram illustrating a video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-prediction processing unit 84, and an intra-BC unit 85. The video decoder 30 may perform a decoding process that is generally inverse to the encoding process described above with respect to the video encoder 20 in connection with FIG. 2. For example, the motion compensation unit 82 may generate prediction data based on motion vectors obtained from the entropy decoding unit 80, while the intra-prediction processing unit 84 may generate prediction data based on an intra-prediction mode indicator obtained from the entropy decoding unit 80.

[0041]

[0047] In some examples, one unit within the video decoder 30 may be assigned the task of performing embodiments of the present application. In some examples, embodiments of the present disclosure may be divided into one or more units within the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in cooperation with other units, such as the motion compensation unit 82, the intra prediction processing unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not have the intra BC unit 85, and the functionality of the intra BC unit 85 may be performed by other components, such as the motion compensation unit 82 within the prediction processing unit 81.

[0042]

[0048] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. The video data stored in video data memory 79 may be obtained, for example, from recording device 32, from a local video source such as a camera via wired or wireless network communication of the video data, or from a physical data storage medium (e.g., a flash drive or hard disk). Video data memory 79 may include a coded picture buffer (CPB) that stores coded video data from the coded video bitstream. A decoded picture buffer (DPB) 92 within video decoder 30 stores reference video data used in decoding video data in video decoder 30 (e.g., in intra-predictive coding mode or inter-predictive coding mode). Video data memory 79 and DPB 92 may be comprised of any of a wide variety of memory devices, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), magneto-resistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For purposes of illustration, video data memory 79 and DPB 92 are shown as two separate components within video decoder 30. However, those skilled in the art will appreciate that video data memory 79 and DPB 92 may be implemented in the same memory device or separate memory devices. In some examples, video data memory 79 may be implemented on-chip with other components within video decoder 30 or may be implemented off-chip relative to the other components.

[0043]

[0049] During the decoding process, video decoder 30 receives an encoded bitstream that indicates multiple video blocks of an encoded video frame and associated syntax elements. Video decoder 30 may receive the syntax elements at the video frame level and / or the video block level. An entropy decoding unit 80 within video decoder 30 performs entropy decoding on the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then sends the motion vectors and other syntax elements to a prediction processor 81.

[0044]

[0050] When a video frame is coded as an intra-prediction coded (I) frame or for intra-coded predictive blocks in other types of frames, an intra-prediction processing unit 84 in the prediction processing unit 81 may generate predictive data for video blocks in the current video frame based on signaled intra-prediction modes and reference data from previously decoded blocks in the current frame.

[0045]

[0051] When a video frame is coded as an inter-predictive (e.g., B or P) frame, a motion compensation unit 82 within prediction processing unit 81 generates one or more prediction blocks for a video block in the current video frame based on the motion vectors and other syntax elements obtained from entropy decoding unit 80. Each of the prediction blocks may be generated from a reference frame in one of a plurality of reference frame lists. Video decoder 30 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on the reference frames stored in DPB 92.

[0046]

[0052] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 within prediction processor 81 generates a prediction block for the current video block based on the block vectors and other syntax elements obtained from entropy decoder 80. The prediction block may be within the same reconstruction region in the picture as the current video block as defined by video encoder 20.

[0047]

[0053] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for video blocks in the current video frame by analyzing the motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block to be decoded. For example, to decode a video block in the current video frame, motion compensation unit 82 uses some of the obtained syntax elements to determine the prediction mode (e.g., intra prediction or inter prediction) used to encode the video block in the video frame, the type of inter-predicted frame (e.g., B or P), configuration information for one or more reference frame lists for the frame, the motion vector for each inter-predictively coded video block in the frame, the inter-prediction state for each inter-predictively coded video block in the frame, and other information.

[0048]

[0054] Similarly, to decode video blocks in the current video frame, intra BC unit 85 may use some of the obtained syntax elements (e.g., flags) to determine that the current video block was predicted using intra BC mode, configuration information indicating which video blocks in the frame are in the reconstruction domain and should be stored in DPB 92, block vectors for each of the intra BC predicted video blocks in the frame, intra BC prediction states for each of the intra BC predicted video blocks in the frame, and other information.

[0049]

[0055] Motion compensation unit 82 may perform the interpolation using an interpolation filter, such as that used by video encoder 20 when encoding the video block, to calculate interpolated values ​​for the sub-integer pixels of the reference block. In this case, motion compensation unit 82 may determine, from the obtained syntax elements, the interpolation filter used by video encoder 20, and use that interpolation filter to generate the prediction block.

[0050]

[0056] An inverse quantization unit 86 performs inverse quantization on the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. An inverse transform unit 88 applies an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct residual blocks in the pixel domain.

[0051]

[0057] After motion compensation unit 82 or intra BC unit 85 generates a predictive block for the current video block based on the vector and other syntax elements, summer 90 reconstructs a decoded video block for the current video block by adding the residual block from inverse transform processor 88 and the corresponding predictive block generated by motion compensation unit 82 and intra BC unit 85. An in-loop filter (not shown) may be disposed between summer 90 and DPB 92 for further processing of the decoded video block. The decoded video blocks in a particular frame are then stored in DPB 92. DPB 92 stores reference frames used for subsequent motion compensation of the next video block. DPB 92 or a memory device separate from DPB 92 may store the decoded video for subsequent display on a display device, such as display device 34 of FIG. 1 .

[0052]

[0058] In a typical video decoding process, a video sequence typically includes a set of ordered frames or pictures. Each frame may include three sample arrays, denoted SL, SCb, and Scr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb color difference (chroma) samples. SCr is a two-dimensional array of Cr color difference (chroma) samples. Alternatively, a frame may be monochromatic, thereby having only a single two-dimensional array of luma samples.

[0053]

[0059] As shown in FIG. 4A, video encoder 20 (or, more specifically, divider 45) generates a coded representation of a frame by first dividing the frame into a set of coding tree units (CTUs). A video frame may contain an integer number of CTUs consecutively ordered in raster scan order from top to bottom and left to right. Each CTU is the largest logical coding unit, and the width and height of the CTUs are signaled by video encoder 20 in a sequence parameter set to ensure that all CTUs in a video sequence have the same size as one of 128x128, 64x64, 32x32, and 16x16. However, it should be noted that this application is not necessarily limited to any particular size. As shown in FIG. 4B, each CTU may have one coding tree block (CTB) consisting of luma samples and two corresponding coding tree blocks consisting of chroma samples, along with syntax elements used to encode the samples in the coding tree blocks. The syntax elements describe the characteristics of different types of units of coding pixel blocks and how a video sequence can be reconstructed by video decoder 30, including inter- or intra-prediction, intra-prediction mode, motion vectors, and other parameters. For monochrome pictures or pictures with three different color planes, a CTU may contain a single coding tree block and the syntax elements used to encode the samples of that coding tree block. A coding tree block may be an NxN block of samples.

[0054]

[0060] To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary-tree partitioning, quad-tree partitioning, or a combination of both, on the coding tree block of a CTU to divide the CTU into smaller coding units (CUs). As shown in FIG. 4C , a 64×64 CTU 400 is first partitioned into four smaller CUs, each with a 32×32 block size. Of these four smaller CUs, CU 410 and CU 420 are each partitioned into four CUs with a 16×16 block size. Two 16×16 CUs, 430 and 440, are each further partitioned into four CUs with an 8×8 block size. Figure 4D is a diagram of a quadtree data structure representing the final result of applying the segmentation process to the CTU 400 shown in Figure 4C, with each leaf node of the quadtree corresponding to a CU with a size ranging from 32x32 to 8x8. Like the CTU shown in Figure 4B, each CU has a coding block (CB) of luma samples and two corresponding coding blocks of chroma samples in a frame of the same size, along with syntax elements used to encode the samples in these coding blocks. For monochrome pictures or pictures with three different color planes, a CU may have a single coding block and syntax structures used to encode the samples in the coding block.

[0055]

[0061] In some embodiments, video encoder 20 may further divide the coding block of a CU into one or more N×N prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples to which the same prediction, i.e., inter or intra, is applied. A prediction unit (PU) of a CU may include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax elements used to predict the prediction blocks. For monochrome pictures or pictures with three different color planes, a PU may include a single prediction block and syntax structures used to predict the prediction block. Video encoder 20 may generate a predicted luma block, a predicted Cb block, and a predicted Cr block for the luma prediction block, the Cb prediction block, and the Cr prediction block in each PU of a CU.

[0056]

[0062] Video encoder 20 may generate the predictive blocks for a PU using intra prediction or inter prediction. When video encoder 20 generates the predictive blocks for a PU using intra prediction, video encoder 20 may generate the predictive blocks for the PU based on decoded samples of a frame related to the PU. When video encoder 20 generates the predictive blocks for a PU using inter prediction, video encoder 20 may generate the predictive blocks for the PU based on decoded samples of one or more frames other than the frame related to the PU.

[0057]

[0063] After the video encoder 20 generates a predicted luma block, a predicted Cb block, and a predicted Cr block for one or more PUs of a CU, the video encoder 20 may generate a luma residual block for the CU by subtracting the predicted luma block of the CU from its original luma coding block, where each sample of the luma residual block of the CU indicates a difference between a luma sample of one of the multiple predicted luma blocks of the CU and a corresponding sample of the original luma coding block of the CU. Similarly, the video encoder 20 may generate a Cb residual block and a Cr residual block for the CU, where each sample of the Cb residual block of the CU indicates a difference between a Cb sample of one of the multiple predicted Cb blocks of the CU and a corresponding sample of the original Cb coding block of the CU, and each sample of the Cr residual block of the CU may indicate a difference between a Cr sample of one of the multiple predicted Cr blocks of the CU and a corresponding sample of the original Cr coding block of the CU.

[0058]

[0064] Furthermore, as shown in FIG. 4C , video encoder 20 may use quadtree partitioning to decompose the luma residual block, Cb residual block, and Cr residual block of a CU into one or more luma transform blocks, Cb transform blocks, and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU may include a transform block of luma samples, two corresponding transform blocks of luma samples, and syntax elements used to transform the samples of the transform blocks. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. For monochrome pictures or pictures with three different color planes, a TU may contain a single transform block and the syntax structures used to transform the samples of that transform block.

[0059]

[0065] The video encoder 20 may apply one or more transforms to a luma transform block of a TU to generate a luma coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. The video encoder 20 may apply one or more transforms to a Cb transform block of a TU to generate a Cb coefficient block for the TU. The video encoder 20 may apply one or more transforms to a Cr transform block of the TU to generate a Cr coefficient block for the TU.

[0060]

[0066] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization typically refers to the process of quantifying transform coefficients so as to reduce the total amount of data used to represent the transform coefficients, thereby providing additional compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy code the syntax elements representing the quantized transform coefficients. For example, video encoder 20 may perform context-adaptive binary arithmetic coding (CABAC) on the syntax elements representing the quantized transform coefficients. Finally, video encoder 20 may output a bitstream representing a sequence of bits comprising a representation of the coded frame and associated data, which may be stored in recording device 32 or transmitted to destination device 14.

[0061]

[0067] After receiving the bitstream generated by video encoder 20, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct frames of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing video data is typically the inverse of the encoding process performed by video encoder 20. For example, video decoder 30 may perform an inverse transform on coefficient blocks associated with TUs of a current CU to reconstruct residual blocks associated with the TUs of the current CU. Video decoder 30 may reconstruct coding blocks of the current CU by adding samples of predictive blocks for PUs in the current CU to corresponding samples of transform blocks of TUs in the current CU. After reconstructing coding blocks for each CU of a frame, video decoder 30 may reconstruct the frame.

[0062]

[0068] As mentioned above, video coding achieves video compression using two main modes: intra-frame prediction (or intra-prediction) and inter-frame prediction (or inter-prediction). It should be noted that IBC can be considered either intra-frame prediction or a third mode. Of the two modes, inter-frame prediction contributes to coding efficiency more than intra-frame prediction because it uses motion vectors to predict a current video block from a reference video block.

[0063]

[0069] However, with the continuous improvement of video data capture technology and the finer video block size to retain more detailed information of the video data, the amount of data required to represent the motion vector for the current frame also increases substantially. One way to overcome this problem is to benefit from the fact that not only do a set of neighboring CUs in both the spatial and temporal domains have similar video data for prediction, but the motion vectors between these neighboring CUs are also similar. Therefore, by finding spatial and temporal correlations, the motion information of spatially neighboring CUs and / or temporally connected CUs can be used as the approximate motion information (e.g., motion vector) of the current CU. This is also called the "motion vector predictor (MVP)" of the current CU.

[0064]

[0070] Instead of encoding the actual motion vector of the current CU determined by motion estimator 42 into the video bitstream as described above with respect to Figure 2, the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to generate a motion vector difference (MVD) for the current CU. In doing so, the motion vector determined by motion estimator 42 for each CU of the frame does not need to be encoded into the video bitstream, significantly reducing the amount of data used to represent motion information in the video bitstream.

[0065]

[0071] Like the process of selecting a prediction block in a reference frame during inter-frame prediction of a codeblock, a set of rules needs to be applied in both video encoder 20 and video decoder 30 to construct a motion vector candidate list for a current CU using candidate motion vectors that may be generated in association with spatially neighboring CUs and / or temporally connected CUs to the current CU, and then select an element from the motion vector candidate list as a motion vector predictor for the current CU. By doing so, the motion vector candidate list itself does not need to be transmitted between video encoder 20 and video decoder 30, and the index of the selected motion vector predictor in the motion vector candidate list is sufficient for video encoder 20 and video decoder 30 to use the same motion vector predictor in the motion vector candidate list to encode and decode the current CU.

[0066]

[0072] In some embodiments, each inter-predicted CU has three motion vector prediction modes, including inter (hereinafter also referred to as "advanced motion vector prediction (AMVP)"), skip, and merge, for constructing a motion vector candidate list. Under each mode, one or more motion vector candidates may be added to the motion vector candidate list according to the algorithm described below. One candidate from the candidate list is finally used as the best motion vector predictor for the inter-predicted CU to be encoded into a video bitstream by video encoder 20 or decoded from a video bitstream by video decoder 30. To find the best motion vector predictor from the candidate list, a motion vector competition (MVC) scheme is introduced, which selects a motion vector from a given candidate set of motion vectors, i.e., the motion vector candidate list including spatial and temporal motion vector candidates.

[0067]

[0073] In addition to extracting motion vector predictor candidates from spatially neighboring or temporally connected CUs, the motion vector predictor candidates can also be extracted from a so-called "history-based motion vector prediction (HMVP)" table. The HMVP table stores a predetermined number of motion vector predictors, and each motion vector predictor is used to encode / decode a specific CU in the same CTU row (or sometimes the same CTU). Due to the spatial / temporal proximity of these CUs, there is a high probability that one motion vector predictor in the HMVP table will be reused to encode / decode different CUs in the same CTU row. Therefore, by incorporating the HMVP table into the process of constructing a motion vector candidate list, higher coding efficiency can be achieved.

[0068]

[0074] In some embodiments, the HMVP table has a fixed length (e.g., 5) and is managed in a quasi-first-in-first-out (quasi-FIFO) manner. For example, a motion vector is reconstructed for a CU when decoding one inter-coded block of the CU. The HMVP table is updated on the fly with the reconstructed motion vector because such a motion vector can be a motion vector predictor for the next CU. When updating the HMVP table, there are two scenarios: (i) the reconstructed motion vector is different from other existing motion vectors in the HMVP table, or (ii) the reconstructed motion vector is the same as one of the existing motion vectors in the HMVP table. For the first scenario, if the HMVP table is not full, the reconstructed motion vector is added to the HMVP table as the latest motion vector. If the HMVP table is already full, the oldest motion vector in the HMVP table is deleted from the HMVP table before the reconstructed motion vector is added as the latest motion vector.In other words, the HMVP table in this case is similar to a FIFO buffer, and the motion vector information at the beginning of the FIFO buffer and related to other past inter-coded blocks is shifted out of the buffer, so that the reconstructed motion vector is added to the back of the FIFO buffer as the latest element in the HMVP table.For the second scenario, before the reconstructed motion vector is added as the latest motion vector, the existing motion vector in the HMVP table that is substantially the same as the reconstructed motion vector is deleted from the HMVP table.If the HMVP table is organized in the form of a FIFO buffer, the motion vector predictor after the same motion vector in the HMVP table is shifted forward by one element to occupy the space remaining after the deleted motion vector, and then the reconstructed motion vector is added to the back of the FIFO buffer as the latest element in the HMVP table.

[0069]

[0075] The motion vectors in the HMVP table can be added to the motion vector candidate list under several different prediction modes, such as AMVP, merge, skip, etc. It has been found that the motion information of past inter-coded blocks that are stored in the HMVP table and are not even adjacent to the current block can be used for more efficient motion vector prediction.

[0070]

[0076] After an MVP candidate from a given candidate set of motion vectors is selected for the current CU, video encoder 20 generates one or more syntax elements for the corresponding MVP candidate and encodes them into the video bitstream so that video decoder 30 can use the syntax elements to obtain the MVP candidate from the video bitstream. Depending on the specific mode used to construct the motion vector candidate set, different modes (e.g., AMVP, merge, skip, etc.) have different sets of syntax elements. For AMVP mode, the syntax elements include an inter-prediction indicator (list 0, list 1, or bi-prediction), a reference index, a motion vector candidate index, and a motion vector prediction residual signal. For skip and merge modes, only the merge index is coded into the bitstream because the current CU inherits other syntax elements, including the inter-prediction indicator, reference index, and motion vector, from the neighboring CU referenced by the coded merge index. For skip-coded CUs, the motion vector prediction residual signal is also ignored.

[0071]

[0077] 5A is a block diagram illustrating spatially neighboring and temporally connected block positions relative to a current CU to be coded / decoded, according to some embodiments of the present disclosure. For a given mode, a motion vector prediction (MVP) candidate list is constructed by first checking the availability of motion vectors associated with spatially neighboring block positions to the left and above, and the availability of motion vectors associated with temporally connected block positions, and then checking the motion vectors in the FDMVP table. During the process of constructing the MVP candidate list, some duplicate MVP candidates are removed from the candidate list, and zero-valued motion vectors are added, if necessary, to make the candidate list have a fixed length (note that different modes may have different fixed lengths). After constructing the MVP candidate list, video encoder 20 can select the best motion vector predictor from the candidate list and encode a corresponding index indicating the selected candidate into the video bitstream.

[0072]

[0078] Using FIG. 5A as an example and assuming the candidate list has a fixed length of 2, the motion vector predictor (MVP) candidate list may be constructed for the current CU by sequentially performing the following steps under AMVP mode: 1) Selection of MVP candidates from multiple spatially neighboring CUs a) Extracting one unscaled MVP candidate from one of the two spatially neighboring CUs on the left, starting from A0 and ending at A1; b) if there is no non-scaled MVP candidate available from the left in the previous step, extracting one scaled MVP candidate from one of the two spatially neighboring CUs on the left, starting from A0 and ending with A1; c) Extracting one unscaled MVP candidate from one of the three spatially neighboring CUs above, starting from B0, passing through B1, and ending at B2; d) when neither A0 nor A1 is available or they are coded in intra mode, extracting one scaled MVP candidate from one of the three spatially neighboring CUs above, starting from B0, passing through B1, and ending with B2; 2) If two MVP candidates are found in the previous step and they are identical, remove one of the two candidates from the MVP candidate list; 3) Selection of MVP candidates from multiple temporally connected CUs a) if the MVP candidate list after the previous step does not contain two MVP candidates, extracting one MVP candidate from the temporally connected CUs; 4) Selecting MVP candidates from the HMVP table a) extracting two history-based MVPs from the HMVP table when the MVP candidate list after the previous step does not contain two MVP candidates; 5) If the MVP candidate list after the previous step does not contain two MVP candidates, add two zero-valued MVPs to the MVP candidate list.

[0073]

[0079] Since there are only two candidates in the AMVP mode MVP candidate list constructed as above, a related syntax element, such as a binary flag, is coded in the bitstream to indicate which of the two MVP candidates in the candidate list is used to decode the current CU.

[0074]

[0080] In some embodiments, in skip mode or merge mode, the MVP candidate list may be constructed for the current CU by sequentially performing similar steps as described above. Note that for skip mode or merge mode, a special type of merge candidate, called a "pair-wise merge candidate," is also incorporated into the MVP candidate list. A pair-wise merge candidate is generated by averaging multiple motion vectors of two previously extracted merge mode motion vector candidates. The size of the merge MVP candidate list (e.g., 1 to 6) is signaled in the slice header of the current CU. In merge mode, the index of the best merge candidate is coded for each CU using truncated unary binarization (TU). The first bin of the merge index is coded with the context, and bypass coding is used for the other bins.

[0075]

[0081] As described above, a history-based MVP can be added to either the AMVP mode MVP candidate list or the merged MVP candidate list after a spatial or temporal MVP. The motion information of past inter-coded CUs is stored in the HMVP table and used as an MVP candidate for the current CU. The HMVP table is maintained during the encoding / decoding process. Whenever a non-subblock inter-coded CU is found, its associated motion vector information is added as a new candidate to the last entry of the HMVP table (if the HMVP table is already full and there are no duplicates of its associated motion vector information), while the motion vector information stored in the first entry of the HMVP table is deleted from it. Alternatively, before its associated motion vector information is added to the last entry of the HMVP table, duplicates of its associated motion vector information are removed from the table.

[0076]

[0082] As described above, intra block copy (IBC) can significantly improve the coding efficiency of displayed content material. Since the IBC mode is implemented as a block-level coding mode, block matching (BM) is performed in the video encoder 20 to find the optimal block vector for each CU, where the block vector is used to indicate the displacement of the current block to a reference block that has already been reconstructed in the current picture. The IBC-coded CU is treated as a third prediction mode other than the intra prediction mode and the inter prediction mode.

[0077]

[0083] At the CU level, the IBC mode can be signaled as IBC·AMVP mode or IBC·Skip / Merge mode below. - IBC·AMVP mode: The block vector difference (BVD) between the actual block vector of a CU and the block vector predictor of that CU selected from the block vector candidates of that CU is coded in the same way as the motion vector differential coding in the AMVP mode described above. The block vector prediction method uses two block vector candidates as predictors, one of which is obtained from the left neighbor and the other from the upper neighbor. If neither neighbor is available, the default block vector will be used as the block vector predictor. A binary flag is signaled to indicate the index of the block vector predictor. The IBC·AMVP candidate list consists of spatial HMVP candidates. - IBC Skip / Merge Mode: A merge candidate index is used to indicate which of the block vector candidates in the merge candidate list from neighboring IBC coded blocks will be used to predict the block vector for the current block. The IBC merge candidate list consists of spatial, HMVP and pairwise candidates.

[0078]

[0084] Another approach to improving the coding efficiency of conventional coding standards is to introduce parallel processing into the video encoding / decoding process, for example, by using a multi-core processor. For example, wavefront parallel processing (WPP) has already been introduced in HEVC as a feature that uses multiple threads to encode or decode multiple CTU rows in parallel.

[0079]

[0085] FIG. 5B is a block diagram illustrating multi-threaded coding of multiple CTU rows in a picture using wavefront parallel processing (WPP) in accordance with some embodiments of the present disclosure. When WPP is enabled, multiple CTU rows can be processed in parallel in a wavefront manner, where there may be a delay of two CTU rows between the beginning of two neighboring wavefronts. For example, to encode picture 500 using WPP, a video coder, such as video encoder 20 and video decoder 30, divides the coding tree units (CTUs) of picture 500 into multiple wavefronts, each wavefront corresponding to a respective CTU row in the picture. The video coder may begin coding the leading wavefront, e.g., using a first coder core or thread. After the video coder has coded two or more CTUs in the leading wavefront, the video coder may begin coding the next-to-leading wavefront in parallel with the coding of the leading wavefront, e.g., using a second parallel coder core or thread. After the video coder has coded two or more CTUs of the second-to-top waveplane, the video coder may begin coding the third-to-top waveplane in parallel with coding the higher-order waveplanes, e.g., using a third parallel coder core or thread. This pattern may continue for multiple waveplanes within picture 500. In this disclosure, a collection of CTUs that the video coder codes concurrently is referred to as a "group of CTUs." Thus, when the video coder codes a picture using WPP, each CTU in the group of CTUs may belong to only one waveplane within the picture, and each CTU may be offset by at least two CTU columns within the picture from a CTU in the waveplane immediately above.

[0080]

[0086] The video coder may initialize a context for a current wave-plane and perform context adaptive binary arithmetic coding (CABAC) for the current wave-plane based on data of the first two blocks of the wave-plane above and one or more elements in a slice header for a slice that includes the first code block of the current wave-plane. The video coder may perform CABAC initialization for a next wave-plane (or row of CTUs) using a context state after coding two CTUs in the row of CTUs above the next row of CTUs. In other words, before starting coding of the current wave-plane, the video coder (or, more specifically, its thread) may perform coding of at least two blocks in a wave-plane above the current wave-plane, assuming that the current wave-plane is not the first row of CTUs in a picture. The video coder may then initialize a CABAC context for the current wave-plane after coding at least two blocks in the wave-plane above the current wave-plane. In this example, each CTU row in picture 500 is a separate partition and has an associated thread (WPP thread 1, WPP thread 2, ...) so that multiple CTU rows in the picture are coded in parallel.

[0081]

[0087] Because current implementations of the HMVP table use a global motion vector (MV) buffer to store previously reconstructed motion vectors, this HMVP table cannot be implemented in the WPP-enabled parallel coding scheme described above with respect to Figure 5B. In particular, the fact that the global MV buffer is shared by all threads in the encoding / decoding process of the video coder prevents subsequent WPP threads after the first WPP thread (e.g., WPP thread 1) from starting because they must wait until the last CTU (e.g., the right-most CTU) in the first WPP thread (e.g., the first CTU row) has completed updating the HMVP table.

[0082]

[0088] To overcome this issue, we propose replacing the global MV buffer shared by WPP threads with a buffer dedicated to multiple CTU rows, so that when WPP is enabled in a video coder, each wavefront of a CTU row has its own buffer for storing the HMVP table corresponding to the CTU row processed by the corresponding WPP thread. It should be noted that each CTU row with its own HMVP table is equivalent to resetting the HMVP table before coding the first CU of that CTU row. Resetting the HMVP table erases all motion vectors in the HMVP table obtained from coding other CTU rows. In one embodiment, the reset process involves setting the size of available motion vector predictors in the HMVP table to zero. In another embodiment, the reset process could involve setting the reference index of all entries in the HMVP table to an invalid value, such as -1. In this way, the MVP candidate list for the current CTU in a particular wavefront is constructed according to the HMVP table associated with the WPP thread processing that particular wavefront, regardless of which of the three modes—AMVP, merge, or skip—is in use. There is no interdependence between different wavefronts other than the delay between the two CTUs mentioned above, and the construction of multiple motion vector candidate lists associated with multiple different wavefronts can proceed in parallel, as in the WPP processing shown in FIG. 5B. In other words, when starting to process a particular wavefront, the HMVP table is reset to an empty state without affecting the coding of other CTU wavefronts by other WPP threads. In some cases, the HMVP table can be reset to an empty state before coding each CTU. In this case, the motion vectors in the HMVP table are limited to a specific CTU, and perhaps the motion vector in that HMVP table is more likely to be selected as the motion vector for the current CU in that specific CTU.

[0083]

[0089] 6 is a flowchart illustrating an example process by a video coder, such as video encoder 20 or video decoder 30, that implements a technique for constructing a motion vector predictor candidate list using at least an HMVP table, in accordance with some embodiments of this disclosure. For illustrative purposes, the flowchart illustrates a video decoding process. First, video decoder 30 obtains (610) an encoded video bitstream that includes data associated with multiple coded pictures. As shown in FIGS. 4A and 4C, each picture includes multiple coding tree unit (CTU) rows, and each CTU includes one or more coding units (CUs). Video decoder 30 extracts multiple different pieces of information, such as syntax elements and pixel values, from the video bitstream to reconstruct a picture row by row.

[0084]

[0090] Before decoding the current CTU row, video decoder 30 first resets the history-based motion vector predictor (HMVP) table for the current CTU row (620). As described above, resetting the HMVP table ensures that video decoder 30 can decode multiple CTU rows in the current picture in parallel, for example, using multithreaded processing, with one thread having its own HMVP table per CTU row or per multi-core processor, or one core having its own HMVP table per CTU row, or both. In yet some other embodiments, before decoding the current CTU, video decoder 30 first resets the history-based motion vector predictor (HMVP) table for the current CTU (620). As described above, resetting the HMVP table ensures that multiple CTUs in the current picture can be decoded in parallel, for example using multi-threaded processing, and that one thread has its own HMVP table per CTU or per multi-core processor, or one core has its own HMVP table per CTU, or both.

[0085]

[0091] While decoding the current CTU row (630), video decoder 30 maintains (630-1) a plurality of motion vector predictors in an HMVP table. As described above, each motion vector predictor stored in the HMVP table has been used to decode at least another CU in the current CTU row. In fact, a motion vector predictor exists in the HMVP table because it may be reused to predict other CUs in the current CTU row when the HMVP table is involved in the process of constructing the motion vector candidate list as described above.

[0086]

[0092]

[0093] For a current CU in a current CTU row, video decoder 30 extracts a prediction mode from the video bitstream (630-3). As described above, a CU may have multiple prediction modes, including advanced motion vector prediction (AMVP) mode, merge mode, skip mode, IBC-AMVP mode, and IBC merge mode. Once video encoder 20 selects an appropriate prediction mode for the CU, the selected prediction mode is signaled in the bitstream. As described above, there are various steps that may be performed in various orders to construct a motion vector candidate list. Here, video decoder 30 constructs a motion vector candidate list based at least in part on multiple motion vector predictors in the HMVP table according to the prediction mode (630-5). From other sources, the motion vector candidate list may include motion vector predictors from spatially neighboring CUs and / or temporally connected CUs to the current CU (if the prediction mode is one of AMVP mode, IBC-AMVP mode, and IBC merge mode), and optionally pairwise motion vector predictors (if the prediction mode is one of merge mode and skip mode). Optionally, one or more zero-valued motion vector predictors may be added to the motion vector candidate list if the motion vector candidate list does not reach a predetermined length.

[0087]

[0094] Next, video decoder 30 selects a motion vector predictor for the current CU from the motion vector candidate list (630-7) and determines a motion vector based at least in part on the selected motion vector predictor and the prediction mode (630-9). As described above, depending on whether the prediction mode is AMVP mode, the selected motion vector predictor may or may not be an estimated motion vector for the current CU. For example, if the prediction mode is AMVP mode, an estimated motion vector is determined by adding a motion vector differential reconstructed from the bitstream to the selected motion vector predictor, and the current CU is then decoded at least in part using the estimated motion vector and the corresponding CU in the reference picture. However, if the prediction mode is merge mode or skip mode, the selected motion vector predictor is already an estimated motion vector and can be used in decoding the current CU together with the corresponding CU in the reference picture. Finally, video decoder 30 updates the HMVP table based on the determined motion vector (630-11). As described above, all elements in the HMVP table have previously been used to decode at least another CU and are retained in the HMVP table to form a motion vector candidate list until they are removed from the HMVP table by the insertion of a motion vector used to decode the next other CU in the current CTU row or by a table reset.

[0088]

[0095] In some embodiments, there are two possible scenarios for inserting a motion vector into the HMVP table based on the comparison result between the motion vector determined for the current CU and the multiple motion vector predictors in the HMVP table: If none of the multiple motion vector predictors in the HMVP table are identical to the determined motion vector, and the HMVP table is full, the earliest or oldest motion vector predictor is deleted from the HMVP table, and the motion vector is added to the table as the latest; If one of the multiple motion vector predictors in the HMVP table is identical to the motion vector, the same motion vector predictor is deleted from the HMVP table, and all other motion vector predictors after the deleted motion vector predictor are moved forward in the HMVP table, so that the motion vector is added to the end of the HMVP table as the latest.

[0089]

[0096] As described above, two or more of the multiple CTU rows may be encoded / decoded in parallel, e.g., using WPP, with each CTU row having an associated HMVP table for storing multiple history-based motion vector predictors used to encode / decode the corresponding CTU row. For example, a thread may be assigned to the decoding of a particular CTU row in the current picture being decoded, such that multiple different CTU rows can be decoded with multiple different associated threads, as described above with respect to FIG. 5B. In some examples, video decoder 30 identifies one or more motion vector predictors in the motion vector candidate list as redundant and removes them from the motion vector candidate list to further improve coding efficiency.

[0090]

[0097] In one or more examples, the functions described above may be implemented in hardware, software, or a combination thereof. When implemented in software, the functions described above may be stored on or transmitted as one or more instructions or code on a computer-readable medium for execution by a hardware-based processing unit. Computer-readable media may include computer-readable media, which correspond to tangible media such as recording media, or communication media, which include any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. As such, computer-readable media may generally correspond to (1) tangible, non-transitory computer-readable media or (2) communication media, such as signals or carrier waves. Data storage media may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures implementing the embodiments described herein. A computer program product may include a computer-readable medium.

[0091]

[0098] The terms used in the description of the embodiments herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. The term "and / or," as used herein, will be understood to mean and include any and all possible combinations of one or more of the associated listed items. Furthermore, the terms "comprises" and / or "comprising," as used herein, will be understood to specify the presence of stated features, components, and / or elements, but will not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0092]

[0099] It will be understood that terms such as "first" and "second" are used herein to describe various elements, but these elements should not be limited by such terms. These terms are used only to distinguish one element from another. For example, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode, without departing from the scope of the embodiments. A first electrode and a second electrode are both electrodes, but are not the same electrode.

[0093]

[0100] The description of this application has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications, variations, and alternative embodiments will be apparent to those skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. The above embodiments have been chosen and described to best explain the principles and practical applications of the invention, and to enable others skilled in the art to understand the invention in its various embodiments and to make full use of the underlying principles and various embodiments with various modifications suited to the particular uses intended. It is, therefore, to be understood that the scope of the appended claims should not be limited to the particular examples of the disclosed embodiments, but that modifications and other embodiments are intended to be encompassed within the scope of the appended claims.

Claims

1. 1. An encoding method comprising: Dividing a current picture into a plurality of coding tree unit (CTU) rows, each CTU including one or more coding units (CUs); resetting a history-based motion vector predictor (HMVP) table before processing a first CU in a current CTU row in the current picture; processing the current CTU row; Equipped with The step of processing the current CTU row includes: maintaining a plurality of motion vector predictors in the HMVP table, each of the motion vector predictors in the HMVP table being used to process at least one CU in the current CTU row; For the current CU of the current CTU row to be processed, constructing a motion vector candidate list based at least in part on the plurality of motion vector predictors in the HMVP table according to a prediction mode; selecting a motion vector predictor from the motion vector candidate list; determining a motion vector for processing the current CU based at least in part on the prediction mode and the selected motion vector predictor; updating the HMVP table based on the determined motion vector; The step of updating the HMVP table based on the determined motion vector includes: comparing the motion vector predictors in the HMVP table with the determined motion vector; In response to a comparison result that one of the plurality of motion vector predictors in the HMVP table is identical to the determined motion vector, removing the same motion vector predictor from the HMVP table; moving each of the motion vector predictors after the removed motion vector predictor forward in the HMVP table; adding the determined motion vector to the HMVP table as a current motion vector; the prediction mode is an inter mode, the motion vector candidate list has a fixed length of 2; The step of constructing the motion vector candidate list comprises: If a history-based motion vector predictor is selected from the HMVP table to construct the motion vector candidate list, the method includes adding two history-based motion vector predictors from the HMVP table to the motion vector candidate list.

2. 10. The method of claim 1, The step of updating the HMVP table based on the determined motion vector includes: In response to determining that none of the plurality of motion vector predictors in the HMVP table is identical to the determined motion vector, removing the earliest motion vector predictor from the HMVP table when the HMVP table is full; adding the determined motion vector to the HMVP table as the latest motion vector. The method further comprises:

3. 10. The method of claim 1, The step of constructing the motion vector candidate list comprises: adding zero or more motion vector predictors from spatially neighboring CUs and / or temporally connected CUs to the current CU to the motion vector candidate list; when a current length of the motion vector candidate list is less than a first predetermined threshold, adding zero or more history-based motion vector predictors from the HMVP table to the motion vector candidate list until the current length of the motion vector candidate list is equal to the first predetermined threshold; The method further comprises:

4. 4. The method of claim 3, The step of constructing the motion vector candidate list comprises: when a current length of the motion vector candidate list is less than a first predetermined threshold, adding zero or more zero-valued motion vector predictors to the motion vector candidate list until the current length of the motion vector candidate list is equal to the first predetermined threshold. The method further comprises:

5. 4. The method of claim 3, adding zero or more motion vector predictors derived from spatially neighboring CUs and / or temporally connected CUs to the current CU to the motion vector candidate list, adding zero or more motion vector predictors from spatially neighboring CUs to the current CU to the motion vector candidate list; adding zero or more motion vector predictors derived from CUs temporally contiguous to the current CU to the motion vector candidate list when the current length of the motion vector candidate list is less than the first predetermined threshold; A method comprising:

6. 6. The method of claim 5, wherein the step of adding zero or more motion vector predictors derived from CUs that are temporally connected to the current CU to the motion vector candidate list includes the step of extracting one candidate motion vector predictor derived from the CU that is temporally connected.

7. 2. The method of claim 1, wherein resetting the HMVP table comprises setting the size of available motion vector predictors in the HMVP table to zero.

8. The method of claim 1 , wherein processing the current CTU row further comprises encoding the prediction mode into a video bitstream.

9. A computing device, one or more processors; a memory coupled to the one or more processors; a plurality of programs stored in the memory; Equipped with A computing device, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform the method of any one of claims 1 to 8.

10. A non-transitory computer-readable recording medium storing a plurality of programs for execution by a computing device having one or more processors, A non-transitory computer-readable recording medium, the plurality of programs, when executed by the one or more processors, causing the computing device to perform the method of any one of claims 1 to 8 and generate a bitstream.

11. 9. A computer program for execution by a computing device having one or more processors, the computer program, when executed by the one or more processors, causing the computing device to perform the method of any one of claims 1 to 8.

12. A method for transmitting a bitstream, comprising: performing an encoding method to generate a bitstream; transmitting the bitstream; Equipped with The encoding method comprises: Dividing a current picture into a plurality of coding tree unit (CTU) rows, each CTU including one or more coding units (CUs); resetting a history-based motion vector predictor (HMVP) table before processing a first CU in a current CTU row in the current picture; processing the current CTU row; Equipped with The step of processing the current CTU row includes: maintaining a plurality of motion vector predictors in the HMVP table, each of the motion vector predictors in the HMVP table being used to process at least one CU in the current CTU row; For the current CU of the current CTU row to be processed, constructing a motion vector candidate list based at least in part on the plurality of motion vector predictors in the HMVP table according to a prediction mode; selecting a motion vector predictor from the motion vector candidate list; determining a motion vector for processing the current CU based at least in part on the prediction mode and the selected motion vector predictor; updating the HMVP table based on the determined motion vector; The step of updating the HMVP table based on the determined motion vector includes: comparing the motion vector predictors in the HMVP table with the determined motion vector; In response to a comparison result that one of the plurality of motion vector predictors in the HMVP table is identical to the determined motion vector, removing the same motion vector predictor from the HMVP table; moving each of the motion vector predictors after the removed motion vector predictor forward in the HMVP table; adding the determined motion vector to the HMVP table as a current motion vector; the prediction mode is an inter mode, the motion vector candidate list has a fixed length of 2; The step of constructing the motion vector candidate list comprises: If a history-based motion vector predictor is selected from the HMVP table to construct the motion vector candidate list, the method includes adding two history-based motion vector predictors from the HMVP table to the motion vector candidate list.

Images