Video Coding Method and Apparatus Using Sub-Block-Based Temporal Motion Vector Prediction

Abstract

To provide a method and an arithmetic device for sub-block-based temporal motion vector prediction.SOLUTION: The method includes acquiring a video bitstream that contains data related to a plurality of coded pictures, selecting one reference picture as a collocated picture of the current picture according to a syntax element signaled in the video bitstream while decoding the current picture in the video bitstream, determining a motion shift amount between the collocated picture and the current picture from motion information of spatially neighboring blocks in a current coding unit (CU) in accordance with a certain order, dividing the current CU into multiple sub-CUs, obtaining a temporal motion vector predictor for each sub-CU from the motion shift amount and motion information of blocks in the collocated picture corresponding to each sub-block of the current picture, and decoding the current CU according to the temporal motion vector predictor.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] This application generally relates to the encoding and decoding of video data, and more particularly to a video coding method and system that uses sub-block-based temporal motion vector prediction.

Background Art

[0002] Digital video is supported by various electronic devices such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video teleconferencing devices, video streaming devices, and the like. Such electronic devices support video coding standards such as MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10 AVC (Advanced Video Cod ing), HEVC (High Efficiency Video Coding), By implementing video compression extension specifications defined in coding standards such as VVC (Versatile Video Coding), digital video data is transmitted, received, encoded, decoded, and / or stored. Generally, video compression includes performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data. For block-based video coding, a video frame is divided into one or more slices, and each slice has a plurality of video blocks that can also be referred to as coding tree units (CTUs). Each CTU may contain one coding unit (CU), or may be recursively divided into smaller CUs until a predetermined minimum CU size is reached. Each CU (also named a leaf CU) contains one or more transform units (TUs) and also includes one or more prediction units (PUs). Each CU can be encoded in either an intra mode, an inter mode, or an IBC mode. Video blocks within an intra-coded (I) slice in a video frame are encoded using spatial prediction against reference samples in neighboring blocks within the same video frame. Video blocks within an inter-coded (P or B) slice in a video frame may use spatial prediction against reference samples in neighboring blocks within the same video frame, or may use temporal prediction against reference samples in other previous and / or other future reference video frames.

[0003] For example, spatial or temporal prediction based on previously encoded reference blocks, such as neighboring blocks, results in a prediction block for the current video block to be encoded. The process of finding the reference block can be achieved by a block matching algorithm. Residual data representing the pixel difference between the current block to be encoded and the prediction block is referred to as a residual block or prediction error. An inter-coded block is encoded according to a motion vector indicating a reference block in a reference frame forming the prediction block and the residual block. The process of determining the motion vector is generally referred to as motion prediction. An intra-coded block is encoded according to an intra-prediction mode and the residual block. For further compression, the residual block can be transformed from the pixel domain to a transform domain, such as the frequency domain, to yield residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, may be scanned to generate a one-dimensional vector of transform coefficients, which is then entropy encoded into a video bitstream to achieve further compression.

[0004] The encoded video bitstream is then stored in a computer-readable recording medium (such as flash memory) accessible by another electronic device with digital video capabilities or transmitted directly to the electronic device, either wired or wirelessly. The electronic device then performs video decompression (a process inverse to the aforementioned video compression) by, for example, parsing the encoded video bitstream to obtain syntax elements from the bitstream and reconstructing the digital video data from the encoded video bitstream into its original format based at least in part on the syntax elements obtained from the bitstream, and rendering the reconstructed digital video data on a display of the electronic device.

[0005] As the quality of digital video migrates from High Definition to 4K×2K or 8K×4K, the amount of video data to be encoded / decoded increases exponentially. This has led to a continuous effort in terms of how video data can be encoded / decoded more efficiently while maintaining the picture quality of the decoded video data.

Summary of the Invention

[0006] This application describes embodiments related to the encoding and decoding of video data, and more particularly, relates to a system and method for encoding and decoding video using sub-block-based temporal motion vector prediction.

Means for Solving the Problems

[0007] According to a first aspect of the present application, a method of sub-block-based temporal motion vector prediction is executed in an arithmetic device having one or more processors and a memory storing a plurality of programs executed by the one or more processors. The arithmetic device acquires a video bitstream including data related to a plurality of encoded pictures. The arithmetic device decodes a current picture in the video bitstream while selecting, as a reference picture as the collocated picture of the current picture, one reference picture according to the syntax elements signaled in the video bitstream, and determining a motion shift amount between the collocated picture and the current picture from the motion information of spatially adjacent blocks of the current coding unit (CU) in a certain order. Next, the arithmetic device divides the current CU into a plurality of sub-CUs, each corresponding to a respective sub-block of the current picture. Then, the arithmetic device obtains a temporal motion vector predictor for each sub-CU of the current CU from (i) the motion shift amount between the collocated picture and the current picture, and (ii) the motion information of the blocks in the collocated picture corresponding to the respective sub-blocks of the current picture, and decodes the current CU according to the temporal motion vector predictors of the plurality of sub-CUs of the current CU.

[0008] According to a second aspect of the present application, the arithmetic unit includes one or more processors, a memory, and a plurality of programs stored in the memory. When the program is executed by one or more processors, the arithmetic unit executes the processing as described above.

[0009] According to a third aspect of the present application, a non-transitory computer-readable recording medium stores a plurality of programs for execution by an arithmetic unit having one or more processors. When the program is executed by one or more processors, the arithmetic unit executes the processing as described above.

[0010] The accompanying drawings, which are included to provide a further understanding of the embodiments, are incorporated in and constitute a part of this specification, illustrate the described embodiments, and together with the description serve to explain the basic principles. Similar reference numerals refer to corresponding parts.

Brief Description of the Drawings

[0011]

Fig. 1

Fig. 2

Fig. 3

Fig. 4A

Fig. 4B

Fig. 4C

Fig. 4D

Fig. 5A

Fig. 5B

Fig. 5C

Fig. 6

DETAILED DESCRIPTION OF THE INVENTION

[0012] Next, specific embodiments are referred to in detail, and those examples are shown in the accompanying drawings. In the following detailed description, many non-limiting and specific details are disclosed to assist in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternative forms can be used without departing from the scope of the claims, and that the subject matter can be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented in many types of electronic devices with digital video capabilities.

[0013] FIG. 1 is a block diagram showing an exemplary system 10 for performing video block encoding and decoding in parallel according to some embodiments of the present disclosure. As shown in FIG. 1, system 10 includes a source device 12 that generates and encodes video data to be later decoded by a destination device 14. The source device 12 and the destination device 14 may comprise any of a variety of electronic devices including desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, and the like. In some embodiments, the source device 12 and the destination device 14 are equipped with wireless communication capabilities.

[0014] In some embodiments, the destination device 14 may receive the encoded video data to be decoded through link 16. Link 16 may comprise any type of communication medium or communication device capable of transferring the encoded video data from the source device 12 to the destination device 14. In one example, link 16 may comprise a communication medium that enables the source device 12 to directly transmit the encoded video data to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and transmitted to the destination device 14. The communication medium may comprise any wireless or wired communication medium such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or other devices that facilitate communication from the source device 12 to the destination device 14.

[0015] In some other embodiments, the encoded video data may be transmitted from the output interface 22 to the recording device 32. Subsequently, the encoded video data in the recording device 32 may be accessed by the destination device 14 via the input interface 28. The recording device 32 may include any of a variety of distributed or locally accessible data recording media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or other digital recording media suitable for storing the encoded video data. In a further example, the recording device 32 may correspond to a file server or another intermediate recording device that may hold the encoded video data generated by the source device 12. The destination device 14 may access the stored video data by streaming or downloading from the recording device 32. The file server may be any type of computer capable of storing the encoded video data or transmitting the encoded video data to the destination device 14. Exemplary file servers include a web server (for example, for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. The destination device 14 may access the encoded video data through any standard data connection including a wireless channel (for example, Wi-Fi connection) suitable for accessing the encoded video data stored on the file server, a wired connection (for example, DSL, cable modem, etc.), or a combination of both. The transmission of the encoded video data from the recording device 32 may be a streaming transmission, a download transmission, or a combination of both.

[0016] As shown in FIG. 1, the information source device 12 includes a video source 18, a video encoder 20, and an output interface 22. The video source 18 can include sources such as, for example, a video camera, a video archive including previously captured video, a video supply interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video, or a combination of such sources, such as a video capture device. As an example, when the video source 18 is a video camera of a security monitoring system, the information source device 12 and the destination device 14 can form a camera phone or a video phone. However, the embodiments described in this application can generally be applicable to video coding and can be applied to wireless and / or wired applications.

[0017] The captured, previously captured, or computer-generated video can be encoded by the video encoder 20. The encoded video data can be directly transmitted to the destination device 14 through the output interface 22 of the information source device 12. The encoded video data can also (or instead) be stored in the recording device 32 for later access by the destination device 14 or other devices for decoding and / or playback. The output interface 22 can further include a modem and / or a transmitter.

[0018] The destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. The input interface 28 can include a receiver and / or a modem and receives the encoded video data through the link 16. The encoded video data communicated through the link 16 or supplied by the recording device 32 can include various syntax elements generated by the video encoder 20 that are used when the video decoder 30 decodes the video data. The encoded video data that can include such syntax elements is transmitted over a communication medium and stored in a recording medium or a file server.

[0019] In some embodiments, the display device 34 that the destination device 14 may include can be an integrated display device and an external display device configured to communicate with the destination device 14. The display device 34 displays the decoded video data to the user and can include any of various display devices such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0020] The video encoder 20 and the video decoder 30 can operate based on intellectual property or industry standards such as VVC, HEVC, MPEG-4 Part 10 AVC (Advanced Video Coding), or extended versions of these standards. It should be understood that this application is not limited to specific video encoding / decoding standards and can be applicable to other video encoding / decoding standards. In general, it is contemplated that the video encoder 20 of the source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is generally contemplated that the video decoder 30 of the destination device 14 can be configured to decode video data according to any of these current or future standards. The video encoder 20 and the video decoder 30 each include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs),

[0021] It can be implemented as any of various suitable encoded circuit configurations such as an integrated circuit, a field programmable gate array (FPGA), discrete logic, software, hardware, firmware, or any combination thereof. When the electronic device is implemented partially in software, instructions regarding the software are stored in a suitable non-transitory computer-readable medium, and the instructions are executed in hardware using one or more processors to perform the video encoding / decoding processes disclosed in this disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and any of them may be integrated as part of an integrated encoder / decoder (CODEC) combined in their respective devices.

[0022] FIG. 2 is a block diagram showing an exemplary video encoder 20 according to some embodiments described in this application. The video encoder 20 may perform intra-prediction encoding and inter-prediction encoding of video blocks inside a video frame. Intra-prediction encoding relies on spatial prediction to reduce or remove spatial redundancy in the video data within a given video frame or picture. Inter-prediction encoding relies on temporal prediction to reduce or remove temporal redundancy in the video data within adjacent video frames or pictures of a video sequence.

[0023] As shown in FIG. 2, the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a conversion processing unit 52, a quantization unit 54, and an entropy encoding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a partitioning unit 45, an intra-prediction processing unit 46, and an intra-block copy (BC) unit 48. Some embodiments In a form, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for reconstructing video blocks. From the reconstructed video, a deblocking filter (not shown) may be arranged between the adder 62 and the DPB 64 to filter the block boundary to remove block distortion. In addition to the deblocking filter, an in-loop filter (not shown) may also be used to filter the output of the adder 62. The video encoder 20 may take the form of a non-changeable or programmable hardware unit, or may be divided among one or more non-changeable or programmable hardware units.

[0024] The video data memory 40 may store video data encoded by components of the video encoder 20. The video data of the video data memory 40 may be obtained, for example, from the video source 18. The DPB 64 is a buffer that records reference video data used to encode video data by the video encoder 20 (for example, in an intra prediction encoding mode or an inter prediction encoding mode). The video data memory 40 and the DPB 64 may be formed by any of various recording devices. In various examples, the video data memory 40 may be on-chip with other components of the video encoder 20, or may be off-chip with respect to those components.

[0025] As shown in FIG. 2, a splitting unit 45 inside the prediction processing unit 41 splits the received video data into video blocks. This splitting may include splitting a video frame into slices, tiles, or other larger coding units (CUs) according to a predetermined splitting structure such as a quadtree structure associated with the video data. The video frame may be split into a plurality of video blocks (or a set of video blocks referred to as tiles). The prediction processing unit 41 may select one of a plurality of possible prediction coding modes, such as one of a plurality of intra prediction coding modes or one of a plurality of inter prediction coding modes, for the current video block based on error results (e.g., coding rate or distortion level). The prediction processing unit 41 may supply the resulting intra prediction coded block or inter prediction coded block to an adder 50 to generate a residual block, and may also supply this coded block to an adder 62 to reconstruct it for later use as part of a reference frame. The prediction processing unit 41 also supplies syntax elements such as motion vectors, intra mode indicators, splitting information, and other such syntax information to an entropy coding unit 56. [[ID=D1]] [[ID=D2]]

[0026] [[ID=D3]]<T To select an appropriate intra prediction coding mode for the current video block, an intra prediction processing unit 46 inside the prediction processing unit 41 may perform intra prediction coding of the current video block with respect to one or more neighboring blocks in the same frame as the current block to be coded, resulting in spatial prediction. A motion estimation unit 42 and a motion compensation unit 44 inside the prediction processing unit 41 perform inter prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames, resulting in temporal prediction. The video encoder 20 may execute a plurality of coding paths, for example, to select an appropriate coding mode for each block of video data. [[ID=D4]] [[ID=D5]]

[0027] [[ID=D6]] In some embodiments, the motion estimation unit 42 determines an inter-prediction mode for the current video frame by generating a motion vector indicating the displacement of a prediction unit (PU) of a video block within the current video frame with respect to a prediction block within a reference video frame according to a predetermined pattern within a series of video frames. The motion prediction executed by the motion estimation unit 42 is a process of generating a motion vector for estimating the motion of a video block. The motion vector is, for example, associated with the current block to be encoded within the current frame (or within another coding unit), and is a video within the current video frame or picture with respect to a prediction block within a reference frame (or within another coding unit). The displacement of the PU of the block may be indicated. The predetermined pattern may specify the video frame as a P-frame or a B-frame in the sequence. The intra BC unit 48 may determine a vector such as a block vector for intra BC encoding in the same manner as the determination of the motion vector by the motion estimation unit 42 for inter-prediction, or may determine the block vector using the motion estimation unit 42.

[0028] The prediction block is a block of the reference frame that is considered to closely correspond to the PU of the video block to be encoded from the perspective of pixel difference, and can be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference criteria. In some embodiments, the video encoder 20 may calculate the values of the sub-integer pixel positions of the reference frame stored in the DPB 64. For example, the video encoder 20 may interpolate the values of the 1 / 4 pixel position, 1 / 8 pixel position, or other fractional pixel positions of the reference frame. Accordingly, the motion estimation unit 42 may perform a motion search regarding the overall pixel position and fractional pixel position and output a motion vector having fractional pixel accuracy.

[0029] The motion estimation unit 42 calculates a motion vector by comparing the position of a prediction block in a reference frame selected from the first reference frame list (list 0) or the second reference frame list (list 1) with the position of the PU of the video block of the inter-predicted coded frame. Here, the first reference frame list or the second reference frame list identifies one or more reference frames stored in the DPB 64, respectively. The motion estimation unit 42 sends the calculated motion vector to the motion compensation unit 44 and then to the entropy coding unit 56.

[0030] The motion compensation executed by the motion compensation unit 44 may include fetching or generating a prediction block based on the motion vector determined by the motion estimation unit 42. When receiving the motion vector for the PU of the current video block, the motion compensation unit 44 searches for the prediction block indicated by the motion vector in one of the reference frame lists, extracts the prediction block from the DPB 64, and transfers the prediction block to the adder 50. Then, the adder 50 forms a residual video block of pixel difference values by subtracting the pixel values of the prediction block provided by the motion compensation unit 44 from the pixel values of the current video block to be coded. The pixel difference values forming the residual video block may include a luminance difference component, a chroma difference component, or both. The motion compensation unit 44 may also generate syntax elements related to the video blocks of the video frame used when the video decoder 30 decodes the video blocks of the video frame. The syntax elements may include, for example, syntax elements defining the motion vector used to identify the prediction block, any flag indicating the prediction mode, or other syntax information described herein. Note that the motion estimation unit 42 and the motion compensation unit 44 can be almost integrated, but are shown separately for conceptual purposes.

[0031] In some embodiments, the intra BC unit 48 may generate vectors and capture prediction blocks in a manner similar to that described above with respect to the motion estimation unit 42 and the motion compensation unit 44, but the prediction blocks are in the same frame as the current block being encoded, and the vectors are referred to as block vectors as opposed to motion vectors. Specifically, the intra BC unit 48 may determine to use an intra prediction mode to encode the current block. In some examples, the intra BC unit 48 may encode the current block using various intra prediction modes, for example, during an individual encoding pass, and analyze the performance of those intra prediction modes by rate-distortion analysis. Next, the intra BC unit 48 may select an appropriate intra prediction mode to use to generate an intra mode indicator among the various intra prediction modes tested. For example, the intra BC unit 48 may use rate-distortion analysis to calculate rate-distortion values for the various intra prediction modes tested and select, as the appropriate intra prediction mode to use, the intra prediction mode having the best rate-distortion characteristics among the modes tested. Rate-distortion analysis generally determines the amount of distortion (or error) between an encoded block and the original pre-encoded block from which the encoded block was generated, along with the bit rate (i.e., number of bits) used to generate those encoded blocks. The intra BC unit 48 may calculate the ratio of distortion to rate for various encoded blocks and determine the intra prediction mode that exhibits the best rate-distortion value for that block.

[0032] In other examples, the intra BC unit 48 may use all or part of the motion estimation unit 42 and the motion compensation unit 44 to perform such functions for intra BC prediction according to the embodiments described herein. In either case, for intra block copy, the prediction block may be a block that is considered to closely correspond to the block to be coded from the perspective of pixel difference, and may be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference metrics. The determination of the prediction block may include the calculation of the values of sub-integer pixel positions.

[0033] Whether the prediction block is from the same frame by intra prediction or from different frames by inter prediction, the video encoder 20 may form a residual video block by subtracting the pixel values of the prediction block from the pixel values of the current video block to be coded, forming pixel difference values. The pixel difference values for forming the residual video block may include both the luminance difference component and the chrominance difference component.

[0034] As described above, the intra prediction processing unit 46 may intra predict the current video block as an alternative to the inter prediction performed by the motion estimation unit 42 and the motion compensation unit 44, or the intra block copy prediction performed by the intra BC unit 48. Specifically, the intra prediction processing unit 46 may determine to use an intra prediction mode to code the current block. To do so, the intra prediction processing unit 46 may, for example, code the current block using various intra prediction modes during an individual coding pass, and the intra prediction processing unit 46 (or in some examples, the mode selection unit) may select an appropriate intra prediction mode to use from the tested intra prediction modes. The intra prediction processing unit 46 may supply information representing the selected intra prediction mode for that block to the entropy coding unit 56. The entropy coding unit 56 may code the information indicating the selected intra prediction mode in the bitstream.

[0035] After the prediction processing unit 41 determines a prediction block for the current video block by either inter prediction or intra prediction, the adder 50 generates a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more transform units (TUs) and is supplied to the transform processing unit 52. The transform processing unit 52 uses a transform such as a discrete cosine transform (DCT) or a conceptually similar transform to transform the residual video data into residual transform coefficients. transform) or a transform such as a conceptually similar transform.

[0036] The transform processing unit 52 may send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be changed by adjusting the quantization parameter. In some examples, the quantization unit 54 may then perform a scan of the matrix containing the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0037] Following quantization, the entropy coding unit 56 performs, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), Using context - adaptive binary arithmetic coding, probability interval partitioning entropy coding (PIPE), or another entropy coding technique or technology, the quantized transform coefficients are entropy - coded into a video bitstream. The coded bitstream can then be transmitted to the video decoder 30, or recorded in the recording device 32 for later transmission to, or retrieval by, the video decoder 30. The entropy coding unit 56 may also entropy - code motion vectors and other syntax elements related to the current video frame to be coded.

[0038] To generate a reference block for predicting other video blocks, the inverse quantization unit 58 applies inverse quantization and the inverse transform processing unit 60 applies inverse transform to reconstruct the residual video block in the pixel region. As described above, the motion compensation unit 44 can generate a motion - compensated prediction block from one or more reference blocks of the frames stored in the DPB 64. The motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub - integer pixel values for use in motion prediction.

[0039] The adder 62 generates a reference block for storing in the DPB 64 by adding the reconstructed residual block to the motion - compensated prediction block generated by the motion compensation unit 44. The reference block can then be used by the intra BC unit 48, the motion estimation unit 42, and the motion compensation unit 44 as a prediction block for inter - predicting another video block in a subsequent video frame.

[0040] FIG. 3 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra prediction processing unit 84, and an intra BC unit 85. The video decoder 30 may perform a decoding process that is overall inverse to the encoding process described with respect to the video encoder 20 in relation to FIG. 2. For example, the motion compensation unit 82 may generate prediction data based on the motion vectors received from the entropy decoding unit 80, while the intra prediction processing unit 84 may generate prediction data based on the intra prediction mode indicator received from the entropy decoding unit 80.

[0041] In some examples, the units of the video decoder 30 may be tasked to execute the embodiments of the present application. Also, in some examples, the embodiments of the present disclosure may be divided among one or more units of the video decoder 30. For example, the intra BC unit 85 may execute the embodiments of the present application alone or in combination with other units such as the motion compensation unit 82, the intra prediction processing unit 84, and the entropy decoding unit 80 of the video decoder 30. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functionality of the intra BC unit 85 may be executed by other components of the prediction processing unit 81 such as the motion compensation unit 82.

[0042] The video data memory 79 may store video data such as an encoded video bitstream decoded by other components of the video decoder 30. The video data stored in the video data memory 79 is from a recording device 32, for example, local video from a camera It can be obtained from a source by means of wired or wireless network communication of video data, or by accessing a physical data recording medium such as a flash drive or a hard disk. The video data memory 79 may include a coded picture buffer (CPB) that stores coded video data from a coded video bitstream. The decoded picture buffer (DPB) 92 of the video decoder 30 stores reference video data used to encode video data by the video decoder 30 (e.g., in an intra prediction coding mode or an inter prediction coding mode). The video data memory 79 and the DPB 92 may be formed by any of various memory devices, including synchronous dynamic random access memory (SDRAM), magneto-resistive RAM (MRAM), resistive random access memory (RRAM), or other types of memory devices. For purposes of illustration, the video data memory 79 and the DPB 92 are shown as two separate components of the video decoder 30 in FIG. 3. However, it will be apparent to those skilled in the art that the video data memory 79 and the DPB 92 may be provided by the same memory device or individual memory devices. In some examples, the video data memory 79 may be on-chip with other components of the video decoder 30 or off-chip with respect to those components.

[0043] During the decoding process, video decoder 30 receives an encoded video bitstream representing an encoded video frame and video blocks of associated syntax elements. Video decoder 30 may receive syntax elements at the video frame level and / or at the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra prediction mode indicators, and other syntax elements. Then, entropy decoding unit 80 transfers the motion vectors and other syntax elements to prediction processing unit 81.

[0044] When the video frame is encoded as an intra prediction encoded (I) frame or for intra encoded prediction blocks in other types of frames, intra prediction processing unit 84 of prediction processing unit 81 may generate prediction data for video blocks of the current video frame based on the signaled intra prediction mode and reference data from previously decoded blocks of the current frame.

[0045] When the video frame is encoded as an inter prediction encoded (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 generates one or more prediction blocks for video blocks of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the prediction blocks may be generated from one of the internal reference frames of a reference frame list. Video decoder 30 may configure a reference frame list, such as list 0 and list 1, using a default configuration technique based on the reference frames stored in DPB 92.

[0046] In some examples, when a video block is encoded according to the intra BC mode described herein, the intra BC unit 85 of the prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from the entropy decoding unit 80. The prediction block may be within the reconstructed area of the same picture as the current video block defined by the video encoder 20.

[0047] The motion compensation unit 82 and / or the intra BC unit 85 analyzes the motion vector and other syntax elements to obtain prediction information regarding the video blocks of the current video frame and then uses the prediction information to generate a prediction block for the current video block to be decoded. For example, the motion compensation unit 82 uses some of the received syntax elements to determine a prediction mode (e.g., intra prediction or inter prediction) used to encode the video blocks of the video frame, an inter prediction frame type (e.g., B or P), configuration information of one or more of the reference frame lists regarding the frame, the motion vector of each inter prediction encoded video block in the frame, the inter prediction state of each inter prediction encoded video block in the frame, and other information for decoding the video blocks in the current video frame.

[0048] Similarly, the intra BC unit 85 may use some of the received syntax elements, such as a flag, to determine that the current video block is predicted using the intra BC mode, configuration information of the video blocks of the frame that should be stored in the DPB 92 within the reconstructed area, the block vector of each intra BC prediction video block in the frame, the intra BC prediction state of each intra BC prediction video block in the frame, and other information for decoding the video blocks in the current video frame.

[0049] The motion compensation unit 82 may also perform interpolation using an interpolation filter such as that used to calculate the sub-integer pixel interpolation values of the reference block during the encoding of the video block by the video encoder 20. In this case, the motion compensation unit 82 may determine the interpolation filter used by the video encoder 20 from the received syntax elements and generate a prediction block using the interpolation filter.

[0050] The inverse quantization unit 86 inverse quantizes the quantized transform coefficients given in the bitstream and entropy decoded by the entropy decoder 80 using the same quantization parameter as that calculated to determine the degree of quantization for each video block in the video frame by the video encoder 20. The inverse transform processing unit 88 applies an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients to reconstruct the residual block in the pixel domain.

[0051] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by summing the residual block from the inverse transform processing unit 88 and the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. An in-loop filter (not shown) may be arranged between the adder 90 and the DPB 92 to further process the decoded video block. Then, the decoded video block in a given frame is stored in the DPB 92 that stores the reference frame used for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for presentation to a display device such as the display device 34 of FIG. 1 later.

[0052] In a typical video coding process, a video sequence typically includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chrominance samples. SCr is a two-dimensional array of Cr chrominance samples. In other cases, a frame may be black and white and therefore include only one two-dimensional array of luma samples.

[0053] As shown in FIG. 4A, video encoder 20 (more specifically, divider 45) first divides a frame into a set of coding tree units (CTUs). The video decoder 30 generates a coded representation. A video frame may contain an integer number of CTUs, sequentially ordered in raster scan order from left to right and top to bottom. Each CTU is the largest logical coding unit, and the width and height of a CTU are signaled by the video encoder 20 in a sequence parameter set so that all CTUs in a video sequence have the same size, which may be one of 128x128, 64x64, 32x32, and 16x16. However, it should be noted that the present application is not necessarily limited to a particular size. As shown in FIG. 4B, each CTU may include one coding tree block (CTB) consisting of luma samples, a corresponding coding tree block consisting of two chrominance samples, and syntax elements used to encode the samples in the coding tree block. The syntax elements describe the characteristics of various types of coding blocks of pixels and how the video sequence may be reconstructed in the video decoder 30, including inter- or intra-prediction, intra-prediction mode, motion vectors, and other parameters. For black and white pictures or pictures with three separate color planes, a CTU may contain a single coding tree block and syntax elements used to encode the samples of the coding tree block, which may be an NxN block of samples.

[0054] To achieve better performance, the video encoder 20 can recursively perform tree partitioning, such as binary-tree partitioning, quad-tree partitioning, or a combination of both, on the coding tree blocks of the CTU to divide the CTU into smaller coding units (CUs). As shown in FIG. 4C, a 64×64 CTU 400 is first divided into four smaller CUs, each having a block size of 32×32. Among the four smaller CUs, CU 410 and CU 420 are each divided into four CUs with a block size of 16×16. Two 16×16 CUs, 430 and 440, are each further divided into four CUs with a block size of 8×8. FIG. 4D represents a quad-tree data structure showing the final result of the partitioning process of the CTU 400 as shown in FIG. 4C, where each leaf node of the quad-tree corresponds to one CU of each size in the range of 32×32 to 8×8. Each CU may include, similar to the CTU shown in FIG. 4B, a coding block (CB) of luminance samples, two corresponding coding blocks of chrominance samples of the same size in the frame, and syntax elements used to code the samples of the coding blocks. For a black-and-white picture or a picture having three separate color planes, the CU may include a single coding block and a syntax structure used to code the samples of the coding block.

[0055] In some embodiments, video coder 20 may further divide the coding block of a CU into one or more M×N prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples to which the same (inter or intra) prediction is applied. The prediction unit (PU) of a CU may include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and the syntax elements used to predict the prediction block. In a monochrome picture or a picture having three separate color planes, the PU may include a single prediction block and the syntax structure used to predict the prediction block. Video coder 20 may generate prediction luma, Cb, and Cr blocks for luma, Cb, and each PU of the CU's Cr prediction blocks.

[0056] Video coder 20 may use intra prediction or inter prediction to generate the prediction block for the PU. When video coder 20 uses intra prediction to generate the prediction block of the PU, video coder 20 may generate the prediction block of the PU based on the decoded samples of the frame associated with the PU. When video coder 20 uses inter prediction to generate the prediction block of the PU, video coder 20 may generate the prediction block of the PU based on the decoded samples of one or more frames other than the frame associated with the PU may.

[0057] After the video encoder 20 generates prediction luminance blocks, prediction Cb blocks, and prediction Cr blocks for one or more PUs in a CU, it can generate a luminance residual block for the CU by subtracting the prediction luminance block of the CU from the original luminance encoded block of the CU such that each sample in the luminance residual block of the CU represents the difference between a luminance sample in one of the prediction luminance blocks of the CU and the corresponding sample in the original luminance encoded block of the CU. Similarly, the video encoder 20 can generate the Cb residual block and the Cr residual block of the CU such that each sample in the Cb residual block of the CU represents the difference between a Cb sample in one of the prediction Cb blocks of the CU and the corresponding sample in the original Cb encoded block of the CU, and each sample in the Cr residual block of the CU can represent the difference between a Cr sample in one of the prediction Cr blocks of the CU and the corresponding sample in the original Cr encoded block of the CU.

[0058] Moreover, as shown in FIG. 4C, the video encoder 20 uses quadtree splitting to decompose the luminance, Cb, and Cr residual blocks of the CU into one or more luminance, Cb, and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. The transform unit (TU) of the CU can include a transform block of luminance samples, two corresponding transform blocks of chrominance samples, and syntax elements used to predict the transform block samples. Thus, each TU of the CU can be associated with a luminance transform block, a Cb transform block, and a Cr transform block. In some examples, the luminance transform block associated with the TU can be a sub-block of the luminance residual block of the CU. The Cb transform block can be a sub-block of the Cb residual block of the CU. The Cr transform block can be a sub-block of the Cr residual block of the CU. In a black-and-white picture or a picture having three separate color planes, the TU can include a single transform block and a syntax structure used to transform the samples of the transform block.

[0059] Video encoder 20 may apply one or more transforms to the luminance transform block of the TU to generate a luminance coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. Video encoder 20 may apply one or more transforms to the Cb transform block of the TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to the Cr transform block of the TU to generate a Cr coefficient block for the TU.

[0060] After generating a coefficient block (e.g., a luminance coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to the process by which transform coefficients are quantized to somehow reduce the amount of data used to represent the transform coefficients, resulting in further compression. After quantizing the coefficient block, video encoder 20 may entropy encode the syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform context-adaptive binary arithmetic coding (CABAC) on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 may output a bitstream including a series of bits forming a representation of the encoded frame and associated data, which is stored in recording device 32 or transmitted to destination device 14.

[0061] After receiving the bitstream generated by video encoder 20, video decoder 30 may analyze the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally the reverse of the encoding process performed by video encoder 20. For example, the video decoder 30 may perform inverse transformation on the coefficient block related to the TU of the current CU to reconstruct the residual block related to the TU of the current CU. The video decoder 30 may also reconstruct the encoded block of the current CU by adding the samples of the prediction block for the current CU's PU to the samples of the transform block of the corresponding current CU's TU. After reconstructing the encoded blocks for each CU of the frame, the video decoder 30 may reconstruct the frame.

[0062] As described above, video coding mainly achieves video compression using two modes: intra-frame prediction (i.e., intra prediction) and inter-frame prediction (i.e., inter prediction). It should be noted that IBC can be regarded as either intra-frame prediction or a third mode. Between these two modes, inter-frame prediction uses motion vectors to predict the current video block from reference video blocks, thus contributing more significantly to the coding efficiency than intra-frame prediction.

[0063] However, due to the continuous improvement of video data capture technology for preserving details in video data and the finer video block sizes, the amount of data required to represent the motion vectors of the current frame also increases significantly. One way to overcome this problem is to take advantage of the fact that groups of neighboring CUs in both the spatial and temporal domains not only have similar video data for prediction purposes but also have similar motion vectors between these neighboring CUs. Therefore, it is possible to use the motion information of spatially neighboring CUs and / or CUs arranged in a temporally related manner as approximate motion information (e.g., motion vectors) of the current CU by detecting the spatial and temporal correlation relationships of these CUs. This is also referred to as the "motion vector predictor (MVP)" of the current CU.

[0064] Rather than encoding the actual motion vector of the current CU determined by the motion estimation unit 42 as described with respect to FIG. 2 into the video bitstream, the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to generate a motion vector difference (MVD) for the current CU. By doing so, it becomes unnecessary to encode the actual motion vector determined by the motion estimation unit 42 for each CU of the frame into the video bitstream, and the amount of data used to represent motion information in the video bitstream can be significantly reduced.

[0065] Similar to the process of selecting a prediction block in the reference frame during inter-frame prediction of a coded block, a list of motion vector candidates for the current CU is constructed using the possible candidate motion vectors related to the spatially neighboring CUs and / or the CUs arranged in a temporally related manner of the current CU, and then a set of rules is required to be adopted by both the video encoder 20 and the video decoder 30 to select one element from the list of motion vector candidates as the motion vector predictor for the current CU. By doing so, it becomes unnecessary to transmit the list of motion vector candidates itself between the video encoder 20 and the video decoder 30, and the index of the motion vector predictor selected within the list of motion vector candidates is sufficient for the video encoder 20 and the video decoder 30 to use the same motion vector predictor within the list of motion vector candidates for encoding and decoding the current CU.

[0066] In some embodiments, each inter-prediction CU has three motion vector prediction modes including an inter-prediction mode (also referred to as the "advanced motion vector prediction" (AMVP) mode), a skip prediction mode, and a merge prediction mode to construct a list of motion vector candidates. Under each mode, according to the algorithms described below, the motion vector One or more motion vector candidates may be added to the candidate list. Ultimately, one of the motion vector candidates in the candidate list is used as the best motion vector predictor for an inter prediction CU that is encoded into the video bitstream by the video encoder 20 or decoded from the video bitstream by the video decoder 30. To find the best motion vector predictor from the candidate list, a motion vector competition (MVC) scheme is introduced to select a motion vector from a given set of motion vectors, i.e., a list of motion vector candidates, including spatial motion vector candidates and temporal motion vector candidates.

[0067] After one MVP candidate is selected for the current CU within a given set of motion vector candidates, the video encoder 20 may generate one or more syntax elements for the corresponding MVP candidate and encode these syntax elements into the video bitstream so that the video decoder 30 can use these syntax elements to retrieve the MVP candidate from the video bitstream. Different modes (e.g., AMVP mode, merge mode, skip mode, etc.) have different sets of syntax elements depending on the specific mode used to construct the set of motion vector candidates. For the AMVP mode, the syntax elements include an inter prediction indicator (e.g., list 0, list 1, or bi-directional prediction), a reference index, a motion vector candidate index, a motion vector difference, and a prediction residual signal, etc. For the skip mode and the merge mode, since the current CU inherits other syntax elements including an inter prediction indicator, a reference index, and a motion vector from a neighboring CU referenced by the encoded merge index, only the merge index is encoded into the bitstream. In the case of a skip-encoded CU, the motion vector prediction residual signal is also omitted.

[0068] FIG. 5A is a block diagram showing spatially adjacent block positions arranged in time relation in the current CU that are encoded / decoded according to some embodiments of the present disclosure. For a given mode (e.g., AMVP mode, merge mode, or skip mode), the motion vector prediction (MVP) candidate list is first constructed by examining the validity of the motion vectors associated with spatially left (A0, A1) and upper (B0, B1, B2) adjacent block positions and the validity of the motion vectors associated with block positions arranged in time relation. During the process of constructing the list of MVP candidates, redundant MVP candidates are removed from the candidate list, and if necessary, zero-valued motion vectors are added to give the candidate list a fixed length (note that different modes may have different fixed lengths). After constructing the list of MVP candidates, the video encoder 20 can select the best motion vector predictor from the candidate list and encode the corresponding index indicating the selected candidate into the video bitstream.

[0069] Using FIG. 5A as an example and assuming that the candidate list has two fixed lengths, FIG. 5B is a flowchart showing an exemplary process by which a list of motion vector candidates is identified according to some embodiments of the present disclosure. Specifically, the list of motion vector prediction (MVP) candidates for the current CU can be constructed by performing the following steps under the AMVP mode as represented in FIG. 5B. 1) Step 505: Select two MVP candidates from five spatially adjacent CUs a) Derive one unscaled MVP candidate from one of the two left adjacent CUs spatially from A0 to A1 b) If there is no valid unscaled MVP candidate on the left in the previous step, derive one scaled MVP candidate from one of the two left adjacent CUs spatially from A0 to A1 c) In order from B0, B1, B2, derive one unscaled MVP candidate from one of the three upper adjacent CUs spatially d) If neither A0 nor A1 is valid, or if both are coded in intra mode, In this case, we derive one scaled MVP candidate from B0, B1, B2, and one of the three spatially upper neighboring CUs. 2) Step 510: Select one MVP candidate from two CUs that are aligned in time relation; 3) Step 515: Remove duplicate MVP candidates found in previous steps from the MVP candidate list; 4) Step 520: Add up to two MVPs with zero values to the MVP candidate list; 5) Step 525: Remove MVP candidates with index greater than 1 from the MVP candidate list; 6) Step 530: Finalize two MVP candidates in the MVP candidate list for the current CU.

[0070] Since there are only two candidates in the MVP candidate list for the AMVP mode constructed above, an associated syntax element, such as a binary flag, is coded into the bitstream to indicate which of the two MVP candidates within the candidate list will be used to decode the current CU.

[0071] In some embodiments, the process of selecting a temporal motion vector predictor for encoding / decoding a current CU is performed at the sub-CU level to improve the accuracy of the decoded picture. This process first identifies a co-located picture for the current picture that includes the current CU, and then determines a temporal vector (also known as a "motion shift amount" in this application). Next, this process divides the current CU into multiple sub-CUs and derives motion information for each sub-CU from a corresponding block in the co-located picture identified by the temporal vector according to a predetermined algorithm also referred to as "subblock-based temporal motion vector prediction (SbTMVP)."

[0072] Figure 5C is a block diagram showing how sub-block based temporal motion vector prediction is performed between the current picture and the collocated picture according to some embodiments of the present disclosure. In this example, the current CU 535-1 is a 64×64 coded block and is divided into 8×8 sub-CUs, and each sub-CU is an 8×8 coded block. To derive the motion information for each sub-CU, the SbTMVP process is divided into the following two main steps. · Step 1: Using the so-called "temporal vector" between the current picture 535 and the collocated picture 540, identify the corresponding block 540-1 in the collocated picture 540. · Step 2: Divide the current CU 535-1 into a plurality of sub-CUs, and obtain the motion information including the motion vector for the sub-CU and the reference index (default is zero) for each sub-CU 535-3 from the corresponding block in the collocated picture 540.

[0073] As described above, it is assumed that the collocated picture 540 for the current picture 535 is known before the execution of the SbTMVP process. For example, the collocated picture is generally one of the two reference picture lists of the current picture, such as list 0, list 1, i.e., one reference picture of the current picture. In some embodiments, the corresponding block is the one at the same relative position as the current CU in the current picture and the collocated picture. In some other embodiments (e.g., Figure 5C), the corresponding block is not necessarily the one at the same relative position as the current CU 535-1 in the current picture and the collocated picture. Instead, there is a temporal vector that associates the center of the current CU 535-1 in the current picture with the center of the corresponding block 540-1 in the collocated picture 540. No. Instead, there is a temporal vector that associates the center of the current CU 535-1 in the current picture with the center of the corresponding block 540-1 in the collocated picture 540.

[0074] In some embodiments, the SbTMVP process is implemented as part of the process of constructing the list of motion vector candidates described above for the current CU during video encoding. In other words, if it is determined that the current CU should be processed using SbTMVP, then accordingly, parameters corresponding to SbTMVP are added to the list of motion vector candidates. In some other embodiments, the SbTMVP process is implemented independently of the process of constructing the list of motion vector candidates described above for the current CU during video encoding. In other words, SbTMVP is treated as a separate inter-prediction mode, like the aforementioned inter-prediction mode. Since the encoding process and the decoding process are symmetric in nature, hereinafter, the use of the decoding of the current CU to show how the SbTMVP process is employed to predict the temporal motion vector of the current CU will be described.

[0075] FIG. 6 is a flowchart showing an exemplary decoding process in which, according to some embodiments of the present disclosure, a video decoder implements a technique for constructing sub-block-based temporal motion vector prediction for a current picture from motion information of a collocated picture.

[0076] First, the video decoder 30 obtains an encoded video bitstream including data related to a plurality of encoded pictures (610). As shown in FIGS. 4A and 4C, each picture includes a plurality of rows of coding tree units (CTUs), and each CTU includes one or more coding units (CUs). The video decoder 30 extracts separate information portions from the video bitstream, such as syntax elements and pixel values, and reconstructs the pictures row by row.

[0077] In this example, video decoder 30 is assumed to decode (630) the current picture 535 shown in FIG. 5C, more specifically, the current CU 535-1 in the current picture 535. As described above, the current picture 535 has a plurality of reference pictures (e.g., list 0 and / or list 1). To predict the temporal motion vector, one of the reference pictures is a so-called "collocated picture" 540 of the current picture 535 as shown in FIG. 5C. Thus, video decoder 30 first determines (630-1) one of the reference pictures as the collocated picture of the current picture according to the syntax elements shown in a predetermined order in the video bitstream. For example, in the derivation of the temporal motion vector candidates, first, an explicit flag (collocated_from_l0_flag) in the slice header is sent to video decoder 30 to indicate whether the collocated picture is selected from, for example, list 0 or list 1. A collocated reference index (collocated_ref_idx) is further sent to video decoder 30 to indicate which reference picture in that list is selected as the collocated picture for deriving the temporal motion vector candidates.

[0078] In either case, after the reference picture of the current picture is specified as the collocated picture, video decoder 30 determines (630-3) the amount of motion shift (also known as the "temporal vector") between the collocated picture and the current picture (e.g., see the temporal vector 537 shown in FIG. 5C) from the motion information of the spatially neighboring blocks of the current CU 535-1 in a certain order. As described above, each CTU (including one or more CUs) has a plurality of spatially neighboring blocks such as A0, A1, B0, B1, B2, etc. Each of the spatially neighboring blocks may have a motion vector indicating the corresponding block inside each reference picture of the current CU. In some embodiments, video decoder 30 searches until one identical to the collocated picture is found. Examine each reference picture associated with one corresponding spatially neighboring block. The order of examination is adaptable. In some embodiments, the order of examination starts from one of List 0 and List 1 according to the low-delay condition (LDC) and the syntax element "collocated_from_10_flag". The LDC is a boolean variable indicating whether all reference pictures have a picture order count (POC) smaller than that of the current picture. For example, List 0 may include at least one reference picture preceding the current picture in time, and optionally may include one or more reference pictures following the current picture in time. List 1 may include at least one reference picture following the current picture in time, or only reference pictures preceding the current picture in time. After the video decoder 30 identifies the reference pictures associated with a particular spatially neighboring block of the current CU, it determines the motion information of the spatially neighboring block and uses this as the motion shift amount between the collocated picture and the current picture. Using the motion shift amount, blocks inside the collocated picture can be identified as corresponding to the current CU. After the video decoder 30 establishes the mapping relationship between the current CU535-1 and the corresponding block 540-1 as shown in FIG. 5C, it can start to construct the temporal motion vector prediction for each sub-block of the current CU535-1.

[0079] Note that the video encoder 20 is assumed to divide the current CU535-1 into a plurality of sub-CUs during the generation of the video bitstream received by the video decoder 30. Therefore, the video decoder 30 can use the same set of syntax elements from the video bitstream to divide the current CU535-1 into a plurality of sub-CUs (630-5). As shown in FIG. 5C, each sub-CU535-3 in the current picture 535 has a corresponding sub-block at the same relative position in the collocated picture 540. The video decoder 30 is responsible for reconstructing the same temporal motion vector prediction as when the video encoder 20 encodes the sub-CU into the video bitstream according to the SbTMVP process for each sub-CU.

[0080] In some embodiments, the video decoder 30 obtains a temporal motion vector predictor for each sub-CU of the current CU based on two pieces of information: the motion shift amount between the collocated picture and the current picture, and the motion information of the blocks in the collocated picture corresponding to the respective sub-blocks of the current picture (630-7). For example, for each sub-CU in the current picture 535, the video decoder 30 identifies a block in the collocated picture at the same relative position as the sub-block of the sub-CU in the current picture according to the motion shift amount between the collocated picture and the current picture. Next, the video decoder 30 determines the motion information of the identified block in the collocated picture, selects a motion vector and a reference index from the determined motion information of the identified block, and derives a temporal motion vector predictor for the sub-CU according to the ratio of the POC difference between the current picture and its reference picture and the POC difference between the collocated picture and the reference picture of the corresponding block.

[0081] On the other hand, the corresponding block 540-1 in the collocated picture 540 may belong to different CUs, CTUs, or even different slices or tiles. It is possible that different sub-blocks in the corresponding block 540-1 may have different prediction modes so that some of the sub-blocks in the corresponding block 540-1 may not have any motion vectors at all. In some embodiments, the video decoder 30 addresses this situation by examining a specific sub-block inside the corresponding block 540-1 to determine whether the specific sub-block has motion information. If the specific sub-block has motion information, when there is no motion information in the corresponding block within the collocated picture of the sub-CU, the video decoder 30 may use the one of the default temporal motion vector predictors for another sub-CU of the current CU for the specific sub-block. For example, before processing any sub-CU in the current CU, the video decoder 30 may first examine the corresponding block of the block (or sample) at or near the center of the corresponding block 540-1 (e.g., the one to the right and directly below the center of the corresponding block 540-1) to check whether this block has motion information. If this block does not have motion information, the video decoder 30 then assumes that the SbTMVP process is not applicable to the current CU and proceeds to process another merge candidate in the current picture. However, if this block has motion information, the video decoder 30 assumes that the SbTMVP process is applicable to the current CU and, when the corresponding block at the same relative position in the collocated picture of the current CU does not have the motion information for constructing the temporal motion vector predictor of the sub-CU, uses this motion information as the default temporal motion vector predictor for any sub-CU of the current CU. The video decoder 30 assumes that the SbTMVP process is applicable to the current CU and accordingly decodes the current CU using the temporal motion vector predictors obtained for a plurality of sub-CUs of the current CU (630-9).

[0082] As described above, intra-block copy (IBC) can significantly improve the encoding efficiency of screen content materials. Since the IBC mode is implemented as a block-level encoding mode, block matching (BM) is performed in the video encoder 20 to find the best block vector for each CU. Here, a block vector is used to indicate the displacement from the current block to a reference block that has already been reconstructed within the current picture. The CU encoded by IBC is treated as a third prediction mode other than the intra prediction mode or the inter prediction mode.

[0083] At the CU level, the IBC mode can be signaled as the IBC·AMVP mode or the IBC skip / merge mode as follows. - IBC·AMVP mode: The block vector difference (BVD) between the actual block vector of the CU and the block vector predictor of the CU selected from the CU's block vector candidates is encoded in the same way as the motion vector difference is encoded under the aforementioned AMVP mode. The block vector prediction method uses two block vector candidates of the left neighboring block vector candidate and the upper neighboring block vector candidate as predictors (when encoded by IBC). When neither of the neighboring block vector candidates is valid, the default block vector is used as the block vector predictor. A binary flag indicating the index of the block vector predictor is signaled. - IBC skip / merge mode: A merge candidate index is used to indicate which of the block vector candidates in the merge candidate list from neighboring IBC-encoded blocks is used to predict the block vector for the current block.

[0084] The video coder may initialize the context for the current wavefront based on the data of the first two blocks of the upper wavefront and one or more elements of the slice header of the slice including the first coded block of the current wavefront to perform context-adaptive binary arithmetic coding (CABAC) of the current wavefront. After coding two CTUs of the upper CTU row of the subsequent CTU row, the video coder may use the context state to perform CABAC initialization of the subsequent wavefront (or CTU row). In other words, assuming that the current wavefront is not the top row of the CTUs of the picture, the video coder may code at least two blocks of the upper wavefront of the current wavefront before starting to code the current wavefront. Then, the video coder may initialize the CABAC context for the current wavefront after coding at least two blocks of the upper wavefront of the current wavefront.

[0085] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable recording medium corresponding to a tangible medium such as a data recording medium, or a communication medium including any medium that facilitates transfer of a computer program from one location to another, for example, according to a communication protocol. Thus, the computer-readable medium generally may correspond to (1) a non-transitory tangible computer-readable recording medium, or (2) a communication medium such as a signal or a carrier wave. The data recording medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the embodiments described in this application. A computer program product may include a computer-readable medium.

[0086] The terminology used in the description of the embodiments herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the term "and / or" refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0087] For the purpose of describing various elements, terms such as first, second, etc. may be used herein, but it will be understood that these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of the embodiments, the first electrode may be referred to as the second electrode, and similarly, the second electrode may be referred to as the first electrode. The first electrode and the second electrode are both electrodes but not the same electrode.

[0088] The description of this application is presented for illustration and explanation purposes and is not intended to be exhaustive or limited to the invention in the disclosed form. Many modifications, variations, and alternative embodiments should be apparent to those skilled in the art who benefit from the teachings presented in the foregoing description and the associated drawings. The embodiments are selected and described in order to best explain the principles of the invention and the practical application thereof, to enable others skilled in the art to understand the invention with respect to various embodiments, and to best utilize the basic principles and various embodiments with various modifications suitable for the intended specific usage. Therefore, it should be understood that the claims are not limited to the specific examples of the disclosed embodiments and their modifications, and that other embodiments are intended to be included within the scope of the appended claims.

Claims

1. A decoding method, comprising: selecting, as a collocated picture of a current picture, one of a plurality of reference pictures of the current picture according to a syntax element signaled in a video bitstream, wherein the plurality of reference pictures include a first reference picture list and a second reference picture list different from the first reference picture list, the syntax element includes a first syntax element and a second syntax element, the first syntax element indicates whether the collocated picture is selected from the first reference picture list or the second reference picture list, and the second syntax element indicates which reference picture in the reference picture list indicated by the first syntax element is the collocated picture; determining a temporal vector between the collocated picture and the current picture; examining reference pictures associated with one spatially neighboring block of a current CU in the current picture in a certain order until one reference picture identical to the collocated picture is found among the plurality of reference pictures associated with the one spatially neighboring block; selecting a motion vector of the one spatially neighboring block that designates the examined reference picture as the temporal vector; including; dividing the current CU into a plurality of sub-CUs, each corresponding to a respective sub-block of the current picture; obtaining a temporal motion vector predictor for each sub-CU of the current CU from (i) the temporal vector between the collocated picture and the current picture and (ii) motion information of blocks in the collocated picture corresponding to the respective sub-blocks of the current picture; decoding the current CU according to the temporal motion vector predictors of the plurality of sub-CUs of the current CU; A method comprising the above steps.

2. The method according to claim 1, wherein the first reference picture list includes at least one reference picture preceding the current picture in time, and the second reference picture list includes at least one reference picture following the current picture in time.

3. The method according to claim 1, wherein the fixed order is a predetermined order of the plurality of reference pictures, and the predetermined order is for processing the first reference picture list before processing the second reference picture list.

4. The method according to claim 1, wherein the fixed order is a predetermined order of the plurality of reference pictures, and the predetermined order is for processing the second reference picture list before processing the first reference picture list.

5. The method according to claim 1, The step of obtaining a temporal motion vector predictor for each sub-CU of the current CU comprises: identifying a block in the collocated picture at the same relative position as the sub-block of the sub-CU in the current picture according to the temporal vector between the collocated picture and the current picture; determining motion information of the identified block in the collocated picture; deriving a motion vector from the determined motion information of the identified block as the temporal motion vector predictor of the sub-CU. The method further comprising the above steps.

6. The method according to claim 1, The step of obtaining a temporal motion vector predictor for each sub-CU of the current CU comprises: identifying a block in the collocated picture corresponding to the sub-block of the sub-CU at or near the center of the current CU in the current picture; determining motion information of the identified block in the collocated picture; deriving a motion vector from the determined motion information of the identified block as the default temporal motion vector predictor of any sub-CU of the current CU, in a step where the corresponding block in the collocated picture of the current CU has no motion information. The method further comprising the above steps.

7. One or more processors, A memory connected to the one or more processors and configured to store instructions executable by the one or more processors and a bitstream to be decoded; A plurality of programs stored in the memory; An arithmetic unit comprising the above, wherein the one or more processors are configured to perform the method according to any one of claims 1 to 6 on the bitstream when executing the instructions. **Claim 8** A non-transitory computer-readable recording medium storing computer-executable instructions and a bitstream to be decoded, wherein when the computer-executable instructions are executed by one or more computer processors, the one or more computer processors are caused to perform the method according to any one of claims 1 to 6 on the bitstream. **Claim 9** A computer program for execution by an arithmetic unit having one or more processors, wherein when the computer program is executed by the one or more processors, the arithmetic unit is caused to perform the steps of the method according to any one of claims 1 to 6. **Claim 10** A method of storing a video bitstream, comprising: Generating the video bitstream by performing an encoding method; and Storing the video bitstream to be decoded by the method according to any one of claims 1 to 6, wherein the encoding method comprises: Selecting, as a collocated picture of the current picture, one of a plurality of reference pictures of the current picture, the plurality of reference pictures including a first reference picture list and a second reference picture list different from the first reference picture list; Determining a temporal vector between the collocated picture and the current picture; Inspecting the reference pictures associated with one spatially adjacent block of the current CU in the current picture in a certain order until one reference picture identical to the collocated picture is found among the plurality of reference pictures associated with the one spatially adjacent block. selecting, as the temporal vector, motion vectors of the one spatially neighboring block that specifies the examined reference picture including the step of dividing the current CU into a plurality of sub-CUs, each corresponding to a respective sub-block of the current picture obtaining a temporal motion vector predictor for each sub-CU of the current CU from (i) the temporal vector between the collocated picture and the current picture and (ii) motion information of blocks in the collocated picture corresponding to the respective sub-blocks of the current picture encoding the current CU according to the temporal motion vector predictors of the plurality of sub-CUs of the current CU to generate a video bitstream, the video bitstream including a first syntax element and a second syntax element, the first syntax element indicating whether the collocated picture is selected from the first reference picture list or the second reference picture list, and the second syntax element indicating which reference picture in the reference picture list indicated by the first syntax element is the collocated picture A method comprising.

Images