Video encoding method and related apparatus

Abstract

The present disclosure provides a video encoding method and related apparatus, obtaining a current picture, for a current block included in the current picture, identifying a reference block included in a reference picture different from the current picture, the current block being divided into a plurality of sub-blocks (CBSB), and the reference block having a plurality of sub-blocks (RBSB), the plurality of RBSB respectively corresponding to the plurality of CBSB. For each CBSB, in response to the corresponding RBSB being encoded in an intra mode or an intra block copy mode, setting a motion vector predictor of the CBSB as a default motion vector. For each CBSB, in response to the corresponding RBSB being encoded in an inter mode, determining the motion vector predictor of the CBSB as equal to a motion vector predictor of the corresponding RBSB. Performing sub-block based temporal motion vector prediction on each CBSB based on the motion vector predictor of each CBSB, and encoding each CBSB to obtain a coded video bitstream.

Description

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 680,468, "METHODS FOR SUB-BLOCK BASED TEMPORAL MOTION VECTOR PREDICTION," filed June 4, 2018, and U.S. Application No. 16 / 210,635, filed December 5, 2018, the entire contents of which are expressly incorporated by reference herein.

[0002] This application is a divisional application of Chinese Patent Application No. 201980033757.1, filed May 30, 2019, entitled "METHODS FOR SUB-BLOCK BASED TEMPORAL MOTION VECTOR PREDICTION," and claims the benefit of priority to U.S. Provisional Application No. 62 / 680,468, "METHODS FOR SUB-BLOCK BASED TEMPORAL MOTION VECTOR PREDICTION," filed June 4, 2018, and U.S. Application No. 16 / 210,635, filed December 5, 2018, the entire contents of which are expressly incorporated by reference herein. TECHNICAL FIELD

[0003] The present disclosure relates to video processing techniques, and more particularly, to methods for sub-block based temporal motion vector prediction and related apparatuses. BACKGROUND

[0004] The background description provided herein is for the purpose of generally presenting the context of the disclosure. The work of the presently named inventors, to the extent the work is described in this background section, as well as aspects of the description that can not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0005] It has been known for several decades to use inter-picture prediction with motion compensation for performing video encoding and decoding. Uncompressed digital video can include a series of pictures, each picture having a spatial dimension, for example 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (also known as frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has a very high data rate. For example, 1080p60 4:2:0 video (1920 x 1080 luminance samples at 60 Hz) at 8 bit per sample requires approximately 1.5 Gbit / s bandwidth. An hour of such video requires more than 600 GB of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal by compression. Compression can help to reduce the bandwidth or storage space needs, as mentioned above, by two orders of magnitude or more in some cases. Lossless compression, lossy compression, and a combination thereof can be employed. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal can not be identical to the original signal, but the distortion between original and reconstructed signal is small enough to make the reconstructed signal usable for the intended application. In the case of video, lossy compression is applied widely. The amount of distortion tolerated depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of a television contribution application. The achievable compression ratio can reflect that: a higher permissible / tolerable distortion can result in a higher compression ratio.

[0007] Motion compensation can be a lossy compression technique and can involve the use of a block of sample data from a previously reconstructed picture or a portion thereof (reference picture) to predict a newly reconstructed picture or picture portion after a spatial shift in a direction indicated by a motion vector (MV henceforth). In some cases, the reference picture can be the same as the picture under reconstruction. An MV can have two dimensions, X and Y, or three dimensions, where the third dimension is an indication of the reference picture in use (the latter can be indirectly a temporal dimension).

[0008] In some video compression techniques, an MV applicable to a certain region of sample data can be predicted from other MVs, as for example from other MVs related to another region of sample data that is spatially adjacent to the region under reconstruction and precedes the MV in decoding order. This can substantially reduce the data amount needed to signal the MV, thereby removing redundancy and increasing compression. MV prediction can work effectively, for example, because in the case of coding an input video signal derived from a camera (known as natural video), there is a statistical likelihood that regions larger than the region to which a single MV is applicable move in a similar direction and, therefore, can in some cases be predicted using a similar motion vector derived from MVs of neighboring regions. This results in the MV found for a given region to be similar or the same as the MV predicted from the surrounding MVs and can in turn be represented at a number of bits less than what would be used had the MV been coded directly after entropy coding. In some cases, MV prediction can be an example of lossless compression of a signal (namely: the MVs) derived from the original signal (namely: the sample stream). In other cases, MV prediction itself can be lossy, for example because of rounding errors when calculating a predictor from several surrounding MVs.

[0009] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Out of the many MV prediction mechanisms offered by H.265, this disclosure describes a technique henceforth referred to as "spatial merge."

[0010] Some forms of inter prediction are performed at the sub-block level. However, sub-block based temporal motion vector prediction modes, such as alternative temporal motion vector prediction (ATMVP) and spatial-temporal motion vector prediction (STMVP), require the corresponding sub-block to be coded in inter mode. However, these temporal motion vector prediction modes are not capable of handling sub-blocks coded in intra mode, such as intra block copy mode. SUMMARY

[0011] An exemplary embodiment of the present disclosure includes a method for video decoding of a decoder. The method includes obtaining a current picture from a coded video bitstream. The method also includes, for a current block included in the current picture, identifying a reference block included in a reference picture different from the current picture, wherein the current block is partitioned into a plurality of sub-blocks (CBSB) and the reference block has a plurality of sub-blocks (RBSB) respectively corresponding to different ones of the plurality of CBSB. The method further includes determining whether a reference picture of the RBSB is the current picture, and in response to determining that the reference picture of the RBSB is the current picture, determining a coding mode of the RBSB as an intra mode. The method also includes, in response to determining that the reference picture of the RBSB is not the current picture, (i) determining, for one of the CBSB, whether a coding mode of the RBSB is one of an intra mode and an inter mode, and (ii) determining a motion vector predictor value for the one of the CBSB based on whether the coding mode of the corresponding RBSB is one of the intra mode and the inter mode.

[0012] An exemplary embodiment of the present disclosure includes an apparatus for video decoding. The apparatus includes an obtaining unit, an identifying unit, a first determining unit, and a second determining unit. The obtaining unit is configured to obtain a current picture from a coded video bitstream. The identifying unit is configured to, for a current block included in the current picture, identify a reference block included in a reference picture different from the current picture, the current block being partitioned into a plurality of sub-blocks (CBSBs), the reference block having a plurality of sub-blocks (RBSBs) corresponding to different ones of the plurality of CBSBs respectively. The first determining unit is configured to determine whether the reference picture of the RBSB is the current picture. The second determining unit is configured to, in response to determining that the reference picture of the RBSB is the current picture, determine a coding mode of the RBSB as an intra mode; and in response to determining that the reference picture of the RBSB is not the current picture, (i) determine, for one of the CBSBs, whether a coding mode of the RBSB is one of an intra mode and an inter mode, and (ii) determine a motion vector predictor of the one of the CBSBs based on whether the coding mode of the corresponding RBSB is one of the intra mode and the inter mode.

[0013] An exemplary embodiment of the present disclosure includes a non-transitory computer-readable medium storing instructions that, when executed by a processor in a video decoder, cause the processor to perform a method. The method includes obtaining a current picture from a coded video bitstream. The method also includes, for a current block included in the current picture, identifying a reference block included in a reference picture different from the current picture, wherein the current block is partitioned into a plurality of sub-blocks (CBSBs) and the reference block has a plurality of sub-blocks (RBSBs) corresponding to different ones of the plurality of CBSBs respectively. The method further includes determining whether the reference picture of the RBSB is the current picture, and in response to determining that the reference picture of the RBSB is the current picture, determining a coding mode of the RBSB as an intra mode. The method also includes, in response to determining that the reference picture of the RBSB is not the current picture, (i) determining, for one of the CBSBs, whether a coding mode of the RBSB is one of an intra mode and an inter mode, and (ii) determining a motion vector predictor of the one of the CBSBs based on whether the coding mode of the corresponding RBSB is one of the intra mode and the inter mode.

[0014] An exemplary embodiment of the present disclosure includes a computer device, the device comprising a processor and a memory; the memory is configured to store program codes and transmit the program codes to the processor; the processor is configured to execute the video decoding method for decoder according to the instructions in the program codes.

[0015] As can be seen from the above solutions, some implementations of the present disclosure can implement time motion vector prediction mode processing to encode sub-blocks in an intra mode such as an intra block copy mode. BRIEF DESCRIPTION OF DRAWINGS

[0016] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and from the accompanying drawings, in which:

[0017] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment;

[0018] Figure 2 is a schematic diagram of a simplified block diagram of a communication system according to another embodiment;

[0019] Figure 3 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment;

[0020] Figure 4 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment;

[0021] Figure 5 shows a block diagram of an encoder according to another embodiment;

[0022] Figure 6 shows a block diagram of a decoder according to another embodiment;

[0023] Figure 7 is a schematic diagram of intra picture block compensation;

[0024] Figure 8 is a schematic diagram of a current block and surrounding spatial merge candidates of the current block;

[0025] Figure 9 is a schematic diagram of sub-blocks of a current block and corresponding sub-blocks of a reference block;

[0026] Figure 10 shows an embodiment of processing performed by an encoder or a decoder;

[0027] Figure 11 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION

[0028] The Chinese interpretation of English abbreviations and the English full names involved in the present disclosure are shown in Appendix A attached at the end of the specification.

[0029] Figure 1A simplified block diagram of a communication system (100) is shown in accordance with one embodiment of the present disclosure. The communication system (100) includes multiple terminal devices that can communicate with one another, e.g., via a network (150). For instance, the communication system (100) includes a first pair of terminal devices (110) and (120) interconnected via the network (150). In Figure 1 In an example, the first pair of terminal devices (110) and (120) performs unidirectional data transmission. For instance, the terminal device (110) can encode video data, e.g., a stream of video pictures captured by the terminal device (110) for transmission to the other terminal device (120) via the network (150). The encoded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device (120) can receive the encoded video data from the network (150), decode the encoded video data to recover the video pictures, and display the video pictures according to the recovered video data. Unidirectional data transmission is common in media serving applications.

[0030] In another example, the communication system (100) includes a second pair of terminal devices (130) and (140) that performs bidirectional transmission of encoded video data, e.g., occurring during a videoconferencing session. For bidirectional data transmission, both the terminal devices (130) and (140) can code video data (e.g., streams of video pictures captured by the terminal devices) for transmission to the other terminal device in the pair of terminal devices (130) and (140) via the network (150). In an example, each terminal device in the pair of terminal devices (130) and (140) can receive the encoded video data transmitted by the other terminal device in the pair of terminal devices (130) and (140), and can decode the encoded video data to recover the video pictures, and can display the video pictures on an accessible display device according to the recovered video data.

[0031] In Figure 1In the example, the terminal devices (110), (120), (130), and (140) can be illustrated as personal computers and smartphones, but principles of the present disclosure can be not limited thereto. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (150) represents any number of networks that convey coded video data among the terminal devices (110), (120), (130), and (140), including for example wire- line connections (wired), and / or wireless connections. The network (150) can exchange data in circuit- switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of the present discussion, the architecture and topology of the network (150) can be immaterial to the operation of the present disclosure unless otherwise stated herein.

[0032] As an example of the application of the disclosed subject matter, Figure 2 The placement of video encoders and video decoders is shown in a streaming environment. The disclosed subject matter can be equally applicable to other support of video, including, for example, video conferencing, digital TV, storing of compressed video on digital media including CD, DVD, memory stick etc.

[0033] A streaming system can include a capture subsystem (213) that can include a video source (201), for example a digital camera, that creates, for example, a stream of video pictures (202) that are uncompressed. In an example, the stream of video pictures (202) includes samples taken by the digital camera. The stream of video pictures (202), depicted as a thick line to emphasize a high data volume when compared to the encoded video data (204) (or coded video bitstream), can be processed by an electronic device (220) that includes a video encoder (203) coupled to the video source (201). The video encoder (203) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data (204) (or coded video bitstream (204)), depicted as a thin line to emphasize the lower data volume when compared to the stream of video pictures (202), can be stored on a streaming server (205) for future use. Coders such as video encoders and decoders (also known as "CODECs") can be implemented as hardware or software with a Figure 2One or more streaming client subsystems of client subsystems (206) and (208) in the example can access the streaming server (205) to retrieve copies (207) and (209) of the encoded video data (204). The client subsystem (206) can include, for example, a video decoder (210) in an electronic device (230). The video decoder (210) decodes the incoming copy (207) of the encoded video data (204) and creates an outgoing stream of video pictures (211) that can be presented on a display (212) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (204), copies (207) and (209) thereof (e.g., video bitstreams) can be encoded according to certain video coding / compression standards. Examples of those standards include ITU-T H.265. In examples, a video coding standard under development is informally known as Versatile Video Coding or VVC. The disclosed subject matter can be used in the context of the VVC.

[0034] It is noted that the electronic device (220) and the electronic device (230) can include other components (not shown). For example, the electronic device (220) can include a video decoder (not shown) and the electronic device (230) can include a video encoder (not shown).

[0035] Figure 3 A block diagram of a video decoder (310) is shown according to an embodiment of the disclosure. The video decoder (310) can be included in an electronic device (330). The electronic device (330) can include a receiver (331) (e.g., receiving circuitry). The video decoder (310) can be used in place of Figure 2 The video decoder (210) in the example.

[0036] The receiver (331) can receive one or more coded video sequences to be decoded by the video decoder (310); in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent from other coded video sequences. The coded video sequences can be received from a channel (301), which can be a hardware / software link into a storage device where the encoded video data is stored. The receiver (331) can receive the encoded video data with other data, e.g., coded audio data and / or ancillary data streams that can be forwarded to their respective consuming entities (not depicted). The receiver (331) can separate the coded video sequence from the other data. To combat network jitter, a buffer memory (315) can be coupled between the receiver (331) and the entropy decoder / pars er (320) (hereinafter “parser (320)”). In some applications, the buffer memory (315) is part of the video decoder (310). In other applications, the buffer memory (315) can be outside the video decoder (310) (not depicted). In still other applications, there can be a buffer memory outside the video decoder (310) (not depicted) to combat network jitter, for example, and a separate buffer memory (315) inside the video decoder (310) to handle playout timing, for example. When the receiver (331) is receiving data from a store / forward device of sufficient capacity and controllability or from an isosychronous network, the buffer memory (315) can not be needed, or can be small. For use on best effort packet networks such as the Internet, the buffer memory (315) can be relatively large and can advantageously be of adaptive size, and can be implemented at least partly in operating system or similar elements outside the video decoder (310) (not depicted).

[0037] The video decoder (310) can include a parser (320) that reconstructs symbols (321) from the coded video sequence. Categories of these symbols (321) include information used to manage operation of the video decoder (310), and potentially information to control a rendering device such as a rendering device (312) (e.g., a display screen) that is not integral but is coupled to the electronic device (330), as is Figure 3Control information for the rendering device(s) can be in the form of supplemental enhancement information (SEI messages) or video usability information (VUI) parameter set fragments (not depicted). The parser (320) can parse / entropy-decode the received coded video sequence. The coding of the coded video sequence can be in accordance with a video coding technology or standard, and can follow various principles, including variable length coding, Huffman coding, arithmetic coding, with or without context sensitivity, etc. The parser (320) can extract from the coded video sequence, based upon at least one parameter corresponding to a group, a group parameter set for at least one of the subgroups of pixels in the video decoder. The subgroups can include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs) and so on. The parser (320) can also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0038] The parser (320) can perform entropy-decoding / parsing operations on the video sequence received from the buffer memory (315), thereby creating symbols (321).

[0039] The reconstruction of the symbols (321) can involve a number of different units, depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks, etc.), and other factors. Which units are involved, and how they are involved, can be controlled by subgroup control information parsed by the parser (320) from the coded video sequence. Such subgroup control information flows between the parser (320) and the following number of units is not described for the sake of brevity.

[0040] Beyond the functional blocks already mentioned, the video decoder (310) can be conceptually subdivided into a number of functional units as described below. In practical implementations operating under business constraints, many of these units interact closely with each other and can be at least partly integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the functional units below is appropriate.

[0041] A first unit is a scaler / inverse transform unit (351). The scaler / inverse transform unit (351) receives quantized transform coefficients as symbols (321) from the parser (320), along with control information including which transform to use, block size, quantization factors, quantization scaling matrices, etc. The scaler / inverse transform unit (351) can output blocks comprising sample values that can be input into the aggregator (355).

[0042] In some cases, the output samples of the scaler / inverse transform unit (351) can belong to an intra coded block; i.e., a block that is not using predictive information from previously reconstructed pictures, but can use predictive information from previously reconstructed parts of the current picture. Such predictive information can be provided by the intra picture prediction unit (352). In some cases, the intra picture prediction unit (352) generates a block of the same size and shape as the block under reconstruction using surrounding already reconstructed information taken from the current picture buffer (358). The current picture buffer (358) in some cases buffers partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (355) adds the predictive information generated by the intra picture prediction unit (352) on a per sample basis to the output sample information provided by the scaler / inverse transform unit (351).

[0043] In other cases, the output samples of the scaler / inverse transform unit (351) can belong to an inter coded and potentially motion compensated block. In such cases, the motion compensated prediction unit (353) can access the reference picture memory (357) to fetch samples used for prediction. After motion compensation of the fetched samples according to the motion vector belonging to the block (321), these samples can be added by the aggregator (355) to the output of the scaler / inverse transform unit (351), referred to as residual samples or residual signal in this case, resulting in the output sample information. The address within the reference picture memory (357) from which the motion compensated prediction unit (353) fetches the prediction samples can be controlled by a motion vector, which is made available to the motion compensated prediction unit (353) in the form of a symbol (321), which can have, for example, X, Y, and reference picture components. Motion compensation can also include interpolation of sample values fetched from the reference picture memory (357) when sub-sample precise motion vectors are used, motion vector prediction mechanisms, etc.

[0044] The output samples of the aggregator (355) can be subject to various loop filtering techniques in the loop filter unit (356). Video compression technologies can include in-loop filter technologies that are controlled by parameters included in the coded video sequence (also referred to as coded video bitstream) and made available to the loop filter unit (356) as symbols (321) from the parser (320), however, video compression technologies can also be responsive to meta-information obtained during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, as well as responsive to previously reconstructed and loop-filtered sample values.

[0045] The output of the loop filter unit (356) can be a stream of samples that can be output to the rendering device (312), as well as stored in the reference picture memory (357) for use in future inter picture predictions.

[0046] Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. For instance, once a coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified (by, for example, parser (320)) as a reference picture, the current picture buffer (358) can become part of the reference picture memory (357), and a fresh current picture buffer (358) can be reallocated before commencing the reconstruction of the following coded picture.

[0047] Video decoder (310) can perform decoding operations according to a predetermined video compression technology in a standard, such as ITU-T H.265. In the sense that the coded video sequence adheres to both the syntax of the video compression technology or standard and the profiles documented in the video compression technology or standard, the coded video sequence can conform to the syntax specified by the video compression technology or standard in effect. Specifically, a profile can select certain tools available in the video compression technology or standard as the only tools in use under the profile. Also required for conformance can be that the complexity of the coded video sequence is within bounds set by the level of the video compression technology or standard. In some cases, levels restrict the maximum picture size, maximum frame rate, maximum reconstruction sample rate (measured in, for example, megasamples per second), maximum reference picture size, and / or the like. Limits set by levels can be further restricted through Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0048] In an embodiment, receiver (331) can receive additional (redundant) data with the encoded video. The additional data can be included as a part of the coded video sequence(s). The additional data can be used by video decoder (310) to properly decode the data and / or to more accurately reconstruct the original video data. Additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

[0049] Figure 4 A block diagram of a video encoder (403) according to an embodiment of the disclosure is shown. The video encoder (403) is included in an electronic device (420). The electronic device (420) includes a transmitter (440) (e.g., transmission circuitry). The video encoder (403) can be used in place of Figure 2 The video encoder (203) in the example.

[0050] The video encoder (403) can receive video data from the video source (401) (and not from a network, as it is the case for the video decoder (310)) and can encode the video data to efficiently represent the video data for the purpose of, e.g., storage, transmission, and the like. Figure 4In one example, a portion of an electronic device (420) receives video samples, and a video source (401) can capture video images to be encoded by a video encoder (403). In another example, the video source (401) is a portion of the electronic device (420).

[0051] A video source (401) can provide a sequence of source video samples in the form of a digital video sample stream to be encoded by a video encoder (403). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit…), any color space (e.g., BT.601YCrCb, RGB…), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (401) can be a storage device storing previously prepared video. In a video conferencing system, the video source (401) can be a camera device capturing local image information as a video sequence. Video data can be provided as multiple individual pictures that are given motion when viewed in sequence. The pictures themselves can be organized as spatial pixel arrays, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0052] According to one embodiment, the video encoder (403) can encode and compress images of a source video sequence into an encoded video sequence (443) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (450). In some embodiments, the controller (450) controls and is functionally coupled to other functional units as described below. This coupling is not depicted for brevity. Parameters set by the controller (450) may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of images (GOP) layout, maximum motion vector search range, etc. The controller (450) can be configured to have other suitable functions belonging to the video encoder (403) optimized for certain system designs.

[0053] In some embodiments, the video encoder (403) is configured to operate in a coding loop. As a very simple description, in one example, the coding loop can include a source coder (430) (e.g., responsible for creating symbols such as a symbol stream based on input pictures to be coded, and reference picture(s)), and a (local) decoder (433) embedded in the video encoder (403). The (local) decoder (433) reconstructs the symbols to create the sample data in a similar manner as a (remote) decoder would create the sample data. Since any compression between symbols and coded video bitstream in the video compression technology considered in the disclosed subject matter is lossless, the content in the reference picture buffer (434) is also bit-exact between the local encoder and the remote encoder. In other words, the reference picture samples that the prediction part of an encoder "sees" are exactly the same as the sample values that a decoder would "see" when using the prediction in the decoding. This reference picture synchrony basic principle (and the drift that arises if the synchrony cannot be maintained, e.g., due to channel errors) also holds for some related art.

[0054] The operation of the (local) decoder (433) can be identical to that of the "remote" decoder of the video decoder (310) that has been described in detail above, e.g., with respect to FIG. 3. Figure 3 The operation of the (local) decoder (433) can be identical to that of the "remote" decoder of the video decoder (310) that has been described in detail above, e.g., with respect to FIG. 3. Figure 3 When symbols are available and the entropy coder (445) and the parser (320) can losslessly encode / decode the symbols into the coded video sequence, the entropy decoding portion of the video decoder (310) including the buffer memory (315) and the parser (320) can not be fully implemented in the local decoder (433).

[0055] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must necessarily be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on the decoder operation. Since the encoder technology is reciprocal to the decoder technology that is described at length, the description of the encoder technology can be simplified. Only in certain areas does a more detailed description become necessary and is provided below.

[0056] During operation, in some examples, the source coder (430) can perform motion compensated predictive coding, which codes an input picture predictively with reference to one or more previously-coded pictures from the video sequence that are designated as "reference pictures". In this manner, coding engine (432) codes the difference between the pixel blocks of an input picture and reference pixel blocks that can be selected as predictions for the input picture from reference picture memory (434).

[0057] The (local) decoder (433) can decode coded video data of a picture that can be designated as a reference picture, based on symbols created by the source coder (430). The operations of the coding engine (432) can advantageously be lossy processes. When the coded video data can be decoded at a video decoder (not shown), the reconstructed video sequence can generally be a replica of the source video sequence with some errors. Figure 4 The (local) decoder (433) replicates the decoding processes that can be performed by a video decoder on reference pictures, and can cause reconstructed reference pictures to be stored in reference picture memory (434). In this manner, the video coder (403) can store copies of reconstructed reference pictures locally that have common content as the reconstructed reference pictures that will be obtained by a far-end video decoder (absent transmission errors).

[0058] The predictor (435) can perform a prediction search for the coding engine (432). That is, for a new picture to be coded, the predictor (435) can search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, and so on, that can be used as an appropriate prediction reference for the new picture. The predictor (435) can operate on a sample block-by-pixel block basis to find appropriate prediction references. In some cases, as determined by search results obtained by the predictor (435), an input picture can have prediction references drawn from multiple reference pictures stored in the reference picture memory (434).

[0059] The controller (450) can manage coding operations of the source coder (430), including, for example, setting of parameters and subgroup parameters for encoding the video data.

[0060] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy coder (445). The entropy coder (445) converts the symbols generated by the various functional units into the coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, and so on, for example.

[0061] The transmitter (440) can buffer the coded video sequence(s) created by the entropy coder (445) to prepare it for transmission via a communication channel (460), which can be a hardware / software link to a storage device which will store the encoded video data. The transmitter (440) can combine the encoded video data from the video coder (403) with other data to be transmitted, for example, audio data and / or ancillary data streams (not shown).

[0062] The controller (450) can manage operation of the video encoder (403). During coding, the controller (450) can assign to each coded picture certain coded picture types that can affect the coding techniques that can be applied to corresponding pictures. For example, pictures typically can be assigned as one of the following picture types:

[0063] Intra Picture (I picture), which can be a picture that is coded and decoded without using any other picture in the sequence as a source of prediction. Some video codecs allow for different types of intra pictures, including, for example Independent Decoder Refresh (“IDR”) Pictures. Persons skilled in the art are aware of the variants of I pictures and their respective applications and features.

[0064] Predictive Picture (P picture), which can be a picture that can be coded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict sample values for each block.

[0065] Bi-directionally Interpolated Picture (B picture), which can be a picture that can be coded and decoded using intra prediction or inter prediction, using at most two motion vectors and reference indices to predict sample values for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0066] Source pictures typically can be spatially subdivided into blocks of samples (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples) and coded block-by-block. The blocks can be predictively coded with reference to other (already coded) blocks, determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I picture can be non-predictively coded, or blocks of an I picture can be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Blocks of a P picture can be predictively coded with reference to one previously coded reference picture, either via spatial prediction or via temporal prediction. Blocks of a B picture can be predictively coded with reference to one or two previously coded reference pictures, either via spatial prediction or via temporal prediction.

[0067] The video encoder (403) can perform encoding operations in accordance with a predetermined video coding technology or standard, such as ITU-T H.265. In its operations, the video encoder (403) can perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data can therefore conform to a syntax specified by the video coding technology or standard in use.

[0068] In an embodiment, the transmitter (440) can transmit additional data with the encoded video. The source coder (430) can include such data as part of the coded video sequence. Additional data can comprise temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and so on.

[0069] Captured video can be in terms of a plurality of source pictures (video pictures) in temporal sequence. Intra-picture prediction (often abbreviated as intra prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits (temporal or other) correlation across pictures. In an example, a particular picture, referred to as the current picture, in encoding / decoding is partitioned into blocks. A block in the current picture can be coded with a reference to a reference block in a reference picture that is previously coded and still buffered, when the block in the current picture is similar to the reference block in the reference picture. The reference to the reference block can be a vector, referred to as a motion vector. The motion vector can have a third dimension to identify the reference picture in cases where multiple reference pictures are used.

[0070] In some embodiments, bi-prediction techniques can be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture that are both preceding the current picture in decoding order (but can be respectively past and future in display order) in the video. A block in the current picture can be coded with a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block can be predicted by a combination of the first reference block and the second reference block.

[0071] In addition, merge mode techniques can be used for inter-picture prediction to improve coding efficiency.

[0072] According to some embodiments of the disclosure, predictions, such as inter-picture predictions and intra-picture predictions, are performed in the unit of blocks. For example, according to the HEVC standard, a picture in a video picture sequence is partitioned into coding tree units (CTU) for compression, the CTUs in a picture have the same size, such as 64 x 64 pixels, 32 x 32 pixels, or 16 x 16 pixels. In general, a CTU includes three coding tree blocks (CTB), namely one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree split into one or multiple coding units (CU). For example, a CTU of 64 x 64 pixels can be split into one CU of 64 x 64 pixels, or 4 CUs of 32 x 32 pixels, or 16 CUs of 16 x 16 pixels. In an example, each CU is analyzed to determine a prediction type for the CU, such as an inter prediction type or an intra prediction type. Depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PU). Generally, each PU includes a luma prediction block (PB), and two chroma PBs. In one embodiment, a prediction operation in encoding (encoding / decoding) is performed in the unit of a prediction block. Using a luma prediction block as an example of a prediction block, a prediction block includes a matrix of pixel values (e.g., luma values), such as 8 x 8 pixels, 16 x 16 pixels, 8 x 16 pixels, 16 x 8 pixels, and / or the like.

[0073] Figure 5 A diagram illustrates a video encoder (503) according to another embodiment of the disclosure. The video encoder (503) is configured to receive a processing block (e.g., a prediction block) of sample values within a current video picture in a video picture sequence, and encode the processing block into a coded picture that is part of a coded video sequence. In an example, the video encoder (503) is used in place of the video encoder (203) in the example video encoding system (200). Figure 2 The video encoder (203) in the example.

[0074] In the HEVC example, the video encoder (503) receives a matrix of sample values for a processing block, e.g., a prediction block of 8x8 samples, etc. The video encoder (503) determines whether the processing block is best coded using intra, inter, or bi-prediction mode using, e.g., rate-distortion optimization. When the processing block is to be coded in intra mode, the video encoder (503) can use an intra prediction technique to encode the processing block into the coded picture; and when the processing block is to be coded in inter or bi-prediction mode, the video encoder (503) can use an inter or bi-prediction technique, respectively, to encode the processing block into the coded picture. In certain video coding techniques, the merge mode can be an inter picture prediction submode in which a motion vector is derived from one or more motion vector predictors without the benefit of a coded motion vector component outside the processing block. In certain other video coding techniques, there can be a motion vector component applicable to the subject block. In an example, the video encoder (503) includes other components, e.g., a mode decision module (not shown) that determines the mode of the processing block.

[0075] In Figure 5 In an example, the video encoder (503) includes an inter encoder (530), an intra encoder (522), a residue calculator (523), a switch (526), a residue encoder (524), a general controller (521), and an entropy encoder (525) coupled together as shown. Figure 5

[0076] The inter encoder (530) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in reference pictures (e.g., blocks in previous and future pictures), generate inter prediction information (e.g., redundancy information description according to inter coding techniques, motion vectors, merge mode information), and compute an inter prediction result (e.g., a predicted block) based on the inter prediction information using any suitable technique.

[0077] The intra encoder (522) is configured to receive samples of a current block (e.g., a processing block), compare the block to already coded blocks in the same picture in some cases, generate quantized coefficients after a transform, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques).

[0078] ​A general-purpose controller (521) is configured to determine general-purpose control data and control other components of the video encoder (503) based on the general-purpose control data. In the example, the general-purpose controller (521) determines the mode of the block and provides control signals to the switch (526) based on the mode. For example, when the mode is intra-frame mode, the general-purpose controller (521) controls the switch (526) to select the intra-frame mode result for use by the residual calculator (523) and controls the entropy encoder (525) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; and when the mode is inter-frame mode, the general-purpose controller (521) controls the switch (526) to select the inter-frame prediction result for use by the residual calculator (523) and controls the entropy encoder (525) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

[0079] A residual calculator (523) is configured to calculate the difference (residual data) between the received block and the prediction selected from the intra-frame encoder (522) or inter-frame encoder (530). A residual encoder (524) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In this example, the residual encoder (524) is configured to transform the residual data in the frequency domain and generate transform coefficients. The transform coefficients are then subjected to quantization to obtain quantized transform coefficients.

[0080] An entropy encoder (525) is configured to format the bitstream to include encoded blocks. The entropy encoder (525) is configured to include various information according to a suitable standard such as the HEVC standard. In this example, the entropy encoder (525) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. It should be noted that, according to the disclosed subject matter, residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0081] Figure 6 A diagram of a video decoder (610) according to another embodiment of the present disclosure is shown. The video decoder (610) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In the example, the video decoder (610) is used instead of Figure 2 The video decoder (210) in the example.

[0082] exist Figure 6 In the example, the video decoder (610) includes, for example, Figure 6 The entropy decoder (671), inter-frame decoder (680), residual decoder (673), reconstruction module (674), and intra-frame decoder (672) are shown coupled together.

[0083] The entropy decoder (671) can be configured to reconstruct, from the coded picture, certain symbols representing the syntax elements that constitute the coded picture, from the coded picture. Such symbols can include, for example, a mode used to code a block (e.g., an intra mode, an inter mode, a bi-predictive mode, the latter two merge submodes or another submode), prediction information that can identify certain samples or metadata that the intra decoder (672) or the inter decoder (680) can use, respectively, for prediction (e.g., intra prediction information or inter prediction information), residual information in the form of, for example, quantized transform coefficients, etc. In an example, when the prediction mode is an inter or bi-predictive mode, the inter prediction information is provided to the inter decoder (680); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (672). The residual information can be subject to inverse quantization and is provided to the residue decoder (673).

[0084] The inter decoder (680) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0085] The intra decoder (672) is configured to receive the intra prediction information and generate a prediction result based on the intra prediction information.

[0086] The residue decoder (673) is configured to perform inverse quantization to extract de-quantized transform coefficients and process the de-quantized transform coefficients to convert a residual from the frequency domain into the spatial domain. The residue decoder (673) can also require certain control information, including the quantizer parameter (QP), and this information can be provided by the entropy decoder (671) (data path not depicted as this is only low amount control information).

[0087] The reconstruction module (674) is configured to combine, in the spatial domain, the residual output by the residue decoder (673) and the prediction result (which can be the case output by the inter or intra prediction module) to form a reconstructed block, which can be part of a reconstructed picture, which in turn can be part of a reconstructed video. It is noted that other suitable operations, such as a deblocking operation and the like, can be performed to improve visual quality.

[0088] It should be noted that the video encoders (203), (403), and (503), and the video decoders (210), (310), and (610) can be implemented using any suitable technique. In one embodiment, the video encoders (203), (403), and (503), and the video decoders (210), (310), and (610) can be implemented using one or more integrated circuits. In another embodiments, the video encoders (203), (403), and (503), and the video decoders (210), (310), and (610) can be implemented using one or more processors that execute software instructions.

[0089] Block-based compensation from different pictures can be referred to as motion compensation. Block-based compensation can also be performed according to previously reconstructed areas within the same picture, which can be referred to as intra-picture block-based compensation or intra-block copy. For example, a displacement vector indicating the offset between a current block and a reference block is referred to as a block vector. According to some embodiments, a block vector points to a reference block that has been reconstructed and is available for reference. Also, for parallel processing considerations, reference areas that extend beyond the tile / slice boundaries or wavefront trapezoidal boundaries can also be excluded from the reference of a block vector. Due to these constraints, a block vector can be different from a motion vector (MV) in motion compensation, where a motion vector can be an arbitrary value (positive or negative in x or y direction).

[0090] Figure 7 Embodiments of intra-picture block-based compensation (e.g., intra-block copy mode) are shown. In Figure 7 In the example, a current picture 700 includes a set of blocks that have been encoded / decoded (i.e., the gray blocks) and a set of blocks that have not been encoded / decoded (i.e., the white blocks). A sub-block 702 of one of the blocks that have not been encoded / decoded can be associated with a block vector 704 that points to another sub-block 706 that has been previously encoded / decoded. Thus, any motion information associated with the sub-block 706 can be used for the encoding / decoding of the sub-block 702.

[0091] According to some embodiments, the encoding of a block vector is explicit. In other embodiments, the encoding of a block vector is implicit. In explicit mode, the difference between a block vector and its predictor is signaled, while in implicit mode, a block vector is recovered from its predictor in a similar way as the motion vector prediction in merge mode. In some embodiments, the resolution of a block vector is limited to integer positions. In other embodiments, a block vector points to fractional positions.

[0092] According to some embodiments, a reference index is used to signal the use of block-level intra-picture block compensation (i.e., intra block copy mode), where the currently decoded picture is treated as a reference picture, which is placed in the last position of the reference picture list. The reference picture can also be managed in the decoded picture buffer (DPB) together with other temporal reference pictures.

[0093] According to some embodiments, the reference block is flipped horizontally or vertically (e.g., flipped intra block copy) before being used to predict the current block. In some embodiments, each compensation unit inside the MxN coding block is either Mxl or l xN row (e.g., row-based intra block copy).

[0094] According to some embodiments, block-level motion compensation is performed, where the current block is a processing unit for which motion compensation is performed using the same motion information. In this regard, given the size of the block, all pixels in the block will use the same motion information to form its prediction block. Examples of block-level motion compensation include the use of spatial merge candidates, temporal candidates, and combinations of motion vectors from existing merge candidates in bi-prediction.

[0095] Referring to Figure 8 , the current block (801) includes samples that have been discovered by the encoder / decoder during a motion search process, which can be predicted from a previously block of the same size that is spatially offset. In some embodiments, instead of directly coding the MV, the MV associated with any of the five surrounding samples labeled A0, Al, and B0, Bl, B2 (corresponding to 802 to 806, respectively) can be directly derived from metadata associated with one or more reference pictures, such as the most recent (in decoding order) reference picture. Blocks A0, Al, B0, Bl, and B2 can be referred to as spatial merge candidates.

[0096] According to some embodiments, pixels at different locations within a motion compensated block (e.g., sub-blocks) can have different motion information. These differences from the block-level motion information can be derived instead of being signaled. This type of motion compensation can be referred to as sub-block level motion compensation, which allows the motion compensation of a block to be less than the block itself. In this regard, each block can have multiple sub-blocks, each of which can contain different motion information.

[0097] Examples of sub-block level motion compensation include sub-block based temporal motion vector prediction, where sub-blocks of the current block have different motion vectors. Another example of sub-block level motion compensation is ATMVP, which is a method that allows each coding block to extract multiple sets of motion information from multiple blocks of a collocated reference picture, which are smaller than the current coding block.

[0098] Another example of sub-block level motion compensation includes spatial / temporal blending with sub-block adjustment, in which for each sub-block in the current block, the motion vector of the sub-block is adjusted according to the motion vectors of the spatial / temporal neighbors of the sub-block. In this mode, for some sub-blocks, motion information from the corresponding sub-block in the temporal reference picture can be needed.

[0099] Another example of sub-block level motion compensation is affine coded motion compensation block, in which the motion vectors at the four corners of the current block are first derived according to the motion vectors of the neighboring blocks. Subsequently, an affine model is used to derive the remaining motion vectors (e.g., sub-block or pixel level) of the current block, such that each sub-block can have different motion vectors compared to its neighbors.

[0100] Another example of sub-block level motion compensation is merge candidate refinement using decoder-side motion vector derivation. In this mode, after obtaining the motion vector predictor(s) of the current block or sub-blocks of the current block, a method such as template matching or bilateral matching can be used to further refine the given motion vector predictor(s). The refined motion vector predictor can be used to perform motion compensation. The same refinement operation can be performed at both the encoder side and the decoder side, such that the decoder does not need additional information about how the refinement is shifted from the original predictor. Furthermore, the skip mode can be considered as a special merge mode, in which, in addition to deriving the motion information of the current block from its neighbors, the prediction residual of the current block is also zero.

[0101] According to some embodiments, in sub-block temporal motion vector prediction, a sub-block of a current block can have different motion vector predictors derived from a temporal reference picture. For example, a set of motion information is identified, which includes a motion vector and an associated reference index of the current block. The motion information can be determined according to a first available spatial merge candidate. Using the motion information, a reference block in a reference picture for the current block is determined. The reference block is also divided into sub-blocks. In some embodiments, for each current block sub-block (CBSB) in the current picture, there is a corresponding reference block sub-block (RBSB) in the reference picture.

[0102] In some embodiments, for each CBSB, if the corresponding RBSB is coded in inter mode using a set of motion information, the motion information is converted (e.g., using a method such as motion vector scaling in temporal motion vector prediction) and used as the motion vector predictor for the CBSB. Methods for handling RBSBs coded in intra mode (e.g., intra block copy mode) are described in further detail below.

[0103] According to some embodiments, when the sub-block based temporal motion vector prediction mode is used, each CBSB is not allowed to be coded in an intra mode such as the intra block copy mode. This can be achieved by considering the intra block copy coded RBSB as an intra mode in the sub-block based temporal motion vector prediction. In particular, regardless of how the intra block copy mode is considered (e.g., as an inter mode, an intra mode, or a third mode), for a CBSB, when its corresponding RBSB is coded in the intra block copy mode, the RBSB is considered as an intra mode in the sub-block based temporal motion vector prediction. Thus, in some embodiments, the RBSB coded in the intra block copy mode is processed according to the default setting in the sub-block based temporal motion vector prediction. For example, when the corresponding RBSB of a CBSB is coded in the intra block copy mode, a default motion vector such as a zero motion vector can be used as the motion vector predictor for the CBSB. In this example, the reference picture for the CBSB will no longer be the current picture, but a temporal reference picture. For example, the temporal reference picture can be a picture shared by all sub-blocks of the current block, the first reference picture in the reference picture list, a co-located picture for TMVP purpose, etc. In another example, for a CBSB, when its corresponding RBSB is coded in an inter mode but the reference picture is the current picture, a default motion vector such as a zero motion vector is assigned to the CBSB. In this regard, even though the RBSB is coded in an inter mode, since the current picture is the same as the reference picture, the RBSB is processed as if it is coded in an intra mode.

[0104] Figure 9 An example of performing sub-block based temporal motion vector prediction is shown. Figure 9 A current picture 900 is shown, which has nine blocks including a current block 900A. The current block 900A is divided into four sub-blocks 1-4. The current picture 900 can be associated with a reference picture 902, which includes nine previously coded / decoded blocks. In addition, as shown, the current block 900A has a motion vector 904 pointing to a reference block 902A. The motion vector 904 can be determined by using motion vectors of one or more neighboring blocks of the current block 900A (e.g., spatial merge candidates). The reference block 902A is divided into four sub-blocks 1-4. The sub-blocks 1-4 of the reference block 902A correspond to the sub-blocks 1-4 of the current block 900A, respectively. If the reference picture 902 is the same as the current picture 900, each RBSB in the block 902A is considered as if these blocks are coded in an intra mode. In this regard, for example, if the sub-block 1 of the block 902A is coded in an inter mode, but the reference picture 902 is the same as the current picture 900, the sub-block 1 of the block 902A is considered as if it is coded in an intra mode, where a default motion vector is assigned to the sub-block 1 of the block 900A. Figure 9 ​

[0105] If the reference picture 902 is different from the current picture 900, sub-blocks 1-4 of the reference block 902A can be used to perform sub-block based temporal motion vector prediction for sub-blocks 1-4 of the block 900A, respectively. For example, based on whether sub-block 1 of the reference block 902A is coded in inter mode or intra mode (e.g., intra block copy mode), a motion vector predictor is determined for sub-block 1 of the current block 900A. If sub-block 1 of the reference block 902A is coded in inter mode, the motion vector of sub-block 1 of the reference block 902A is used to determine the motion vector of sub-block 1 of the current block 900A. However, if sub-block 1 of the reference block 902A is coded in intra mode, the motion vector of sub-block 1 of the current block 900A is set to a zero motion vector.

[0106] Figure 10 An embodiment of a process that can be performed by an encoder or decoder, such as the intra encoder 522 or the intra decoder 672, is shown. The process can begin at step S1000, where a current picture is obtained from a coded video bitstream. For example, referring to FIG. 6, the intra decoder 672 can obtain the current picture 900 from the coded video bitstream 650. Figure 9 The current picture 900 can be obtained from the coded video bitstream. The process proceeds to step S1002, where, for a current block in the current picture, a reference block from a reference picture is identified. For example, referring to FIG. 6, the intra decoder 672 can identify the reference block 902A from the reference picture 902. Figure 9 The reference picture 902 can be retrieved from a reference picture list associated with the current block 900A. The motion vector 904 can be used to identify the reference block 902A of the reference picture 902 when performing sub-block temporal motion vector prediction for the current block 900A.

[0107] The process proceeds to step S1004, where it is determined whether the reference picture is the same as the current picture. If the reference picture is different from the current picture, the process proceeds to step S1006, where an encoding mode of an RBSB corresponding to a CBSB is determined. For example, referring to FIG. 6, the intra decoder 672 can determine the encoding mode of the RBSB 904A corresponding to the CBSB 900A. Figure 9determining a coding mode of an RBSB 1 of the reference block 902A, the RBSB 1 corresponding to a CBSB 1 of the current block 900A. Processing proceeds to step S1008, where it is determined whether the coding mode of the RBSB is an inter mode. If the coding mode of the RBSB is an inter mode, processing proceeds to step S1010, where a motion vector predictor of the CBSB is determined based on a motion vector predictor of the RBSB. For example, if the coding mode of the RBSB 1 of the reference block 902A is an inter mode, a motion vector predictor of the CBSB 1 of the current block 900A is determined based on a motion vector predictor of the RBSB 1 of the reference block 902A. For example, the motion vector predictor of the RBSB 1 of the reference block 902A is converted (e.g., using a method such as motion vector scaling in temporal motion vector prediction) and used as the motion vector predictor of the CBSB 1 of the current block 900A.

[0108] Returning to step S1008, if the coding mode of the RBSB is not an inter mode (e.g., the coding mode of the RBSB is an intra mode), processing proceeds to step S1012, where a motion vector predictor of the CBSB is set to a default motion vector. For example, if the RBSB 1 of the reference block 902A is coded in an intra mode, a motion vector predictor of the CBSB 1 of the current block 900A is set to a default motion vector, such as a zero motion vector.

[0109] Returning to step S1004, if the reference picture is the same as the current picture, processing proceeds to step S1012, where a motion vector predictor of the CBSB is set to a default motion vector. In this regard, when the reference picture is the same as the current picture, the coding mode of the RBSB is determined to be an intra mode, where a motion vector predictor for the corresponding CBSB is set to a default motion vector. In this regard, even if the RBSB is an inter mode, the RBSB can be treated as coded in an intra mode by setting the motion vector predictor of the CBSB to a default motion vector. Steps S1004 through S1012 can be repeated for each sub-block in the current block 900A.

[0110] The techniques described above, can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 11 A computer system (1100) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0111] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or the like, and executed on a personal computer, smart phone, gaming device, or the like.

[0112] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, internet of things devices, and the like.

[0113] Figure 11 The components shown in the computer system (1100) are exemplary and not intended to suggest any limitation as to the scope of use or functionality of the employing computer software. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of a computer system (1100).

[0114] The computer system (1100) can include certain human interface input devices. Such a human interface input device can be responsive to user input data being generated anywhere within a particular vicinity of the computer system (1100). The user input

[0115] Input human interface devices can include one or more of (just one of each described): keyboard (1101), mouse (1102), touchpad (1103), touchscreen (1110), data g love (not shown), joystick (1105), microphone (1106), scanner (1107), camera (1108).

[0116] ​Computer system (1100) can also include certain human interface output devices. Such human interface output devices can be stimulating one or more of the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices can include tactile output devices (for example, speakers (1109), haptic feedback devices, etc.), audio output devices, visual output devices (for example, screens (1110), one or more retinas of an augmented reality eye-piece, etc.), and printers (not depicted).

[0117] Computer system (1100) can also include human accessible storage devices and their associated media such as optical media including CD / DVD ROM / RW (1120) with CD / DVD

[0118] Those skilled in the art will further appreciate that the term "computer-readable medium" used in connection with the currently disclosed subject matter does not encompass transmission media, carrier waves, or other transient media.

[0119] The computer system (1100) can also include a network interface (1154) to one or more communications networks (1155). The networks (1155) can be wired, wireless, optical, and so forth. The networks (1155) can be local, wide-area, metropolitan, and so forth. The networks (1155) can be terrestrial, space-based, etc. Examples of networks (1155) include local area networks such as Ethernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LTE and the like, TV wireline or wireless wide area digital networks to include broadband-over-cable, broadband-over- fiber, broadband-over-DSL, wireless networks to include WiMAX and 5G, satellite

[0120] The aforementioned human interface devices, human-accessible storage devices, and network interfaces (1154) can be attached to the core (1140) of the computer system (1100).

[0121] The core (1140) can include one or more Central Processing Units (CPU) (1141), Graphics Processing Units (GPU) (1142), specialized programmable processing units in the form of Field Programmable Gate Arrays (FPGA) (1143), hardware accelerators for certain tasks (1144), and so forth. These devices, along with Read-only memory (ROM) (1145), Random-access memory (1146), internal mass storage such as internal non-user accessible hard drives, SSDs, and the like (1147), can be connected together by a system bus (1148), which can be configured for passing

[0122] CPUs (1141), GPUs (1142), FPGAs (1143), and accelerators (1144) can execute certain instructions that can be combined to constitute the above-mentioned computer code. That computer code can be stored in ROM (1145) or RAM (1146). Transitional data can be also stored in RAM (1146), whereas permanent data can be stored for example, in the internal mass storage (1147). Fast storage and retrieval can be enabled through the use of cache memory, which can be closely associated with one or many CPU (1141), GPU (1142), mass storage (1147), ROM (1145), RAM (1146), etc.

[0123] The computer software can be implemented as computer code that is executable on a computer system. The computer software can be stored on a computer readable medium that is transitory or non-transitory.

[0124] By way of example, and not limitation, one embodiment of the computer system having architecture (1100), specifically the core (1140) can provide functionality as a result of processor(s) (including CPUs, GPUs, FPGA, accelerators, and the like) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as introduced above, as well as certain storage of the core (1140) that are of a non-transitory nature, such as core-internal mass storage (1147) or ROM (1145). The software implemented to

[0125] Appendix A: Acronyms

[0126] Motion Vector (MV)

[0127] High Efficiency Video Coding (HEVC)

[0128] Supplementary Enhancement Information (SEI)

[0129] Video Usability Information (VUI)

[0130] Group of Pictures (GOP)

[0131] Transform Unit (TU)

[0132] prediction unit (PU)

[0133] coding tree unit (CTU)

[0134] coding tree block (CTB)

[0135] prediction block (PB)

[0136] hypothetical reference decoder (HRD)

[0137] signal-to-noise ratio (SNR)

[0138] central processing unit (CPU)

[0139] graphics processing unit (GPU)

[0140] cathode ray tube (CRT)

[0141] liquid-crystal display (LCD)

[0142] organic light-emitting diode (OLED)

[0143] compact disc (CD)

[0144] digital video disc (DVD)

[0145] read-only memory (ROM)

[0146] random access memory (RAM)

[0147] application-specific integrated circuit (ASIC)

[0148] programmable logic device (PLD)

[0149] Local Area Network (LAN)

[0150] Global System for Mobile communications (GSM)

[0151] Long-Term Evolution (LTE)

[0152] Controller Area Network Bus (CANBus)

[0153] Universal Serial Bus (USB)

[0154] Peripheral Component Interconnect (PCI)

[0155] Field Programmable Gate Areas (FPGA)

[0156] solid-state drive (SSD)

[0157] Integrated Circuit (IC)

[0158] Coding Unit (CU)

[0159] While several example embodiments have been described, changes, substitutions, and various alternatives and equivalents can exist that fall within the scope of the disclosure. It will be appreciated that many of those changes, substitutions, and various alternatives and equivalents can be resorted to, by those skilled in the art, without departing from the spirit and scope of the disclosure.

[0160] The present disclosure provides a method for video decoding of a decoder, comprising: obtaining a current picture from a coded video bitstream; identifying, for a current block included in the current picture, a reference block included in a reference picture different from the current picture, the current block being partitioned into a plurality of sub-blocks (CBSBs), the reference block having a plurality of sub-blocks (RBSBs), the plurality of RBSBs respectively corresponding to different CBSBs in the plurality of CBSBs; determining whether a reference picture of the RBSB is the current picture; in response to determining that the reference picture of the RBSB is the current picture, determining a coding mode of the RBSB as an intra mode; and in response to determining that the reference picture of the RBSB is not the current picture, (i) determining, for one of the CBSBs, whether a coding mode of the RBSB is one of an intra mode and an inter mode, and (ii) determining a motion vector predictor of the one of the CBSBs based on whether the coding mode of the corresponding RBSB is the one of the intra mode and the inter mode.

[0161] In a possible implementation, if it is determined that the coding mode of the RBSB is the intra mode, the method further comprises, in response to determining that the coding mode of the corresponding RBSB is the intra mode, setting the motion vector predictor of the one of the CBSBs to a default motion vector, the default motion vector causing a reference picture to change to a temporally reference picture.

[0162] In a possible implementation, the default motion vector is one of (i) a zero motion vector, and (ii) an offset between the CBSB and the RBSB.

[0163] In a possible implementation, in response to determining that the coding mode of the corresponding RBSB is the inter mode, the motion vector predictor of the one of the CBSBs is based on a motion vector predictor associated with the corresponding RBSB.

[0164] In a possible implementation, the motion vector predictor of the one of the CBSBs is a scaled version of the motion vector predictor associated with the corresponding RBSB.

[0165] In a possible implementation, the method further comprises, in response to determining that the reference picture of the RBSB is the current picture and the coding mode of the RBSB is the inter mode, setting the motion vector predictor of the one of the CBSBs to a default motion vector to determine the coding mode of the RBSB as the intra mode, the default motion vector causing a reference picture to change to a temporally reference picture.

[0166] In a possible implementation, the method further includes decoding each CBSB by performing sub-block based temporal motion vector prediction on each CBSB based on a motion vector predictor of each CBSB.

[0167] In a possible implementation, the reference block is identified according to a motion vector predictor associated with a block neighboring the current block.

[0168] In a possible implementation, the reference picture is a first reference picture from a sequence of reference pictures, the sequence of reference pictures being associated with the current picture.

[0169] An exemplary embodiment of the present disclosure includes an apparatus for video decoding. The apparatus includes an obtaining unit, an identifying unit, a first determining unit and a second determining unit. The obtaining unit is configured to obtain a current picture from a coded video bitstream; the identifying unit is configured to identify, for a current block included in the current picture, a reference block included in a reference picture different from the current picture, the current block being divided into a plurality of sub-blocks (CBSBs), the reference block having a plurality of sub-blocks (RBSBs) corresponding to different CBSBs in the plurality of CBSBs respectively; the first determining unit is configured to determine whether the reference picture of the RBSB is the current picture; the second determining unit is configured to determine, in response to determining that the reference picture of the RBSB is the current picture, a coding mode of the RBSB as an intra mode; and in response to determining that the reference picture of the RBSB is not the current picture, (i) determine, for one of the CBSBs, whether a coding mode of the RBSB is one of an intra mode and an inter mode, and (ii) determine a motion vector predictor of the one of the CBSBs based on whether the coding mode of the corresponding RBSB is one of the intra mode and the inter mode.

[0170] The present disclosure provides a non-transitory computer readable medium storing instructions that, when executed by a processor in a video decoder, cause the processor to perform any of the aforementioned methods for video decoding by a decoder.

Claims

1. A method of video coding, the method comprising: The method comprises: obtaining a current picture from a video sequence to be encoded; identifying, for a current block included in the current picture, a reference block included in a reference picture different from the current picture, the current block being divided into a plurality of sub-blocks, the reference block having a plurality of sub-reference blocks respectively corresponding to the plurality of sub-blocks; for each sub-block, setting a motion vector predictor of the sub-block to a default motion vector in response to the corresponding sub-reference block being encoded in an intra mode or an intra block copy mode, the default motion vector being a zero motion vector; for each sub-block, determining the motion vector predictor of the sub-block to be equal to a motion vector predictor of the corresponding sub-reference block in response to the corresponding sub-reference block being encoded in an inter mode; performing a sub-block based temporal motion vector prediction on each sub-block based on the motion vector predictor of each sub-block, and encoding each sub-block to obtain an encoded video bitstream.

2. The method of claim 1, wherein, The motion vector predictor of the sub-block is a scaled version of the motion vector predictor associated with the corresponding sub-reference block.

3. The method according to any of claims 1-2, characterized in that, The identifying the reference block included in the reference picture different from the current picture comprises: identifying the reference block according to a motion vector predictor associated with a block neighboring the current block.

4. The method according to any one of claims 1-2, characterized in that, The reference picture is a first reference picture from a reference picture sequence associated with the current picture.

5. An apparatus for video coding, the apparatus comprising: The apparatus comprises an obtaining unit, an identifying unit, a first determining unit and a second determining unit: The obtaining unit is configured to obtain a current picture from a video sequence to be encoded; The identifying unit is configured to identify, for a current block included in the current picture, a reference block included in a reference picture different from the current picture, the current block being divided into a plurality of sub-blocks, the reference block having a plurality of sub-reference blocks respectively corresponding to the plurality of sub-blocks; The first determining unit is configured to, for each sub-block, set a motion vector predictor of the sub-block to a default motion vector in response to the corresponding sub-reference block being encoded in an intra mode or an intra block copy mode, the default motion vector being a zero motion vector; The second determining unit is configured to, for each sub-block, determine the motion vector predictor of the sub-block to be equal to a motion vector predictor of the corresponding sub-reference block in response to the corresponding sub-reference block being encoded in an inter mode; The apparatus is further configured to perform a sub-block based temporal motion vector prediction on each sub-block based on the motion vector predictor of each sub-block, and encode each sub-block to obtain an encoded video bitstream.

6. The apparatus of claim 5, wherein, The motion vector predictor of the sub-block is a scaled version of the motion vector predictor associated with the corresponding sub-reference block.

7. The device of any of claims 5-6, wherein, The identifying unit is configured to: identify the reference block according to a motion vector predictor associated with a block neighboring the current block.

8. The device of any of claims 5-6, wherein, The reference picture is a first reference picture from a reference picture sequence associated with the current picture.

9. A non-transitory computer-readable medium storing instructions, the method comprising: The instructions, when executed by a processor in a video encoder, cause the processor to perform the method of any one of claims 1-4.

10. A computer device, comprising: The device comprises a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1-4 according to instructions in the program code.

11. A method of storing a bitstream, characterized by, The method according to any one of claims 1-4 generates a bitstream; and stores the bitstream.

12. A method of transmitting a bitstream, the method comprising: The method according to any one of claims 1-4 generates a bitstream; and transmits the bitstream.

13. A computer readable storage medium having stored thereon computer programs / instructions and a bitstream, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1-4 to generate the bitstream.

Images