Method for sub-block based temporal motion vector prediction and apparatus thereof, and computer program

Abstract

To provide a method for sub-block based temporal motion vector prediction.SOLUTION: A video decoding method includes the steps of: obtaining a current picture; and identifying a reference block included in a reference picture different from the current picture with respect to a current block included in the current picture, in which the current block is divided into a plurality of sub blocks (CBSBs), and the reference block has the plurality of sub blocks (RBSBs). The method also includes steps of: determining whether the reference picture for the RBSB is the current picture; and, in response to determining that the reference picture for the RBSB is the current picture, determining a coding mode of the RBSB to be an intra-frame mode. The method also includes a step of, in response to determining that the reference picture of the RBSB is not the current picture, determining a motion vector prediction value for the one of the CBSBs based on whether or not the coding mode of the corresponding RBSB is one of the intra-frame modes, and an inter-frame mode.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] This disclosure claims priority to U.S. Provisional Application No. 62 / 680,468, entitled "METHODS FOR SUB-BLOCK BASED TEMPORAL MOTION VECTOR PREDICTION," filed June 4, 2018, the entire contents of which are incorporated herein by reference.

[0002] This disclosure describes embodiments generally relating to video encoding. [Background technology]

[0003] The background description provided in this specification is intended to provide a general understanding of the background of the present disclosure. In light of the extent of the work described in the background section, the work of the currently signed inventors and aspects not otherwise limited as prior art at the time of submission are not expressly or implicitly admitted as prior art to the present disclosure.

[0004] Since the last few decades, it has been known to perform video encoding and decoding by inter-picture prediction with motion compensation. Uncompressed digital video comprises a sequence of pictures, each with spatial dimensions, e.g., 1920x1080 luminance samples and associated chrominance samples. The sequence of pictures may have a fixed or variable picture rate (also informally called frame rate), e.g., 60 pictures per second or 60 Hz. Uncompressed video has high bitrate requirements. For example, 1080p60 4:2:0 video (60 A video of 1920x1080 luminance samples at a 1080Hz frame rate requires a bandwidth of approximately 1.5 Gbit / s. One hour of such video would require over 600 GB of storage space.

[0005] One goal of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression contributes to reducing the bandwidth or storage space requirements mentioned above, in some cases by more than two orders of magnitude. Lossless compression, lossy compression, and combinations of both are available. Lossless compression refers to techniques that reconstruct an exact copy of the original signal from a compressed version. When lossy compression is used, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signal is small enough that the reconstructed signal is useful for its intended application. In the case of video, lossy compression is widely used. The amount of acceptable distortion depends on the application; for example, users of consumer streaming applications tolerate higher distortion than users of television streaming applications. Achievable compression ratios reflect the fact that the higher the acceptable / tolerable distortion, the higher the compression ratio.

[0006] Motion compensation may be a lossy compression technique, in which blocks of sample data from a previously constructed picture or part thereof (reference picture) are used to predict a newly reconstructed picture or part thereof after being spatially shifted in a direction indicated by a motion vector (hereinafter referred to as MV). In some situations, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or it may have three dimensions, the third of which is an indication of the reference picture being used (the latter may indirectly be a temporal dimension).

[0007] In some video compression techniques, the motion vector (MV) to be applied to a region of sample data can be predicted from other motion vectors, such as those for other regions of sample data spatially adjacent to the region being reconstructed, in decoding order. In this way, the amount of data required to encode the MV is significantly reduced, redundancy is eliminated, and compression is increased. MV prediction works effectively because, for example, when encoding a video input signal derived from a camera (called natural video), there is a statistical possibility that regions larger than the region to which a single MV is applicable will move in a similar direction and therefore, in some situations, can be predicted by similar motion vectors derived from the MVs of neighboring regions. This allows the MV found for a particular region to be similar or identical to the MV predicted from the surrounding MVs and, after entropy coding, can be represented with fewer bits than would be used to encode the MV directly. In some situations, MV prediction may be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other situations, the MV prediction itself may be lossy, for example due to rounding errors when calculating the prediction from several surrounding MVs.

[0008] H.265 / HEVC (ITU-T Recommendation H.265, “High Efficiency Video Encoding / Decoding (High Various MV prediction mechanisms are described in "H.265: Efficiency Video Coding," December 2016. Among the various MV prediction mechanisms provided by H.265, the one described in this application is a technique hereinafter referred to as "spatial merging." Summary of the Invention [Problem to be solved by the invention]

[0009] Some forms of inter-frame prediction are performed at the sub-block level, but sub-block-based temporal motion vector prediction modes, such as alternate temporal motion vector prediction (ATMVP) and spatio-temporal motion vector prediction (STMVP), require the corresponding sub-blocks to be coded in an inter-frame mode, but these temporal motion vector prediction modes cannot handle sub-blocks coded in an intra-frame mode, such as the intra-frame block copy mode. [Means for solving the problem]

[0010] An illustrative embodiment of the present disclosure includes a method for video decoding in a decoder, the method including: obtaining a current picture from an encoded video bitstream; identifying, for a current block included in the current picture, a reference block included in a reference picture different from the current picture, the current block being divided into a plurality of sub-blocks (CBSBs), the reference block having a plurality of sub-blocks (RBSBs), each sub-block corresponding to a different CBSB among the plurality of CBSBs; determining whether a reference picture of the RBSBs is the current picture; and determining a coding mode of the RBSBs as an intra-frame mode in response to determining that the reference picture of the RBSBs is the current picture. The method further includes, in response to determining that the reference picture of the RBSB is not the current picture, (i) for one of the CBSBs, determining whether the coding mode of the corresponding RBSB is an intra-frame mode or an inter-frame mode, and (ii) determining a motion vector prediction value for the one of the CBSBs based on whether the coding mode of the corresponding RBSB is an intra-frame mode or an inter-frame mode.

[0011] An illustrative embodiment of the present disclosure includes a video decoder for video decoding, the video decoder having a processing circuit configured to obtain a current picture from an encoded video bitstream. The processing circuit is further configured to identify, for a current block included in the current picture, a reference block included in a reference picture different from the current picture, where the current block is divided into a plurality of sub-blocks (CBSBs), and the reference block has a plurality of sub-blocks (RBSBs), each sub-block corresponding to a different CBSB among the plurality of CBSBs. The processing circuit is further configured to determine whether the reference picture of the RBSBs is the current picture, and, in response to determining that the reference picture of the RBSBs is the current picture, determine a coding mode of the RBSBs as an intra-frame mode. The processing circuit further, in response to determining that the reference picture of the RBSB is not the current picture, (i) determines, for one of the CBSBs, whether the encoding mode of the corresponding RBSB is an intra-frame mode or an inter-frame mode, and (ii) determines a motion vector prediction value for the one of the CBSBs based on whether the encoding mode of the corresponding RBSB is an intra-frame mode or an inter-frame mode.

[0012] An illustrative embodiment of the present disclosure includes a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor in a video decoder, cause the processor to perform a method. The method includes obtaining a current picture from an encoded video bitstream. The method further includes, for a current block included in the current picture, identifying a reference block included in a reference picture different from the current picture, where the current block is divided into a plurality of sub-blocks (CBSBs), and the reference block has a plurality of sub-blocks (RBSBs), each sub-block corresponding to a different CBSB among the plurality of CBSBs. The method further includes determining whether the reference picture of the RBSBs is the current picture, and, in response to determining that the reference picture of the RBSBs is the current picture, determining a coding mode of the RBSBs as an intra-frame mode. The method further includes, in response to determining that the reference picture of the RBSB is not the current picture, (i) for one of the CBSBs, determining whether the coding mode of the corresponding RBSB is an intra-frame mode or an inter-frame mode, and (ii) determining a motion vector prediction value for the one of the CBSBs based on whether the coding mode of the corresponding RBSB is an intra-frame mode or an inter-frame mode. [Brief explanation of the drawings]

[0013] Other features, properties and advantages of the disclosed subject matter will become more apparent from the following detailed description and drawings.

[0014] [Figure 1] FIG. 1 is a simplified block diagram of a communication system (100) according to one embodiment. [Figure 2] FIG. 2 is a simplified block diagram of a communication system (200) according to one embodiment. [Figure 3] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 5] 10 shows a block diagram of an encoder according to another embodiment. [Figure 6] 10 shows a block diagram of a decoder according to another embodiment; [Figure 7] FIG. 1 is a schematic diagram of intra-frame picture block compensation; [Figure 8] FIG. 2 is a schematic diagram of a current block and surrounding spatial merge candidates for the current block; [Figure 9] 2 is a schematic diagram of sub-blocks of a current block and corresponding sub-blocks of a reference block; FIG. [Figure 10] 1 illustrates an example of a process performed by an encoder or decoder. [Figure 11] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to one embodiment of the present disclosure. The communication system (100) includes multiple terminal devices that can communicate with each other, for example, via a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) connected to each other via the network (150). In the example of FIG. 1, the pair of terminal devices (110) and (120) perform unidirectional data transmission. For example, the terminal device (110) encodes video data (e.g., a video picture stream captured by the terminal device (110)) and transmits it to the other terminal device (120) via the network (150). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. The terminal device (120) receives the encoded video data from the network (150), decodes the encoded video data to recover the video pictures, and displays the video pictures based on the recovered video data. One-way data transmission is common in media service applications.

[0016] In another example, the communication system 100 includes a second pair of terminal devices 130, 140 for performing bidirectional transmission of encoded video data, such as that generated during a video conference. For bidirectional data transmission, in this example, each of the terminal devices 130, 140 encodes video data (e.g., a video picture stream captured by the terminal device) for transmission to another of the terminal devices 130, 140 via the network 150. Each of the terminal devices 130, 140 can further receive encoded video data transmitted from another of the terminal devices 130, 140, decode the encoded video data to recover the video pictures, and display the video pictures on an accessible display device based on the recovered video data.

[0017] In the example of FIG. 1 , terminal devices 110, 120, 130, and 140 are shown as a server, a personal computer, and a smartphone, respectively, although the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure may also apply to laptop computers, tablets, media players, and / or specialized videoconferencing equipment. Network 150 may represent any number of networks, including, for example, wired and / or wireless communication networks, for transmitting encoded video data between terminal devices 110, 120, 130, and 140. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, unless otherwise specified, the architecture and topology of network 150 are not important to the operation of the present disclosure.

[0018] As an example application of the disclosed subject matter, Figure 2 shows a video encoder and decoder arrangement in a streaming transmission environment. The disclosed subject matter is equally applicable to other applications with video capabilities, such as video conferencing, digital television, and storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0019] The streaming transmission system includes a capture subsystem (213) that includes a video source (201), such as a digital camera, for constructing an uncompressed video picture stream (202). In the illustrated example, the video picture stream (202) includes samples captured by the digital camera. The video picture stream (202), depicted as a thick line to emphasize its large amount of data compared to the encoded video data (204) (or encoded video bitstream), is processed by an electronic device (220) that includes a video encoder (203) coupled to the video source (201). The video encoder (203) may include hardware, software, or a combination thereof to realize or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (204) (or encoded video bitstream (204)), depicted as a thin line to emphasize its small amount of data compared to the video picture stream (202), is stored in a streaming server (205) for later use. One or more streaming client subsystems, such as the client subsystems 206 and 208 in FIG. 2, can access the streaming server 205 to retrieve copies 207 and 209 of the encoded video data 204. The client subsystem 206 includes a video decoder 210, for example, in an electronic device 230. The video decoder 210 decodes the incoming copy 207 of the encoded video data and constructs an output video picture stream 211 for display on a display 212 (e.g., a screen) or other display device (not shown). In a streaming transmission system, the encoded video data 204, 207, and 209 (e.g., a video bitstream) can be encoded according to a video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In the example, the developing video encoding standard is informally referred to as Versatile Video Coding, or VVC.The topic of disclosure applies to the VVC context.

[0020] Note that the electronics 220 and 230 may include other components (not shown). For example, the electronics 220 may include a video decoder (not shown), and the electronics 230 may include a video encoder (not shown).

[0021] 3 shows a block diagram of a video decoder (310) according to one embodiment of the present disclosure. The video decoder (310) is included in electronic equipment (330). The electronic equipment (330) may include a receiver (331) (e.g., receiving circuitry). The video decoder (310) may replace the video decoder (210) in the example of FIG. 2.

[0022] The receiver (331) can receive one or more coded video sequences to be decoded by the video decoder (310), and in the same or other embodiments, can receive one coded video sequence at a time, with each coded video sequence being decoded independently of the other coded video sequences. The coded video sequences can be received from a channel (301), which can be a hardware or software link to a storage device for storing the coded video data. The receiver (331) can receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which can be forwarded to a respective utilization entity (not shown). The receiver (331) can separate the coded video sequences from other data. To prevent network jitter, a buffer memory (315) is coupled between the receiver (331) and the entropy decoder / parser (320) (hereinafter referred to as "parser (320)"). In some applications, the buffer memory 315 is part of the video decoder 310. In other applications, the buffer memory 315 may be external to the video decoder 310 (not shown). Still other applications may include a buffer memory (not shown) external to the video decoder 310, for example, to prevent network jitter, and another buffer memory 315 internal to the video decoder 310, for example, to handle broadcast timing. When the receiver 331 receives data from a store-and-forward device or an isochronous network with sufficient bandwidth and controllability, the buffer memory 315 may not be required, or may be small. For example, when used with a best-effort packet network such as the Internet, the buffer memory 315 may be required, but may be significantly larger, advantageously adaptively sized, and implemented, at least in part, in an operator system or similar element (not shown) external to the video decoder 310.

[0023] The video decoder (310) includes a parser (320) that reconstructs codes (321) based on the encoded video sequence. These code categories include information for managing the operation of the video decoder (310) and implicit information for controlling a display device, such as a display device (312) (e.g., a screen), which is not an integral part of the electronic device (330) but is coupled to it, as shown in FIG. 3. The control information used by the display device(s) may be in the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set segments (not shown). The parser (320) performs parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence is based on a video coding technique or standard and may include variable length codes, Huffman codes, and other encoding techniques. The parser (320) extracts a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (320) can further extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0024] The parser (320) can construct a code (321) by performing an entropy decoding / parsing operation on the video sequence received from the buffer memory (315).

[0025] Depending on the type of coded video picture or portion thereof (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of the code (321) involves several different units. Which units are involved, and how, can be controlled by subgroup control information parsed by the parser (320) from the coded video sequence. For the sake of brevity, such subgroup control information streams between the parser (320) and the following units are not described:

[0026] In addition to the functional blocks already mentioned, the video decoder (310) can be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, several of these units may closely interact with each other and may be at least partially integrated with each other. However, for purposes of illustrating the subject matter of the disclosure, it is appropriate to conceptually subdivide the video decoder (310) into the following functional units:

[0027] The first unit is a scalar / inverse transform unit (351). The scalar / inverse transform unit (351) receives quantized transform coefficients as code(s) (321) from the parser (320) and control information including the type of transform scheme to use, block size, quantization factor, quantization scaling matrix, etc. The scalar / inverse transform unit (351) can output blocks containing sample values that are input to an aggregator (355).

[0028] In some situations, the output samples of the scaler / inverse transform unit (351) may belong to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information is provided by the intra picture prediction unit (352). In some situations, the intra picture prediction unit (352) generates blocks of the same size and shape as the block being reconstructed using surrounding, already reconstructed information fetched from the current picture buffer (358). For example, the current picture buffer (358) buffers a partially reconstructed and / or fully reconstructed current picture. In some situations, the aggregator (355) adds, based on each sample, the prediction information generated by the intra prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351).

[0029] In other situations, the output samples of the scalar / inverse transform unit (351) may belong to an inter-frame coded, possibly motion-compensated block. In such situations, the motion-compensated prediction unit (353) can access the reference picture memory (357) to fetch the samples used for prediction. After performing motion compensation on the fetched samples based on the code (321) for that block, the samples are added by the aggregator (355) to the output of the scalar / inverse transform unit (351) (in this case, referred to as residual samples or a residual signal) to generate output sample information. The addresses in the reference picture memory (357) from which the motion-compensated prediction unit (353) fetches the prediction samples can be controlled by a motion vector, which is provided to the motion-compensated prediction unit (353) in the form of a code (321), which may have, for example, X, Y, and reference picture components. Motion compensation may further include interpolation of sample values fetched from the reference picture memory (357) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0030] The output samples of the aggregator (355) can utilize various loop filtering techniques in the loop filter unit (356). Video compression techniques can include in-loop filtering techniques that are controlled by parameters included in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (356) as codes (321) from the parser (320), but can also respond to meta-information obtained while decoding a coded picture or previous portion of the coded video sequence (in decoding order), as well as to previously reconstructed loop-filtered sample values.

[0031] The output of the loop filter unit (356) can be a sample stream that can be output to a display device (312) or stored in a reference picture memory (357) for use in later inter-frame picture prediction.

[0032] Once fully reconstructed, a coded picture can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and identified as a reference picture (e.g., by the parser (320)), the current picture buffer (358) becomes part of the reference picture memory (357), and a new current picture buffer is allocated before reconstructing a subsequent coded picture.

[0033] The video decoder (310) can perform decoding operations based on a given video compression technique in a standard, such as ITU-T Recommendation H.265. The encoded video sequence conforms to the grammar specified by the video compression technique or standard in use, in the sense that the encoded video sequence conforms to both the grammar of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file can select some tools from all available tools in the video compression technique or standard as the only tools available in the configuration file. Compliance requires that the complexity of the encoded video sequence be within the limits defined by the level of the video compression technique or standard. In some situations, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some situations, the limits set by the level are further constrained via the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled in the encoded video sequence.

[0034] In one embodiment, the receiver (331) can receive additional (redundant) data along with the encoded video. The additional data can be included as part of the encoded video sequence(s). The additional data can be utilized by the video decoder (310) to properly decode the data and / or more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0035] 4 shows a block diagram of a video encoder (403) according to one embodiment of the present disclosure. The video encoder (403) is included in electronic equipment (420). The electronic equipment (420) includes a transmitter (440) (e.g., a transmission circuit). The video encoder (403) may replace the video encoder (203) in the example of FIG. 2.

[0036] The video encoder (403) can receive video samples from a video source (401) (not part of the electronics (420) in the example of FIG. 4) that can capture video images to be encoded by the video encoder (403). In other examples, the video source (401) is part of the electronics (420).

[0037] The video source (401) provides a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (403), the digital video sample stream being of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling configuration (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (401) may be a storage device for storing previously prepared video. In a video conferencing system, the video source (401) may be a camera for capturing local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The picture itself is organized as a spatial pixel array, with each pixel containing one or more samples depending on the sampling configuration, color space, etc. in use. The relationship between pixels and samples is readily apparent to those skilled in the art. The following description focuses on samples.

[0038] According to one embodiment, the video encoder (403) encodes and compresses pictures of a source video sequence into an encoded video sequence (443) in real time or under other time constraints required by the application. Enforcing an appropriate encoding rate is one function of the controller (450). In some embodiments, the controller (450) controls and is functionally coupled to other functional units described below. For simplicity, such coupling is not shown. Parameters set by the controller (450) may include parameters related to rate control (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) placement, maximum motion vector search range, etc. The controller (450) may also be configured with other appropriate functions for optimizing the video encoder (403) for a particular system design.

[0039] In some embodiments, the video encoder (403) is configured to operate in a coding loop. As a very simple description, in one example, the coding loop includes a source encoder (430) (e.g., responsible for constructing a code, such as a codestream, based on an input picture to be coded and one or more reference pictures) and a (local) decoder (433) embedded in the video encoder (403). The decoder (433) reconstructs the code and constructs sample data in the same way that a (remote) decoder constructs sample data (because in the video compression techniques considered in the disclosed subject matter, compression between the code and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (434). Because decoding of the codestream produces bit-exact results independent of the decoder location (local or remote), the contents of the reference picture memory (434) are bit-exact between the local encoder and the remote encoder. In other words, the predicted parts that the encoder "sees" as reference picture samples are exactly the same sample values that the decoder "sees" when it tries to use the prediction during decoding. The basic principles of reference picture synchrony (and the drift that occurs when synchrony cannot be maintained, e.g., from channel error) also apply in related fields.

[0040] The operation of the "local" decoder (433) may be similar to that of the "remote" decoder of the video decoder (310), for example, as described in detail in connection with Figure 3. However, with brief reference to Figure 3, because codes are available and the encoding / decoding of the codes into an encoded video sequence by the entropy coder (445) and parser (320) may be lossless, the entropy decoding portion of the video decoder (310), including the buffer memory (315) and parser (320), may not be implemented entirely in the local decoder (433).

[0041] In this case, any decoder techniques other than analysis / entropy decoding present in the decoder necessarily must exist in essentially the same functional form in the corresponding encoder. For this reason, the subject of this disclosure focuses on the operation of the decoder. The description of the encoder technique may be simplified, since it is the inverse of the fully described decoder technique. Only in certain areas is a more detailed explanation required, which is provided below.

[0042] In operation, in some examples, the source encoder (430) can perform motion-compensated predictive encoding, which predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence, designated as "reference pictures." In this manner, the encoding engine (432) encodes differences between pixel blocks of the input picture and pixel blocks of one or more reference pictures that can be selected as prediction basis(s) for the input picture.

[0043] The local video decoder (433) can decode the coded video data of pictures that can be designated as reference pictures based on the codes constructed by the source encoder (430). The operation of the coding engine (432) is preferably a lossy process. When the coded video data can be decoded by a video decoder (not shown in FIG. 4), the reconstructed video sequence can be a copy of the source video sequence, typically with some error. The local video decoder (433) replicates the decoding process that the video decoder performs on the reference pictures and stores the reconstructed reference pictures in a reference picture cache (434). In this manner, the video encoder (403) locally stores copies of the reconstructed reference pictures, which share common content (i.e., are free of transmission errors) with the reference pictures of the reconstructions obtained by the remote video decoder.

[0044] The predictor (435) may perform a predictive search for the coding engine (432). That is, for a new picture to be coded, the predictor (435) may search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (435) can find suitable prediction references by operating on a pixel block-by-pixel block basis based on the sample blocks. In some situations, an input picture may have prediction references obtained from multiple reference pictures stored in the reference picture memory (434), as determined based on the search results obtained by the predictor (435).

[0045] The controller (450) can manage the encoding operations of the source encoder (430), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0046] The outputs of all the functional units mentioned above can be entropy coded in the entropy coder (445), which performs lossless compression on the codes generated by the various functional units and converts the codes into coded video sequences using techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0047] The sender (440) buffers the encoded video sequence(s) constructed by the entropy encoder (445) to prepare them for transmission over a communication channel (460), which may be a hardware / software link to a storage device for storing the encoded video data. The sender (440) merges the encoded video data from the video encoder (403) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0048] The controller (450) can manage the operation of the video encoder (403). During encoding, the controller (450) assigns each coded picture a number of coding picture types, which may affect the coding technique applied to the respective picture. For example, pictures are typically assigned one of the following picture types:

[0049] An intraframe picture (I-picture) may be a picture that is coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intraframe pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variants of I-pictures and their respective uses and characteristics.

[0050] A predicted picture (P-picture) may be a picture that is coded and decoded using intra-frame or inter-frame prediction, using at most one motion vector and reference index to predict the sample values of each block.

[0051] A bidirectionally predicted picture (B-picture) may be a picture that is encoded and decoded using intra-frame or inter-frame prediction, using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, a multi-predicted picture may use more than two reference pictures and associated metadata to reconstruct a single block.

[0052] A source picture can generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples), and encoding is performed block by block. Blocks can be predictively coded with reference to other (coded) blocks determined by the coding assignment applied to their respective pictures. For example, blocks of an I-picture can be non-predictively coded, or they can be predictively coded (spatial or intra-frame) with reference to already coded blocks of the same picture. Pixel blocks of a P-picture can be predictively coded via spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture can be predictively coded via spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0053] The video encoder (403) may perform encoding operations based on a given video encoding technology or standard, such as ITU-T Recommendation H.265. During this operation, the video encoder (403) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the grammar specified by the video encoding technology or standard used.

[0054] In one embodiment, the transmitter (440) can transmit additional data along with the encoded video. The source encoder (430) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other types of redundant data such as redundant pictures and slices, Supplemental Enhancement Information (SEI) messages, Video Usability Information (VUI) parameter set segments, etc.

[0055] A captured video may consist of multiple source pictures (video pictures) that represent a time sequence. Intra-frame picture prediction (often abbreviated as intra-frame prediction) exploits spatial correlation within a particular picture, while inter-frame picture prediction exploits correlation (temporal or otherwise) between pictures. In the example, a particular picture in encoding / decoding, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously encoded and still buffered reference picture in the video, that block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension for identifying the reference picture when multiple reference pictures are used.

[0056] In some embodiments, a bidirectional prediction technique is used for inter-frame picture prediction. Based on the bidirectional prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are before the current picture in the video in decoding order (but may be past and future, respectively, in display order). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0057] Also, merged mode techniques can be applied to inter-frame picture prediction to improve coding efficiency.

[0058] According to some embodiments of the present disclosure, prediction such as inter-frame picture prediction and intra-frame picture prediction is performed for each block. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), each including one luminance CTB and two chrominance CTBs. Each CTU is recursively partitioned into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU is partitioned into one CU of 64×64 pixels, four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In the example, each CU is analyzed to determine a prediction type for the CU, such as an inter-frame prediction type or an intra-frame prediction type. Depending on temporal and / or spatial predictability, a CU is divided into one or more prediction units (PUs). Generally, each PU includes a luminance prediction block (PB) and two chrominance PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed for each prediction block. An example of a prediction block is a luminance prediction block, which includes a matrix of pixel values (e.g., luminance values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0059] 5 shows a diagram of a video decoder (503) according to another embodiment of the present disclosure. The video encoder (503) is configured to receive a processed block (e.g., a predictive block) of sample values in a current video picture in a video picture sequence and to encode the processed block into an encoded picture as part of the encoded video sequence. In the example, the video encoder (503) is used in place of the video encoder (203) in the example of FIG. 2.

[0060] In an HEVC example, a video encoder (503) receives a processing block, e.g., a matrix of sample values, such as a predictive block of 8x8 samples. The video encoder (503) determines, e.g., by rate-distortion optimization, whether the processing block is best coded using intra-frame mode, inter-frame mode, or bidirectional predictive mode. When coding the processing block in intra-frame mode, the video encoder (503) codes the processing block into a coded picture using intra-frame prediction techniques; when coding the processing block in inter-frame mode or bidirectional predictive mode, the video encoder (503) can code the processing block into a coded picture using inter-frame prediction or bidirectional prediction techniques, respectively. In some video coding techniques, the merged mode can be an inter-frame picture prediction submode in which motion vectors are derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In some other video coding techniques, there can be motion vector components applied to the theme block. In the illustrated example, the video encoder (503) includes other components, such as a mode decision module (not shown) for determining the mode of the processing blocks.

[0061] In the example of FIG. 5, the video encoder (503) includes an interframe encoder (530), an intraframe encoder (522), a residual calculator (523), a switch (526), a residual encoder (524), a general controller (521), and an entropy encoder (525), concatenated as shown in FIG. 5.

[0062] The inter-frame encoder (530) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., a description of redundancy information, motion vectors, and merge mode information based on an inter-frame coding technique), and calculate an inter-frame prediction result (e.g., a block of predictions) based on the inter-frame prediction information using any suitable technique.

[0063] The intraframe encoder (522) is configured to receive samples of a current block (e.g., a processing block), optionally compare the block with previously coded blocks in the same picture, generate transformed and quantized coefficients, and optionally also generate intraframe prediction information (e.g., intraframe prediction direction information based on one or more intraframe coding techniques).

[0064] The general-purpose controller (521) is configured to determine general-purpose control data and control other components of the video encoder (503) based on the general-purpose control data. In the illustrated example, the general-purpose controller (521) determines the mode of the block and provides a control signal to the switch (526) based on the mode. For example, if the mode is intra-frame mode, the general-purpose controller (521) controls the switch (526) to select the intra-frame mode result for use by the residual calculator (523) and controls the entropy encoder (525) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream. If the mode is inter-frame mode, the general-purpose controller (521) controls the switch (526) to select the inter-frame prediction result for use by the residual calculator (523) and controls the entropy encoder (525) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

[0065] The residual calculator (523) is configured to calculate the difference (residual data) between the received block and a prediction result selected from the intraframe encoder (522) or the interframe encoder (530). The residual encoder (524) is configured to operate on the residual data to encode the residual data and generate transform coefficients. In the example, the residual encoder (524) is configured to transform the residual data in the frequency domain to generate transform coefficients, which are then quantized to obtain quantized transform coefficients.

[0066] The entropy encoder (525) is configured to format a bitstream to include the coded block. The entropy encoder (525) is configured to include various information based on an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (525) is configured to include general control data, selected prediction information (e.g., intraframe prediction information or interframe prediction information), residual information, and other appropriate information in the bitstream. Note that, according to the disclosed subject matter, residual information is not present when coding a block in an interframe mode or a merged submode of the bidirectional prediction mode.

[0067] 6 shows a diagram of a video decoder (610) according to another embodiment of the present disclosure. The video decoder (610) is configured to receive coded pictures as part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In the example, the video decoder (610) is used in place of the video decoder (210) in the example of FIG. 2.

[0068] In the example of FIG. 6, the video decoder (610) includes an entropy decoder (671), an interframe decoder (680), a residual decoder (673), a reconstruction module (674), and an intraframe decoder (672), concatenated as shown in FIG. 6.

[0069] The entropy decoder (671) is configured to reconstruct, based on the coded picture, specific codes indicative of the grammar elements that make up the coded picture. Such codes include, for example, prediction information (e.g., intraframe prediction information or interframe prediction information) that identifies the mode for coding the block (e.g., intraframe mode, interframe mode, bidirectional prediction mode, a combined submode of the latter two, or other submodes), specific samples or metadata used for prediction by the intraframe decoder (672) or the interframe decoder (680), respectively, and residual information in the form of, for example, quantized transform coefficients. In the illustrated example, if the prediction mode is an interframe or bidirectional prediction mode, the interframe prediction information is provided to the interframe decoder (680), and if the prediction type is an intraframe prediction type, the intraframe prediction information is provided to the intraframe decoder (672). The residual information is provided to the residual decoder (673) via inverse quantization.

[0070] The inter-frame decoder (680) is configured to receive the inter-frame prediction information and to generate an inter-frame prediction result based on the inter-frame prediction information.

[0071] The intraframe decoder (672) is configured to receive intraframe prediction information and generate a prediction result based on the intraframe prediction information.

[0072] The residual decoder (673) is configured to perform inverse quantization to extract dequantized transform coefficients and to transform the residual from the frequency domain to the spatial domain by processing the dequantized transform coefficients. The residual decoder (673) may require some control information (to include quantizer parameters (QP)), which is provided by the entropy decoder (671) (data path not shown since this is little control information).

[0073] The reconstruction module (674) is configured to combine the residual output from the residual decoder (673) with a prediction result (which may be output from an inter-frame prediction module or an intra-frame prediction module) in the spatial domain to form a reconstructed block, which may be part of a reconstructed picture, which may in turn be part of a reconstructed video, and to perform other appropriate operations, such as a deblocking operation, to improve visual quality.

[0074] It should be noted that the video encoder (203), video encoder (403), video encoder (503), and video decoder (210), video decoder (310), and video decoder (610) may be implemented using any suitable technology. In one embodiment, the video encoder (203), video encoder (403), video encoder (503), and video decoder (210), video decoder (310), and video decoder (610) may be implemented using one or more integrated circuits. In another embodiment, the video decoders (203), (403), (403), and video decoders (210), (310), and (610) are implemented by one or more processors for executing software instructions.

[0075] Compensation based on blocks from different pictures is called motion compensation. Block compensation can also be performed based on previously constructed regions within the same picture, which is called intra-picture block compensation or intra-frame block copying. For example, a displacement vector indicating the offset between a current block and a reference block is called a block vector. In some embodiments, a block vector refers to a reference block that has already been reconstructed and is used for reference. Similarly, from the perspective of parallel processing, reference regions beyond tile / slice boundaries or wavefront trapezoid boundaries may also be excluded from reference by block vectors. Due to these constraints, block vectors may differ from motion vectors (MVs) in motion compensation, and in motion compensation, motion vectors may be any value (positive or negative in the x or y direction).

[0076] 7 illustrates an example of intra-frame picture block compensation (e.g., intra-frame block copy mode). In FIG. 7, a current picture 700 has a set of coded / decoded blocks (i.e., gray blocks) and a set of uncoded / decoded blocks (i.e., white blocks). A sub-block 702 of one of the uncoded / decoded blocks may be associated with a block vector 704 that points to another sub-block 706 that was previously coded / decoded. Thus, any motion information associated with the sub-block 706 can be used in coding / decoding the sub-block 702.

[0077] In some embodiments, the coding of the block vectors is explicit. In other embodiments, the coding of the block vectors is implicit. In explicit mode, the difference between the block vector and its predicted value is signaled, and in implicit mode, the block vector is recovered from the predicted value of the block vector in a manner similar to motion vector prediction in merged mode. In some embodiments, the resolution of the block vectors is limited to integer positions. In other embodiments, the block vectors point to fractional positions.

[0078] In some embodiments, the reference index signals intra-frame picture block compensation using block level (i.e., intra-frame block copy mode), where the currently decoded picture is regarded as a reference picture and placed at the end of the reference picture list, which may further be managed in a decoded picture buffer (DPB) together with other temporal reference pictures.

[0079] According to some embodiments, the reference block is flipped horizontally or vertically (e.g., flipped intra-frame block copy) before being used in prediction for the current block. In some embodiments, each compensation unit within an M×N coding block is M×1 or 1×N rows (e.g., row-based intra-frame block copy).

[0080] According to some embodiments, block-level motion compensation is performed, and the current block is the processing unit for performing motion compensation with the same motion information. Therefore, once the size of a block is specified, all pixels in the block will use the same motion information to form its prediction block. Examples of block-level motion compensation include using spatial merge candidates, temporal candidates, and combining motion vectors from existing multiple merge candidates in bidirectional prediction.

[0081] Referring to Figure 8, a current block (801) contains samples that the encoder / decoder has already found in the motion estimation process to be predictable from a previous block of the same size but spatially shifted. In some embodiments, instead of directly encoding the motion vectors, the motion vectors can be derived from metadata associated with one or more reference pictures, such as from the nearest reference picture (in decoding order), using motion vectors associated with any of five surrounding samples, denoted A0, A1, B0, B1, and B2 (corresponding to 802-806, respectively). Blocks A0, A1, B0, B1, and B2 are referred to as spatial merge candidates.

[0082] According to some embodiments, pixels at different locations within a motion compensation block (e.g., sub-blocks) may have different motion information. The difference between these block-level motion information is not signaled but derived. This type of motion compensation is called sub-block-level motion compensation, which allows the motion compensation of a block to be smaller than the block itself. In this regard, each block may have multiple sub-blocks, where each sub-block may contain different motion information.

[0083] An example of sub-block level motion compensation includes sub-block-based temporal motion vector prediction, where sub-blocks of a current block have different motion vectors. Another example of sub-block level motion compensation is ATMVP, which is a technique that allows each coding block to fetch multiple sets of motion information from multiple blocks smaller than the current coding block from a collocated reference picture.

[0084] Another example of sub-block-level motion compensation includes spatial / temporal blending with sub-block adjustment, which adjusts the motion vector of each sub-block in the current block based on the motion vectors of its spatial / temporal neighbors. In this mode, some sub-blocks may require motion information from corresponding sub-blocks in a temporal reference picture.

[0085] Another example of sub-block level motion compensation is affine coding motion compensation, which first derives motion vectors at the four corners of the current block based on the motion vectors of neighboring blocks, and then derives other motion vectors for the current block (e.g., at the sub-block or pixel level) according to the affine model, so that each sub-block can have a different motion vector from its neighbors.

[0086] Another example of subblock-level motion compensation is merge candidate refinement using motion vector derivation at the decoder side. In this mode, after obtaining one or more motion vector predictors for the current block or a subblock of the current block, the given motion vector predictors can be further refined using methods such as template matching or bilateral matching. The refined motion vectors are used to perform motion compensation. By performing the same refinement operation on both the encoder and decoder sides, the decoder does not need additional information on how the refinement is displaced from the original prediction. Also, skip mode can be considered as a special merge mode, in which, in addition to deriving the motion information of the current block from its neighbors, the prediction residual of the current block is also zero.

[0087] According to some embodiments, in sub-block temporal motion vector prediction, sub-blocks of a current block may have different motion vector predictors derived from a temporal reference picture. For example, a set of motion information including a motion vector of the current block and an associated reference index is identified. The motion information is determined from a first available spatial merge candidate. The motion information is used to determine a reference block in a reference picture for the current block. The reference block is also divided into sub-blocks. In some embodiments, for each current block sub-block (CBSB) in the current picture, there is a corresponding reference block sub-block (RBSB) in the reference picture.

[0088] In some embodiments, for each CBSB, when the corresponding RBSB is coded in inter-frame mode using a set of motion information, the motion information is transformed (e.g., using a method such as motion vector scaling in temporal motion vector prediction) and used as a prediction value of the motion vector for that CBSB. Below, we will describe in more detail how to handle RBSB coded in intra-frame mode (e.g., intra-frame block copy mode).

[0089] According to some embodiments, when using a subblock-based temporal motion vector prediction mode, it is not permitted for each CBSB to be coded in an intraframe mode such as an intraframe block copy mode. This is achieved by regarding an RBSB coded in intraframe block copy as an intraframe mode. In particular, regardless of how the intraframe block copy mode is considered (e.g., whether it is considered as an interframe mode, an intraframe mode, or a third mode), if a corresponding RBSB for a CBSB is coded in an intraframe block copy mode, the RBSB is considered to be an intraframe mode in subblock-based temporal motion vector prediction. Therefore, in some embodiments, an RBSB coded in an intraframe block copy mode is treated according to a default setting in subblock-based temporal motion vector prediction. For example, when coding a corresponding RBSB of a CBSB in an intraframe block copy mode, a default motion vector such as a zero motion vector is used as a predicted value for the CBSB. In this example, the reference picture used for the CBSB is a temporal reference picture, not the current picture. For example, a temporal reference picture may be a picture shared by all sub-blocks of the current block, the first reference picture in a reference picture list, a co-located picture for TMVP purposes, etc. In another example, for a CBSB, if the corresponding RBSB is coded in inter mode but the reference picture is the current picture, a default motion vector, e.g., a zero motion vector, is assigned to the CBSB. In this regard, even if the RBSB is coded in inter mode, the RBSB is treated as if it were coded in intra mode because the current picture and the reference picture are the same.

[0090] FIG. 9 illustrates an example of performing sub-block-based temporal motion vector prediction. FIG. 9 shows a current picture 900 having nine blocks, including a current block 900A. The current block 900A is divided into four sub-blocks 1 to 4. The current picture 900 can be associated with a reference picture 902, which has nine previously coded / decoded blocks. Also shown in FIG. 9, the current block 900A has a motion vector 904 pointing to a reference block 902A. The motion vector 904 can be determined using motion vectors of one or more neighboring blocks (e.g., spatial merge candidates) of the current block 900A. The reference block 902A is divided into four sub-blocks 1 to 4. Sub-blocks 1 to 4 of the reference block 902A correspond to sub-blocks 1 to 4 of the current block 900A, respectively. If the reference picture 902 and the current picture 900 are the same, each RBSB in the block 902A is treated as if these blocks were coded in intraframe mode. In this regard, for example, if subblock 1 of block 902A is coded in inter-frame mode, but reference picture 902 and current picture 900 are the same, then subblock 1 of block 902A is treated as if it were coded in intra-frame mode, and a default motion vector is assigned to subblock 1 of block 900A.

[0091] When the reference picture 902 and the current picture 900 are different, sub-blocks 1 to 4 of the reference block 902A are used to perform sub-block-based temporal motion vector prediction for sub-blocks 1 to 4 of the current block 900A, respectively. For example, the motion vector prediction value for sub-block 1 of the current block 900A is determined based on whether sub-block 1 of the reference block 902A is coded in inter-frame mode or intra-frame mode (e.g., intra-frame block copy mode). If sub-block 1 of the reference block 902A is coded in inter-frame mode, the motion vector for sub-block 1 of the reference block 902A is used to determine the motion vector for sub-block 1 of the current block 900A. If sub-block 1 of the reference block 902A is coded in intra-frame mode, the motion vector for sub-block 1 of the current block 900A is set to a zero motion vector.

[0092] 10 shows an example of a process performed by an encoder or decoder, such as intraframe encoder 522 or intraframe decoder 672. The process begins in step S1000, where a current picture is obtained from an encoded video bitstream. For example, see FIG. 9, where current picture 900 is obtained from the encoded video bitstream. The process proceeds to step S1002, where a reference block from a reference picture is identified for a current block in the current picture. For example, see FIG. 9, where reference picture 902 is searched for from a reference picture list associated with current block 900A. When performing sub-block temporal motion vector prediction for current block 900A, motion vector 904 may be used to identify reference block 902A in reference picture 902.

[0093] The process proceeds to step S1004, where it is determined whether the reference picture is the same as the current picture. If the reference picture is different from the current picture, the process proceeds to step S1006, where it determines the coding mode of the RBSB corresponding to the CBSB. For example, referring to FIG. 9, the CBSB of the current block 900A is The process then proceeds to step S1008, where it is determined whether the coding mode of the RBSB is an inter-frame mode. If the coding mode of the RBSB is an inter-frame mode, the process proceeds to step S1010, where it determines a motion vector prediction value for the CBSB based on the motion vector prediction value for the RBSB. For example, if the coding mode of the RBSB of the reference block 902A is an inter-frame mode, the process proceeds to step S1010, where it determines a motion vector prediction value for the CBSB based on the motion vector prediction value for the RBSB. If the coding mode of 1 is the inter-frame mode, the RBSB of the reference block 902A For example, the motion vector prediction value of RBSB 1 of the reference block 902A is transformed (e.g., using a method such as motion vector scaling in temporal motion vector prediction) to determine the motion vector prediction value of CBSB 1 of the current block 900A. It is used as a motion vector predictor for 1.

[0094] Returning to step S1008, if the coding mode of RBSB is not the inter-frame mode (for example, the coding mode of RBSB is the intra-frame mode), the process proceeds to step S1012, where the motion vector prediction value of CBSB is set to the default motion vector. For example, the RBSB of the reference block 902A If 1 is coded in intraframe mode, the CBSB of the current block 900A A motion vector predictor value of 1 is set to a default motion vector, such as the zero motion vector.

[0095] Returning to step S1004, if the reference picture and the current picture are the same, the process proceeds to step S1012, where the motion vector predictor for CBSB is set to the default motion vector. In this regard, if the reference picture and the current picture are the same, the coding mode for RBSB is determined to be intra mode, and the motion vector predictor for the corresponding CBSB is set to the default motion vector. In this regard, even if RBSB is in inter mode, setting the motion vector predictor for CBSB to the default motion vector allows RBSB to be treated as if it were coded in intra mode. Steps S1004 to S1012 may be repeated for each sub-block in the current block 900A.

[0096] The techniques may be implemented as computer software by computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 11 illustrates a computer system (1100) for implementing some embodiments of the disclosed subject matter.

[0097] Computer software may be encoded in any suitable machine code or computer language, which may be edited, compiled, linked, or otherwise constructed to contain instructions that are executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or the like, or may be interpreted, executed by microcode, or the like.

[0098] The instructions may be executed by various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0099] The components of computer system 1100 shown in Figure 11 are exemplary in nature and are not intended to limit the scope or functionality of use of the example computer software for implementing the present disclosure. Neither the arrangement of components should be interpreted as having a dependency or requirement regarding any one or combination of components shown in the example embodiment of computer system 1100.

[0100] The computer system 1100 may include several human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users, for example, through tactile input (e.g., keystrokes, slides, data glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface devices may also capture media not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capture devices), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0101] The input human-machine interface device may include one or more of a keyboard (1101), a mouse (1102), a touchpad (1103), a touch panel (1110), a data glove (not shown), a joystick (1105), a microphone (1106), a scanner (1107), and an imaging device (1108) (only one of each listed).

[0102] The computer system (1100) may also include human-machine interface output devices. Such human-machine interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1110), data gloves (not shown), or joystick (1105), although haptic feedback devices that are not used as input devices also exist), audio output devices (e.g., speakers (1109), headphones (not shown)), visual output devices (e.g., a screen (1110), including a CRT screen, LCD screen, plasma screen, or OLED screen, each of which may or may not have touchscreen input and haptic feedback capabilities, some of which may provide two-dimensional visual output or three-dimensional or higher-dimensional output, such as through stereoscopic output, including virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0103] The computer system (1100) may further include a human-accessible storage device and associated media, such as a CD / DVD drive (1121). These include optical media including ROM / RW (1120), thumb drives (1122), removable hard drives or solid state drives (1123), traditional magnetic media such as magnetic tape and floppy disks (not shown), and dedicated ROM / ASIC / PLD based devices such as dongles (not shown).

[0104] Those skilled in the art will appreciate that in conjunction with the presently disclosed subject matter, the term "computer-readable medium" as used does not include transmission media, carrier waves or other transient signals.

[0105] The computer system 1100 may further include interfaces for one or more communication networks. The networks may be, for example, wireless, wired, or optical. The networks may also be local, wide-area, metropolitan, vehicular, industrial, real-time, or delay-tolerant networks. Examples of networks include local area networks such as Ethernet; cellular networks, including WLAN, GSM, 3G, 4G, 5G, and LTE; television wired or wireless wide-area digital networks, including cable television, satellite television, and terrestrial television; and vehicular and industrial networks, including CANbus. Some networks typically require an external network interface adapter coupled to a general-purpose data port or peripheral bus 1149 (e.g., a USB port on the computer system 1100). Other networks are typically integrated into the core of the computer system 1100 by coupling to a system bus, described below (e.g., an Ethernet interface for a PC computer system or a cellular network interface for a smartphone computer system). Through any of these networks, the computer system 1100 can communicate with other entities. Such communications may be one-way and receive only (e.g., broadcast television), one-way and transmit only (e.g., a CANbus to a CANbus device), or two-way (e.g., to another computer system over a local or wide area digital network). Specific protocols and protocol stacks may be utilized for each of these networks and network interfaces described above.

[0106] The human-machine interface device, human-accessible storage device, and network interface can be coupled to a core (1140) of the computer system (1100).

[0107] The core (1140) includes one or more central processing units (CPUs) (1141), graphics processing units (GPUs) (1142), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1143), hardware accelerators for certain tasks (1144), etc. These devices, along with read-only memory (ROM) (1145), random access memory (1146), and internal mass storage devices (1147) such as internal non-user-accessible hard disk drives, SSDs, etc., are connected via a system bus (1148). In some computer systems, expansion is possible with additional CPUs, GPUs, etc., by accessing the system bus (1148) in the form of one or more physical plugs. Peripheral devices are coupled to the core's system bus (1148) directly or via a peripheral bus (1149). Peripheral bus architectures include PCI, USB, etc.

[0108] The CPU (1141), GPU (1142), FPGA (1143), and accelerator (1144) can execute a number of instructions, which, when combined, constitute the computer code referred to above. The computer code is stored in ROM (1145) or RAM (1146). Transient data is stored in RAM (1146), and permanent data may be stored, for example, in an internal mass storage device (1147). A cache memory can be used to quickly store and retrieve any of the memory devices, and the cache memory may be closely associated with one or more of the CPU (1141), GPU (1142), mass storage device (1147), ROM (1145), RAM (1146), etc.

[0109] The computer-readable medium contains computer code for performing various computer-implemented operations, and the medium and computer code may be specially designed and constructed for the purposes of this disclosure, or may be of the type known and available to those skilled in the art of computer software.

[0110] By way of example and not limitation, a computer system having architecture 1100, and in particular core 1140, can provide functionality by having one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software embodied in one or more tangible computer-readable media. Such computer-readable media may include the user-accessible mass storage media discussed above, as well as storage devices of the core 1140 that are non-transitory in nature, such as the core's internal mass storage device 1147 or ROM 1145. Software for implementing the present disclosure in various embodiments is stored on such devices and executed by the core 1140. Depending on particular needs, the computer-readable media may include one or more storage devices or chips. The software may cause the cores (1140), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or portions of specific processes described herein, define data structures stored in RAM (1146), and modify such data structures based on the software-defined processes. Additionally or alternatively, the computer system may provide functionality embodied in logically hardwired or otherwise implemented circuitry (e.g., accelerators (1144)), which may operate in place of or in conjunction with software to execute specific processes or portions of specific processes described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) on which software is stored for execution, circuitry embodying logic for execution, or both. The present disclosure includes any appropriate combination of hardware and software.

[0111] While certain illustrative embodiments have been described in this disclosure, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. Thus, many systems and methods that embody the principles of this disclosure and are within the spirit and scope of this disclosure will be apparent to those skilled in the art, even if they are not explicitly described herein.

[0112] (1) A method for video decoding in a decoder, comprising: obtaining a current picture from an encoded video bitstream; identifying a reference block included in a reference picture different from the current picture for a current block included in the current picture; the current block being divided into a plurality of sub-blocks (CBSBs); the reference block having a plurality of sub-blocks (RBSBs), each of the plurality of sub-blocks (RBSBs) corresponding to a different CBSB of the plurality of CBSBs; determining whether a reference picture of the RBSB is the current picture; In response to determining that the reference picture is the current picture, determine the coding mode of the RBSB as an intra-frame mode; and in response to determining that the reference picture of the RBSB is not the current picture, (i) determine, for one of the plurality of CBSBs, whether the coding mode of the corresponding RBSB is an intra-frame mode or an inter-frame mode, and (ii) determine a motion vector prediction value for the one CBSB of the plurality of CBSBs based on whether the coding mode of the corresponding RBSB is the intra-frame mode or the inter-frame mode.

[0113] (2) The method according to (1), wherein in response to determining that the coding mode of the corresponding RBSB is the intra-frame mode, the motion vector predictor determined for the one CBSB is set to a default motion vector.

[0114] (3) The method according to (2), wherein the default motion vector is (i) a zero motion vector or (ii) a deviation between the CBSB and the RBSB.

[0115] (4) The method according to any one of features (1) to (3), wherein in response to determining that the coding mode of the corresponding RBSB is the inter-frame mode, the motion vector prediction value determined for one CBSB is based on the motion vector prediction value associated with the corresponding RBSB.

[0116] (5) The method according to feature (4), wherein the determined motion vector predictor is a scaled version of the motion vector predictor associated with the corresponding RBSB.

[0117] (6) The method according to any one of features (1) to (5), wherein the reference block is identified according to a motion vector prediction value associated with a block adjacent to the current block.

[0118] (7) The method according to any one of features (1) to (6), wherein the reference picture is a first reference picture from a reference picture sequence related to the current picture.

[0119] (8) A video decoder for video decoding, comprising a processing circuit, for obtaining a current picture from an encoded video bitstream, and for a current block included in the current picture, identifying a reference block included in a reference picture different from the current picture, the current block being divided into a plurality of sub-blocks (CBSBs), the reference block having a plurality of sub-blocks (RBSBs), the plurality of sub-blocks (RBSBs) respectively corresponding to different CBSBs of the plurality of CBSBs, determining whether a reference picture of the RBSBs is the current picture, and In response to determining that a picture is the current picture, determine the coding mode of the RBSB as an intra-frame mode; and in response to determining that the reference picture of the RBSB is not the current picture, (i) determine, for one of the plurality of CBSBs, whether the coding mode of the corresponding RBSB is an intra-frame mode or an inter-frame mode, and (ii) determine a motion vector prediction value for the one CBSB of the CBSBs based on whether the coding mode of the corresponding RBSB is the intra-frame mode or the inter-frame mode.

[0120] (9) The video decoder of feature (8), wherein the processing circuit is configured to set the motion vector prediction value for the one CBSB to a default motion vector in response to determining that the coding mode of the corresponding RBSB is the intra-frame mode.

[0121] (10) The video decoder of (9), wherein the default motion vector is (i) a zero motion vector or (ii) a deviation between the CBSB and the RBSB.

[0122] (11) A video decoder described in any one of features (8) to (10), wherein the processing circuit is configured to determine a motion vector prediction value for one CBSB based on a motion vector prediction value associated with the corresponding RBSB in response to determining that the coding mode of the corresponding RBSB is the inter-frame mode.

[0123] (12) The video decoder of feature (11), wherein the determined motion vector predictor is a scaled version of the motion vector predictor associated with the corresponding RBSB.

[0124] (13) The video decoder according to any one of features (8) to (12), wherein the processing circuit identifies the reference block based on a motion vector prediction value associated with a block adjacent to the current block.

[0125] (14) The video decoder according to any one of features (8) to (13), wherein the reference picture is a first reference picture from a reference picture sequence related to the current picture.

[0126] (15) A computer program having instructions, which, when executed by a processor, cause the processor to perform a method, the method including: obtaining a current picture from an encoded video bitstream; identifying, for a current block included in the current picture, a reference block included in a reference picture different from the current picture; the current block being divided into a plurality of sub-blocks (CBSBs); the reference block having a plurality of sub-blocks (RBSBs), each of the plurality of sub-blocks (RBSBs) corresponding to a different CBSB of the plurality of CBSBs; and determining whether the reference picture of the RBSB is the current picture. and in response to determining that the reference picture of the RBSB is not the current picture, (i) for one of the plurality of CBSBs, determining whether the coding mode of the corresponding RBSB is an intra mode or an inter mode, and (ii) determining a motion vector prediction value for the one CBSB of the plurality of CBSBs based on whether the coding mode of the corresponding RBSB is the intra mode or the inter mode.

[0127] (16) The computer program according to (15), wherein the motion vector predictor determined for one CBSB is set to a default motion vector in response to determining that the coding mode of the corresponding RBSB is the intra-frame mode.

[0128] (17) The computer program according to (16), wherein the default motion vector is (i) a zero motion vector or (ii) a deviation between the CBSB and the RBSB.

[0129] (18) A computer program according to any one of features (15) to (17), wherein in response to determining that the coding mode of the corresponding RBSB is the inter-frame mode, a motion vector prediction value determined for one CBSB is based on a motion vector prediction value associated with the corresponding RBSB.

[0130] (19) The non-transitory computer-readable medium of (18), wherein the determined motion vector predictor is a scaled version of the motion vector predictor associated with the corresponding RBSB.

[0131] (20) A non-transitory computer-readable medium according to any one of features (15) to (19), wherein the reference block is identified according to a motion vector prediction value associated with a block adjacent to the current block.

[0132] Appendix A: Acronyms MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Replenishment Enhancement Information VUI: Video Usability Information GOP: Group of Pictures TU: Conversion unit PU: Prediction Unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted Block HRD: Hypothetical Reference Decoder SNR: Signal-to-Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Array SSD: Solid State Drive IC: Integrated Circuit CU: Coding Unit

Claims

1. A method for video decoding performed by a decoder, Steps to obtain the current picture from the encoded video bitstream, A step of identifying a reference block contained in a reference picture with respect to a current block contained in the current picture, wherein the reference picture differs from the current picture, the current block is divided into a plurality of current subblocks (CBSBs) coded in intermode, and the reference block has a plurality of reference subblocks (RBSBs), each CBSB corresponding to at least one RBSB. The steps include determining whether the corresponding RBSB is coded in intra-predictive mode or intra-block copy mode, In response to the fact that the corresponding RBSB of each of the plurality of CBSBs is coded in intra-prediction mode or intra-block copy mode, the steps include setting the first motion vector prediction value of each CBSB to a zero motion vector, A method comprising the step of decoding the plurality of CBSBs by performing subblock-based time motion vector prediction based on first motion vector prediction values ​​of the plurality of CBSBs.

2. The method according to claim 1, wherein, in response to the fact that the corresponding RBSB of each CBSB is coded in intermode, the first motion vector prediction value of each CBSB is set using the second motion vector prediction value of the corresponding RBSB.

3. The method of claim 2, wherein the second motion vector prediction value of the corresponding RBSB is scaled.

4. The method according to any one of claims 1 to 3, wherein the reference block is identified according to a motion vector prediction value related to a block adjacent to the current block.

5. The method according to any one of claims 1 to 4, wherein the reference picture is a temporal reference picture.

6. The method according to claim 5, wherein the temporal reference picture is a first reference picture from a reference picture sequence associated with the current picture.

7. The method according to claim 5, wherein the aforementioned time reference picture is a collated picture.

8. The method according to claim 5, wherein the temporal reference picture is a picture shared by all subblocks of the current block.

9. A video decoder for video decoding, One or more memory locations containing computer programs, One or more processors, It has, The computer program causes one or more processors to perform the method according to any one of claims 1 to 8. Video decoder.

10. A computer program having instructions, wherein, when executed by a processor, the instructions cause the processor to perform the method according to any one of claims 1 to 8.

Images