Method, apparatus, and computer program for predicting temporal motion vectors based on subblocks

Abstract

To provide a method for sub-block based temporal motion vector prediction.SOLUTION: A video decoding method includes the steps of: obtaining a current picture; and identifying a reference block included in a reference picture different from the current picture with respect to a current block included in the current picture, in which the current block is divided into a plurality of sub blocks (CBSBs), and the reference block has the plurality of sub blocks (RBSBs). The method also includes steps of: determining whether the reference picture for the RBSB is the current picture; and, in response to determining that the reference picture for the RBSB is the current picture, determining a coding mode of the RBSB to be an intra-frame mode. The method also includes a step of, in response to determining that the reference picture of the RBSB is not the current picture, determining a motion vector prediction value for the one of the CBSBs based on whether or not the coding mode of the corresponding RBSB is one of the intra-frame modes, and an inter-frame mode.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] This disclosure claims priority to U.S. Provisional Application No. 62 / 680,468, filed on June 4, 2018, entitled "METHODS FOR SUB-BLOCK BASED TEMPORAL MOTION VECTOR PREDICTION", and all the contents of that application are incorporated herein by reference.

[0002] This disclosure generally describes embodiments related to video encoding.

Background Art

[0003] The background description provided herein is for the purpose of generally embodying the background of the present disclosure. From the work level described in this background art section, the work of the currently named inventors and aspects not limited as prior art descriptions at the time of filing are not explicitly or implicitly recognized as prior art to the present disclosure.

[0004] For decades, it has been known to perform video encoding and decoding by inter-picture prediction with motion compensation. Uncompressed digital video has a series of pictures, each picture having spatial dimensions, for example, 1920×1080 luminance samples and associated chrominance samples. The series of pictures may have, for example, a fixed or variable picture rate of 60 pictures per second or 60 Hz (also informally called the frame rate). Uncompressed video has high bitrate requirements. For example, 1080p60 4:2:0 video (resolution of 1920×1080 luminance samples at a 60 Hz frame rate) with 8 bits per sample requires a bandwidth of about 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0005] Video encoding and decoding aim to reduce redundancy in the video signal input through compression. Compression contributes to reducing the bandwidth or storage space requirements mentioned above, sometimes by more than two orders of magnitude. Lossless compression, lossy compression, and combinations thereof are available. Lossless compression refers to the technique of reconstructing an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for its intended application. In the case of video, lossy compression is applied extensively. The amount of distortion that can be tolerated depends on the application; for example, users of consumer streaming applications may tolerate higher distortion than users of television-contributing applications. The achievable compression ratio reflects that a higher allowable / acceptable distortion results in a higher compression ratio.

[0006] Motion compensation may be a lossy compression technique and relates to the following technique, namely, a technique in which a block of sample data from a previously constructed picture or a part of it (a reference picture) is used to predict a newly reconstructed picture or part of a picture after it has been spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV). In some situations, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, where the third dimension indicates the reference picture being used (the latter may indirectly be the time dimension).

[0007] In some video compression techniques, the motion vector (MV) applied to a region of sample data can be predicted from other MVs, such as those relating to other regions of sample data spatially adjacent to the region being reconstructed, and the MV is predicted in the order of decoding. In this way, the amount of data required to encode the MV is greatly reduced, redundancy is eliminated, and compression is enhanced. MV prediction works effectively because, for example, when encoding a video input signal derived from a camera (called natural video), there is a statistical probability that regions larger than the region to which a single MV can be applied will move in a similar direction, and therefore, in some situations, can be predicted by similar motion vectors derived from the MVs of adjacent regions. As a result, the MV found for a particular region will be similar to or the same as the MV predicted from surrounding MVs, and can be represented with fewer bits than would be used when directly encoding the MV after entropy coding. In some situations, MV prediction may be an example of lossless compression of the signal (i.e., MV) derived from the original signal (i.e., sample stream). In other situations, the MV prediction itself may be irreversible, for example, due to rounding errors when calculating the predicted value from several peripheral MVs.

[0008] H.265 / HEVC (ITU-T Recommendation H.265, “High Efficiency Video Coding / Decoding (High The H.265 standard (Efficiency Video Coding), December 2016, describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this application describes the technique referred to below as "spatial merging." [Overview of the project] [Problems that the invention aims to solve]

[0009] Some forms of inter-frame prediction are performed at the sub-block level. However, sub-block-based temporal motion vector prediction modes, such as Alternative Temporal Motion Vector Prediction (ATMVP) and Spatiotemporal Motion Vector Prediction (STMVP), require the corresponding sub-blocks to be encoded in inter-frame mode. However, these temporal motion vector prediction modes cannot handle sub-blocks encoded in intra-frame modes, such as intra-frame block copy mode. [Means for solving the problem]

[0010] An exemplary embodiment of the foregoing disclosure includes a method for decoding video using a decoder. The method includes obtaining a current picture from an encoded video bitstream. The method further includes identifying a reference block in a reference picture different from the current picture for a current block contained in the current picture, wherein the current block is divided into a plurality of subblocks (CBSBs), and the reference block has a plurality of subblocks (RBSBs), each corresponding to a different CBSB among the plurality of CBSBs. The method further includes determining whether the reference picture of the RBSB is the current picture, and in response to determining that the reference picture of the RBSB is the current picture, determining the encoding mode of the RBSB as an in-frame mode. The method further includes, in response to the determination that the reference picture of the RBSB is not a picture, (i) determining for one of the CBSBs whether the encoding mode of the corresponding RBSB is in-frame mode or inter-frame mode, and (ii) determining a motion vector prediction value for the one of the CBSBs based on whether the encoding mode of the corresponding RBSB is in-frame mode or inter-frame mode.

[0011] An exemplary embodiment of the foregoing disclosure includes a video decoder for video decoding. The video decoder has a processing circuit configured to obtain a current picture from an encoded video bitstream. The processing circuit is further configured to identify a reference block in a reference picture different from the current picture for a current block contained in the current picture, the current block being divided into a plurality of subblocks (CBSBs), and the reference block having a plurality of subblocks (RBSBs), each corresponding to one different CBSB from the plurality of CBSBs. The processing circuit is further configured to determine whether the reference picture of the RBSB is the current picture, and in response to determining that the reference picture of the RBSB is the current picture, to determine the encoding mode of the RBSB as an in-frame mode. The processing circuit further determines, in response to the determination that the reference picture of the RBSB is not currently a picture, (i) for one of the CBSBs, whether the encoding mode of the corresponding RBSB is in-frame mode or inter-frame mode, and (ii) based on whether the encoding mode of the corresponding RBSB is in-frame mode or inter-frame mode, it determines a motion vector prediction value for the one of the CBSBs.

[0012] An exemplary embodiment of the content of this disclosure includes a non-temporary computer-readable medium on which instructions are stored, and when the instructions are executed by a processor in a video decoder, the processor causes the processor to perform a method. The method includes obtaining a current picture from an encoded video bitstream. The method further includes, for a current block contained in the current picture, identifying a reference block contained in a reference picture different from the current picture, wherein the current block is divided into a plurality of subblocks (CBSBs), and the reference block has a plurality of subblocks (RBSBs), each corresponding to one different CBSB from the plurality of CBSBs. The method further includes determining whether the reference picture of the RBSB is the current picture, and in response to determining that the reference picture of the RBSB is the current picture, determining the encoding mode of the RBSB as an in-frame mode. The method further includes, in response to the determination that the reference picture of the RBSB is not currently a picture, (i) determining for one of the CBSBs whether the encoding mode of the corresponding RBSB is in-frame mode or inter-frame mode, and (ii) determining a motion vector prediction value for the one of the CBSBs based on whether the encoding mode of the corresponding RBSB is in-frame mode or inter-frame mode. [Brief explanation of the drawing]

[0013] Other features, properties, and advantages of the topic will become clearer based on the detailed description and drawings below.

[0014] [Figure 1] This is a schematic diagram of a simplified block diagram of a communication system (100) based on one embodiment. [Figure 2] This is a schematic diagram of a simplified block diagram of a communication system (200) based on one embodiment. [Figure 3] This is a schematic diagram of a simplified block diagram of a decoder based on one embodiment. [Figure 4] This is a schematic diagram of a simplified block diagram of an encoder based on one embodiment. [Figure 5] A block diagram of the encoder based on another embodiment is shown. [Figure 6] A block diagram of a decoder based on another embodiment is shown. [Figure 7] This is a schematic diagram of in-frame picture block compensation. [Figure 8] This is a schematic diagram of the current block and the candidate mergers in the surrounding space of the current block. [Figure 9] This is a schematic diagram of the subblocks of the current block and the corresponding subblocks of the referenced block. [Figure 10] An example of a process performed by an encoder or decoder is shown. [Figure 11] This is a schematic diagram of a computer system based on one embodiment. [Modes for carrying out the invention]

[0015] Figure 1 shows a simplified block diagram of a communication system (100) based on one embodiment of the present disclosure. The communication system (100) has a plurality of terminal devices that can communicate with each other, for example, via a network (150). For example, the communication system (100) has a first pair of terminal devices (110), (120) connected to each other via the network (150). In the example of Figure 1, the first pair of terminal devices (110), (120) perform one-way data transmission. For example, terminal device (110) encodes video data (for example, a video picture stream captured by terminal device (110)) and transmits it to other terminal device (120) via the network (150). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. Terminal device (120) receives the encoded video data from the network (150), decodes the encoded video data to recover the video picture, and displays the video picture based on the recovered video data. One-way data transmission is commonly seen in media service applications and other similar applications.

[0016] In another example, the communication system (100) includes a second pair of terminal devices (130), (140) for performing bidirectional transmission of encoded video data generated, for example, during a video conference. For bidirectional data transmission, in the example, each of the terminal devices (130), (140) encodes video data (e.g., a video picture stream captured by the terminal device) and transmits it to the other terminal device in the terminal devices (130), (140) via the network (150). Each of the terminal devices (130), (140) further receives the encoded video data transmitted from the other terminal device in the terminal devices (130), (140), decodes the encoded video data to recover the video picture, and can display the video picture on an accessible display device based on the recovered video data.

[0017] In the example of FIG. 1, the terminal devices (110), (120), (130), and (140) are shown as servers, personal computers, and smartphones, but the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure are applicable to laptop computers, tablets, media players, and / or dedicated video conferencing devices. The network (150) represents any number of networks, such as a wired connection (wired) and / or a wireless communication network, for transmitting encoded video data among the terminal devices (110), the terminal device (120), the terminal device (130), and the terminal device (140). The communication network (150) can exchange data in a circuit-switched and / or packet-switched channel. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, unless otherwise specified, the architecture and topology of the network (150) are not critical to the operation of the present disclosure.

[0018] As an example of the application of the disclosed theme, Figure 2 shows a configuration of a video encoder and decoder in a streaming transmission environment. The disclosed theme is equally applicable to other applications with video capabilities, such as video conferencing, digital television, and the storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0019] The streaming transmission system includes a capture subsystem (213), which includes a video source (201), such as a digital camera, for constructing an uncompressed video picture stream (202). In this example, the video picture stream (202) includes a sample captured by the digital camera. The video picture stream (202), which is drawn as a thick line to emphasize its larger data volume compared to encoded video data (204) (or encoded video bitstream), is processed by electronic equipment (220) including a video encoder (203) coupled to the video source (201). The video encoder (203) includes hardware, software, or a combination thereof to implement or carry out each aspect of the themes of the disclosure described in more detail below. The encoded video data (204) (or encoded video bitstream (204)), which is drawn as a thin line to emphasize its smaller data volume compared to the video picture stream (202), is stored in a streaming server (205) for later use. One or more streaming client subsystems, such as client subsystems (206) and (208) in Figure 2, can access a streaming server (205) to retrieve copies (207) and (209) of the encoded video data (204). Client subsystem (206) includes, for example, a video decoder (210) in an electronic device (230). The video decoder (210) decodes the incoming copy of the encoded video data (207) and constructs an output video picture stream (211) that is displayed on a display (212) (e.g., a screen) or other display device (not shown). In a streaming transmission system, the encoded video data (204), (207), and (209) (e.g., video bitstream) can be encoded based on a video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In examples, video encoding standards under development are informally referred to as multipurpose video coding or VVC.The disclosed theme is applied to the context of VVC.

[0020] Note that the electronic device (220) and the electronic device (230) may include other components (not shown). For example, the electronic device (220) may include a video decoder (not shown), and the electronic device (230) may include a video encoder (not shown).

[0021] FIG. 3 shows a block diagram of a video decoder (310) based on one embodiment of the present disclosure. The video decoder (310) is included in the electronic device (330). The electronic device (330) may include a receiver (331) (e.g., a receiving circuit). The video decoder (310) may be used instead of the video decoder (210) in the example of FIG. 2.

[0022] The receiver (331) can receive one or more encoded video sequences to be decoded by the video decoder (310), and in the same embodiment or other embodiments, it may receive one encoded video sequence at a time, and the decoding of each encoded video sequence is independent of other encoded video sequences. It may also receive encoded video sequences from channel (301), which may be a hardware / software link to a storage device for storing encoded video data. The receiver (331) can receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams, which may be transferred to their respective user entities (not shown). The receiver (331) can separate the encoded video sequences from the other data. To prevent network jitter, a buffer memory (315) is connected between the receiver (331) and the entropy decoder / parser (320) (hereinafter referred to as "parser (320)"). In some applications, the buffer memory (315) is part of the video decoder (310). In other applications, the buffer memory (315) may be located outside the video decoder (310) (not shown). Furthermore, in other applications, a buffer memory (not shown) located outside the video decoder (310) can, for example, prevent network jitter, and a separate buffer memory (315) located inside the video decoder (310) can, for example, handle broadcast timing. When the receiver (331) receives data from a storage / transfer device or isochronous network with sufficient bandwidth and controllability, the buffer memory (315) may not be required, or it may be small. For example, when used in a best-effort packet network such as the Internet, the buffer memory (315) is required, made considerably large, and advantageously adaptively sized, and may be implemented, at least partially, in an operator system or similar element (not shown) located outside the video decoder (310).

[0023] The video decoder (310) has a parser (320) that reconstructs codes (321) based on the encoded video sequence. These codes include information for managing the operation of the video decoder (310) and latent information for controlling display devices such as a display device (312) (e.g., a screen), which, as shown in Figure 3, is not an integral part of the electronic device (330) but is linked to the electronic device (330). The control information used for (one or more) display devices may be in the form of supplementary enhancement information (SEI messages) or video usability information (VUI) parameter set segments (not shown). The parser (320) performs parsing / entropy decoding on the received, encoded video sequence. The encoding of the encoded video sequence is based on video encoding techniques or standards, and may include variable-length codes, Huffman codes, etc. The parser (320) follows various principles, such as arithmetic codes that are context-sensitive or not. Based on at least one parameter corresponding to a group, the parser (320) extracts a set of subgroup parameters for at least one subgroup of pixel subgroups in the video decoder from the encoded video sequence. Subgroups include picture groups (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), and predictive units (PU). The parser (320) can further extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, and motion vectors.

[0024] The parser (320) can construct a code (321) by performing an entropy decoding / analysis operation on the video sequence received from the buffer memory (315).

[0025] Depending on the type of the encoded video picture or parts thereof (e.g., interframe and intraframe pictures, interframe and intraframe blocks) and other factors, the reconstruction of the code (321) involves several different units. Which units are involved and how they are involved can be controlled by subgroup control information analyzed by the parser (320) from the encoded video sequence. For brevity, such subgroup control information streams between the parser (320) and the following multiple units are not described.

[0026] In addition to the functional blocks already mentioned, the video decoder (310) can be conceptually subdivided into several functional units described below. In actual implementations operating under commercial constraints, multiple units in these units may interact closely with each other and may be at least partially integrated with each other. However, for the purpose of illustrating the theme of the disclosure, it is appropriate to subdivide it conceptually into the following functional units.

[0027] The first unit is the scaler / inverse unit (351). The scaler / inverse unit (351) receives from the parser (320) one or more codes (321) containing the quantization conversion coefficients and control information including the conversion scheme to be used, the block size, quantization factors, and the quantization scaling matrix. The scaler / inverse unit (351) can output a block containing the sample values ​​that will be input to the aggregator (355).

[0028] In some situations, the output samples of the scaler / inverse unit (351) may belong to an in-frame coded block, i.e., a block that does not utilize prediction information from a previously reconstructed picture, but can utilize prediction information from a previously reconstructed portion of the current picture. Such prediction information is provided by the in-frame picture prediction unit (352). In some situations, the in-frame picture prediction unit (352) generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information fetched from the current picture buffer (358). For example, the current picture buffer (358) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some situations, the aggregator (355) adds the prediction information generated by the in-frame prediction unit (352) to the output sample information provided by the scaler / inverse unit (351), based on each sample.

[0029] In other situations, the output samples of the scaler / inverse unit (351) may belong to a potentially motion-compensated block that has been interframe-encoded. In such situations, the motion-compensated prediction unit (353) can access the reference picture memory (357) to fetch the samples used for prediction. After motion compensation is performed on the fetched samples based on the code (321) relating to the block, these samples are added to the output of the scaler / inverse unit (351) by the aggregator (355) (in this case, called residual samples or residual signals) to generate output sample information. The address in the reference picture memory (357) from which the motion-compensated prediction unit (353) fetches the predicted samples can be controlled by a motion vector, which is used by the motion-compensated prediction unit (353) in the form of code (321), which may have, for example, X, Y, and reference picture components. Motion compensation may further include interpolation of sample values ​​fetched from reference picture memory (357) when using the precise motion vectors of subsamples, motion vector prediction mechanisms, and so on.

[0030] The output samples of the aggregator (355) can be used by various loop filtering techniques in the loop filtering unit (356). The video compression technique may include an in-loop filtering technique, which is controlled by parameters included in the encoded video sequence (also called the encoded video bitstream) and made available to the loop filtering unit (356) as a code (321) from the parser (320), but can also respond to metadata obtained while decoding earlier parts of the encoded picture or encoded video sequence (in the order of decoding), as well as to previously reconstructed and loop-filtered sample values.

[0031] The output of the loop filter unit (356) can be a sample stream that can be output to the display device (312), and can also be stored in the reference picture memory (357) for use in subsequent inter-frame picture prediction.

[0032] Once fully reconstructed, an encoded picture can be used as a reference picture for later predictions. For example, when an encoded picture corresponding to the current picture is fully reconstructed and that encoded picture is identified as a reference picture (e.g., by a parser (320)), the current picture buffer (358) becomes part of the reference picture memory (357), and a new current picture buffer is allocated before any subsequent encoded pictures are reconstructed.

[0033] The video decoder (310) can perform decoding operations based on a specified video compression technique in a standard such as ITU-T Recommendation H.265. The encoded video sequence conforms to the grammar specified by the video compression technique or standard in use, in the sense that it conforms to both the grammar of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file can select several tools from all the tools available in the video compression technique or standard, with the configuration file being the only tool available. Compliance requires that the complexity of the encoded video sequence be within the limits defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples / second), maximum reference picture size, etc. In some cases, the limits set by the level are further restricted through the specification of the Hypothetical Reference Decoder (HRD) and metadata for HRD buffer management signaled in the encoded video sequence.

[0034] In one embodiment, the receiver (331) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of one or more encoded video sequences. The additional data can be used by the video decoder (310) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, time, space, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0035] Figure 4 shows a block diagram of a video encoder (403) based on one embodiment of the present disclosure. The video encoder (403) is contained within an electronic device (420). The electronic device (420) has a transmitter (440) (e.g., a transmitting circuit). The video encoder (403) may replace the video encoder (203) in the example of Figure 2.

[0036] The video encoder (403) can receive video samples from a video source (401) (not part of the electronic device (420) in the example in Figure 4) that can capture video images encoded by the video encoder (403). In other examples, the video source (401) is part of the electronic device (420).

[0037] The video source (401) provides a source video sequence, which is in the form of a digital video sample stream, encoded by the video encoder (403), the digital video sample stream having any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits…), any color space (e.g., BT.601 Y CrCB, RGB…), and any suitable sampling configuration (e.g., Y CrCb 4:2:0, Y CrCb It may have a 4:4:4 (sequence). In a media service system, the video source (401) may be a storage device for storing previously prepared video. In a video conferencing system, the video source (401) may be a capture device for capturing local image information as a video sequence. The video data may be provided as a plurality of individual pictures that convey motion when viewed sequentially. The pictures themselves may be organized as a spatial pixel array, and each pixel may contain one or more samples depending on the sampling configuration, color space, etc., used. The relationship between pixels and samples will be easily understood by those skilled in the art. The following description focuses on samples.

[0038] Based on one embodiment, the video encoder (403) encodes and compresses pictures from a source video sequence as an encoded video sequence (443) in real time or under other time constraints required by the application. Enforcing an appropriate encoding rate is one function of the controller (450). In some embodiments, the controller (450) controls and is functionally coupled to other functional units described below. For simplicity, such coupling is not illustrated. Parameters set by the controller (450) may include rate control parameters (picture skip, quantizer, λ value of rate distortion optimization technique, etc.), picture size, picture group (GOP) arrangement, maximum motion vector search range, etc. The controller (450) may be configured to have other appropriate functions for the video encoder (403) optimized for a particular system design.

[0039] In some embodiments, the video encoder (403) is configured to operate in an encoding loop. In a very brief description, in one example, the encoding loop includes a source encoder (430) (for example, responsible for constructing a code, such as a code stream, based on the input picture to be encoded and one or more reference pictures), and a (local) decoder (433) embedded in the video encoder (403). The decoder (433) reconstructs the code and constructs sample data in the same manner as a (remote) decoder constructs sample data (because in the video compression techniques considered in the theme of disclosure, both compression between the code and the encoded video bitstream are reversible). The reconstructed sample stream (sample data) is input to a reference picture memory (434). Since decoding the code stream produces a bit-accurate result independent of the decoder location (local or remote), the contents in the reference picture memory (434) are bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion that the encoder "sees" as a reference picture sample is exactly the same as the sample value that the decoder "sees" when attempting to utilize the prediction during decoding. The fundamental principles of the synchronization of the reference picture (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) also apply to the relevant fields.

[0040] The operation of the “local” decoder (433) may be similar to that of the “remote” decoder of the video decoder (310), for example, as described in detail in relation to Figure 3. However, referring briefly to Figure 3, since the codes are available and the encoding / decoding of the codes to the encoded video sequence by the entropy encoder (445) and parser (320) can be reversible, the entropy decoding portion of the video decoder (310), including the buffer memory (315) and parser (320), does not need to be fully implemented in the local decoder (433).

[0041] In this case, any decoder techniques other than analysis / entropy decoding present in the decoder must also be present in the corresponding encoder in essentially the same functional form. For this reason, the theme of the disclosure focuses on the operation of the decoder. Since the encoder techniques are the inverse of the decoder techniques described in full, the description of the encoder techniques can be simplified. More detailed explanations are required only in specific areas and are provided below.

[0042] In operation, in some examples, the source encoder (430) can perform motion-compensated predictive coding, which predictively encodes the input picture by referencing one or more previously encoded pictures from a video sequence, designated as “reference pictures”. In this scheme, the encoding engine (432) encodes the difference between the pixel blocks of the input picture and the pixel blocks of one or more reference pictures that can be selected as prediction criteria for the input picture.

[0043] A local video decoder (433) can decode encoded video data of a picture that can be designated as a reference picture, based on the code constructed by the source encoder (430). The operation of the encoding engine (432) is preferably lossy. Once the encoded video data can be decoded by a video decoder (not shown in Figure 4), the reconstructed video sequence may generally be a copy of the source video sequence with some degree of error. The local video decoder (433) copies the decoding process that the video decoder performs on the reference picture and stores the reconstructed reference picture in a reference picture cache (434). In this scheme, the video encoder (403) stores a copy of the reconstructed reference picture locally, and this copy has the same content (no transmission error) as the reconstructed reference picture obtained by the remote video decoder.

[0044] The predictor (435) can perform a predictive search for the coding engine (432). That is, for a new picture to be coded, the predictor (435) can search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., which can serve as appropriate predictive references for the new picture. Based on the sample blocks, the predictor (435) can find appropriate predictive references by operating on each pixel block. In some situations, the input picture may have predictive references obtained from multiple reference pictures stored in the reference picture memory (434), as determined based on the search results obtained by the predictor (435).

[0045] The controller (450) can manage the encoding operations of the source encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[0046] The entropy encoder (445) can perform entropy coding on the outputs of all the functional units mentioned above. Based on techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding, the entropy encoder (445) performs lossless compression on the codes generated from the various functional units to convert the codes into an encoded video sequence.

[0047] The transmitter (440) buffers (one or more) encoded video sequences constructed by the entropy encoder (445) to prepare them for transmission over a communication channel (460), which may be a hardware / software link leading to a storage device for storing the encoded video data. The transmitter (440) merges the encoded video data from the video encoder (403) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0048] The controller (450) can manage the operation of the video encoder (403). In encoding, the controller (450) assigns each encoded picture to one of several encoded picture types, which may affect the encoding technique applied to that picture. For example, a picture is typically assigned one of the following picture types:

[0049] An in-frame picture (I-picture) may be a picture that is encoded and decoded without using any other pictures in the sequence as a source for prediction. Some video codecs allow different types of in-frame pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are familiar with the variations of I-pictures, as well as their respective uses and characteristics.

[0050] A prediction picture (P-picture) may be a picture that uses at most one motion vector and a reference index to predict the sample values ​​of each block, and performs encoding and decoding using intra-frame prediction or inter-frame prediction.

[0051] A bidirectional predictive picture (B-picture) can be a picture that uses at most two motion vectors and reference indices to predict the sample values ​​of each block, and uses intra-frame or inter-frame prediction for encoding and decoding. Similarly, a multiple predictive picture can use more than two reference pictures and associated metadata to reconstruct a single block.

[0052] A source picture can generally be subdivided in space into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples), and each block can be coded. Blocks can be coded predictively by referencing other (coded) blocks determined by the coding assignment applied to each picture in those blocks. For example, blocks in an I-picture can be coded non-predictively, or blocks in an I-picture can be coded predictively (spatial or intra-frame) by referencing already coded blocks of the same picture. Pixel blocks in a P-picture can be coded predictively via spatial or temporal prediction by referencing one previously coded reference picture. Blocks in a B-picture can be coded predictively via spatial or temporal prediction by referencing one or two previously coded reference pictures.

[0053] The video encoder (403) can perform encoding operations based on a given video encoding technique or standard, such as ITU-T Recommendation H.265. In these operations, the video encoder (403) can perform various compression operations, including predictive encoding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the grammar specified by the video encoding technique or standard used.

[0054] In one embodiment, the transmitter (440) may transmit additional data along with the encoded video. The source encoder (430) may include such data as part of the encoded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, video usability information (VUI) parameter set segments, and the like.

[0055] The captured video may consist of multiple source pictures (video pictures) that present a temporal sequence. Intra-frame picture prediction (often abbreviated as intra-frame prediction) utilizes spatial correlations in a particular picture, while inter-frame picture prediction utilizes (temporal or other) correlations between pictures. In the example, a particular picture in encoding / decoding, called the current picture, is divided into blocks. If a block in the current picture is analogous to a reference block in a previously encoded and still-buffered reference picture in the video, that block in the current picture can be encoded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and, if multiple reference pictures are used, may have a third dimension for identifying the reference picture.

[0056] In some embodiments, bidirectional prediction techniques are used for inter-frame picture prediction. Based on bidirectional prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which, in decoding order, precede the current picture in the video (however, in display order, they may be past and future, respectively). The block in the current picture can be encoded by a first motion vector pointing to the first reference block in the first reference picture and a second motion vector pointing to the second reference block in the second reference picture. The block can be predicted by the combination of the first and second reference blocks.

[0057] Furthermore, the combined mode technique can be applied to inter-frame picture prediction to improve coding efficiency.

[0058] Based on several embodiments of this disclosure, predictions such as inter-frame picture prediction and intra-frame picture prediction are performed block by block. For example, according to the HEVC standard, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs) that include one luminance CTB and two chrominance CTBs. Each CTU is recursively divided into one or more coding units (CUs) in a quadtree. For example, a CTU that is 64x64 pixels is divided into one CU that is 64x64 pixels, or four CUs that are 32x32 pixels, or sixteen CUs that are 16x16 pixels. In the examples, by analyzing each CU, a prediction type for the CU is determined, such as an inter-frame prediction type or an intra-frame prediction type. Depending on the temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Generally, each PU includes a luminance prediction block (PB) and two chrominance PBs. In one embodiment, the prediction operation in encoding (encoding / decoding) is performed for each prediction block. An example of a prediction block is a luminance prediction block, which includes a matrix of pixel values ​​(e.g., luminance values) such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0059] Figure 5 shows a diagram of a video decoder (503) based on another embodiment of the present disclosure. The video encoder (503) is configured to receive a processing block (e.g., a prediction block) of sample values ​​in the current video picture in a video picture sequence and to encode the processing block into an encoded picture as part of an encoded video sequence. In this example, the video encoder (503) is used in place of the video encoder (203) in the example of Figure 2.

[0060] In the HEVC example, the video encoder (503) receives a matrix of sample values, such as a processing block, for example, an 8x8 sample prediction block. The video encoder (503) determines, for example by rate distortion optimization, whether the processing block is best encoded using intra-frame mode, inter-frame mode, or bidirectional prediction mode. When attempting to encode the processing block in intra-frame mode, the video encoder (503) encodes the processing block into an encoded picture using an intra-frame prediction technique. When attempting to encode the processing block in inter-frame mode or bidirectional prediction mode, the video encoder (503) can encode the processing block into an encoded picture using an inter-frame prediction technique or a bidirectional prediction technique, respectively. In one video encoding technique, the combined mode may be an inter-frame picture prediction submode that derives a motion vector from one or more motion vector prediction values ​​without benefiting from an encoded motion vector component outside the prediction values. In another video encoding technique, there may be a motion vector component applied to the theme block. In the example, the video encoder (503) includes other components, such as a mode determination module (not shown) for determining the mode of the processing block.

[0061] In the example shown in Figure 5, the video encoder (503) includes a concatenated interframe encoder (530), an intraframe encoder (522), a residual calculator (523), a switch (526), ​​a residual encoder (524), a general-purpose controller (521), and an entropy encoder (525), as shown in Figure 5.

[0062] The interframe encoder (530) is configured to receive a sample of the current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in the previous and subsequent pictures), generate interframe prediction information (e.g., a description of redundant information, motion vectors, and merged mode information based on interframe coding techniques), and compute an interframe prediction result (e.g., a block of predictions) using any appropriate technique based on the interframe prediction information.

[0063] The in-frame encoder (522) is configured to receive a sample of the current block (e.g., a processing block), optionally compare the block with an already encoded block in the same picture, convert it, generate quantization coefficients, and optionally generate further in-frame prediction information (e.g., in-frame prediction direction information based on one or more in-frame coding techniques).

[0064] The general-purpose controller (521) is configured to determine general-purpose control data and, based on that data, control other components of the video encoder (503). In the example, the general-purpose controller (521) determines the mode of a block and, based on that mode, provides control signals to the switch (526). For example, if the mode is an in-frame mode, the general-purpose controller (521) controls the switch (526) to select the in-frame mode result for use by the residual calculator (523) and controls the entropy encoder (525) to select the in-frame prediction information and include it in the bitstream. If the mode is an inter-frame mode, the general-purpose controller (521) controls the switch (526) to select the inter-frame prediction result for use by the residual calculator (523) and controls the entropy encoder (525) to select the inter-frame prediction information and include it in the bitstream.

[0065] The residual calculator (523) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the in-frame encoder (522) or the inter-frame encoder (530). The residual encoder (524) is configured to operate on the residual data to encode the residual data and generate conversion coefficients. In the example, the residual encoder (524) is configured to transform the residual data in the frequency domain and generate conversion coefficients. The conversion coefficients are then quantized to obtain quantized conversion coefficients.

[0066] The entropy encoder (525) is configured to format the bitstream to include the encoded blocks. The entropy encoder (525) is configured to include various types of information based on an appropriate standard such as the HEVC standard. In the example, the entropy encoder (525) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other appropriate information in the bitstream. Note that, according to the theme of disclosure, residual information is not present when encoding blocks in inter-frame mode or a combined sub-mode of bidirectional prediction mode.

[0067] Figure 6 shows a diagram of a video decoder (610) based on another embodiment of the present disclosure. The video decoder (610) is configured to receive an encoded picture as part of an encoded video sequence and to decode the encoded picture to produce a reconstructed picture. In this example, the video decoder (610) is used in place of the video decoder (210) in the example of Figure 2.

[0068] In the example shown in Figure 6, the video decoder (610) includes a coupled entropy decoder (671), interframe decoder (680), residual decoder (673), reconstruction module (674), and intraframe decoder (672), as shown in Figure 6.

[0069] The entropy decoder (671) is configured to reconstruct specific codes that represent the grammatical elements constituting the encoded picture, based on the encoded picture. Such codes include, for example, a mode for encoding a block (e.g., an in-frame mode, an inter-frame mode, a bidirectional prediction mode, a combined submode of the latter, or other submodes), prediction information (e.g., in-frame prediction information or inter-frame prediction information) that can recognize specific samples or metadata used for prediction by the in-frame decoder (672) or inter-frame decoder (680), respectively, and residual information, for example, in the form of quantization conversion coefficients. In the example, if the prediction mode is an inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (680), and if the prediction type is an in-frame prediction type, in-frame prediction information is provided to the in-frame decoder (672). Residual information is provided to the residual decoder (673) via inverse quantization.

[0070] The interframe decoder (680) is configured to receive interframe prediction information and generate interframe prediction results based on the interframe prediction information.

[0071] The in-frame decoder (672) is configured to receive in-frame prediction information and generate prediction results based on the in-frame prediction information.

[0072] The residual decoder (673) is configured to perform inverse quantization to extract the inversely quantized transformation coefficients and to process the inversely quantized transformation coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (673) may require some control information (including quantizer parameters (QP)), which is provided by the entropy decoder (671) (this is little control information, so the data path is not shown).

[0073] The reconstruction module (674) is configured to form a reconstructed block in the spatial domain by combining the residual output from the residual decoder (673) with the prediction result (which may be output from the inter-frame prediction module or the intra-frame prediction module). This reconstructed block can be part of a reconstructed picture, and the reconstructed picture can be part of a reconstructed video. Visual quality can be improved by performing other appropriate operations, such as deblocking.

[0074] Furthermore, the video encoder (203), video encoder (403), video encoder (503), and video decoder (210), video decoder (310), and video decoder (610) can be implemented using any suitable technology. In one embodiment, the video encoder (203), video encoder (403), video encoder (503), and video decoder (210), video decoder (310), and video decoder (610) can be implemented using one or more integrated circuits. In other embodiments, the video decoders (203), (403), (403), and video decoders (210), (310), and (610) are implemented by one or more processors for executing software instructions.

[0075] Block-based compensation from different pictures is called motion compensation. Block compensation may also be based on a previously constructed region within the same picture, which is called in-frame picture block compensation or in-frame block copying. For example, a displacement vector used to indicate the displacement between the current block and a reference block is called a block vector. Based on some embodiments, the block vector refers to a reference block that has already been reconstructed and used for reference. Similarly, from a parallel processing perspective, reference regions beyond tile / slice boundaries or wavefront trapezoidal boundaries may also be excluded from the block vector's reference. Due to these constraints, the block vector may differ from the motion vector (MV) in motion compensation, in which the motion vector may be of any value (positive or negative in the x or y direction).

[0076] Figure 7 shows an example of in-frame picture block compensation (e.g., in-frame block copy mode). In Figure 7, the picture 700 currently has one set of encoded / decoded blocks (i.e., gray blocks) and one set of unencoded / decoded blocks (i.e., white blocks). A subblock 702 of one of the unencoded / decoded blocks may be associated with a block vector 704 that points to the other subblock 706 that was previously encoded / decoded. Thus, any motion information associated with subblock 706 can be used to encode / decode subblock 702.

[0077] Based on some embodiments, the encoding of block vectors is explicit. In other embodiments, the encoding of block vectors is implicit. In explicit mode, the difference between the block vector and its predicted value is signaled, and in implicit mode, the block vector is recovered from its predicted value in a manner similar to motion vector prediction in combined mode. In some embodiments, the resolution of the block vector is limited to integer positions. In other embodiments, the block vector points to fractional positions.

[0078] Based on several embodiments, a reference index signals in-frame picture block compensation (i.e., in-frame block copy mode) that utilizes the block level, and the picture currently being decoded is considered a reference picture, which is then placed at the end of the reference picture list. This reference picture may also be managed in the decoded picture buffer (DPB) along with other temporal reference pictures.

[0079] Based on several embodiments, the reference block is flipped horizontally or vertically (e.g., an in-frame block copy of the flip) before being used for prediction for the current block. In some embodiments, each compensation unit within an M×N coded block is M×1 or 1×N rows (e.g., an in-frame block copy based on rows).

[0080] Based on several embodiments, block-level motion compensation is performed, and currently, a block is the processing unit for performing motion compensation with the same motion information. Therefore, once the size of a block is determined, all pixels in that block use the same motion information to form its predicted block. An example of block-level motion compensation includes using spatial merge candidates, temporal candidates, and a combination of motion vectors from multiple existing merge candidates in bidirectional prediction.

[0081] Referring to Figure 8, the current block (801) contains a sample that has already been found by the encoder / decoder in the motion search process to be predictable from a previous block of the same size that has been spatially shifted. In some embodiments, instead of directly encoding the MV, the MV can be derived from metadata associated with one or more reference pictures, such as from the most recent reference picture (in the decoding order), using the MV associated with one of the five surrounding samples denoted A0, A1, B0, B1, B2 (corresponding to 802-806, respectively). Blocks A0, A1, B0, B1, B2 are called spatial merge candidates.

[0082] Based on several embodiments, pixels located at different positions within a motion-compensated block (e.g., subblocks) may have different motion information. The difference between this block-level motion information and the subblocks is not signaled but derived. This type of motion compensation is called subblock-level motion compensation and allows the motion compensation of a block to be smaller than that of the block itself. In this regard, each block may have multiple subblocks, each of which may contain different motion information.

[0083] An example of subblock-level motion compensation involves predicting a temporal motion vector based on a subblock, where the subblocks of the current block have different motion vectors. Another example of subblock-level motion compensation is ATMVP, a technique that allows each encoded block to fetch multiple sets of motion information from multiple blocks smaller than the current encoded block from a juxtaposed reference picture.

[0084] Another example of subblock-level motion compensation includes spatial / temporal fusion with subblock adjustment, which adjusts the motion vector of each subblock in the current block based on the motion vectors of its spatial / temporal neighbors. In this mode, some subblocks may require motion information from their corresponding subblocks in a temporal reference picture.

[0085] Another example of subblock-level motion compensation is an affine-coded motion-compensated block, which first derives motion vectors at the four corners of the current block based on the motion vectors of adjacent blocks. Then, by deriving other motion vectors of the current block (e.g., at the subblock or pixel level) using an affine model, each subblock can have different motion vectors than its adjacent subblocks.

[0086] Another example of subblock-level motion compensation is merge candidate refinement using motion vector derivation at the decoder side. In this mode, after obtaining one or more motion vector predictions for the current block or its subblocks, the given one or more motion vector predictions can be further refined using methods such as template matching or bilateral matching. The refined motion vectors are then used to perform motion compensation. By performing the same refinement operation at both the encoder and decoder sides, the decoder does not need additional information on how the refinement deviates from the original predictions. The skip mode can also be considered a special merge mode in which, in addition to deriving motion information for the current block from its neighbors, the prediction residuals for the current block are also zero.

[0087] Based on several embodiments, in subblock temporal motion vector prediction, the subblocks of the current block may have different motion vector prediction values ​​derived from the temporal reference picture. For example, a set of motion information is identified that includes the motion vector of the current block and its associated reference index. The motion information is determined from a first available spatial merge candidate. The reference block in the reference picture for the current block is determined using this motion information. The reference block is also divided into subblocks. In some embodiments, for each current block subblock (CBSB) in the current picture, the reference picture has a corresponding reference block subblock (RBSB).

[0088] In some embodiments, for each CBSB, the corresponding RBSB is encoded in interframe mode using a set of motion information. This motion information is then transformed (e.g., using methods such as motion vector scaling in temporal motion vector prediction) and used as the predicted value of the motion vector of the CBSB. The following describes in more detail how to handle RBSBs encoded in intraframe mode (e.g., intraframe block copy mode).

[0089] According to some embodiments, when using a subblock-based temporal motion vector prediction mode, each CBSB is not allowed to be encoded in an in-frame mode, such as an in-frame block copy mode. This is achieved by treating an RBSB encoded in an in-frame block copy as an in-frame mode. In particular, regardless of how the in-frame block copy mode is considered (e.g., as an inter-frame mode, an in-frame mode, or a third mode), for a CBSB, if the corresponding RBSB is encoded in an in-frame block copy mode, that RBSB is considered an in-frame mode in subblock-based temporal motion vector prediction. Therefore, in some embodiments, an RBSB encoded in an in-frame block copy mode is treated according to the default settings in subblock-based temporal motion vector prediction. For example, when encoding the corresponding RBSB of a CBSB in an in-frame block copy mode, a default motion vector, such as a zero motion vector, is used as the predicted value for that CBSB. In this example, the reference picture used for the CBSB is a temporal reference picture, not a current picture. For example, a temporal reference picture may be a picture shared by all subblocks of the current block, the first reference picture in the reference picture list, or a colocation picture for TMVP purposes. In another example, if a CBSB has a corresponding RBSB encoded in inter-frame mode, but the reference picture is the current picture, then a default motion vector, such as a zero motion vector, is assigned to the CBSB. In this regard, even if an RBSB is encoded in inter-frame mode, since the current picture and the reference picture are the same, the RBSB is treated as if it were encoded in intra-frame mode.

[0090] Figure 9 illustrates an example of performing a temporal motion vector prediction based on subblocks. Figure 9 shows a current picture 900 which has nine blocks, including current block 900A. Current block 900A is divided into four subblocks 1-4. Current picture 900 can be associated with reference picture 902, which has nine previously encoded / decoded blocks. Also, as shown in Figure 9, current block 900A has a motion vector 904 pointing to reference block 902A. Motion vector 904 can be determined using the motion vectors of one or more adjacent blocks in current block 900A (e.g., spatial merge candidates). Reference block 902A is divided into four subblocks 1-4. Subblocks 1-4 of reference block 902A correspond to subblocks 1-4 of current block 900A, respectively. If reference picture 902 and current picture 900 are the same, each RBSB in block 902A is treated as if these blocks were encoded in in-frame mode. In this regard, for example, if subblock 1 of block 902A is encoded in interframe mode, but the reference picture 902 and the current picture 900 are the same, subblock 1 of block 902A is treated as if it were encoded in intraframe mode, and the default motion vector is assigned to subblock 1 of block 900A.

[0091] If the reference picture 902 and the current picture 900 are different, subblocks 1-4 of reference block 902A are used to perform subblock-based temporal motion vector prediction for subblocks 1-4 of block 900A, respectively. For example, the motion vector prediction value of subblock 1 of the current block 900A is determined based on whether subblock 1 of reference block 902A is encoded in inter-frame mode or in intra-frame mode (e.g., intra-frame block copy mode). If subblock 1 of reference block 902A is encoded in inter-frame mode, the motion vector of subblock 1 of reference block 902A is used to determine the motion vector of subblock 1 of the current block 900A. If subblock 1 of reference block 902A is encoded in intra-frame mode, the motion vector of subblock 1 of the current block 900A is set to a zero motion vector.

[0092] Figure 10 shows an example of a process performed by an encoder or decoder, such as an in-frame encoder 522 or an in-frame decoder 672. The process begins in step S1000, in which the current picture is obtained from the encoded video bitstream. For example, referring to Figure 9, the current picture 900 is obtained from the encoded video bitstream. The process proceeds to step S1002, in which a reference block from a reference picture is identified for the current block in the current picture. For example, referring to Figure 9, the reference picture 902 is found in the list of reference pictures associated with the current block 900A. If a subblock temporal motion vector prediction is performed for the current block 900A, the motion vector 904 may be used to identify the reference block 902A of the reference picture 902.

[0093] The process proceeds to step S1004, in which it determines whether the reference picture is the same as the current picture. If the reference picture is different from the current picture, the process proceeds to step S1006, in which it determines the encoding mode of the RBSB corresponding to the CBSB. For example, see Figure 9, the current CBSB of block 900A The encoding mode of RBSB 1 of reference block 902A corresponding to 1 is determined. The process proceeds to step S1008, in which it is determined whether the encoding mode of the RBSB is interframe mode. If the encoding mode of the RBSB is interframe mode, the process proceeds to step S1010, in which the motion vector prediction value of the CBSB is determined based on the motion vector prediction value of the RBSB. For example, the RBSB of reference block 902A If the encoding mode of 1 is interframe mode, then the RBSB of reference block 902A Based on the motion vector prediction value of 1, the motion vector prediction value of CBSB 1 of block 900A is determined. For example, the motion vector prediction value of RBSB 1 of reference block 902A is transformed (for example, using a method such as motion vector scaling in temporal motion vector prediction), and the CBSB of block 900A is determined. It is used as the motion vector prediction value for 1.

[0094] Returning to step S1008, if the encoding mode of the RBSB is not interframe mode (for example, the encoding mode of the RBSB is intraframe mode), the process proceeds to step S1012, in which the motion vector prediction value of the CBSB is set to the default motion vector. For example, the RBSB of reference block 902A If 1 is encoded in in-frame mode, then currently block 900A CBSB A motion vector prediction value of 1 is set to the default motion vector, such as a zero motion vector.

[0095] Returning to step S1004, if the reference picture and the current picture are the same, the process proceeds to step S1012, in which the motion vector prediction value of the CBSB is set to the default motion vector. In connection with this, if the reference picture and the current picture are the same, the encoding mode of the RBSB is determined to be in-frame mode, and the motion vector prediction value for the corresponding CBSB is set to the default motion vector. In connection with this, even if the RBSB is in inter-frame mode, by setting the motion vector prediction value of the CBSB to the default motion vector, the RBSB can be treated as if it were encoded in in-frame mode. Steps S1004 to S1012 can be repeated for each subblock in the current block 900A.

[0096] The aforementioned technology may be implemented as computer software by computer-readable instructions and may be stored physically on one or more computer-readable media. For example, Figure 11 shows a computer system (1100) for implementing some embodiments of the themes of the disclosure.

[0097] Computer software is encoded in any appropriate machine code or computer language, and the machine code or computer language constructs code containing instructions through mechanisms such as editing, compiling, and linking, and these instructions are executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or by interpretation, microcode execution, etc.

[0098] The aforementioned instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things devices.

[0099] The components of the computer system (1100) shown in Figure 11 are illustrative and not limit the scope or functionality of the computer software used in the embodiments for realizing the present disclosure. The arrangement of the components should not be interpreted as having any dependency or requirement on any component or combination thereof in the exemplary embodiments of the computer system (1100).

[0100] The computer system (1100) may include several human-machine interface input devices. Such human-machine interface input devices can respond to input from one or more human users, for example, through tactile input (e.g., keystrokes, slides, data glove movements), audio input (e.g., voices, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface devices can further capture certain media, such as audio (e.g., voices, music, ambient sounds), images (e.g., scanned images, photographic images acquired from a still image capture device), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0101] The input human-machine interface device may have one or more of the following: keyboard (1101), mouse (1102), touchpad (1103), touch panel (1110), data glove (not shown), joystick (1105), microphone (1106), scanner (1107), and imaging device (1108) (only one of each of the listed).

[0102] The computer system (1100) may further have human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light and smell / taste. Such human-machine interface output devices include tactile output devices (e.g., tactile feedback via touch panels (1110), data gloves (not shown) or joysticks (1105), although tactile feedback devices may also exist that are not used as input devices), audio output devices (e.g., speakers (1109), headphones (not shown)), visual output devices (e.g., screens (1110), CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touch panel input capability and tactile feedback capability, some of which output two-dimensional vision or three-dimensional or more-dimensional vision by means such as stereoscopic image output, and include virtual reality glasses (not shown), holographic displays and smoke tanks (not shown)), and printers (not shown).

[0103] The computer system (1100) may further have a memory device and related media accessible to humankind, for example, a CD / DVD having media such as CD / DVD (1121). This includes optical media including ROM / RW (1120), thumb drives (1122), removable hard drives or solid-state drives (1123), traditional magnetic media such as magnetic tape and floppy disks (not shown), and devices based on dedicated ROM / ASIC / PLD, such as dongles (not shown).

[0104] Those skilled in the art will understand, by combining the themes of the present disclosure, that the term “computer-readable medium” as used does not include a transmission medium, carrier wave, or other instantaneous signal.

[0105] The computer system (1100) may further have interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. The networks may also be local, wide-area, urban, vehicle, industrial, real-time, or latency-tolerant networks. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital television networks including wired television, satellite television, and terrestrial television, and vehicle and industrial networks including CANBus. Some networks generally require an external network interface adapter connected to a general-purpose data port or peripheral bus (1149) (e.g., a USB port on the computer system (1100)), while others are generally integrated into the core of the computer system (1100) by being connected to system buses described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Through any of these networks, the computer system (1100) can communicate with other entities. Such communications can be one-way and receive only (e.g., broadcast television), one-way and transmit only (e.g., CANbus to a CANbus device), or two-way (e.g., to another computer system via a local area or wide area digital network). Specific protocols and protocol stacks can be used for each of these networks and network interfaces described above.

[0106] The aforementioned human-machine interface devices, human-accessible memory devices, and network interfaces can be connected to the core (1140) of the computer system (1100).

[0107] A core (1140) includes one or more central processing units (CPUs) (1141), graphics processing units (GPUs) (1142), field-programmable gate arrays (FPGAs) (1143), hardware accelerators (1144) for specific tasks, and so on. These devices, along with read-only memory (ROM) (1145), random-access memory (1146), and internal mass storage devices such as hard disk drives and SSDs (1147) that are inaccessible to the internal user, are connected via a system bus (1148). In a computer system, it can be expanded by other CPUs, GPUs, etc., by accessing the system bus (1148) in the form of one or more physical plugs. Peripheral devices are connected to the core's system bus (1148) directly or via peripheral buses (1149). Peripheral bus architectures include PCI, USB, etc.

[0108] The CPU (1141), GPU (1142), FPGA (1143), and accelerator (1144) can execute several instructions, and these instructions are combined to form the computer code mentioned above. This computer code is stored in ROM (1145) or RAM (1146). Transition data is stored in RAM (1146), and permanent data may be stored in, for example, an internal mass storage device (1147). Cache memory enables fast storage and retrieval of any of the memory devices, and the cache memory can be closely associated with one or more CPUs (1141), GPUs (1142), mass storage devices (1147), ROMs (1145), RAM (1146), etc.

[0109] A computer-readable medium contains computer code for performing various operations that a computer can perform. The medium and computer code may be a medium and computer code specifically designed and constructed for the purposes of this disclosure, or they may be of a type known and available to those skilled in the art of computer software.

[0110] To the present and not to the extent of an example, a computer system having architecture (1100), particularly a core (1140), can provide functionality by having (one or more) processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software embodied in one or more tangible computer-readable media. Such computer-readable media may be media relating to user-accessible mass storage devices as described above, and non-temporary storage devices of the core (1140), such as internal mass storage devices (1147) or ROM (1145) within the core. Software of various embodiments for realizing the present disclosure is stored in such devices and executed by the core (1140). Depending on specific requirements, the computer-readable media may include one or more storage devices or chips. The software includes causing the core (1140), in particular the processor (including CPU, GPU, FPGA, etc.), to execute specific processes or specific parts of specific processes as described herein, limiting the data configuration stored in RAM (1146), and modifying such data configuration based on the processes limited by the software. Alternatively, the computer system may provide functionality by being embodied in circuits (e.g., accelerators (1144)) by logical fixed connections or other means, and such circuits may execute specific processes or specific parts of specific processes as described herein, either in place of or in conjunction with the software. Where appropriate, the software referred to may include logic, and conversely, the logic referred to may include software. Where appropriate, the computer-readable medium referred to may include circuits (e.g., integrated circuits (ICs)) on which software for execution is stored, circuits embodying logic for execution, or both. The disclosure includes any appropriate combination of hardware and software.

[0111] While this disclosure includes some illustrative examples, there are many variations, substitutions, and alternative equivalents that fall within the scope of this disclosure. Therefore, many systems and methods that embody the principles of this disclosure and fall within the spirit and scope of this disclosure, although not explicitly described herein, can be conceived by those skilled in the art.

[0112] (1) A method for decoding video using a decoder, wherein the method involves obtaining a current picture from an encoded video bitstream, identifying a reference block in a reference picture different from the current picture for a current block contained in the current picture, the current block being divided into a plurality of subblocks (CBSBs), the reference block having a plurality of subblocks (RBSBs), the plurality of subblocks (RBSBs) corresponding to different CBSBs of the plurality of CBSBs, determining whether the reference picture of the RBSB is the current picture, and the RBSBs In response to the determination that the reference picture is the current picture, the encoding mode of the RBSB is determined to be the in-frame mode, and in response to the determination that the reference picture of the RBSB is not the current picture, (i) for one of the plurality of CBSBs, the encoding mode of the corresponding RBSB is determined to be either the in-frame mode or the inter-frame mode, and (ii) based on whether the encoding mode of the corresponding RBSB is the in-frame mode or the inter-frame mode, the motion vector prediction value for one of the plurality of CBSBs is determined.

[0113] (2) The method of feature (1) wherein, in response to the determination that the encoding mode of the corresponding RBSB is the in-frame mode, the motion vector prediction value determined for one CBSB is set to the default motion vector.

[0114] (3) The method according to feature (2), wherein the default motion vector is (i) a zero motion vector, or (ii) a deviation between the CBSB and the RBSB.

[0115] (4) The method according to any one of features (1) to (3), wherein, in response to the determination that the coding mode of the corresponding RBSB is the interframe mode, the motion vector prediction value determined for one CBSB is based on the motion vector prediction value associated with the corresponding RBSB.

[0116] (5) The method according to feature (4), wherein the determined motion vector prediction value is a scaled version of the motion vector prediction value associated with the corresponding RBSB.

[0117] (6) The method according to any one of (1) to (5) of the feature that identifies the reference block according to the motion vector prediction value related to the block adjacent to the current block.

[0118] (7) The method according to any one of the features (1) to (6), wherein the reference picture is a first reference picture from a reference picture sequence associated with the current picture.

[0119] (8) A video decoder for video decoding having a processing circuit, wherein the processing circuit obtains a current picture from an encoded video bitstream, identifies a reference block in a reference picture different from the current picture for a current block contained in the current picture, the current block is divided into a plurality of subblocks (CBSBs), the reference block has a plurality of subblocks (RBSBs), the plurality of subblocks (RBSBs) correspond to different CBSBs of the plurality of CBSBs, and determines whether the reference picture of the RBSB is the current picture, and the reference of the RBSB In response to the determination that the picture is the current picture, the encoding mode of the RBSB is determined to be the in-frame mode, and in response to the determination that the reference picture of the RBSB is not the current picture, the system is configured to (i) determine whether the encoding mode of the corresponding RBSB for one of the plurality of CBSBs is the in-frame mode or the inter-frame mode, and (ii) determine the motion vector prediction value for one of the CBSBs based on whether the encoding mode of the corresponding RBSB is the in-frame mode or the inter-frame mode.

[0120] (9) The video decoder according to feature (8), wherein the processing circuit is configured to set the motion vector prediction value for one CBSB to a default motion vector in response to the determination that the encoding mode of the corresponding RBSB is the in-frame mode.

[0121] (10) The video decoder according to feature (9), wherein the default motion vector is (i) a zero motion vector, or (ii) a deviation between the CBSB and the RBSB.

[0122] (11) The video decoder according to any one of features (8) to (10), wherein the processing circuit is configured to determine, in response to the determination that the encoding mode of the corresponding RBSB is the interframe mode, a motion vector prediction value for one CBSB is determined based on a motion vector prediction value associated with the corresponding RBSB.

[0123] (12) The video decoder according to feature (11), wherein the determined motion vector prediction is a scaled version of the motion vector prediction associated with the corresponding RBSB.

[0124] (13) The processing circuit identifies the reference block based on a motion vector prediction value related to a block adjacent to the current block, according to any one of the features (8) to (12).

[0125] (14) The video decoder according to any one of the features (8) to (13), wherein the reference picture is a first reference picture from a reference picture sequence associated with the current picture.

[0126] (15) A computer program having instructions, the instructions, when executed by a processor, cause the processor to execute a method, the method of obtaining a current picture from an encoded video bitstream, identifying a reference block in a reference picture different from the current picture for a current block contained in the current picture, the current block being divided into a plurality of subblocks (CBSBs), the reference block having a plurality of subblocks (RBSBs), the plurality of subblocks (RBSBs) corresponding to different CBSBs of the plurality of CBSBs, and the reference picture of the RBSB being the current picture. The process includes determining whether the reference picture of the RBSB is the current picture, determining the encoding mode of the RBSB as in-frame mode, and in response to determining that the reference picture of the RBSB is not the current picture, (i) determining whether the encoding mode of the corresponding RBSB for one of the plurality of CBSBs is in-frame mode or inter-frame mode, and (ii) determining the motion vector prediction value for one of the plurality of CBSBs based on whether the encoding mode of the corresponding RBSB is in-frame mode or inter-frame mode.

[0127] (16) A computer program according to feature (15) that sets the motion vector prediction value determined for one CBSB to a default motion vector in response to the determination that the encoding mode of the corresponding RBSB is the in-frame mode.

[0128] (17) The computer program described in feature (16) wherein the default motion vector is (i) a zero motion vector, or (ii) a deviation between the CBSB and the RBSB.

[0129] (18) A computer program according to any one of the features (15) to (17), wherein, in response to the determination that the encoding mode of the corresponding RBSB is the interframe mode, the motion vector prediction value determined for one CBSB is based on the motion vector prediction value associated with the corresponding RBSB.

[0130] (19) The motion vector prediction value to be determined is a scaled version of the motion vector prediction value associated with the corresponding RBSB in a non-transient computer-readable medium as described in feature (18).

[0131] (20) A non-temporary computer-readable medium according to any one of the features (15) to (19) that identifies the reference block according to a motion vector prediction value related to a block adjacent to the current block.

[0132] Appendix A: Acronyms MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Replenishment Enhancement Information VUI: Video Usability Information GOP: Picture Group TU: Conversion Unit PU: Prediction Unit CTU: Encoding Tree Unit CTB: Encoded Tree Block PB: Prediction Block HRD: Virtual Reference Decoder SNR: Signal-to-Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-only memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral component interconnection FPGA: Field-Programmable Gate Array SSD: Solid State Drive IC: Integrated Circuit CU: Encoding Unit

Claims

1. A method for video decoding performed by a decoder, Steps to obtain the current picture from the encoded video bitstream, A step of identifying a reference block contained in a reference picture with respect to a current block contained in the current picture, wherein the reference picture differs from the current picture, the current block is divided into a plurality of current subblocks (CBSBs) coded in intermode, and the reference block has a plurality of reference subblocks (RBSBs), each CBSB corresponding to at least one RBSB. The steps include determining whether the corresponding RBSB is coded in intra-predictive mode or intra-block copy mode, In response to the fact that the corresponding RBSB of each of the plurality of CBSBs is coded in intra-prediction mode or intra-block copy mode, the steps include setting the first motion vector prediction value of each CBSB to a zero motion vector, A method comprising the step of decoding the plurality of CBSBs by performing subblock-based time motion vector prediction based on first motion vector prediction values ​​of the plurality of CBSBs.

2. The method according to claim 1, wherein, in response to the fact that the corresponding RBSB of each CBSB is coded in intermode, the first motion vector prediction value of each CBSB is set using the second motion vector prediction value of the corresponding RBSB.

3. The method of claim 2, wherein the second motion vector prediction value of the corresponding RBSB is scaled.

4. The method according to any one of claims 1 to 3, wherein the reference block is identified according to a motion vector prediction value related to a block adjacent to the current block.

5. The method according to any one of claims 1 to 4, wherein the reference picture is a temporal reference picture.

6. The method according to claim 5, wherein the temporal reference picture is a first reference picture from a reference picture sequence associated with the current picture.

7. The method according to claim 5, wherein the aforementioned time reference picture is a collated picture.

8. The method according to claim 5, wherein the temporal reference picture is a picture shared by all subblocks of the current block.

9. A video decoder for video decoding, One or more memory locations containing computer programs, One or more processors, It has, The computer program causes one or more processors to perform the method according to any one of claims 1 to 8. Video decoder.

10. A computer program having instructions, wherein, when executed by a processor, the instructions cause the processor to perform the method according to any one of claims 1 to 8.

Images